Nov 03 13:04:06.025775 ip-10-0-130-198 systemd[1]: Starting Kubernetes Kubelet... Nov 03 13:04:06.444388 ip-10-0-130-198 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 13:04:06.444388 ip-10-0-130-198 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 13:04:06.444388 ip-10-0-130-198 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 13:04:06.444388 ip-10-0-130-198 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 13:04:06.449447 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449370 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:04:06.449447 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449447 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:04:06.449447 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449450 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:04:06.449552 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449454 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:04:06.449552 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449459 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:04:06.449552 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449464 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:04:06.449552 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449479 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:04:06.449552 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449482 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:04:06.449552 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449485 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:04:06.449552 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449488 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:04:06.449552 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449491 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:04:06.449552 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449494 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:04:06.449552 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449496 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:04:06.449552 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449499 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:04:06.449552 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449502 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:04:06.449552 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449505 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:04:06.449552 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449507 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:04:06.449552 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449510 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:04:06.449552 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449512 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:04:06.449552 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449516 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:04:06.449552 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449521 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:04:06.449552 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449524 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:04:06.449987 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449527 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:04:06.449987 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449530 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:04:06.449987 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449541 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:04:06.449987 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449544 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:04:06.449987 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449547 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:04:06.449987 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449549 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:04:06.449987 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449552 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:04:06.449987 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449555 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:04:06.449987 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449558 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:04:06.449987 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449561 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:04:06.449987 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449565 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:04:06.449987 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449568 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:04:06.449987 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449570 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:04:06.449987 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449573 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:04:06.449987 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449576 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:04:06.449987 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449578 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:04:06.449987 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449581 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:04:06.449987 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449584 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:04:06.449987 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449586 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:04:06.449987 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449589 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:04:06.450457 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449591 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:04:06.450457 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449594 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:04:06.450457 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449596 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:04:06.450457 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449599 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:04:06.450457 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449602 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:04:06.450457 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449604 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:04:06.450457 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449607 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:04:06.450457 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449609 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:04:06.450457 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449612 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:04:06.450457 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449614 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:04:06.450457 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449617 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:04:06.450457 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449619 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:04:06.450457 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449623 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:04:06.450457 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449628 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:04:06.450457 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449630 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:04:06.450457 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449633 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:04:06.450457 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449636 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:04:06.450457 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449638 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:04:06.450457 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449641 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:04:06.450457 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449644 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:04:06.450943 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449646 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:04:06.450943 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449649 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:04:06.450943 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449652 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:04:06.450943 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449654 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:04:06.450943 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449657 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:04:06.450943 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449660 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:04:06.450943 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449662 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:04:06.450943 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449664 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:04:06.450943 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449667 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:04:06.450943 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449669 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:04:06.450943 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449672 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:04:06.450943 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449675 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:04:06.450943 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449678 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:04:06.450943 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449680 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:04:06.450943 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449683 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:04:06.450943 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449686 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:04:06.450943 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449688 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:04:06.450943 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449691 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:04:06.450943 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449693 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:04:06.451383 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449696 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:04:06.451383 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449699 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:04:06.451383 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449701 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:04:06.451383 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449704 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:04:06.451383 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449707 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:04:06.451383 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449710 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:04:06.451383 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449712 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:04:06.451383 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449715 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:04:06.451383 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449717 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:04:06.451383 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449720 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:04:06.451383 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449723 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:04:06.451383 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449725 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:04:06.451383 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449727 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:04:06.451383 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449730 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:04:06.451383 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449733 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:04:06.451383 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449735 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:04:06.451383 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449738 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:04:06.451383 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449740 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:04:06.451383 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449742 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:04:06.451914 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449745 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:04:06.451914 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449748 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:04:06.451914 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449751 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:04:06.451914 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449754 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:04:06.451914 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449756 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:04:06.451914 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.449760 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:04:06.452976 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.452962 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:04:06.452976 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.452975 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:04:06.453037 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.452979 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:04:06.453037 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.452982 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:04:06.453037 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.452985 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:04:06.453037 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.452987 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:04:06.453037 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.452990 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:04:06.453037 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.452993 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:04:06.453037 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.452996 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:04:06.453037 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.452998 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:04:06.453037 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453001 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:04:06.453037 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453004 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:04:06.453037 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453006 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:04:06.453037 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453009 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:04:06.453037 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453011 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:04:06.453037 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453015 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:04:06.453037 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453017 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:04:06.453037 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453020 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:04:06.453037 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453023 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:04:06.453037 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453027 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:04:06.453037 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453030 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:04:06.453037 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453033 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:04:06.453531 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453036 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:04:06.453531 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453039 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:04:06.453531 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453042 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:04:06.453531 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453044 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:04:06.453531 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453047 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:04:06.453531 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453050 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:04:06.453531 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453053 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:04:06.453531 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453056 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:04:06.453531 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453058 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:04:06.453531 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453060 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:04:06.453531 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453063 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:04:06.453531 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453067 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:04:06.453531 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453070 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:04:06.453531 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453072 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:04:06.453531 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453075 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:04:06.453531 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453077 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:04:06.453531 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453080 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:04:06.453531 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453082 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:04:06.453531 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453085 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:04:06.453531 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453087 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:04:06.454013 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453090 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:04:06.454013 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453092 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:04:06.454013 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453095 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:04:06.454013 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453103 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:04:06.454013 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453106 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:04:06.454013 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453108 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:04:06.454013 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453111 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:04:06.454013 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453113 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:04:06.454013 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453116 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:04:06.454013 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453118 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:04:06.454013 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453121 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:04:06.454013 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453124 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:04:06.454013 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453126 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:04:06.454013 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453130 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:04:06.454013 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453133 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:04:06.454013 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453136 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:04:06.454013 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453138 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:04:06.454013 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453141 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:04:06.454013 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453144 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:04:06.454013 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453147 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:04:06.454509 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453149 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:04:06.454509 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453152 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:04:06.454509 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453156 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:04:06.454509 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453160 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:04:06.454509 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453163 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:04:06.454509 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453166 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:04:06.454509 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453168 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:04:06.454509 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453171 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:04:06.454509 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453173 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:04:06.454509 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453176 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:04:06.454509 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453178 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:04:06.454509 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453181 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:04:06.454509 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453183 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:04:06.454509 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453186 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:04:06.454509 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453188 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:04:06.454509 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453191 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:04:06.454509 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453193 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:04:06.454509 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453196 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:04:06.454509 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453198 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:04:06.454967 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453201 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:04:06.454967 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453203 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:04:06.454967 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453206 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:04:06.454967 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453209 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:04:06.454967 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453212 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:04:06.454967 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453214 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:04:06.454967 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453218 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:04:06.454967 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453221 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:04:06.454967 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453223 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:04:06.454967 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453227 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:04:06.454967 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453229 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:04:06.454967 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453232 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:04:06.454967 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453234 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:04:06.454967 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453237 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:04:06.454967 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453239 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:04:06.454967 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453242 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:04:06.454967 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453244 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:04:06.454967 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453247 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:04:06.454967 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453249 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:04:06.455417 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453252 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:04:06.455417 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453254 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:04:06.455417 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453257 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:04:06.455417 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453259 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:04:06.455417 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453262 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:04:06.455417 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453264 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:04:06.455417 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453756 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:04:06.455417 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453762 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:04:06.455417 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453764 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:04:06.455417 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453767 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:04:06.455417 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453770 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:04:06.455417 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453773 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:04:06.455417 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453776 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:04:06.455417 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453778 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:04:06.455417 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453782 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:04:06.455417 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453785 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:04:06.455417 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453788 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:04:06.455417 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453791 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:04:06.455417 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453793 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:04:06.455417 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453796 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:04:06.455899 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453799 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:04:06.455899 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453802 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:04:06.455899 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453804 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:04:06.455899 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453807 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:04:06.455899 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453810 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:04:06.455899 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453812 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:04:06.455899 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453815 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:04:06.455899 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453817 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:04:06.455899 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453820 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:04:06.455899 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453822 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:04:06.455899 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453825 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:04:06.455899 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453828 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:04:06.455899 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453831 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:04:06.455899 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453833 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:04:06.455899 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453836 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:04:06.455899 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453838 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:04:06.455899 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453841 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:04:06.455899 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453843 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:04:06.455899 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453846 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:04:06.456355 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453850 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:04:06.456355 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453853 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:04:06.456355 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453856 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:04:06.456355 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453859 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:04:06.456355 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453862 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:04:06.456355 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453864 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:04:06.456355 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453867 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:04:06.456355 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453869 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:04:06.456355 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453872 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:04:06.456355 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453875 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:04:06.456355 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453877 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:04:06.456355 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453880 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:04:06.456355 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453883 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:04:06.456355 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453885 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:04:06.456355 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453887 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:04:06.456355 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453890 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:04:06.456355 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453893 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:04:06.456355 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453895 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:04:06.456355 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453898 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:04:06.456827 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453901 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:04:06.456827 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453903 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:04:06.456827 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453906 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:04:06.456827 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453908 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:04:06.456827 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453911 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:04:06.456827 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453914 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:04:06.456827 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453916 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:04:06.456827 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453919 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:04:06.456827 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453922 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:04:06.456827 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453924 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:04:06.456827 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453929 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:04:06.456827 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453931 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:04:06.456827 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453934 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:04:06.456827 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453937 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:04:06.456827 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453940 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:04:06.456827 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453942 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:04:06.456827 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453945 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:04:06.456827 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453947 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:04:06.456827 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453950 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:04:06.457283 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453953 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:04:06.457283 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453957 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:04:06.457283 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453961 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:04:06.457283 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453963 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:04:06.457283 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453966 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:04:06.457283 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453968 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:04:06.457283 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453971 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:04:06.457283 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453974 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:04:06.457283 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453976 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:04:06.457283 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453979 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:04:06.457283 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453981 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:04:06.457283 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453984 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:04:06.457283 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453987 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:04:06.457283 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453989 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:04:06.457283 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453992 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:04:06.457283 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453994 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:04:06.457283 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453996 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:04:06.457283 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.453999 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:04:06.457283 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.454001 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:04:06.457283 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.454008 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:04:06.457816 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.454011 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:04:06.457816 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.454014 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:04:06.457816 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.454017 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:04:06.457816 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.454021 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:04:06.457816 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.454024 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:04:06.457816 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.454027 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:04:06.457816 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.454029 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:04:06.457816 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.454033 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:04:06.457816 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.454035 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:04:06.457816 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.454038 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:04:06.457816 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.454041 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:04:06.457816 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.454043 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:04:06.457816 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.454046 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:04:06.457816 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.454049 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:04:06.457816 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.454051 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:04:06.457816 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.454984 2580 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 13:04:06.457816 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.454996 2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 13:04:06.457816 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455002 2580 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 13:04:06.457816 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455007 2580 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 13:04:06.457816 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455012 2580 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 13:04:06.458308 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455015 2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 13:04:06.458308 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455020 2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 13:04:06.458308 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455024 2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 13:04:06.458308 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455027 2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 13:04:06.458308 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455030 2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 13:04:06.458308 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455034 2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 13:04:06.458308 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455037 2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 13:04:06.458308 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455040 2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 13:04:06.458308 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455043 2580 flags.go:64] FLAG: --cgroup-root="" Nov 03 13:04:06.458308 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455046 2580 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 13:04:06.458308 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455050 2580 flags.go:64] FLAG: --client-ca-file="" Nov 03 13:04:06.458308 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455053 2580 flags.go:64] FLAG: --cloud-provider="external" Nov 03 13:04:06.458308 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455058 2580 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 13:04:06.458308 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455063 2580 flags.go:64] FLAG: --cluster-domain="" Nov 03 13:04:06.458308 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455066 2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 13:04:06.458308 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455070 2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 13:04:06.458308 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455073 2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 13:04:06.458308 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455079 2580 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 13:04:06.458308 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455083 2580 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 13:04:06.458308 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455086 2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 13:04:06.458308 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455089 2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 13:04:06.458308 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455092 2580 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 13:04:06.458308 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455096 2580 flags.go:64] FLAG: --contention-profiling="false" Nov 03 13:04:06.458308 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455098 2580 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 13:04:06.458883 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455102 2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 13:04:06.458883 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455105 2580 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 13:04:06.458883 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455108 2580 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 13:04:06.458883 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455112 2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 13:04:06.458883 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455115 2580 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 13:04:06.458883 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455118 2580 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 13:04:06.458883 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455121 2580 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 13:04:06.458883 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455124 2580 flags.go:64] FLAG: --enable-server="true" Nov 03 13:04:06.458883 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455127 2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 13:04:06.458883 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455131 2580 flags.go:64] FLAG: --event-burst="100" Nov 03 13:04:06.458883 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455135 2580 flags.go:64] FLAG: --event-qps="50" Nov 03 13:04:06.458883 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455137 2580 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 13:04:06.458883 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455140 2580 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 13:04:06.458883 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455144 2580 flags.go:64] FLAG: --eviction-hard="" Nov 03 13:04:06.458883 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455148 2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 13:04:06.458883 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455151 2580 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 13:04:06.458883 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455154 2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 13:04:06.458883 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455157 2580 flags.go:64] FLAG: --eviction-soft="" Nov 03 13:04:06.458883 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455160 2580 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 13:04:06.458883 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455163 2580 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 13:04:06.458883 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455166 2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 13:04:06.458883 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455169 2580 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 13:04:06.458883 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455173 2580 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 13:04:06.458883 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455176 2580 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 13:04:06.458883 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455179 2580 flags.go:64] FLAG: --feature-gates="" Nov 03 13:04:06.459505 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455184 2580 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 13:04:06.459505 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455188 2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 13:04:06.459505 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455191 2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 13:04:06.459505 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455195 2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 13:04:06.459505 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455197 2580 flags.go:64] FLAG: --healthz-port="10248" Nov 03 13:04:06.459505 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455201 2580 flags.go:64] FLAG: --help="false" Nov 03 13:04:06.459505 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455204 2580 flags.go:64] FLAG: --hostname-override="ip-10-0-130-198.us-west-2.compute.internal" Nov 03 13:04:06.459505 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455207 2580 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 13:04:06.459505 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455210 2580 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 13:04:06.459505 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455213 2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 13:04:06.459505 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455217 2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 13:04:06.459505 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455220 2580 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 13:04:06.459505 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455224 2580 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 13:04:06.459505 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455227 2580 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 13:04:06.459505 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455230 2580 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 13:04:06.459505 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455233 2580 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 13:04:06.459505 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455236 2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 13:04:06.459505 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455239 2580 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 13:04:06.459505 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455242 2580 flags.go:64] FLAG: --kube-reserved="" Nov 03 13:04:06.459505 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455245 2580 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 13:04:06.459505 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455248 2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 13:04:06.459505 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455251 2580 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 13:04:06.459505 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455254 2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 13:04:06.460048 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455257 2580 flags.go:64] FLAG: --lock-file="" Nov 03 13:04:06.460048 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455260 2580 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 13:04:06.460048 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455263 2580 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 13:04:06.460048 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455266 2580 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 13:04:06.460048 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455272 2580 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 13:04:06.460048 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455275 2580 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 13:04:06.460048 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455278 2580 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 13:04:06.460048 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455281 2580 flags.go:64] FLAG: --logging-format="text" Nov 03 13:04:06.460048 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455284 2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 13:04:06.460048 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455289 2580 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 13:04:06.460048 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455292 2580 flags.go:64] FLAG: --manifest-url="" Nov 03 13:04:06.460048 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455294 2580 flags.go:64] FLAG: --manifest-url-header="" Nov 03 13:04:06.460048 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455299 2580 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 13:04:06.460048 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455302 2580 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 13:04:06.460048 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455306 2580 flags.go:64] FLAG: --max-pods="110" Nov 03 13:04:06.460048 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455309 2580 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 13:04:06.460048 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455312 2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 13:04:06.460048 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455315 2580 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 13:04:06.460048 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455349 2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 13:04:06.460048 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455353 2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 13:04:06.460048 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455356 2580 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 13:04:06.460048 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455359 2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 13:04:06.460048 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455367 2580 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 13:04:06.460048 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455370 2580 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 13:04:06.460048 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455374 2580 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 13:04:06.460672 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455377 2580 flags.go:64] FLAG: --pod-cidr="" Nov 03 13:04:06.460672 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455380 2580 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 13:04:06.460672 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455383 2580 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 13:04:06.460672 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455386 2580 flags.go:64] FLAG: --pods-per-core="0" Nov 03 13:04:06.460672 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455388 2580 flags.go:64] FLAG: --port="10250" Nov 03 13:04:06.460672 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455391 2580 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 13:04:06.460672 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455395 2580 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-072d2c3726ae77995" Nov 03 13:04:06.460672 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455398 2580 flags.go:64] FLAG: --qos-reserved="" Nov 03 13:04:06.460672 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455401 2580 flags.go:64] FLAG: --read-only-port="10255" Nov 03 13:04:06.460672 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455404 2580 flags.go:64] FLAG: --register-node="true" Nov 03 13:04:06.460672 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455407 2580 flags.go:64] FLAG: --register-with-taints="" Nov 03 13:04:06.460672 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455411 2580 flags.go:64] FLAG: --registry-burst="10" Nov 03 13:04:06.460672 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455415 2580 flags.go:64] FLAG: --registry-qps="5" Nov 03 13:04:06.460672 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455418 2580 flags.go:64] FLAG: --reserved-cpus="" Nov 03 13:04:06.460672 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455422 2580 flags.go:64] FLAG: --reserved-memory="" Nov 03 13:04:06.460672 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455426 2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 13:04:06.460672 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455430 2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 13:04:06.460672 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455433 2580 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 13:04:06.460672 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455436 2580 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 13:04:06.460672 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455439 2580 flags.go:64] FLAG: --runonce="false" Nov 03 13:04:06.460672 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455442 2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 13:04:06.460672 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455445 2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 13:04:06.460672 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455448 2580 flags.go:64] FLAG: --seccomp-default="false" Nov 03 13:04:06.460672 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455451 2580 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 13:04:06.460672 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455454 2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 13:04:06.460672 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455457 2580 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 13:04:06.461317 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455460 2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 13:04:06.461317 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455463 2580 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 13:04:06.461317 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455466 2580 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 13:04:06.461317 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455479 2580 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 13:04:06.461317 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455482 2580 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 13:04:06.461317 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455487 2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 13:04:06.461317 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455490 2580 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 13:04:06.461317 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455493 2580 flags.go:64] FLAG: --system-cgroups="" Nov 03 13:04:06.461317 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455496 2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 13:04:06.461317 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455502 2580 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 13:04:06.461317 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455505 2580 flags.go:64] FLAG: --tls-cert-file="" Nov 03 13:04:06.461317 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455508 2580 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 13:04:06.461317 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455513 2580 flags.go:64] FLAG: --tls-min-version="" Nov 03 13:04:06.461317 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455516 2580 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 13:04:06.461317 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455518 2580 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 13:04:06.461317 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455521 2580 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 13:04:06.461317 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455524 2580 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 13:04:06.461317 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455527 2580 flags.go:64] FLAG: --v="2" Nov 03 13:04:06.461317 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455532 2580 flags.go:64] FLAG: --version="false" Nov 03 13:04:06.461317 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455537 2580 flags.go:64] FLAG: --vmodule="" Nov 03 13:04:06.461317 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455541 2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 13:04:06.461317 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.455545 2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 13:04:06.461317 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455704 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:04:06.461317 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455709 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:04:06.461317 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455712 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:04:06.461953 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455715 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:04:06.461953 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455718 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:04:06.461953 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455721 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:04:06.461953 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455724 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:04:06.461953 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455727 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:04:06.461953 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455729 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:04:06.461953 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455732 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:04:06.461953 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455735 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:04:06.461953 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455738 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:04:06.461953 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455740 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:04:06.461953 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455743 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:04:06.461953 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455745 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:04:06.461953 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455749 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:04:06.461953 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455752 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:04:06.461953 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455755 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:04:06.461953 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455757 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:04:06.461953 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455760 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:04:06.461953 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455763 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:04:06.461953 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455765 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:04:06.462399 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455768 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:04:06.462399 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455770 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:04:06.462399 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455773 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:04:06.462399 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455776 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:04:06.462399 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455778 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:04:06.462399 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455781 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:04:06.462399 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455784 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:04:06.462399 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455786 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:04:06.462399 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455789 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:04:06.462399 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455791 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:04:06.462399 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455796 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:04:06.462399 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455798 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:04:06.462399 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455801 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:04:06.462399 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455803 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:04:06.462399 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455806 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:04:06.462399 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455808 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:04:06.462399 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455811 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:04:06.462399 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455814 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:04:06.462399 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455817 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:04:06.462899 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455819 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:04:06.462899 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455821 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:04:06.462899 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455824 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:04:06.462899 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455827 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:04:06.462899 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455829 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:04:06.462899 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455832 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:04:06.462899 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455836 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:04:06.462899 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455839 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:04:06.462899 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455841 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:04:06.462899 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455844 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:04:06.462899 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455846 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:04:06.462899 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455849 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:04:06.462899 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455852 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:04:06.462899 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455856 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:04:06.462899 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455860 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:04:06.462899 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455863 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:04:06.462899 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455867 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:04:06.462899 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455870 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:04:06.462899 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455873 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:04:06.462899 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455876 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:04:06.463370 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455878 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:04:06.463370 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455881 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:04:06.463370 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455883 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:04:06.463370 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455886 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:04:06.463370 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455889 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:04:06.463370 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455892 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:04:06.463370 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455894 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:04:06.463370 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455897 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:04:06.463370 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455899 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:04:06.463370 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455902 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:04:06.463370 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455905 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:04:06.463370 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455907 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:04:06.463370 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455910 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:04:06.463370 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455913 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:04:06.463370 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455915 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:04:06.463370 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455918 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:04:06.463370 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455920 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:04:06.463370 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455923 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:04:06.463370 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455927 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:04:06.463370 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455930 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:04:06.463878 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455932 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:04:06.463878 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455935 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:04:06.463878 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455938 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:04:06.463878 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455940 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:04:06.463878 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455944 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:04:06.463878 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455947 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:04:06.463878 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455949 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:04:06.463878 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455952 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:04:06.463878 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455955 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:04:06.463878 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455957 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:04:06.463878 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455960 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:04:06.463878 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455964 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:04:06.463878 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455968 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:04:06.463878 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455971 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:04:06.463878 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455974 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:04:06.463878 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455976 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:04:06.463878 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455979 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:04:06.463878 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455985 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:04:06.463878 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455988 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:04:06.464348 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455991 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:04:06.464348 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455993 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:04:06.464348 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455996 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:04:06.464348 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.455998 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:04:06.464348 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456001 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:04:06.464348 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456003 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:04:06.464348 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.456033 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 13:04:06.464348 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456069 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:04:06.464348 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456073 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:04:06.464348 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456077 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:04:06.464348 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456080 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:04:06.464348 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456083 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:04:06.464348 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456088 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:04:06.464348 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456091 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:04:06.464348 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456094 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:04:06.464348 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456097 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:04:06.464747 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456099 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:04:06.464747 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456104 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:04:06.464747 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456107 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:04:06.464747 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456109 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:04:06.464747 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456112 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:04:06.464747 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456115 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:04:06.464747 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456117 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:04:06.464747 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456120 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:04:06.464747 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456123 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:04:06.464747 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456126 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:04:06.464747 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456128 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:04:06.464747 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456131 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:04:06.464747 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456141 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:04:06.464747 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456144 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:04:06.464747 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456147 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:04:06.464747 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456150 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:04:06.464747 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456153 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:04:06.464747 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456156 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:04:06.464747 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456158 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:04:06.465189 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456161 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:04:06.465189 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456163 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:04:06.465189 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456166 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:04:06.465189 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456169 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:04:06.465189 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456171 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:04:06.465189 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456174 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:04:06.465189 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456176 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:04:06.465189 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456179 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:04:06.465189 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456182 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:04:06.465189 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456185 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:04:06.465189 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456188 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:04:06.465189 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456190 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:04:06.465189 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456193 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:04:06.465189 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456196 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:04:06.465189 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456200 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:04:06.465189 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456202 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:04:06.465189 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456205 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:04:06.465189 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456208 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:04:06.465189 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456210 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:04:06.465189 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456213 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:04:06.465688 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456215 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:04:06.465688 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456218 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:04:06.465688 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456220 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:04:06.465688 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456223 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:04:06.465688 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456225 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:04:06.465688 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456228 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:04:06.465688 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456231 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:04:06.465688 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456233 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:04:06.465688 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456237 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:04:06.465688 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456241 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:04:06.465688 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456244 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:04:06.465688 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456247 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:04:06.465688 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456250 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:04:06.465688 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456252 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:04:06.465688 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456255 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:04:06.465688 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456257 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:04:06.465688 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456260 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:04:06.465688 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456262 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:04:06.465688 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456265 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:04:06.466145 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456268 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:04:06.466145 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456270 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:04:06.466145 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456274 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:04:06.466145 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456277 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:04:06.466145 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456280 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:04:06.466145 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456282 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:04:06.466145 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456285 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:04:06.466145 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456289 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:04:06.466145 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456292 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:04:06.466145 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456295 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:04:06.466145 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456297 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:04:06.466145 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456300 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:04:06.466145 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456303 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:04:06.466145 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456306 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:04:06.466145 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456308 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:04:06.466145 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456311 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:04:06.466145 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456313 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:04:06.466145 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456316 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:04:06.466145 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456319 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:04:06.466145 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456321 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:04:06.466635 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456324 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:04:06.466635 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456327 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:04:06.466635 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456329 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:04:06.466635 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456332 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:04:06.466635 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456335 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:04:06.466635 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456337 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:04:06.466635 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456340 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:04:06.466635 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456344 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:04:06.466635 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456347 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:04:06.466635 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456351 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:04:06.466635 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456354 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:04:06.466635 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456356 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:04:06.466635 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456360 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:04:06.466635 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456362 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:04:06.466635 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456366 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:04:06.466635 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456370 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:04:06.466635 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456372 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:04:06.466635 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456375 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:04:06.466635 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.456378 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:04:06.467083 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.463287 2580 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 13:04:06.467083 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.463309 2580 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 13:04:06.467083 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.464022 2580 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 13:04:06.467083 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.464028 2580 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 13:04:06.467083 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464091 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:04:06.467083 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464096 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:04:06.467083 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464100 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:04:06.467083 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464103 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:04:06.467083 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464107 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:04:06.467083 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464110 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:04:06.467083 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464113 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:04:06.467083 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464116 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:04:06.467083 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464119 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:04:06.467083 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464122 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:04:06.467083 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464124 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:04:06.467083 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464127 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:04:06.467083 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464129 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:04:06.467083 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464138 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:04:06.467083 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464143 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:04:06.467083 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464146 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:04:06.467829 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464148 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:04:06.467829 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464151 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:04:06.467829 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464153 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:04:06.467829 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464156 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:04:06.467829 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464158 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:04:06.467829 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464161 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:04:06.467829 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464164 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:04:06.467829 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464166 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:04:06.467829 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464169 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:04:06.467829 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464171 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:04:06.467829 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464175 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:04:06.467829 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464177 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:04:06.467829 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464179 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:04:06.467829 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464190 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:04:06.467829 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464193 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:04:06.467829 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464196 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:04:06.467829 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464199 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:04:06.467829 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464202 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:04:06.467829 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464204 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:04:06.468614 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464207 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:04:06.468614 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464209 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:04:06.468614 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464212 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:04:06.468614 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464214 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:04:06.468614 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464217 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:04:06.468614 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464219 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:04:06.468614 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464222 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:04:06.468614 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464224 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:04:06.468614 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464227 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:04:06.468614 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464230 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:04:06.468614 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464232 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:04:06.468614 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464235 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:04:06.468614 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464237 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:04:06.468614 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464240 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:04:06.468614 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464242 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:04:06.468614 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464245 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:04:06.468614 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464247 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:04:06.468614 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464250 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:04:06.468614 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464252 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:04:06.468614 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464255 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:04:06.469124 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464257 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:04:06.469124 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464259 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:04:06.469124 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464262 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:04:06.469124 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464264 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:04:06.469124 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464267 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:04:06.469124 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464269 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:04:06.469124 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464272 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:04:06.469124 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464281 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:04:06.469124 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464283 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:04:06.469124 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464286 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:04:06.469124 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464289 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:04:06.469124 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464291 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:04:06.469124 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464294 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:04:06.469124 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464297 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:04:06.469124 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464300 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:04:06.469124 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464302 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:04:06.469124 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464305 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:04:06.469124 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464307 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:04:06.469124 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464310 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:04:06.469124 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464312 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:04:06.469651 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464315 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:04:06.469651 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464318 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:04:06.469651 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464322 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:04:06.469651 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464326 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:04:06.469651 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464329 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:04:06.469651 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464332 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:04:06.469651 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464334 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:04:06.469651 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464337 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:04:06.469651 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464340 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:04:06.469651 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464342 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:04:06.469651 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464345 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:04:06.469651 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464347 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:04:06.469651 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464350 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:04:06.469651 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464353 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:04:06.469651 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464356 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:04:06.469651 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464358 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:04:06.469651 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464361 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:04:06.469651 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464364 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:04:06.469651 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464367 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:04:06.470094 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464370 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:04:06.470094 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464378 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:04:06.470094 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464380 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:04:06.470094 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464383 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:04:06.470094 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464385 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:04:06.470094 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464388 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:04:06.470094 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464397 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:04:06.470094 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464401 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:04:06.470094 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464405 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:04:06.470094 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464409 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:04:06.470094 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464412 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:04:06.470094 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464415 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:04:06.470094 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.464442 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 13:04:06.470094 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464654 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:04:06.470094 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464659 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:04:06.470453 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464662 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:04:06.470453 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464665 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:04:06.470453 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464668 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:04:06.470453 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464670 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:04:06.470453 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464673 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:04:06.470453 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464675 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:04:06.470453 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464678 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:04:06.470453 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464680 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:04:06.470453 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464683 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:04:06.470453 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464686 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:04:06.470453 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464690 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:04:06.470453 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464693 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:04:06.470453 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464696 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:04:06.470453 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464699 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:04:06.470453 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464702 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:04:06.470453 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464704 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:04:06.470453 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464707 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:04:06.470453 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464709 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:04:06.470453 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464712 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:04:06.470908 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464720 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:04:06.470908 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464723 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:04:06.470908 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464725 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:04:06.470908 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464728 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:04:06.470908 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464730 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:04:06.470908 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464733 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:04:06.470908 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464735 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:04:06.470908 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464738 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:04:06.470908 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464740 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:04:06.470908 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464742 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:04:06.470908 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464745 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:04:06.470908 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464747 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:04:06.470908 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464750 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:04:06.470908 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464753 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:04:06.470908 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464755 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:04:06.470908 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464758 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:04:06.470908 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464761 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:04:06.470908 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464764 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:04:06.470908 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464766 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:04:06.470908 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464769 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:04:06.471381 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464772 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:04:06.471381 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464774 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:04:06.471381 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464777 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:04:06.471381 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464779 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:04:06.471381 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464782 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:04:06.471381 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464784 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:04:06.471381 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464788 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:04:06.471381 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464793 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:04:06.471381 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464796 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:04:06.471381 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464798 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:04:06.471381 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464801 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:04:06.471381 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464803 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:04:06.471381 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464806 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:04:06.471381 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464814 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:04:06.471381 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464817 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:04:06.471381 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464819 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:04:06.471381 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464822 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:04:06.471381 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464824 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:04:06.471381 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464827 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:04:06.471381 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464829 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:04:06.472019 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464832 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:04:06.472019 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464834 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:04:06.472019 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464837 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:04:06.472019 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464839 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:04:06.472019 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464842 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:04:06.472019 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464844 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:04:06.472019 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464847 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:04:06.472019 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464849 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:04:06.472019 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464852 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:04:06.472019 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464855 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:04:06.472019 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464857 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:04:06.472019 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464859 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:04:06.472019 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464862 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:04:06.472019 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464864 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:04:06.472019 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464867 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:04:06.472019 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464869 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:04:06.472019 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464872 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:04:06.472019 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464874 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:04:06.472019 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464876 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:04:06.472019 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464879 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:04:06.472569 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464881 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:04:06.472569 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464884 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:04:06.472569 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464886 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:04:06.472569 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464889 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:04:06.472569 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464891 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:04:06.472569 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464894 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:04:06.472569 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464905 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:04:06.472569 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464908 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:04:06.472569 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464911 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:04:06.472569 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464913 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:04:06.472569 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464915 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:04:06.472569 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464918 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:04:06.472569 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464920 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:04:06.472569 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464923 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:04:06.472569 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464925 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:04:06.472569 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464928 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:04:06.472569 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464930 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:04:06.472569 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464933 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:04:06.472569 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464936 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:04:06.473026 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464938 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:04:06.473026 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464941 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:04:06.473026 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464943 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:04:06.473026 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464946 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:04:06.473026 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464948 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:04:06.473026 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.464950 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:04:06.473026 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.464974 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 13:04:06.473026 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465014 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:04:06.473026 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465019 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:04:06.473026 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465022 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:04:06.473026 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465025 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:04:06.473026 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465028 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:04:06.473026 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465031 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:04:06.473026 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465034 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:04:06.473026 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465036 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:04:06.473026 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465038 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:04:06.473399 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465041 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:04:06.473399 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465044 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:04:06.473399 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465046 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:04:06.473399 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465049 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:04:06.473399 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465057 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:04:06.473399 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465060 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:04:06.473399 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465062 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:04:06.473399 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465065 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:04:06.473399 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465067 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:04:06.473399 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465070 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:04:06.473399 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465072 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:04:06.473399 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465075 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:04:06.473399 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465077 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:04:06.473399 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465080 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:04:06.473399 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465084 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:04:06.473399 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465088 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:04:06.473399 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465091 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:04:06.473399 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465094 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:04:06.473399 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465096 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:04:06.473869 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465099 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:04:06.473869 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465101 2580 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:04:06.473869 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465104 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:04:06.473869 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465106 2580 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:04:06.473869 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465109 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:04:06.473869 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465111 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:04:06.473869 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465113 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:04:06.473869 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465116 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:04:06.473869 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465119 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:04:06.473869 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465122 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:04:06.473869 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465124 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:04:06.473869 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465126 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:04:06.473869 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465129 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:04:06.473869 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465131 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:04:06.473869 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465134 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:04:06.473869 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465136 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:04:06.473869 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465139 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:04:06.473869 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465141 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:04:06.473869 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465149 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:04:06.473869 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465152 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:04:06.473869 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465154 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:04:06.474357 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465157 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:04:06.474357 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465159 2580 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:04:06.474357 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465162 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:04:06.474357 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465164 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:04:06.474357 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465167 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:04:06.474357 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465169 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:04:06.474357 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465172 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:04:06.474357 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465174 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:04:06.474357 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465177 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:04:06.474357 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465180 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:04:06.474357 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465182 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:04:06.474357 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465184 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:04:06.474357 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465187 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:04:06.474357 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465189 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:04:06.474357 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465192 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:04:06.474357 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465195 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:04:06.474357 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465197 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:04:06.474357 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465200 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:04:06.474357 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465202 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:04:06.474846 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465205 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:04:06.474846 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465207 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:04:06.474846 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465210 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:04:06.474846 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465212 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:04:06.474846 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465215 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:04:06.474846 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465217 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:04:06.474846 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465219 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:04:06.474846 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465222 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:04:06.474846 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465225 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:04:06.474846 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465227 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:04:06.474846 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465230 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:04:06.474846 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465238 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:04:06.474846 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465240 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:04:06.474846 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465243 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:04:06.474846 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465247 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:04:06.474846 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465250 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:04:06.474846 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465252 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:04:06.474846 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465255 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:04:06.474846 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465257 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:04:06.475280 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465259 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:04:06.475280 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465262 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:04:06.475280 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465264 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:04:06.475280 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465267 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:04:06.475280 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465269 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:04:06.475280 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465272 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:04:06.475280 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465275 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:04:06.475280 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465278 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:04:06.475280 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465281 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:04:06.475280 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465284 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:04:06.475280 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465287 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:04:06.475280 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465290 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:04:06.475280 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465293 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:04:06.475280 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465295 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:04:06.475280 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465298 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:04:06.475280 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465300 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:04:06.475280 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465302 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:04:06.475280 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465305 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:04:06.475280 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:06.465309 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:04:06.475782 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.465374 2580 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 13:04:06.475782 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.469287 2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 13:04:06.475782 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.470275 2580 server.go:1014] "Starting client certificate rotation" Nov 03 13:04:06.475782 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.470372 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 13:04:06.475782 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.470417 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 13:04:06.475782 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.473033 2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 13:04:06.476531 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.476463 2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 13:04:06.487520 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.487504 2580 log.go:25] "Validated CRI v1 runtime API" Nov 03 13:04:06.490922 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.490910 2580 log.go:25] "Validated CRI v1 image API" Nov 03 13:04:06.491831 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.491805 2580 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 13:04:06.496673 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.496653 2580 fs.go:135] Filesystem UUIDs: map[1724f46f-a6a0-4519-9880-204f9dc2cbfc:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 9ac2c955-d6f5-4042-b768-daf11305ec3d:/dev/nvme0n1p4] Nov 03 13:04:06.496730 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.496673 2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 13:04:06.502483 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.502356 2580 manager.go:217] Machine: {Timestamp:2026-11-03 13:04:06.500863786 +0000 UTC m=+0.352869966 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3119340 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25f2ad0e51ce23d76bf1e6f6fe8c75 SystemUUID:ec25f2ad-0e51-ce23-d76b-f1e6f6fe8c75 BootID:57b3a02b-23d5-4098-84e1-f37f4f8f675f Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:d7:16:ec:3f Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:d7:16:ec:3f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ce:d5:f3:32:f4:d3 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 13:04:06.503032 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.503021 2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 13:04:06.503122 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.503107 2580 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 13:04:06.506958 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.506935 2580 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 13:04:06.507087 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.506961 2580 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-198.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 13:04:06.507128 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.507089 2580 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 13:04:06.507128 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.507097 2580 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 13:04:06.507128 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.507108 2580 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 13:04:06.507128 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.507122 2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 13:04:06.507992 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.507982 2580 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 13:04:06.511836 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.511823 2580 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 13:04:06.511949 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.511939 2580 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 13:04:06.512667 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.512657 2580 kubelet.go:493] "Attempting to sync node with API server" Nov 03 13:04:06.512707 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.512671 2580 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 13:04:06.512707 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.512681 2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 13:04:06.512707 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.512691 2580 kubelet.go:405] "Adding apiserver pod source" Nov 03 13:04:06.512707 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.512699 2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 13:04:06.513202 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.513189 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 13:04:06.515525 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.515507 2580 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 13:04:06.522163 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.522147 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 13:04:06.523309 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.523298 2580 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 13:04:06.523363 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.523328 2580 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 13:04:06.523495 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.523484 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 13:04:06.523528 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.523501 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 13:04:06.523528 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.523508 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 13:04:06.523528 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.523514 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 13:04:06.523528 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.523520 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 13:04:06.523528 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.523525 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 13:04:06.524404 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.524396 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 13:04:06.524404 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.524405 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 13:04:06.524463 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.524412 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 13:04:06.524463 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.524418 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 13:04:06.524463 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.524427 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 13:04:06.524463 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.524437 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 13:04:06.526323 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.526313 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 13:04:06.526323 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.526324 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 13:04:06.530061 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.530038 2580 server.go:1263] "Started kubelet" Nov 03 13:04:06.530269 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.530216 2580 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 13:04:06.530362 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.530214 2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 13:04:06.530415 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.530369 2580 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 13:04:06.530936 ip-10-0-130-198 systemd[1]: Started Kubernetes Kubelet. Nov 03 13:04:06.531786 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.531766 2580 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 13:04:06.532995 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.532982 2580 server.go:317] "Adding debug handlers to kubelet server" Nov 03 13:04:06.538375 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:06.538358 2580 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 13:04:06.538756 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.538739 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 13:04:06.539279 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.539265 2580 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 13:04:06.539382 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:06.539362 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-198.us-west-2.compute.internal\" not found" Nov 03 13:04:06.539904 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.539885 2580 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 13:04:06.539904 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.539910 2580 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 13:04:06.540063 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.539887 2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 13:04:06.540108 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.540068 2580 reconstruct.go:97] "Volume reconstruction finished" Nov 03 13:04:06.540108 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.540079 2580 reconciler.go:29] "Reconciler: start to sync state" Nov 03 13:04:06.540278 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.540256 2580 factory.go:55] Registering systemd factory Nov 03 13:04:06.540278 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.540271 2580 factory.go:223] Registration of the systemd container factory successfully Nov 03 13:04:06.540465 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.540452 2580 factory.go:153] Registering CRI-O factory Nov 03 13:04:06.540465 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.540466 2580 factory.go:223] Registration of the crio container factory successfully Nov 03 13:04:06.540604 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.540576 2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 13:04:06.540604 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.540601 2580 factory.go:103] Registering Raw factory Nov 03 13:04:06.540695 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.540611 2580 manager.go:1199] Started watching for new ooms in manager Nov 03 13:04:06.541784 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.541768 2580 manager.go:319] Starting recovery of all containers Nov 03 13:04:06.551518 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.548083 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 13:04:06.553555 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.553519 2580 manager.go:324] Recovery completed Nov 03 13:04:06.557858 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:06.557289 2580 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Nov 03 13:04:06.560509 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.560492 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 13:04:06.562526 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.562504 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-198.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 13:04:06.562595 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.562546 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-198.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 13:04:06.562595 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.562557 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-198.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 13:04:06.563095 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.563081 2580 cpu_manager.go:226] "Starting" policy="none" Nov 03 13:04:06.563095 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.563092 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 13:04:06.563227 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.563094 2580 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 13:04:06.563227 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.563131 2580 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 13:04:06.563801 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.563786 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-198.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 13:04:06.563888 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.563814 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-198.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 13:04:06.563888 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.563827 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-198.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 13:04:06.565287 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.565273 2580 policy_none.go:50] "Start" Nov 03 13:04:06.565348 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.565290 2580 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 13:04:06.565348 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.565300 2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 13:04:06.567353 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.567340 2580 policy_none.go:44] "Start" Nov 03 13:04:06.607215 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.607179 2580 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 13:04:06.612781 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.612766 2580 manager.go:341] "Starting Device Plugin manager" Nov 03 13:04:06.612872 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:06.612799 2580 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 13:04:06.612872 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.612808 2580 server.go:87] "Starting device plugin registration server" Nov 03 13:04:06.613095 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.613084 2580 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 13:04:06.613136 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.613102 2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 13:04:06.613269 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.613255 2580 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 13:04:06.613380 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.613350 2580 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 13:04:06.613380 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.613364 2580 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 13:04:06.613907 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:06.613879 2580 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 13:04:06.614001 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:06.613949 2580 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-198.us-west-2.compute.internal\" not found" Nov 03 13:04:06.667533 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:06.667500 2580 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-198.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 13:04:06.669569 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.667612 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-198.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 13:04:06.669569 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:06.667501 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-198.us-west-2.compute.internal.18e48ab17e1d914f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-198.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-198.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 13:04:06.530011471 +0000 UTC m=+0.382017650,LastTimestamp:2026-11-03 13:04:06.530011471 +0000 UTC m=+0.382017650,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-198.us-west-2.compute.internal,}" Nov 03 13:04:06.676680 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.676653 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 13:04:06.676680 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.676683 2580 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 13:04:06.676826 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.676704 2580 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 13:04:06.676826 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:06.676743 2580 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 03 13:04:06.713530 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.713439 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 13:04:06.714389 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.714373 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-198.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 13:04:06.714457 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.714404 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-198.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 13:04:06.714457 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.714419 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-198.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 13:04:06.714457 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.714452 2580 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-198.us-west-2.compute.internal" Nov 03 13:04:06.757053 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.757022 2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 13:04:06.757192 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.757065 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-z9ddh" Nov 03 13:04:06.777102 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.777072 2580 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-198.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-198.us-west-2.compute.internal"] Nov 03 13:04:06.777239 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.777142 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 13:04:06.778499 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.778465 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-198.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 13:04:06.778582 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.778511 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-198.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 13:04:06.778582 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.778521 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-198.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 13:04:06.780695 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.780683 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 13:04:06.780841 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.780825 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-198.us-west-2.compute.internal" Nov 03 13:04:06.780881 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.780857 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 13:04:06.781395 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.781381 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-198.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 13:04:06.781451 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.781394 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-198.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 13:04:06.781451 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.781406 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-198.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 13:04:06.781451 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.781416 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-198.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 13:04:06.781451 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.781420 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-198.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 13:04:06.781451 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.781435 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-198.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 13:04:06.783709 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.783694 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-198.us-west-2.compute.internal" Nov 03 13:04:06.783751 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.783720 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 13:04:06.784430 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.784413 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-198.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 13:04:06.784495 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.784441 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-198.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 13:04:06.784495 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.784451 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-198.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 13:04:06.806830 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:06.806810 2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-198.us-west-2.compute.internal\" not found" node="ip-10-0-130-198.us-west-2.compute.internal" Nov 03 13:04:06.809599 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:06.809574 2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-198.us-west-2.compute.internal\" not found" node="ip-10-0-130-198.us-west-2.compute.internal" Nov 03 13:04:06.817602 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.817586 2580 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 13:04:06.819266 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.819253 2580 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 13:04:06.842075 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.842040 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f76315e331bcfab33ea20c406e740706-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-198.us-west-2.compute.internal\" (UID: \"f76315e331bcfab33ea20c406e740706\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-198.us-west-2.compute.internal" Nov 03 13:04:06.842175 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.842077 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/069487ba3512639ba31fb8a59a1ce2dd-config\") pod \"kube-apiserver-proxy-ip-10-0-130-198.us-west-2.compute.internal\" (UID: \"069487ba3512639ba31fb8a59a1ce2dd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-198.us-west-2.compute.internal" Nov 03 13:04:06.842175 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.842095 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f76315e331bcfab33ea20c406e740706-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-198.us-west-2.compute.internal\" (UID: \"f76315e331bcfab33ea20c406e740706\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-198.us-west-2.compute.internal" Nov 03 13:04:06.912428 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.912400 2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 13:04:06.913690 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.913672 2580 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-198.us-west-2.compute.internal" Nov 03 13:04:06.913774 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:06.913700 2580 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-198.us-west-2.compute.internal\": node \"ip-10-0-130-198.us-west-2.compute.internal\" not found" Nov 03 13:04:06.913774 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.913741 2580 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 13:04:06.939652 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.939629 2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-198.us-west-2.compute.internal" Nov 03 13:04:06.942891 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.942869 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f76315e331bcfab33ea20c406e740706-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-198.us-west-2.compute.internal\" (UID: \"f76315e331bcfab33ea20c406e740706\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-198.us-west-2.compute.internal" Nov 03 13:04:06.942965 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.942899 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f76315e331bcfab33ea20c406e740706-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-198.us-west-2.compute.internal\" (UID: \"f76315e331bcfab33ea20c406e740706\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-198.us-west-2.compute.internal" Nov 03 13:04:06.942965 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.942916 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/069487ba3512639ba31fb8a59a1ce2dd-config\") pod \"kube-apiserver-proxy-ip-10-0-130-198.us-west-2.compute.internal\" (UID: \"069487ba3512639ba31fb8a59a1ce2dd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-198.us-west-2.compute.internal" Nov 03 13:04:06.943062 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.942980 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/069487ba3512639ba31fb8a59a1ce2dd-config\") pod \"kube-apiserver-proxy-ip-10-0-130-198.us-west-2.compute.internal\" (UID: \"069487ba3512639ba31fb8a59a1ce2dd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-198.us-west-2.compute.internal" Nov 03 13:04:06.943062 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.942997 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f76315e331bcfab33ea20c406e740706-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-198.us-west-2.compute.internal\" (UID: \"f76315e331bcfab33ea20c406e740706\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-198.us-west-2.compute.internal" Nov 03 13:04:06.943062 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.942991 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f76315e331bcfab33ea20c406e740706-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-198.us-west-2.compute.internal\" (UID: \"f76315e331bcfab33ea20c406e740706\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-198.us-west-2.compute.internal" Nov 03 13:04:06.995645 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.995578 2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 13:04:06.996523 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:06.996510 2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-198.us-west-2.compute.internal" Nov 03 13:04:07.052025 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.052000 2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 13:04:07.111749 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.111723 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-198.us-west-2.compute.internal" Nov 03 13:04:07.113903 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.113866 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-198.us-west-2.compute.internal" Nov 03 13:04:07.469930 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.469744 2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 13:04:07.470477 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.470116 2580 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 13:04:07.470477 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.470137 2580 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 13:04:07.470477 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:07.470068 2580 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a16aeb6c627d247dd9f516a896748b1c-7174c5b4c9fb5a11.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-130-198.us-west-2.compute.internal.18e48ab1800dbff1\": read tcp 10.0.130.198:33826->3.22.160.232:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-130-198.us-west-2.compute.internal.18e48ab1800dbff1 default 5512 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-198.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-198.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-198.us-west-2.compute.internal,},FirstTimestamp:2026-11-03 13:04:06 +0000 UTC,LastTimestamp:2026-11-03 13:04:06.778495745 +0000 UTC m=+0.630501924,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-198.us-west-2.compute.internal,}" Nov 03 13:04:07.470477 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.470116 2580 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 13:04:07.515326 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.515268 2580 apiserver.go:52] "Watching apiserver" Nov 03 13:04:07.539428 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.539404 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 13:04:07.762597 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.762543 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 12:59:06 +0000 UTC" deadline="2028-08-06 19:03:11.31806624 +0000 UTC" Nov 03 13:04:07.762597 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.762588 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15413h59m3.555483487s" Nov 03 13:04:07.790631 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.790562 2580 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 13:04:07.790906 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.790885 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-zzbpd","openshift-multus/network-metrics-daemon-dczcf","openshift-network-diagnostics/network-check-target-9v5mk","openshift-ovn-kubernetes/ovnkube-node-frppp","kube-system/global-pull-secret-syncer-48vd8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-198.us-west-2.compute.internal","openshift-multus/multus-additional-cni-plugins-zzbqs","openshift-multus/multus-nkdmz","kube-system/kube-apiserver-proxy-ip-10-0-130-198.us-west-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4qjz","openshift-image-registry/node-ca-zrwrz","openshift-network-operator/iptables-alerter-999mg","kube-system/konnectivity-agent-ghgsf","openshift-cluster-node-tuning-operator/tuned-k78zw"] Nov 03 13:04:07.793477 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.793458 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zzbpd" Nov 03 13:04:07.795720 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.795699 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dczcf" Nov 03 13:04:07.795810 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:07.795785 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dczcf" podUID="8a9bc0be-51b3-4a52-83ae-580c3a2f342c" Nov 03 13:04:07.797964 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.797949 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9v5mk" Nov 03 13:04:07.798012 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:07.798002 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9v5mk" podUID="58a9c629-dff8-4f48-99b2-385d5d13e188" Nov 03 13:04:07.800245 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.800226 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:07.802476 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.802458 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48vd8" Nov 03 13:04:07.802561 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:07.802542 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48vd8" podUID="2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895" Nov 03 13:04:07.804939 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.804924 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zzbqs" Nov 03 13:04:07.807210 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.807196 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nkdmz" Nov 03 13:04:07.809599 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.809582 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4qjz" Nov 03 13:04:07.815447 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.815432 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zrwrz" Nov 03 13:04:07.817615 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.817601 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-999mg" Nov 03 13:04:07.819752 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.819735 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ghgsf" Nov 03 13:04:07.821718 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.821701 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:04:07.841219 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.841204 2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 13:04:07.848677 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.848661 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c663a29c-713d-4184-9105-e4a442a894c6-iptables-alerter-script\") pod \"iptables-alerter-999mg\" (UID: \"c663a29c-713d-4184-9105-e4a442a894c6\") " pod="openshift-network-operator/iptables-alerter-999mg" Nov 03 13:04:07.848743 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.848685 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c663a29c-713d-4184-9105-e4a442a894c6-host-slash\") pod \"iptables-alerter-999mg\" (UID: \"c663a29c-713d-4184-9105-e4a442a894c6\") " pod="openshift-network-operator/iptables-alerter-999mg" Nov 03 13:04:07.848743 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.848703 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-host-cni-bin\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:07.848743 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.848729 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-etc-systemd\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:04:07.848897 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.848826 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-systemd-units\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:07.848935 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.848910 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z298l\" (UniqueName: \"kubernetes.io/projected/0f7bd8dd-6293-414d-8b9d-064f04678e61-kube-api-access-z298l\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:07.848982 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.848948 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-host-var-lib-cni-multus\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:04:07.849023 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.848987 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-sys\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:04:07.849056 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.849042 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c7868eea-4b46-49e6-afb8-7fb4bb472fca-etc-tuned\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:04:07.849089 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.849059 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vdxd\" (UniqueName: \"kubernetes.io/projected/3083cbfe-7833-4b37-8325-cb18c20a7bea-kube-api-access-9vdxd\") pod \"node-resolver-zzbpd\" (UID: \"3083cbfe-7833-4b37-8325-cb18c20a7bea\") " pod="openshift-dns/node-resolver-zzbpd" Nov 03 13:04:07.849089 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.849085 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7-socket-dir\") pod \"aws-ebs-csi-driver-node-q4qjz\" (UID: \"d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4qjz" Nov 03 13:04:07.849164 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.849100 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-566w9\" (UniqueName: \"kubernetes.io/projected/c663a29c-713d-4184-9105-e4a442a894c6-kube-api-access-566w9\") pod \"iptables-alerter-999mg\" (UID: \"c663a29c-713d-4184-9105-e4a442a894c6\") " pod="openshift-network-operator/iptables-alerter-999mg" Nov 03 13:04:07.849164 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.849115 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-cnibin\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:04:07.849164 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.849143 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-host\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:04:07.849292 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.849194 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3083cbfe-7833-4b37-8325-cb18c20a7bea-hosts-file\") pod \"node-resolver-zzbpd\" (UID: \"3083cbfe-7833-4b37-8325-cb18c20a7bea\") " pod="openshift-dns/node-resolver-zzbpd" Nov 03 13:04:07.849292 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.849210 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjhfh\" (UniqueName: \"kubernetes.io/projected/8a9bc0be-51b3-4a52-83ae-580c3a2f342c-kube-api-access-xjhfh\") pod \"network-metrics-daemon-dczcf\" (UID: \"8a9bc0be-51b3-4a52-83ae-580c3a2f342c\") " pod="openshift-multus/network-metrics-daemon-dczcf" Nov 03 13:04:07.849292 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.849269 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7-etc-selinux\") pod \"aws-ebs-csi-driver-node-q4qjz\" (UID: \"d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4qjz" Nov 03 13:04:07.849452 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.849300 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-multus-cni-dir\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:04:07.849452 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.849327 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgwqj\" (UniqueName: \"kubernetes.io/projected/0ec0b81c-6052-45f2-bd5b-6096ebb72223-kube-api-access-sgwqj\") pod \"node-ca-zrwrz\" (UID: \"0ec0b81c-6052-45f2-bd5b-6096ebb72223\") " pod="openshift-image-registry/node-ca-zrwrz" Nov 03 13:04:07.849452 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.849355 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-etc-sysconfig\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:04:07.849452 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.849379 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-etc-sysctl-conf\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:04:07.849452 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.849414 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zzbqs\" (UID: \"06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1\") " pod="openshift-multus/multus-additional-cni-plugins-zzbqs" Nov 03 13:04:07.849452 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.849447 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7-sys-fs\") pod \"aws-ebs-csi-driver-node-q4qjz\" (UID: \"d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4qjz" Nov 03 13:04:07.849726 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.849516 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7-device-dir\") pod \"aws-ebs-csi-driver-node-q4qjz\" (UID: \"d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4qjz" Nov 03 13:04:07.849726 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.849543 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-host-slash\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:07.849726 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.849628 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-log-socket\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:07.849726 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.849643 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895-dbus\") pod \"global-pull-secret-syncer-48vd8\" (UID: \"2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895\") " pod="kube-system/global-pull-secret-syncer-48vd8" Nov 03 13:04:07.849726 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.849669 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a83707c3-8143-4aad-9131-59986a79968b-konnectivity-ca\") pod \"konnectivity-agent-ghgsf\" (UID: \"a83707c3-8143-4aad-9131-59986a79968b\") " pod="kube-system/konnectivity-agent-ghgsf" Nov 03 13:04:07.849726 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.849720 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-node-log\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:07.849905 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.849755 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1-system-cni-dir\") pod \"multus-additional-cni-plugins-zzbqs\" (UID: \"06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1\") " pod="openshift-multus/multus-additional-cni-plugins-zzbqs" Nov 03 13:04:07.849905 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.849815 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1-cnibin\") pod \"multus-additional-cni-plugins-zzbqs\" (UID: \"06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1\") " pod="openshift-multus/multus-additional-cni-plugins-zzbqs" Nov 03 13:04:07.849905 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.849888 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-host-var-lib-kubelet\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:04:07.850006 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.849954 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-multus-daemon-config\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:04:07.850006 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.849982 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0ec0b81c-6052-45f2-bd5b-6096ebb72223-serviceca\") pod \"node-ca-zrwrz\" (UID: \"0ec0b81c-6052-45f2-bd5b-6096ebb72223\") " pod="openshift-image-registry/node-ca-zrwrz" Nov 03 13:04:07.850087 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.850016 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3083cbfe-7833-4b37-8325-cb18c20a7bea-tmp-dir\") pod \"node-resolver-zzbpd\" (UID: \"3083cbfe-7833-4b37-8325-cb18c20a7bea\") " pod="openshift-dns/node-resolver-zzbpd" Nov 03 13:04:07.850087 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.850050 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtq2f\" (UniqueName: \"kubernetes.io/projected/d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7-kube-api-access-qtq2f\") pod \"aws-ebs-csi-driver-node-q4qjz\" (UID: \"d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4qjz" Nov 03 13:04:07.850170 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.850108 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzl6r\" (UniqueName: \"kubernetes.io/projected/58a9c629-dff8-4f48-99b2-385d5d13e188-kube-api-access-kzl6r\") pod \"network-check-target-9v5mk\" (UID: \"58a9c629-dff8-4f48-99b2-385d5d13e188\") " pod="openshift-network-diagnostics/network-check-target-9v5mk" Nov 03 13:04:07.850170 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.850136 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7-registration-dir\") pod \"aws-ebs-csi-driver-node-q4qjz\" (UID: \"d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4qjz" Nov 03 13:04:07.850246 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.850207 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-run-openvswitch\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:07.850294 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.850243 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0f7bd8dd-6293-414d-8b9d-064f04678e61-ovnkube-config\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:07.850294 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.850267 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-system-cni-dir\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:04:07.850392 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.850311 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-host-var-lib-cni-bin\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:04:07.850392 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.850341 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-multus-conf-dir\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:04:07.850392 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.850377 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a83707c3-8143-4aad-9131-59986a79968b-agent-certs\") pod \"konnectivity-agent-ghgsf\" (UID: \"a83707c3-8143-4aad-9131-59986a79968b\") " pod="kube-system/konnectivity-agent-ghgsf" Nov 03 13:04:07.850522 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.850404 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0f7bd8dd-6293-414d-8b9d-064f04678e61-env-overrides\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:07.850522 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.850421 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0f7bd8dd-6293-414d-8b9d-064f04678e61-ovn-node-metrics-cert\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:07.850522 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.850461 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-multus-socket-dir-parent\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:04:07.850651 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.850527 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-etc-openvswitch\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:07.850651 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.850557 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0f7bd8dd-6293-414d-8b9d-064f04678e61-ovnkube-script-lib\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:07.850651 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.850592 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895-original-pull-secret\") pod \"global-pull-secret-syncer-48vd8\" (UID: \"2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895\") " pod="kube-system/global-pull-secret-syncer-48vd8" Nov 03 13:04:07.850651 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.850609 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-host-run-netns\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:04:07.850651 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.850639 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd649\" (UniqueName: \"kubernetes.io/projected/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-kube-api-access-nd649\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:04:07.850835 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.850668 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-etc-modprobe-d\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:04:07.850835 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.850683 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c7868eea-4b46-49e6-afb8-7fb4bb472fca-tmp\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:04:07.850835 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.850698 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a9bc0be-51b3-4a52-83ae-580c3a2f342c-metrics-certs\") pod \"network-metrics-daemon-dczcf\" (UID: \"8a9bc0be-51b3-4a52-83ae-580c3a2f342c\") " pod="openshift-multus/network-metrics-daemon-dczcf" Nov 03 13:04:07.850835 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.850732 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-run-systemd\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:07.850835 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.850764 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:07.850835 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.850790 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-etc-sysctl-d\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:04:07.851045 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.850838 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-lib-modules\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:04:07.851045 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.850870 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-host-kubelet\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:07.851045 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.850896 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-host-run-netns\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:07.851045 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.850911 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-var-lib-openvswitch\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:07.851045 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.850936 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-host-cni-netd\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:07.851045 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.850968 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-run\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:04:07.851045 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.851014 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-var-lib-kubelet\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:04:07.851269 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.851068 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-run-ovn\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:07.851269 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.851093 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895-kubelet-config\") pod \"global-pull-secret-syncer-48vd8\" (UID: \"2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895\") " pod="kube-system/global-pull-secret-syncer-48vd8" Nov 03 13:04:07.851269 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.851124 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1-cni-binary-copy\") pod \"multus-additional-cni-plugins-zzbqs\" (UID: \"06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1\") " pod="openshift-multus/multus-additional-cni-plugins-zzbqs" Nov 03 13:04:07.851269 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.851145 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zzbqs\" (UID: \"06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1\") " pod="openshift-multus/multus-additional-cni-plugins-zzbqs" Nov 03 13:04:07.851269 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.851186 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-cni-binary-copy\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:04:07.851269 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.851217 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-host-run-k8s-cni-cncf-io\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:04:07.851269 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.851255 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-host-run-ovn-kubernetes\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:07.851646 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.851297 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-etc-kubernetes\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:04:07.851646 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.851327 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0ec0b81c-6052-45f2-bd5b-6096ebb72223-host\") pod \"node-ca-zrwrz\" (UID: \"0ec0b81c-6052-45f2-bd5b-6096ebb72223\") " pod="openshift-image-registry/node-ca-zrwrz" Nov 03 13:04:07.851646 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.851358 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-hostroot\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:04:07.851646 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.851382 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-host-run-multus-certs\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:04:07.851646 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.851408 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-etc-kubernetes\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:04:07.851646 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.851628 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q4qjz\" (UID: \"d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4qjz" Nov 03 13:04:07.851940 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.851658 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zzbqs\" (UID: \"06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1\") " pod="openshift-multus/multus-additional-cni-plugins-zzbqs" Nov 03 13:04:07.851940 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.851720 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj6m5\" (UniqueName: \"kubernetes.io/projected/06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1-kube-api-access-mj6m5\") pod \"multus-additional-cni-plugins-zzbqs\" (UID: \"06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1\") " pod="openshift-multus/multus-additional-cni-plugins-zzbqs" Nov 03 13:04:07.851940 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.851763 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8df59\" (UniqueName: \"kubernetes.io/projected/c7868eea-4b46-49e6-afb8-7fb4bb472fca-kube-api-access-8df59\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:04:07.851940 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.851798 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 13:04:07.851940 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.851856 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 13:04:07.852200 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.851941 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8v28q\"" Nov 03 13:04:07.854095 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.854071 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 13:04:07.854187 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.854110 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 13:04:07.854187 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.854117 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 13:04:07.854542 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.854310 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 13:04:07.854732 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.854717 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 13:04:07.855146 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.855110 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lbs9x\"" Nov 03 13:04:07.855496 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.855477 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 13:04:07.858779 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.858760 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 13:04:07.858878 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.858804 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 13:04:07.858878 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.858852 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 13:04:07.858990 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.858931 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 13:04:07.858990 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.858948 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 13:04:07.859067 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.858987 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-g4lr6\"" Nov 03 13:04:07.860138 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.860119 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 13:04:07.860405 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.860392 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-5ntlc\"" Nov 03 13:04:07.862795 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.862784 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 13:04:07.862948 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.862934 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 13:04:07.896339 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.896312 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 13:04:07.896339 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.896335 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 13:04:07.896503 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.896401 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 13:04:07.896503 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.896402 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-85kzz\"" Nov 03 13:04:07.897410 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.897389 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 13:04:07.897410 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.897388 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 13:04:07.897589 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.897498 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 13:04:07.903696 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.903682 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 13:04:07.904097 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.903976 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 13:04:07.904097 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.904014 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-s6pph\"" Nov 03 13:04:07.904097 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.904061 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5wxnl\"" Nov 03 13:04:07.904097 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.904077 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 13:04:07.904097 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.904088 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 13:04:07.904353 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.904094 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-l9fl5\"" Nov 03 13:04:07.904353 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.904059 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 13:04:07.904353 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.904120 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-sctm8\"" Nov 03 13:04:07.952156 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.952133 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-cni-binary-copy\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:04:07.952246 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.952161 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-host-run-k8s-cni-cncf-io\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:04:07.952246 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.952184 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-host-run-ovn-kubernetes\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:07.952246 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.952239 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-host-run-ovn-kubernetes\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:07.952380 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.952245 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-host-run-k8s-cni-cncf-io\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:04:07.952437 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.952373 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-etc-kubernetes\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:04:07.952437 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.952409 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0ec0b81c-6052-45f2-bd5b-6096ebb72223-host\") pod \"node-ca-zrwrz\" (UID: \"0ec0b81c-6052-45f2-bd5b-6096ebb72223\") " pod="openshift-image-registry/node-ca-zrwrz" Nov 03 13:04:07.952568 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.952440 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-hostroot\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:04:07.952568 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.952458 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-etc-kubernetes\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:04:07.952568 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.952467 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-host-run-multus-certs\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:04:07.952568 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.952502 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0ec0b81c-6052-45f2-bd5b-6096ebb72223-host\") pod \"node-ca-zrwrz\" (UID: \"0ec0b81c-6052-45f2-bd5b-6096ebb72223\") " pod="openshift-image-registry/node-ca-zrwrz" Nov 03 13:04:07.952568 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.952505 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-etc-kubernetes\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:04:07.952568 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.952533 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-host-run-multus-certs\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:04:07.952568 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.952550 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q4qjz\" (UID: \"d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4qjz" Nov 03 13:04:07.952892 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.952579 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zzbqs\" (UID: \"06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1\") " pod="openshift-multus/multus-additional-cni-plugins-zzbqs" Nov 03 13:04:07.952892 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.952587 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-etc-kubernetes\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:04:07.952892 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.952591 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-hostroot\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:04:07.952892 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.952606 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj6m5\" (UniqueName: \"kubernetes.io/projected/06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1-kube-api-access-mj6m5\") pod \"multus-additional-cni-plugins-zzbqs\" (UID: \"06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1\") " pod="openshift-multus/multus-additional-cni-plugins-zzbqs" Nov 03 13:04:07.952892 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.952632 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q4qjz\" (UID: \"d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4qjz" Nov 03 13:04:07.952892 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.952652 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8df59\" (UniqueName: \"kubernetes.io/projected/c7868eea-4b46-49e6-afb8-7fb4bb472fca-kube-api-access-8df59\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:04:07.952892 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.952683 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c663a29c-713d-4184-9105-e4a442a894c6-iptables-alerter-script\") pod \"iptables-alerter-999mg\" (UID: \"c663a29c-713d-4184-9105-e4a442a894c6\") " pod="openshift-network-operator/iptables-alerter-999mg" Nov 03 13:04:07.952892 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.952720 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c663a29c-713d-4184-9105-e4a442a894c6-host-slash\") pod \"iptables-alerter-999mg\" (UID: \"c663a29c-713d-4184-9105-e4a442a894c6\") " pod="openshift-network-operator/iptables-alerter-999mg" Nov 03 13:04:07.952892 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.952747 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-host-cni-bin\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:07.952892 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.952754 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zzbqs\" (UID: \"06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1\") " pod="openshift-multus/multus-additional-cni-plugins-zzbqs" Nov 03 13:04:07.952892 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.952773 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-etc-systemd\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:04:07.952892 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.952791 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-cni-binary-copy\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:04:07.952892 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.952817 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c663a29c-713d-4184-9105-e4a442a894c6-host-slash\") pod \"iptables-alerter-999mg\" (UID: \"c663a29c-713d-4184-9105-e4a442a894c6\") " pod="openshift-network-operator/iptables-alerter-999mg" Nov 03 13:04:07.952892 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.952840 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-etc-systemd\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:04:07.952892 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.952840 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-host-cni-bin\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:07.952892 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.952850 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-systemd-units\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:07.952892 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.952885 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-systemd-units\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:07.953682 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.952896 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z298l\" (UniqueName: \"kubernetes.io/projected/0f7bd8dd-6293-414d-8b9d-064f04678e61-kube-api-access-z298l\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:07.953682 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.952927 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-host-var-lib-cni-multus\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:04:07.953682 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.952985 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-sys\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:04:07.953682 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.953011 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c7868eea-4b46-49e6-afb8-7fb4bb472fca-etc-tuned\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:04:07.953682 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.953025 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-host-var-lib-cni-multus\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:04:07.953682 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.953048 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vdxd\" (UniqueName: \"kubernetes.io/projected/3083cbfe-7833-4b37-8325-cb18c20a7bea-kube-api-access-9vdxd\") pod \"node-resolver-zzbpd\" (UID: \"3083cbfe-7833-4b37-8325-cb18c20a7bea\") " pod="openshift-dns/node-resolver-zzbpd" Nov 03 13:04:07.953682 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.953074 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7-socket-dir\") pod \"aws-ebs-csi-driver-node-q4qjz\" (UID: \"d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4qjz" Nov 03 13:04:07.953682 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.953092 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-sys\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:04:07.953682 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.953101 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-566w9\" (UniqueName: \"kubernetes.io/projected/c663a29c-713d-4184-9105-e4a442a894c6-kube-api-access-566w9\") pod \"iptables-alerter-999mg\" (UID: \"c663a29c-713d-4184-9105-e4a442a894c6\") " pod="openshift-network-operator/iptables-alerter-999mg" Nov 03 13:04:07.953682 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.953218 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7-socket-dir\") pod \"aws-ebs-csi-driver-node-q4qjz\" (UID: \"d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4qjz" Nov 03 13:04:07.953682 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.953261 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-cnibin\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:04:07.953682 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.953290 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-host\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:04:07.953682 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.953315 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3083cbfe-7833-4b37-8325-cb18c20a7bea-hosts-file\") pod \"node-resolver-zzbpd\" (UID: \"3083cbfe-7833-4b37-8325-cb18c20a7bea\") " pod="openshift-dns/node-resolver-zzbpd" Nov 03 13:04:07.953682 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.953320 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c663a29c-713d-4184-9105-e4a442a894c6-iptables-alerter-script\") pod \"iptables-alerter-999mg\" (UID: \"c663a29c-713d-4184-9105-e4a442a894c6\") " pod="openshift-network-operator/iptables-alerter-999mg" Nov 03 13:04:07.953682 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.953342 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjhfh\" (UniqueName: \"kubernetes.io/projected/8a9bc0be-51b3-4a52-83ae-580c3a2f342c-kube-api-access-xjhfh\") pod \"network-metrics-daemon-dczcf\" (UID: \"8a9bc0be-51b3-4a52-83ae-580c3a2f342c\") " pod="openshift-multus/network-metrics-daemon-dczcf" Nov 03 13:04:07.953682 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.953386 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-cnibin\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:04:07.953682 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.953401 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7-etc-selinux\") pod \"aws-ebs-csi-driver-node-q4qjz\" (UID: \"d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4qjz" Nov 03 13:04:07.953682 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.953404 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3083cbfe-7833-4b37-8325-cb18c20a7bea-hosts-file\") pod \"node-resolver-zzbpd\" (UID: \"3083cbfe-7833-4b37-8325-cb18c20a7bea\") " pod="openshift-dns/node-resolver-zzbpd" Nov 03 13:04:07.954678 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.953414 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-host\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:04:07.954678 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.953432 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-multus-cni-dir\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:04:07.954678 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.953439 2580 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 13:04:07.954678 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.953459 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgwqj\" (UniqueName: \"kubernetes.io/projected/0ec0b81c-6052-45f2-bd5b-6096ebb72223-kube-api-access-sgwqj\") pod \"node-ca-zrwrz\" (UID: \"0ec0b81c-6052-45f2-bd5b-6096ebb72223\") " pod="openshift-image-registry/node-ca-zrwrz" Nov 03 13:04:07.954678 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.953485 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-etc-sysconfig\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:04:07.954678 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.953543 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7-etc-selinux\") pod \"aws-ebs-csi-driver-node-q4qjz\" (UID: \"d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4qjz" Nov 03 13:04:07.954678 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.953583 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-multus-cni-dir\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:04:07.954678 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.953597 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-etc-sysctl-conf\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:04:07.954678 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.953645 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-etc-sysconfig\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:04:07.954678 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.953649 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zzbqs\" (UID: \"06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1\") " pod="openshift-multus/multus-additional-cni-plugins-zzbqs" Nov 03 13:04:07.954678 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.953695 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7-sys-fs\") pod \"aws-ebs-csi-driver-node-q4qjz\" (UID: \"d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4qjz" Nov 03 13:04:07.954678 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.953742 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-etc-sysctl-conf\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:04:07.954678 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.953749 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7-device-dir\") pod \"aws-ebs-csi-driver-node-q4qjz\" (UID: \"d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4qjz" Nov 03 13:04:07.954678 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.953801 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7-device-dir\") pod \"aws-ebs-csi-driver-node-q4qjz\" (UID: \"d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4qjz" Nov 03 13:04:07.954678 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.953804 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-host-slash\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:07.954678 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.953822 2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 13:04:07.954678 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.953835 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-host-slash\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:07.954678 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.953845 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-log-socket\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:07.955743 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.953871 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895-dbus\") pod \"global-pull-secret-syncer-48vd8\" (UID: \"2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895\") " pod="kube-system/global-pull-secret-syncer-48vd8" Nov 03 13:04:07.955743 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.953891 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7-sys-fs\") pod \"aws-ebs-csi-driver-node-q4qjz\" (UID: \"d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4qjz" Nov 03 13:04:07.955743 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.953892 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-sr5s6" Nov 03 13:04:07.955743 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.953900 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a83707c3-8143-4aad-9131-59986a79968b-konnectivity-ca\") pod \"konnectivity-agent-ghgsf\" (UID: \"a83707c3-8143-4aad-9131-59986a79968b\") " pod="kube-system/konnectivity-agent-ghgsf" Nov 03 13:04:07.955743 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.953934 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-log-socket\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:07.955743 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.953959 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-node-log\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:07.955743 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.954008 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-node-log\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:07.955743 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.954026 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895-dbus\") pod \"global-pull-secret-syncer-48vd8\" (UID: \"2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895\") " pod="kube-system/global-pull-secret-syncer-48vd8" Nov 03 13:04:07.955743 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.954059 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1-system-cni-dir\") pod \"multus-additional-cni-plugins-zzbqs\" (UID: \"06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1\") " pod="openshift-multus/multus-additional-cni-plugins-zzbqs" Nov 03 13:04:07.955743 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.954086 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1-cnibin\") pod \"multus-additional-cni-plugins-zzbqs\" (UID: \"06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1\") " pod="openshift-multus/multus-additional-cni-plugins-zzbqs" Nov 03 13:04:07.955743 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.954094 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zzbqs\" (UID: \"06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1\") " pod="openshift-multus/multus-additional-cni-plugins-zzbqs" Nov 03 13:04:07.955743 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.954112 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-host-var-lib-kubelet\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:04:07.955743 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.954137 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-multus-daemon-config\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:04:07.955743 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.954166 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0ec0b81c-6052-45f2-bd5b-6096ebb72223-serviceca\") pod \"node-ca-zrwrz\" (UID: \"0ec0b81c-6052-45f2-bd5b-6096ebb72223\") " pod="openshift-image-registry/node-ca-zrwrz" Nov 03 13:04:07.955743 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.954174 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1-cnibin\") pod \"multus-additional-cni-plugins-zzbqs\" (UID: \"06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1\") " pod="openshift-multus/multus-additional-cni-plugins-zzbqs" Nov 03 13:04:07.955743 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.954190 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3083cbfe-7833-4b37-8325-cb18c20a7bea-tmp-dir\") pod \"node-resolver-zzbpd\" (UID: \"3083cbfe-7833-4b37-8325-cb18c20a7bea\") " pod="openshift-dns/node-resolver-zzbpd" Nov 03 13:04:07.955743 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.954216 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtq2f\" (UniqueName: \"kubernetes.io/projected/d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7-kube-api-access-qtq2f\") pod \"aws-ebs-csi-driver-node-q4qjz\" (UID: \"d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4qjz" Nov 03 13:04:07.955743 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.954231 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1-system-cni-dir\") pod \"multus-additional-cni-plugins-zzbqs\" (UID: \"06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1\") " pod="openshift-multus/multus-additional-cni-plugins-zzbqs" Nov 03 13:04:07.956674 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.954248 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzl6r\" (UniqueName: \"kubernetes.io/projected/58a9c629-dff8-4f48-99b2-385d5d13e188-kube-api-access-kzl6r\") pod \"network-check-target-9v5mk\" (UID: \"58a9c629-dff8-4f48-99b2-385d5d13e188\") " pod="openshift-network-diagnostics/network-check-target-9v5mk" Nov 03 13:04:07.956674 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.954276 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7-registration-dir\") pod \"aws-ebs-csi-driver-node-q4qjz\" (UID: \"d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4qjz" Nov 03 13:04:07.956674 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.954317 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-run-openvswitch\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:07.956674 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.954347 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0f7bd8dd-6293-414d-8b9d-064f04678e61-ovnkube-config\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:07.956674 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.954372 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-system-cni-dir\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:04:07.956674 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.954397 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-host-var-lib-cni-bin\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:04:07.956674 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.954422 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-multus-conf-dir\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:04:07.956674 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.954450 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a83707c3-8143-4aad-9131-59986a79968b-agent-certs\") pod \"konnectivity-agent-ghgsf\" (UID: \"a83707c3-8143-4aad-9131-59986a79968b\") " pod="kube-system/konnectivity-agent-ghgsf" Nov 03 13:04:07.956674 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.954475 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0f7bd8dd-6293-414d-8b9d-064f04678e61-env-overrides\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:07.956674 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.954500 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0f7bd8dd-6293-414d-8b9d-064f04678e61-ovn-node-metrics-cert\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:07.956674 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.954526 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-multus-socket-dir-parent\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:04:07.956674 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.954578 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-etc-openvswitch\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:07.956674 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.954604 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0f7bd8dd-6293-414d-8b9d-064f04678e61-ovnkube-script-lib\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:07.956674 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.954633 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895-original-pull-secret\") pod \"global-pull-secret-syncer-48vd8\" (UID: \"2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895\") " pod="kube-system/global-pull-secret-syncer-48vd8" Nov 03 13:04:07.956674 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.954629 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a83707c3-8143-4aad-9131-59986a79968b-konnectivity-ca\") pod \"konnectivity-agent-ghgsf\" (UID: \"a83707c3-8143-4aad-9131-59986a79968b\") " pod="kube-system/konnectivity-agent-ghgsf" Nov 03 13:04:07.956674 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.954680 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3083cbfe-7833-4b37-8325-cb18c20a7bea-tmp-dir\") pod \"node-resolver-zzbpd\" (UID: \"3083cbfe-7833-4b37-8325-cb18c20a7bea\") " pod="openshift-dns/node-resolver-zzbpd" Nov 03 13:04:07.956674 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.954713 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7-registration-dir\") pod \"aws-ebs-csi-driver-node-q4qjz\" (UID: \"d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4qjz" Nov 03 13:04:07.957709 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.954731 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0ec0b81c-6052-45f2-bd5b-6096ebb72223-serviceca\") pod \"node-ca-zrwrz\" (UID: \"0ec0b81c-6052-45f2-bd5b-6096ebb72223\") " pod="openshift-image-registry/node-ca-zrwrz" Nov 03 13:04:07.957709 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.954757 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-multus-daemon-config\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:04:07.957709 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.954765 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-run-openvswitch\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:07.957709 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.954823 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-host-var-lib-kubelet\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:04:07.957709 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.954841 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-host-run-netns\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:04:07.957709 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.954875 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-host-var-lib-cni-bin\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:04:07.957709 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.954882 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd649\" (UniqueName: \"kubernetes.io/projected/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-kube-api-access-nd649\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:04:07.957709 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.954907 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-etc-modprobe-d\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:04:07.957709 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.954933 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-host-run-netns\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:04:07.957709 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.954928 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c7868eea-4b46-49e6-afb8-7fb4bb472fca-tmp\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:04:07.957709 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.954975 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-etc-openvswitch\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:07.957709 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.954980 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a9bc0be-51b3-4a52-83ae-580c3a2f342c-metrics-certs\") pod \"network-metrics-daemon-dczcf\" (UID: \"8a9bc0be-51b3-4a52-83ae-580c3a2f342c\") " pod="openshift-multus/network-metrics-daemon-dczcf" Nov 03 13:04:07.957709 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.955007 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-run-systemd\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:07.957709 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.955034 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:07.957709 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.955062 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-etc-sysctl-d\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:04:07.957709 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.955087 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-lib-modules\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:04:07.957709 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.955098 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-multus-conf-dir\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:04:07.957709 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.955111 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-host-kubelet\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:07.958463 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.955136 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-host-run-netns\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:07.958463 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.955162 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-var-lib-openvswitch\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:07.958463 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.955204 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-host-cni-netd\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:07.958463 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.955230 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-run\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:04:07.958463 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.955255 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-var-lib-kubelet\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:04:07.958463 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.955281 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-run-ovn\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:07.958463 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.955311 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895-kubelet-config\") pod \"global-pull-secret-syncer-48vd8\" (UID: \"2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895\") " pod="kube-system/global-pull-secret-syncer-48vd8" Nov 03 13:04:07.958463 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.955340 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1-cni-binary-copy\") pod \"multus-additional-cni-plugins-zzbqs\" (UID: \"06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1\") " pod="openshift-multus/multus-additional-cni-plugins-zzbqs" Nov 03 13:04:07.958463 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.955367 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zzbqs\" (UID: \"06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1\") " pod="openshift-multus/multus-additional-cni-plugins-zzbqs" Nov 03 13:04:07.958463 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.955727 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0f7bd8dd-6293-414d-8b9d-064f04678e61-ovnkube-config\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:07.958463 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.955846 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-multus-socket-dir-parent\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:04:07.958463 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.956029 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zzbqs\" (UID: \"06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1\") " pod="openshift-multus/multus-additional-cni-plugins-zzbqs" Nov 03 13:04:07.958463 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.954937 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-system-cni-dir\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:04:07.958463 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:07.956125 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 13:04:07.958463 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.956172 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0f7bd8dd-6293-414d-8b9d-064f04678e61-env-overrides\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:07.958463 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.956240 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-etc-modprobe-d\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:04:07.958463 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.956250 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0f7bd8dd-6293-414d-8b9d-064f04678e61-ovnkube-script-lib\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:07.958463 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:07.956340 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 13:04:07.959175 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.956382 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:07.959175 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:07.956421 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895-original-pull-secret podName:2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895 nodeName:}" failed. No retries permitted until 2026-11-03 13:04:08.456382181 +0000 UTC m=+2.308388376 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895-original-pull-secret") pod "global-pull-secret-syncer-48vd8" (UID: "2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895") : object "kube-system"/"original-pull-secret" not registered Nov 03 13:04:07.959175 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.956460 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-var-lib-openvswitch\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:07.959175 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.956337 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-run-systemd\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:07.959175 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.956489 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-etc-sysctl-d\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:04:07.959175 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.956595 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-lib-modules\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:04:07.959175 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.956645 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-host-kubelet\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:07.959175 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.956670 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-host-run-netns\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:07.959175 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.956698 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-var-lib-kubelet\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:04:07.959175 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.956742 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895-kubelet-config\") pod \"global-pull-secret-syncer-48vd8\" (UID: \"2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895\") " pod="kube-system/global-pull-secret-syncer-48vd8" Nov 03 13:04:07.959175 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.956748 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-host-cni-netd\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:07.959175 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.956789 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-run\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:04:07.959175 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.956808 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-run-ovn\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:07.959175 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:07.956846 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a9bc0be-51b3-4a52-83ae-580c3a2f342c-metrics-certs podName:8a9bc0be-51b3-4a52-83ae-580c3a2f342c nodeName:}" failed. No retries permitted until 2026-11-03 13:04:08.456824743 +0000 UTC m=+2.308830916 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a9bc0be-51b3-4a52-83ae-580c3a2f342c-metrics-certs") pod "network-metrics-daemon-dczcf" (UID: "8a9bc0be-51b3-4a52-83ae-580c3a2f342c") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 13:04:07.959175 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.958038 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1-cni-binary-copy\") pod \"multus-additional-cni-plugins-zzbqs\" (UID: \"06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1\") " pod="openshift-multus/multus-additional-cni-plugins-zzbqs" Nov 03 13:04:07.959175 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.958094 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c7868eea-4b46-49e6-afb8-7fb4bb472fca-etc-tuned\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:04:07.959734 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.958237 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0f7bd8dd-6293-414d-8b9d-064f04678e61-ovn-node-metrics-cert\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:07.959734 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.959310 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c7868eea-4b46-49e6-afb8-7fb4bb472fca-tmp\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:04:07.959734 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:07.959603 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a83707c3-8143-4aad-9131-59986a79968b-agent-certs\") pod \"konnectivity-agent-ghgsf\" (UID: \"a83707c3-8143-4aad-9131-59986a79968b\") " pod="kube-system/konnectivity-agent-ghgsf" Nov 03 13:04:08.011773 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:08.011743 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 13:04:08.011773 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:08.011774 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 13:04:08.011986 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:08.011788 2580 projected.go:196] Error preparing data for projected volume kube-api-access-kzl6r for pod openshift-network-diagnostics/network-check-target-9v5mk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 13:04:08.011986 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:08.011867 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58a9c629-dff8-4f48-99b2-385d5d13e188-kube-api-access-kzl6r podName:58a9c629-dff8-4f48-99b2-385d5d13e188 nodeName:}" failed. No retries permitted until 2026-11-03 13:04:08.511847117 +0000 UTC m=+2.363853303 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-kzl6r" (UniqueName: "kubernetes.io/projected/58a9c629-dff8-4f48-99b2-385d5d13e188-kube-api-access-kzl6r") pod "network-check-target-9v5mk" (UID: "58a9c629-dff8-4f48-99b2-385d5d13e188") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 13:04:08.012593 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:08.012565 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj6m5\" (UniqueName: \"kubernetes.io/projected/06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1-kube-api-access-mj6m5\") pod \"multus-additional-cni-plugins-zzbqs\" (UID: \"06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1\") " pod="openshift-multus/multus-additional-cni-plugins-zzbqs" Nov 03 13:04:08.012801 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:08.012713 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgwqj\" (UniqueName: \"kubernetes.io/projected/0ec0b81c-6052-45f2-bd5b-6096ebb72223-kube-api-access-sgwqj\") pod \"node-ca-zrwrz\" (UID: \"0ec0b81c-6052-45f2-bd5b-6096ebb72223\") " pod="openshift-image-registry/node-ca-zrwrz" Nov 03 13:04:08.013241 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:08.013205 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vdxd\" (UniqueName: \"kubernetes.io/projected/3083cbfe-7833-4b37-8325-cb18c20a7bea-kube-api-access-9vdxd\") pod \"node-resolver-zzbpd\" (UID: \"3083cbfe-7833-4b37-8325-cb18c20a7bea\") " pod="openshift-dns/node-resolver-zzbpd" Nov 03 13:04:08.013543 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:08.013503 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-566w9\" (UniqueName: \"kubernetes.io/projected/c663a29c-713d-4184-9105-e4a442a894c6-kube-api-access-566w9\") pod \"iptables-alerter-999mg\" (UID: \"c663a29c-713d-4184-9105-e4a442a894c6\") " pod="openshift-network-operator/iptables-alerter-999mg" Nov 03 13:04:08.014019 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:08.013995 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8df59\" (UniqueName: \"kubernetes.io/projected/c7868eea-4b46-49e6-afb8-7fb4bb472fca-kube-api-access-8df59\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:04:08.015752 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:08.015718 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd649\" (UniqueName: \"kubernetes.io/projected/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-kube-api-access-nd649\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:04:08.015858 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:08.015748 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z298l\" (UniqueName: \"kubernetes.io/projected/0f7bd8dd-6293-414d-8b9d-064f04678e61-kube-api-access-z298l\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:08.015858 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:08.015782 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtq2f\" (UniqueName: \"kubernetes.io/projected/d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7-kube-api-access-qtq2f\") pod \"aws-ebs-csi-driver-node-q4qjz\" (UID: \"d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4qjz" Nov 03 13:04:08.015858 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:08.015778 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjhfh\" (UniqueName: \"kubernetes.io/projected/8a9bc0be-51b3-4a52-83ae-580c3a2f342c-kube-api-access-xjhfh\") pod \"network-metrics-daemon-dczcf\" (UID: \"8a9bc0be-51b3-4a52-83ae-580c3a2f342c\") " pod="openshift-multus/network-metrics-daemon-dczcf" Nov 03 13:04:08.139079 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:08.138998 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zzbpd" Nov 03 13:04:08.175090 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:08.175064 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:08.191782 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:08.191760 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zzbqs" Nov 03 13:04:08.191909 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:08.191848 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nkdmz" Nov 03 13:04:08.202461 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:08.202445 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4qjz" Nov 03 13:04:08.218780 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:08.218759 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zrwrz" Nov 03 13:04:08.240941 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:08.240912 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-999mg" Nov 03 13:04:08.258164 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:08.258126 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ghgsf" Nov 03 13:04:08.263097 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:08.263076 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:04:08.343485 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:08.343415 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf76315e331bcfab33ea20c406e740706.slice/crio-04419437a7c583e8767facf7b6dd8ac270e24b70dd856e7d9a1bcd0e8b71bb22 WatchSource:0}: Error finding container 04419437a7c583e8767facf7b6dd8ac270e24b70dd856e7d9a1bcd0e8b71bb22: Status 404 returned error can't find the container with id 04419437a7c583e8767facf7b6dd8ac270e24b70dd856e7d9a1bcd0e8b71bb22 Nov 03 13:04:08.350206 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:08.350186 2580 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 13:04:08.351038 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:08.351020 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 13:04:08.354488 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:08.354467 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod069487ba3512639ba31fb8a59a1ce2dd.slice/crio-6de1e4a21fe31cf1358127b395a713134fadd5a4cf986e8b1b3e4fe8cea83e27 WatchSource:0}: Error finding container 6de1e4a21fe31cf1358127b395a713134fadd5a4cf986e8b1b3e4fe8cea83e27: Status 404 returned error can't find the container with id 6de1e4a21fe31cf1358127b395a713134fadd5a4cf986e8b1b3e4fe8cea83e27 Nov 03 13:04:08.459726 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:08.459567 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895-original-pull-secret\") pod \"global-pull-secret-syncer-48vd8\" (UID: \"2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895\") " pod="kube-system/global-pull-secret-syncer-48vd8" Nov 03 13:04:08.459726 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:08.459710 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a9bc0be-51b3-4a52-83ae-580c3a2f342c-metrics-certs\") pod \"network-metrics-daemon-dczcf\" (UID: \"8a9bc0be-51b3-4a52-83ae-580c3a2f342c\") " pod="openshift-multus/network-metrics-daemon-dczcf" Nov 03 13:04:08.459963 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:08.459649 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 13:04:08.459963 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:08.459793 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895-original-pull-secret podName:2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895 nodeName:}" failed. No retries permitted until 2026-11-03 13:04:09.459779763 +0000 UTC m=+3.311785931 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895-original-pull-secret") pod "global-pull-secret-syncer-48vd8" (UID: "2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895") : object "kube-system"/"original-pull-secret" not registered Nov 03 13:04:08.459963 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:08.459829 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 13:04:08.459963 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:08.459873 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a9bc0be-51b3-4a52-83ae-580c3a2f342c-metrics-certs podName:8a9bc0be-51b3-4a52-83ae-580c3a2f342c nodeName:}" failed. No retries permitted until 2026-11-03 13:04:09.45986112 +0000 UTC m=+3.311867288 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a9bc0be-51b3-4a52-83ae-580c3a2f342c-metrics-certs") pod "network-metrics-daemon-dczcf" (UID: "8a9bc0be-51b3-4a52-83ae-580c3a2f342c") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 13:04:08.560285 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:08.560254 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzl6r\" (UniqueName: \"kubernetes.io/projected/58a9c629-dff8-4f48-99b2-385d5d13e188-kube-api-access-kzl6r\") pod \"network-check-target-9v5mk\" (UID: \"58a9c629-dff8-4f48-99b2-385d5d13e188\") " pod="openshift-network-diagnostics/network-check-target-9v5mk" Nov 03 13:04:08.560863 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:08.560381 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 13:04:08.560863 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:08.560395 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 13:04:08.560863 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:08.560407 2580 projected.go:196] Error preparing data for projected volume kube-api-access-kzl6r for pod openshift-network-diagnostics/network-check-target-9v5mk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 13:04:08.560863 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:08.560456 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58a9c629-dff8-4f48-99b2-385d5d13e188-kube-api-access-kzl6r podName:58a9c629-dff8-4f48-99b2-385d5d13e188 nodeName:}" failed. No retries permitted until 2026-11-03 13:04:09.5604437 +0000 UTC m=+3.412449869 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-kzl6r" (UniqueName: "kubernetes.io/projected/58a9c629-dff8-4f48-99b2-385d5d13e188-kube-api-access-kzl6r") pod "network-check-target-9v5mk" (UID: "58a9c629-dff8-4f48-99b2-385d5d13e188") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 13:04:08.681450 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:08.681395 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-198.us-west-2.compute.internal" event={"ID":"069487ba3512639ba31fb8a59a1ce2dd","Type":"ContainerStarted","Data":"6de1e4a21fe31cf1358127b395a713134fadd5a4cf986e8b1b3e4fe8cea83e27"} Nov 03 13:04:08.682468 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:08.682448 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-198.us-west-2.compute.internal" event={"ID":"f76315e331bcfab33ea20c406e740706","Type":"ContainerStarted","Data":"04419437a7c583e8767facf7b6dd8ac270e24b70dd856e7d9a1bcd0e8b71bb22"} Nov 03 13:04:08.795118 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:08.795042 2580 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 13:04:08.920927 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:08.920890 2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 13:04:08.960077 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:08.960024 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 12:59:07 +0000 UTC" deadline="2028-06-27 04:32:23.718603288 +0000 UTC" Nov 03 13:04:08.960077 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:08.960067 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14439h28m14.758539514s" Nov 03 13:04:09.129400 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:09.129366 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3083cbfe_7833_4b37_8325_cb18c20a7bea.slice/crio-9a91b9c57cf303820186cee25a3e09aa472d0822dbae53e22020ed5505ea4fa9 WatchSource:0}: Error finding container 9a91b9c57cf303820186cee25a3e09aa472d0822dbae53e22020ed5505ea4fa9: Status 404 returned error can't find the container with id 9a91b9c57cf303820186cee25a3e09aa472d0822dbae53e22020ed5505ea4fa9 Nov 03 13:04:09.144038 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:09.144015 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f7bd8dd_6293_414d_8b9d_064f04678e61.slice/crio-1ba727bf8ab0e0dbb6df93b3ac18c1f3f0af05d767a6a66729782a995c797fc0 WatchSource:0}: Error finding container 1ba727bf8ab0e0dbb6df93b3ac18c1f3f0af05d767a6a66729782a995c797fc0: Status 404 returned error can't find the container with id 1ba727bf8ab0e0dbb6df93b3ac18c1f3f0af05d767a6a66729782a995c797fc0 Nov 03 13:04:09.155641 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:09.155613 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8eeb7e5_6595_4f5d_9e5c_ce6348551ca7.slice/crio-ff1e220fab575099d63cac650b412e7123bbdba006089d74e309a9535e7600d1 WatchSource:0}: Error finding container ff1e220fab575099d63cac650b412e7123bbdba006089d74e309a9535e7600d1: Status 404 returned error can't find the container with id ff1e220fab575099d63cac650b412e7123bbdba006089d74e309a9535e7600d1 Nov 03 13:04:09.181799 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:09.181772 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06e0a591_ec3f_4cdc_bb51_13cc7d0f88e1.slice/crio-d665fd1da21957030f49e490695e1a57f3438cc4f5b6592c77dca06f03345d8a WatchSource:0}: Error finding container d665fd1da21957030f49e490695e1a57f3438cc4f5b6592c77dca06f03345d8a: Status 404 returned error can't find the container with id d665fd1da21957030f49e490695e1a57f3438cc4f5b6592c77dca06f03345d8a Nov 03 13:04:09.185664 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:09.185627 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cd62d34_ebc1_45cb_8a9c_6936d52bc7f0.slice/crio-97e7b232bd9bde5b04704bff068be4e42c0c8b238cbda87d3cf80eea9248717a WatchSource:0}: Error finding container 97e7b232bd9bde5b04704bff068be4e42c0c8b238cbda87d3cf80eea9248717a: Status 404 returned error can't find the container with id 97e7b232bd9bde5b04704bff068be4e42c0c8b238cbda87d3cf80eea9248717a Nov 03 13:04:09.188078 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:09.188053 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc663a29c_713d_4184_9105_e4a442a894c6.slice/crio-022573a872e57bdae8e145e998cc17c3d120a445311e823f943e50e7c76cc596 WatchSource:0}: Error finding container 022573a872e57bdae8e145e998cc17c3d120a445311e823f943e50e7c76cc596: Status 404 returned error can't find the container with id 022573a872e57bdae8e145e998cc17c3d120a445311e823f943e50e7c76cc596 Nov 03 13:04:09.211037 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:09.211011 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7868eea_4b46_49e6_afb8_7fb4bb472fca.slice/crio-87367e2b902feaa23d558c8f202838eaaec44f4bc0187081eed75391f425ae27 WatchSource:0}: Error finding container 87367e2b902feaa23d558c8f202838eaaec44f4bc0187081eed75391f425ae27: Status 404 returned error can't find the container with id 87367e2b902feaa23d558c8f202838eaaec44f4bc0187081eed75391f425ae27 Nov 03 13:04:09.251824 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:09.251776 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ec0b81c_6052_45f2_bd5b_6096ebb72223.slice/crio-400ccad898a899b91604c1a67253b559529031211d7f42dbae40a19cc322a55d WatchSource:0}: Error finding container 400ccad898a899b91604c1a67253b559529031211d7f42dbae40a19cc322a55d: Status 404 returned error can't find the container with id 400ccad898a899b91604c1a67253b559529031211d7f42dbae40a19cc322a55d Nov 03 13:04:09.273373 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:09.273346 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda83707c3_8143_4aad_9131_59986a79968b.slice/crio-4c54177f6ce7c0b369791011f7103908e9f3a45261bef440845bddd12613fbab WatchSource:0}: Error finding container 4c54177f6ce7c0b369791011f7103908e9f3a45261bef440845bddd12613fbab: Status 404 returned error can't find the container with id 4c54177f6ce7c0b369791011f7103908e9f3a45261bef440845bddd12613fbab Nov 03 13:04:09.467972 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:09.467856 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895-original-pull-secret\") pod \"global-pull-secret-syncer-48vd8\" (UID: \"2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895\") " pod="kube-system/global-pull-secret-syncer-48vd8" Nov 03 13:04:09.467972 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:09.467913 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a9bc0be-51b3-4a52-83ae-580c3a2f342c-metrics-certs\") pod \"network-metrics-daemon-dczcf\" (UID: \"8a9bc0be-51b3-4a52-83ae-580c3a2f342c\") " pod="openshift-multus/network-metrics-daemon-dczcf" Nov 03 13:04:09.468207 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:09.468084 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 13:04:09.468207 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:09.468086 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 13:04:09.468207 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:09.468156 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a9bc0be-51b3-4a52-83ae-580c3a2f342c-metrics-certs podName:8a9bc0be-51b3-4a52-83ae-580c3a2f342c nodeName:}" failed. No retries permitted until 2026-11-03 13:04:11.46813503 +0000 UTC m=+5.320141216 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a9bc0be-51b3-4a52-83ae-580c3a2f342c-metrics-certs") pod "network-metrics-daemon-dczcf" (UID: "8a9bc0be-51b3-4a52-83ae-580c3a2f342c") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 13:04:09.468207 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:09.468177 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895-original-pull-secret podName:2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895 nodeName:}" failed. No retries permitted until 2026-11-03 13:04:11.468165713 +0000 UTC m=+5.320171880 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895-original-pull-secret") pod "global-pull-secret-syncer-48vd8" (UID: "2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895") : object "kube-system"/"original-pull-secret" not registered Nov 03 13:04:09.569503 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:09.569204 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzl6r\" (UniqueName: \"kubernetes.io/projected/58a9c629-dff8-4f48-99b2-385d5d13e188-kube-api-access-kzl6r\") pod \"network-check-target-9v5mk\" (UID: \"58a9c629-dff8-4f48-99b2-385d5d13e188\") " pod="openshift-network-diagnostics/network-check-target-9v5mk" Nov 03 13:04:09.569503 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:09.569386 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 13:04:09.569503 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:09.569407 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 13:04:09.569503 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:09.569419 2580 projected.go:196] Error preparing data for projected volume kube-api-access-kzl6r for pod openshift-network-diagnostics/network-check-target-9v5mk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 13:04:09.569503 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:09.569471 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58a9c629-dff8-4f48-99b2-385d5d13e188-kube-api-access-kzl6r podName:58a9c629-dff8-4f48-99b2-385d5d13e188 nodeName:}" failed. No retries permitted until 2026-11-03 13:04:11.569452154 +0000 UTC m=+5.421458322 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-kzl6r" (UniqueName: "kubernetes.io/projected/58a9c629-dff8-4f48-99b2-385d5d13e188-kube-api-access-kzl6r") pod "network-check-target-9v5mk" (UID: "58a9c629-dff8-4f48-99b2-385d5d13e188") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 13:04:09.677415 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:09.677376 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9v5mk" Nov 03 13:04:09.677620 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:09.677521 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9v5mk" podUID="58a9c629-dff8-4f48-99b2-385d5d13e188" Nov 03 13:04:09.677692 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:09.677621 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48vd8" Nov 03 13:04:09.677735 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:09.677700 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48vd8" podUID="2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895" Nov 03 13:04:09.683924 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:09.678192 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dczcf" Nov 03 13:04:09.683924 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:09.678387 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dczcf" podUID="8a9bc0be-51b3-4a52-83ae-580c3a2f342c" Nov 03 13:04:09.687848 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:09.687815 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nkdmz" event={"ID":"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0","Type":"ContainerStarted","Data":"97e7b232bd9bde5b04704bff068be4e42c0c8b238cbda87d3cf80eea9248717a"} Nov 03 13:04:09.689824 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:09.689794 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4qjz" event={"ID":"d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7","Type":"ContainerStarted","Data":"ff1e220fab575099d63cac650b412e7123bbdba006089d74e309a9535e7600d1"} Nov 03 13:04:09.691313 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:09.691285 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zzbpd" event={"ID":"3083cbfe-7833-4b37-8325-cb18c20a7bea","Type":"ContainerStarted","Data":"9a91b9c57cf303820186cee25a3e09aa472d0822dbae53e22020ed5505ea4fa9"} Nov 03 13:04:09.692828 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:09.692797 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ghgsf" event={"ID":"a83707c3-8143-4aad-9131-59986a79968b","Type":"ContainerStarted","Data":"4c54177f6ce7c0b369791011f7103908e9f3a45261bef440845bddd12613fbab"} Nov 03 13:04:09.694483 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:09.694441 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zrwrz" event={"ID":"0ec0b81c-6052-45f2-bd5b-6096ebb72223","Type":"ContainerStarted","Data":"400ccad898a899b91604c1a67253b559529031211d7f42dbae40a19cc322a55d"} Nov 03 13:04:09.696022 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:09.695997 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-k78zw" event={"ID":"c7868eea-4b46-49e6-afb8-7fb4bb472fca","Type":"ContainerStarted","Data":"87367e2b902feaa23d558c8f202838eaaec44f4bc0187081eed75391f425ae27"} Nov 03 13:04:09.697764 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:09.697739 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-999mg" event={"ID":"c663a29c-713d-4184-9105-e4a442a894c6","Type":"ContainerStarted","Data":"022573a872e57bdae8e145e998cc17c3d120a445311e823f943e50e7c76cc596"} Nov 03 13:04:09.699000 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:09.698960 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zzbqs" event={"ID":"06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1","Type":"ContainerStarted","Data":"d665fd1da21957030f49e490695e1a57f3438cc4f5b6592c77dca06f03345d8a"} Nov 03 13:04:09.700169 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:09.700142 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-frppp" event={"ID":"0f7bd8dd-6293-414d-8b9d-064f04678e61","Type":"ContainerStarted","Data":"1ba727bf8ab0e0dbb6df93b3ac18c1f3f0af05d767a6a66729782a995c797fc0"} Nov 03 13:04:09.960775 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:09.960657 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 12:59:07 +0000 UTC" deadline="2028-05-31 00:05:07.884166001 +0000 UTC" Nov 03 13:04:09.960775 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:09.960725 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13787h0m57.923445219s" Nov 03 13:04:11.487371 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:11.486859 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895-original-pull-secret\") pod \"global-pull-secret-syncer-48vd8\" (UID: \"2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895\") " pod="kube-system/global-pull-secret-syncer-48vd8" Nov 03 13:04:11.487371 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:11.486941 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a9bc0be-51b3-4a52-83ae-580c3a2f342c-metrics-certs\") pod \"network-metrics-daemon-dczcf\" (UID: \"8a9bc0be-51b3-4a52-83ae-580c3a2f342c\") " pod="openshift-multus/network-metrics-daemon-dczcf" Nov 03 13:04:11.487371 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:11.487128 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 13:04:11.487371 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:11.487196 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a9bc0be-51b3-4a52-83ae-580c3a2f342c-metrics-certs podName:8a9bc0be-51b3-4a52-83ae-580c3a2f342c nodeName:}" failed. No retries permitted until 2026-11-03 13:04:15.4871754 +0000 UTC m=+9.339181589 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a9bc0be-51b3-4a52-83ae-580c3a2f342c-metrics-certs") pod "network-metrics-daemon-dczcf" (UID: "8a9bc0be-51b3-4a52-83ae-580c3a2f342c") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 13:04:11.487371 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:11.487277 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 13:04:11.487371 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:11.487315 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895-original-pull-secret podName:2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895 nodeName:}" failed. No retries permitted until 2026-11-03 13:04:15.487302706 +0000 UTC m=+9.339308880 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895-original-pull-secret") pod "global-pull-secret-syncer-48vd8" (UID: "2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895") : object "kube-system"/"original-pull-secret" not registered Nov 03 13:04:11.588878 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:11.588838 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzl6r\" (UniqueName: \"kubernetes.io/projected/58a9c629-dff8-4f48-99b2-385d5d13e188-kube-api-access-kzl6r\") pod \"network-check-target-9v5mk\" (UID: \"58a9c629-dff8-4f48-99b2-385d5d13e188\") " pod="openshift-network-diagnostics/network-check-target-9v5mk" Nov 03 13:04:11.589100 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:11.589081 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 13:04:11.589177 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:11.589108 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 13:04:11.589177 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:11.589127 2580 projected.go:196] Error preparing data for projected volume kube-api-access-kzl6r for pod openshift-network-diagnostics/network-check-target-9v5mk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 13:04:11.589298 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:11.589192 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58a9c629-dff8-4f48-99b2-385d5d13e188-kube-api-access-kzl6r podName:58a9c629-dff8-4f48-99b2-385d5d13e188 nodeName:}" failed. No retries permitted until 2026-11-03 13:04:15.589171358 +0000 UTC m=+9.441177557 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-kzl6r" (UniqueName: "kubernetes.io/projected/58a9c629-dff8-4f48-99b2-385d5d13e188-kube-api-access-kzl6r") pod "network-check-target-9v5mk" (UID: "58a9c629-dff8-4f48-99b2-385d5d13e188") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 13:04:11.677108 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:11.677034 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9v5mk" Nov 03 13:04:11.677264 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:11.677165 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9v5mk" podUID="58a9c629-dff8-4f48-99b2-385d5d13e188" Nov 03 13:04:11.677264 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:11.677235 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48vd8" Nov 03 13:04:11.677378 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:11.677306 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48vd8" podUID="2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895" Nov 03 13:04:11.677378 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:11.677359 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dczcf" Nov 03 13:04:11.677493 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:11.677433 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dczcf" podUID="8a9bc0be-51b3-4a52-83ae-580c3a2f342c" Nov 03 13:04:12.726245 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:12.725571 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-198.us-west-2.compute.internal" event={"ID":"069487ba3512639ba31fb8a59a1ce2dd","Type":"ContainerStarted","Data":"7c197bda48c555b29f8a01d33782f981e659a0e4c91acdbcaf266d0f5c32bb86"} Nov 03 13:04:12.737563 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:12.737441 2580 generic.go:356] "Generic (PLEG): container finished" podID="f76315e331bcfab33ea20c406e740706" containerID="fd188fb652c965ce60014fab5c8b8f58ee6de26cfd2185c7c01b9e6e39660839" exitCode=0 Nov 03 13:04:12.737563 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:12.737509 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-198.us-west-2.compute.internal" event={"ID":"f76315e331bcfab33ea20c406e740706","Type":"ContainerDied","Data":"fd188fb652c965ce60014fab5c8b8f58ee6de26cfd2185c7c01b9e6e39660839"} Nov 03 13:04:12.940227 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:12.938884 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-198.us-west-2.compute.internal" podStartSLOduration=5.938862736 podStartE2EDuration="5.938862736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:04:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:04:12.831872651 +0000 UTC m=+6.683878841" watchObservedRunningTime="2026-11-03 13:04:12.938862736 +0000 UTC m=+6.790868924" Nov 03 13:04:13.677488 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:13.676998 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9v5mk" Nov 03 13:04:13.677488 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:13.677133 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9v5mk" podUID="58a9c629-dff8-4f48-99b2-385d5d13e188" Nov 03 13:04:13.677488 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:13.677213 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48vd8" Nov 03 13:04:13.677488 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:13.677294 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48vd8" podUID="2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895" Nov 03 13:04:13.677488 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:13.677354 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dczcf" Nov 03 13:04:13.677488 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:13.677433 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dczcf" podUID="8a9bc0be-51b3-4a52-83ae-580c3a2f342c" Nov 03 13:04:13.748358 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:13.748277 2580 generic.go:356] "Generic (PLEG): container finished" podID="06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1" containerID="0f4eb43afa1d166d0ce1d07fe458aea5ffe18ce10220af393eec86b984a5783a" exitCode=0 Nov 03 13:04:13.748834 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:13.748374 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zzbqs" event={"ID":"06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1","Type":"ContainerDied","Data":"0f4eb43afa1d166d0ce1d07fe458aea5ffe18ce10220af393eec86b984a5783a"} Nov 03 13:04:13.754142 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:13.754113 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-198.us-west-2.compute.internal" event={"ID":"f76315e331bcfab33ea20c406e740706","Type":"ContainerStarted","Data":"8aa2894f700864fbc3427299e2cfb8e67e78daa6839ac1dc5ff467281a75861c"} Nov 03 13:04:13.969280 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:13.969184 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-198.us-west-2.compute.internal" podStartSLOduration=6.969163321 podStartE2EDuration="6.969163321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:04:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:04:13.968757672 +0000 UTC m=+7.820763862" watchObservedRunningTime="2026-11-03 13:04:13.969163321 +0000 UTC m=+7.821169509" Nov 03 13:04:14.759845 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:14.759593 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ghgsf" event={"ID":"a83707c3-8143-4aad-9131-59986a79968b","Type":"ContainerStarted","Data":"740f467352a892912bcaef57bc65f7992000248e564c16960e5ea45e18486398"} Nov 03 13:04:15.525630 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:15.525559 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895-original-pull-secret\") pod \"global-pull-secret-syncer-48vd8\" (UID: \"2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895\") " pod="kube-system/global-pull-secret-syncer-48vd8" Nov 03 13:04:15.525630 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:15.525608 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a9bc0be-51b3-4a52-83ae-580c3a2f342c-metrics-certs\") pod \"network-metrics-daemon-dczcf\" (UID: \"8a9bc0be-51b3-4a52-83ae-580c3a2f342c\") " pod="openshift-multus/network-metrics-daemon-dczcf" Nov 03 13:04:15.525846 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:15.525756 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 13:04:15.525846 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:15.525815 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a9bc0be-51b3-4a52-83ae-580c3a2f342c-metrics-certs podName:8a9bc0be-51b3-4a52-83ae-580c3a2f342c nodeName:}" failed. No retries permitted until 2026-11-03 13:04:23.525798538 +0000 UTC m=+17.377804707 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a9bc0be-51b3-4a52-83ae-580c3a2f342c-metrics-certs") pod "network-metrics-daemon-dczcf" (UID: "8a9bc0be-51b3-4a52-83ae-580c3a2f342c") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 13:04:15.525955 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:15.525884 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 13:04:15.525955 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:15.525917 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895-original-pull-secret podName:2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895 nodeName:}" failed. No retries permitted until 2026-11-03 13:04:23.525906178 +0000 UTC m=+17.377912349 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895-original-pull-secret") pod "global-pull-secret-syncer-48vd8" (UID: "2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895") : object "kube-system"/"original-pull-secret" not registered Nov 03 13:04:15.626000 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:15.625965 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzl6r\" (UniqueName: \"kubernetes.io/projected/58a9c629-dff8-4f48-99b2-385d5d13e188-kube-api-access-kzl6r\") pod \"network-check-target-9v5mk\" (UID: \"58a9c629-dff8-4f48-99b2-385d5d13e188\") " pod="openshift-network-diagnostics/network-check-target-9v5mk" Nov 03 13:04:15.626187 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:15.626114 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 13:04:15.626187 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:15.626137 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 13:04:15.626187 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:15.626149 2580 projected.go:196] Error preparing data for projected volume kube-api-access-kzl6r for pod openshift-network-diagnostics/network-check-target-9v5mk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 13:04:15.626361 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:15.626206 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58a9c629-dff8-4f48-99b2-385d5d13e188-kube-api-access-kzl6r podName:58a9c629-dff8-4f48-99b2-385d5d13e188 nodeName:}" failed. No retries permitted until 2026-11-03 13:04:23.626189662 +0000 UTC m=+17.478195847 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-kzl6r" (UniqueName: "kubernetes.io/projected/58a9c629-dff8-4f48-99b2-385d5d13e188-kube-api-access-kzl6r") pod "network-check-target-9v5mk" (UID: "58a9c629-dff8-4f48-99b2-385d5d13e188") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 13:04:15.676956 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:15.676919 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48vd8" Nov 03 13:04:15.677130 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:15.677060 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48vd8" podUID="2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895" Nov 03 13:04:15.677130 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:15.677124 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dczcf" Nov 03 13:04:15.677272 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:15.677218 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dczcf" podUID="8a9bc0be-51b3-4a52-83ae-580c3a2f342c" Nov 03 13:04:15.677272 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:15.677264 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9v5mk" Nov 03 13:04:15.677375 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:15.677325 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9v5mk" podUID="58a9c629-dff8-4f48-99b2-385d5d13e188" Nov 03 13:04:15.772637 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:15.772605 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zrwrz" event={"ID":"0ec0b81c-6052-45f2-bd5b-6096ebb72223","Type":"ContainerStarted","Data":"3b1f71c85e765dd399f0f057eebb20e71e448cbaa86b09f7b47a63bc1edb42a6"} Nov 03 13:04:15.885827 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:15.885765 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ghgsf" podStartSLOduration=4.136465978 podStartE2EDuration="8.885747983s" totalImagesPullingTime="4.749282005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:04:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:04:14.867465765 +0000 UTC m=+8.719471955" watchObservedRunningTime="2026-11-03 13:04:15.885747983 +0000 UTC m=+9.737754172" Nov 03 13:04:16.783899 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:16.783824 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4qjz" event={"ID":"d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7","Type":"ContainerStarted","Data":"f336a31ba86661bcd9ebd4da423a49a6440c2da07bda0251b9d820c8b89f4d4a"} Nov 03 13:04:17.678107 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:17.677732 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9v5mk" Nov 03 13:04:17.678107 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:17.677849 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9v5mk" podUID="58a9c629-dff8-4f48-99b2-385d5d13e188" Nov 03 13:04:17.678107 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:17.677930 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48vd8" Nov 03 13:04:17.678107 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:17.677977 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48vd8" podUID="2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895" Nov 03 13:04:17.678107 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:17.678017 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dczcf" Nov 03 13:04:17.678107 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:17.678064 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dczcf" podUID="8a9bc0be-51b3-4a52-83ae-580c3a2f342c" Nov 03 13:04:17.792481 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:17.791709 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nkdmz" event={"ID":"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0","Type":"ContainerStarted","Data":"4d718c23dcb482c96e96c22f61d605bec81b82bc8ef3ee9d6764a474d7d4fe0b"} Nov 03 13:04:17.931964 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:17.931907 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zrwrz" podStartSLOduration=5.426472867 podStartE2EDuration="10.931891221s" totalImagesPullingTime="5.505418354s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:04:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:04:15.886520718 +0000 UTC m=+9.738526908" watchObservedRunningTime="2026-11-03 13:04:17.931891221 +0000 UTC m=+11.783897403" Nov 03 13:04:17.932715 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:17.932671 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-nkdmz" podStartSLOduration=2.575042372 podStartE2EDuration="10.932659082s" totalImagesPullingTime="8.35761671s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:04:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:04:17.931796667 +0000 UTC m=+11.783802855" watchObservedRunningTime="2026-11-03 13:04:17.932659082 +0000 UTC m=+11.784665271" Nov 03 13:04:18.008800 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:18.008774 2580 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 13:04:18.173182 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:18.173153 2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ghgsf" Nov 03 13:04:18.174006 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:18.173988 2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ghgsf" Nov 03 13:04:18.619801 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:18.619639 2580 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T13:04:18.008797377Z","UUID":"8210d703-40e3-47fe-a39f-3f029f608e0b","Handler":null,"Name":"","Endpoint":""} Nov 03 13:04:18.621233 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:18.621202 2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 13:04:18.621359 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:18.621243 2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 13:04:18.795851 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:18.795809 2580 generic.go:356] "Generic (PLEG): container finished" podID="06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1" containerID="706a116018ce9b1048168c4649d08853a7aa36dbc181cf7a6861de402fce6f41" exitCode=0 Nov 03 13:04:18.796344 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:18.796103 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zzbqs" event={"ID":"06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1","Type":"ContainerDied","Data":"706a116018ce9b1048168c4649d08853a7aa36dbc181cf7a6861de402fce6f41"} Nov 03 13:04:18.798809 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:18.798744 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4qjz" event={"ID":"d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7","Type":"ContainerStarted","Data":"c528215ac54bba0633e9b3287a8f00b4fa531f7b40cf0308e3ba741e5183595b"} Nov 03 13:04:18.800446 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:18.800424 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zzbpd" event={"ID":"3083cbfe-7833-4b37-8325-cb18c20a7bea","Type":"ContainerStarted","Data":"c0ea74a8e912595ad35e6111ccf758075afc32c80eecad181de8f754a118ef7d"} Nov 03 13:04:18.800943 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:18.800926 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ghgsf" Nov 03 13:04:18.801273 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:18.801255 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ghgsf" Nov 03 13:04:19.015209 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:19.014957 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-zzbpd" podStartSLOduration=3.255824391 podStartE2EDuration="12.014939066s" totalImagesPullingTime="8.759114675s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:04:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:04:19.014849177 +0000 UTC m=+12.866855364" watchObservedRunningTime="2026-11-03 13:04:19.014939066 +0000 UTC m=+12.866945311" Nov 03 13:04:19.677263 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:19.677237 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9v5mk" Nov 03 13:04:19.677263 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:19.677259 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48vd8" Nov 03 13:04:19.677401 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:19.677342 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48vd8" podUID="2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895" Nov 03 13:04:19.677488 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:19.677400 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9v5mk" podUID="58a9c629-dff8-4f48-99b2-385d5d13e188" Nov 03 13:04:19.677488 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:19.677436 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dczcf" Nov 03 13:04:19.677592 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:19.677506 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dczcf" podUID="8a9bc0be-51b3-4a52-83ae-580c3a2f342c" Nov 03 13:04:19.804513 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:19.804424 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-999mg" event={"ID":"c663a29c-713d-4184-9105-e4a442a894c6","Type":"ContainerStarted","Data":"3dc93409169d4e4a117eae491555acb35d05148304176d624e3dac2ccc41f3ba"} Nov 03 13:04:19.807325 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:19.807297 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4qjz" event={"ID":"d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7","Type":"ContainerStarted","Data":"19eb60ba39a8449ac73b43a22d7c6b3f02b141206b4ed46c5ff6fec51e4fa799"} Nov 03 13:04:20.811624 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:20.811529 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-k78zw" event={"ID":"c7868eea-4b46-49e6-afb8-7fb4bb472fca","Type":"ContainerStarted","Data":"cf1b270276e822cb6a89a692012ba1912ac4d91f8c5fdadba44d1046810c9a7b"} Nov 03 13:04:20.813771 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:20.813742 2580 generic.go:356] "Generic (PLEG): container finished" podID="06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1" containerID="ced9e3a3b5ed9dbbb36de460971869c91ea88503569f70e4147e98c99cb277f3" exitCode=0 Nov 03 13:04:20.813895 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:20.813853 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zzbqs" event={"ID":"06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1","Type":"ContainerDied","Data":"ced9e3a3b5ed9dbbb36de460971869c91ea88503569f70e4147e98c99cb277f3"} Nov 03 13:04:20.919791 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:20.919737 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-999mg" podStartSLOduration=5.122351571 podStartE2EDuration="13.919720045s" totalImagesPullingTime="8.797368474s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:04:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:04:19.914674931 +0000 UTC m=+13.766681118" watchObservedRunningTime="2026-11-03 13:04:20.919720045 +0000 UTC m=+14.771726232" Nov 03 13:04:20.919962 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:20.919941 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-k78zw" podStartSLOduration=2.911540875 podStartE2EDuration="13.919932867s" totalImagesPullingTime="11.008391992s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:04:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:04:20.919831529 +0000 UTC m=+14.771837699" watchObservedRunningTime="2026-11-03 13:04:20.919932867 +0000 UTC m=+14.771939052" Nov 03 13:04:21.139207 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:21.139101 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4qjz" podStartSLOduration=4.023235665 podStartE2EDuration="14.139080502s" totalImagesPullingTime="10.115844837s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:04:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:04:21.138315608 +0000 UTC m=+14.990321799" watchObservedRunningTime="2026-11-03 13:04:21.139080502 +0000 UTC m=+14.991086693" Nov 03 13:04:21.676991 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:21.676959 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9v5mk" Nov 03 13:04:21.677163 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:21.676968 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48vd8" Nov 03 13:04:21.677163 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:21.677072 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9v5mk" podUID="58a9c629-dff8-4f48-99b2-385d5d13e188" Nov 03 13:04:21.677163 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:21.677102 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dczcf" Nov 03 13:04:21.677292 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:21.677170 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48vd8" podUID="2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895" Nov 03 13:04:21.677292 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:21.677255 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dczcf" podUID="8a9bc0be-51b3-4a52-83ae-580c3a2f342c" Nov 03 13:04:22.822162 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:22.821871 2580 generic.go:356] "Generic (PLEG): container finished" podID="06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1" containerID="ad81f969e0651c5fae5288d9b5e8b5bb83c666efb3d8a19cf99e91bb830a6605" exitCode=0 Nov 03 13:04:22.822162 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:22.821979 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zzbqs" event={"ID":"06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1","Type":"ContainerDied","Data":"ad81f969e0651c5fae5288d9b5e8b5bb83c666efb3d8a19cf99e91bb830a6605"} Nov 03 13:04:23.590490 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:23.590441 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895-original-pull-secret\") pod \"global-pull-secret-syncer-48vd8\" (UID: \"2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895\") " pod="kube-system/global-pull-secret-syncer-48vd8" Nov 03 13:04:23.590663 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:23.590515 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a9bc0be-51b3-4a52-83ae-580c3a2f342c-metrics-certs\") pod \"network-metrics-daemon-dczcf\" (UID: \"8a9bc0be-51b3-4a52-83ae-580c3a2f342c\") " pod="openshift-multus/network-metrics-daemon-dczcf" Nov 03 13:04:23.590663 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:23.590582 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 13:04:23.590663 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:23.590646 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895-original-pull-secret podName:2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895 nodeName:}" failed. No retries permitted until 2026-11-03 13:04:39.590630656 +0000 UTC m=+33.442636825 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895-original-pull-secret") pod "global-pull-secret-syncer-48vd8" (UID: "2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895") : object "kube-system"/"original-pull-secret" not registered Nov 03 13:04:23.590663 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:23.590645 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 13:04:23.590830 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:23.590699 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a9bc0be-51b3-4a52-83ae-580c3a2f342c-metrics-certs podName:8a9bc0be-51b3-4a52-83ae-580c3a2f342c nodeName:}" failed. No retries permitted until 2026-11-03 13:04:39.590681716 +0000 UTC m=+33.442687902 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a9bc0be-51b3-4a52-83ae-580c3a2f342c-metrics-certs") pod "network-metrics-daemon-dczcf" (UID: "8a9bc0be-51b3-4a52-83ae-580c3a2f342c") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 13:04:23.677410 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:23.677378 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9v5mk" Nov 03 13:04:23.677578 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:23.677443 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dczcf" Nov 03 13:04:23.677578 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:23.677566 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48vd8" Nov 03 13:04:23.677691 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:23.677584 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9v5mk" podUID="58a9c629-dff8-4f48-99b2-385d5d13e188" Nov 03 13:04:23.677691 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:23.677674 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48vd8" podUID="2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895" Nov 03 13:04:23.677766 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:23.677749 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dczcf" podUID="8a9bc0be-51b3-4a52-83ae-580c3a2f342c" Nov 03 13:04:23.691132 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:23.691106 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzl6r\" (UniqueName: \"kubernetes.io/projected/58a9c629-dff8-4f48-99b2-385d5d13e188-kube-api-access-kzl6r\") pod \"network-check-target-9v5mk\" (UID: \"58a9c629-dff8-4f48-99b2-385d5d13e188\") " pod="openshift-network-diagnostics/network-check-target-9v5mk" Nov 03 13:04:23.691271 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:23.691252 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 13:04:23.691321 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:23.691278 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 13:04:23.691321 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:23.691293 2580 projected.go:196] Error preparing data for projected volume kube-api-access-kzl6r for pod openshift-network-diagnostics/network-check-target-9v5mk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 13:04:23.691403 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:23.691350 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58a9c629-dff8-4f48-99b2-385d5d13e188-kube-api-access-kzl6r podName:58a9c629-dff8-4f48-99b2-385d5d13e188 nodeName:}" failed. No retries permitted until 2026-11-03 13:04:39.691332686 +0000 UTC m=+33.543338858 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-kzl6r" (UniqueName: "kubernetes.io/projected/58a9c629-dff8-4f48-99b2-385d5d13e188-kube-api-access-kzl6r") pod "network-check-target-9v5mk" (UID: "58a9c629-dff8-4f48-99b2-385d5d13e188") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 13:04:24.829549 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:24.829372 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-frppp" event={"ID":"0f7bd8dd-6293-414d-8b9d-064f04678e61","Type":"ContainerStarted","Data":"3f47f9e59391d43568ca334a635eb3f8dbe6dc0dd217d00dd0c8af193d2070d0"} Nov 03 13:04:24.829896 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:24.829586 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-frppp" event={"ID":"0f7bd8dd-6293-414d-8b9d-064f04678e61","Type":"ContainerStarted","Data":"27830a456a723033942900bfb3a21b61537189216936a024dda9c342b99acfd7"} Nov 03 13:04:24.829896 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:24.829618 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-frppp" event={"ID":"0f7bd8dd-6293-414d-8b9d-064f04678e61","Type":"ContainerStarted","Data":"b6d13a5f855beab118b3287531731e0661cea487b717456747e671b6fe7b2024"} Nov 03 13:04:24.829896 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:24.829650 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-frppp" event={"ID":"0f7bd8dd-6293-414d-8b9d-064f04678e61","Type":"ContainerStarted","Data":"2e5c69132368b9972c96e9a94340ede6ecfb551ca4c9ef08d1d0f0d6a689789c"} Nov 03 13:04:24.829896 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:24.829675 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-frppp" event={"ID":"0f7bd8dd-6293-414d-8b9d-064f04678e61","Type":"ContainerStarted","Data":"14f6c817f3d0be4fce76a6149037840be14fa9a3d71bd3e074246a249f0d07a7"} Nov 03 13:04:24.829896 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:24.829703 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-frppp" event={"ID":"0f7bd8dd-6293-414d-8b9d-064f04678e61","Type":"ContainerStarted","Data":"34dfd8f76fca3d53d65307880ff5657b3083c9f77069cf3a4c47750276e26ed1"} Nov 03 13:04:25.677649 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:25.677603 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9v5mk" Nov 03 13:04:25.677649 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:25.677644 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48vd8" Nov 03 13:04:25.677873 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:25.677654 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dczcf" Nov 03 13:04:25.677873 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:25.677712 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9v5mk" podUID="58a9c629-dff8-4f48-99b2-385d5d13e188" Nov 03 13:04:25.677873 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:25.677802 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48vd8" podUID="2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895" Nov 03 13:04:25.677971 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:25.677886 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dczcf" podUID="8a9bc0be-51b3-4a52-83ae-580c3a2f342c" Nov 03 13:04:25.833708 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:25.833679 2580 generic.go:356] "Generic (PLEG): container finished" podID="06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1" containerID="a3532f7d52d5b86e95969d5e54d622b946b8fee936c164924abfd55520ccfc3d" exitCode=0 Nov 03 13:04:25.833708 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:25.833715 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zzbqs" event={"ID":"06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1","Type":"ContainerDied","Data":"a3532f7d52d5b86e95969d5e54d622b946b8fee936c164924abfd55520ccfc3d"} Nov 03 13:04:26.839576 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:26.839538 2580 generic.go:356] "Generic (PLEG): container finished" podID="06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1" containerID="4ae8b45c1c72ab262def8c944455f637b7a80a201fc6b07a9e1a8ecb06bd9541" exitCode=0 Nov 03 13:04:26.840182 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:26.839583 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zzbqs" event={"ID":"06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1","Type":"ContainerDied","Data":"4ae8b45c1c72ab262def8c944455f637b7a80a201fc6b07a9e1a8ecb06bd9541"} Nov 03 13:04:26.843196 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:26.843152 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-frppp" event={"ID":"0f7bd8dd-6293-414d-8b9d-064f04678e61","Type":"ContainerStarted","Data":"c0091749f6a2f4302dda39465679c47bfa34846f686f33250b9227502c1912d6"} Nov 03 13:04:27.677680 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:27.677647 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48vd8" Nov 03 13:04:27.677680 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:27.677670 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9v5mk" Nov 03 13:04:27.677871 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:27.677764 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48vd8" podUID="2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895" Nov 03 13:04:27.677871 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:27.677806 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9v5mk" podUID="58a9c629-dff8-4f48-99b2-385d5d13e188" Nov 03 13:04:27.677871 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:27.677839 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dczcf" Nov 03 13:04:27.677965 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:27.677933 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dczcf" podUID="8a9bc0be-51b3-4a52-83ae-580c3a2f342c" Nov 03 13:04:27.816652 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:27.816614 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ghgsf_a83707c3-8143-4aad-9131-59986a79968b/konnectivity-agent/0.log" limit=1024 Nov 03 13:04:27.850902 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:27.850869 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zzbqs" event={"ID":"06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1","Type":"ContainerStarted","Data":"0bf6c6b0abbd66c57273422450f7f1715f83a14b3011e9b079b4d4c76055e891"} Nov 03 13:04:28.010888 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:28.010788 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-zzbqs" podStartSLOduration=7.678793494 podStartE2EDuration="21.010775646s" totalImagesPullingTime="13.331982152s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:04:07 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:04:28.010532149 +0000 UTC m=+21.862538340" watchObservedRunningTime="2026-11-03 13:04:28.010775646 +0000 UTC m=+21.862781834" Nov 03 13:04:28.856942 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:28.856918 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-frppp" event={"ID":"0f7bd8dd-6293-414d-8b9d-064f04678e61","Type":"ContainerStarted","Data":"08d1d3deadd3502bf9f80e7933d022d7910881b60eec7655efb553df6b5c9416"} Nov 03 13:04:29.677661 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:29.677490 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48vd8" Nov 03 13:04:29.677799 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:29.677532 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dczcf" Nov 03 13:04:29.677799 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:29.677728 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48vd8" podUID="2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895" Nov 03 13:04:29.677860 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:29.677818 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dczcf" podUID="8a9bc0be-51b3-4a52-83ae-580c3a2f342c" Nov 03 13:04:29.677860 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:29.677541 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9v5mk" Nov 03 13:04:29.677935 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:29.677898 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9v5mk" podUID="58a9c629-dff8-4f48-99b2-385d5d13e188" Nov 03 13:04:29.859106 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:29.859074 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:29.859554 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:29.859155 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:29.859554 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:29.859240 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:29.874894 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:29.874868 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:29.877891 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:29.877875 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:04:30.117577 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:30.117449 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-frppp" podStartSLOduration=7.912494775 podStartE2EDuration="23.117434075s" totalImagesPullingTime="15.2049393s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:04:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:04:30.099008172 +0000 UTC m=+23.951014359" watchObservedRunningTime="2026-11-03 13:04:30.117434075 +0000 UTC m=+23.969440263" Nov 03 13:04:31.676907 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:31.676877 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9v5mk" Nov 03 13:04:31.677266 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:31.676991 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48vd8" Nov 03 13:04:31.677266 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:31.677003 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dczcf" Nov 03 13:04:31.677266 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:31.677005 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9v5mk" podUID="58a9c629-dff8-4f48-99b2-385d5d13e188" Nov 03 13:04:31.677266 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:31.677063 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48vd8" podUID="2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895" Nov 03 13:04:31.677266 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:31.677126 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dczcf" podUID="8a9bc0be-51b3-4a52-83ae-580c3a2f342c" Nov 03 13:04:31.695324 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:31.695302 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zzbpd_3083cbfe-7833-4b37-8325-cb18c20a7bea/dns-node-resolver/0.log" Nov 03 13:04:32.695018 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:32.694996 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zrwrz_0ec0b81c-6052-45f2-bd5b-6096ebb72223/node-ca/0.log" Nov 03 13:04:33.677653 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:33.677619 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9v5mk" Nov 03 13:04:33.677918 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:33.677661 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48vd8" Nov 03 13:04:33.677918 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:33.677710 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dczcf" Nov 03 13:04:33.677918 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:33.677729 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9v5mk" podUID="58a9c629-dff8-4f48-99b2-385d5d13e188" Nov 03 13:04:33.677918 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:33.677794 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dczcf" podUID="8a9bc0be-51b3-4a52-83ae-580c3a2f342c" Nov 03 13:04:33.677918 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:33.677892 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48vd8" podUID="2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895" Nov 03 13:04:35.677129 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:35.677095 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9v5mk" Nov 03 13:04:35.677587 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:35.677146 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48vd8" Nov 03 13:04:35.677587 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:35.677157 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dczcf" Nov 03 13:04:35.677587 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:35.677232 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9v5mk" podUID="58a9c629-dff8-4f48-99b2-385d5d13e188" Nov 03 13:04:35.677587 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:35.677349 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48vd8" podUID="2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895" Nov 03 13:04:35.677587 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:35.677423 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dczcf" podUID="8a9bc0be-51b3-4a52-83ae-580c3a2f342c" Nov 03 13:04:37.677039 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:37.677006 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9v5mk" Nov 03 13:04:37.677416 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:37.677082 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dczcf" Nov 03 13:04:37.677416 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:37.677181 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48vd8" Nov 03 13:04:37.677416 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:37.677175 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9v5mk" podUID="58a9c629-dff8-4f48-99b2-385d5d13e188" Nov 03 13:04:37.677416 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:37.677247 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dczcf" podUID="8a9bc0be-51b3-4a52-83ae-580c3a2f342c" Nov 03 13:04:37.677416 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:37.677320 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48vd8" podUID="2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895" Nov 03 13:04:39.599919 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:39.599879 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895-original-pull-secret\") pod \"global-pull-secret-syncer-48vd8\" (UID: \"2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895\") " pod="kube-system/global-pull-secret-syncer-48vd8" Nov 03 13:04:39.599919 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:39.599924 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a9bc0be-51b3-4a52-83ae-580c3a2f342c-metrics-certs\") pod \"network-metrics-daemon-dczcf\" (UID: \"8a9bc0be-51b3-4a52-83ae-580c3a2f342c\") " pod="openshift-multus/network-metrics-daemon-dczcf" Nov 03 13:04:39.600315 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:39.600017 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 13:04:39.600315 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:39.600044 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 13:04:39.600315 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:39.600076 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895-original-pull-secret podName:2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895 nodeName:}" failed. No retries permitted until 2026-11-03 13:05:11.600060931 +0000 UTC m=+65.452067098 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895-original-pull-secret") pod "global-pull-secret-syncer-48vd8" (UID: "2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895") : object "kube-system"/"original-pull-secret" not registered Nov 03 13:04:39.600315 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:39.600090 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a9bc0be-51b3-4a52-83ae-580c3a2f342c-metrics-certs podName:8a9bc0be-51b3-4a52-83ae-580c3a2f342c nodeName:}" failed. No retries permitted until 2026-11-03 13:05:11.600083517 +0000 UTC m=+65.452089683 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a9bc0be-51b3-4a52-83ae-580c3a2f342c-metrics-certs") pod "network-metrics-daemon-dczcf" (UID: "8a9bc0be-51b3-4a52-83ae-580c3a2f342c") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 13:04:39.677581 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:39.677546 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48vd8" Nov 03 13:04:39.677741 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:39.677612 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9v5mk" Nov 03 13:04:39.677741 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:39.677698 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dczcf" Nov 03 13:04:39.677741 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:39.677703 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48vd8" podUID="2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895" Nov 03 13:04:39.677875 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:39.677777 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9v5mk" podUID="58a9c629-dff8-4f48-99b2-385d5d13e188" Nov 03 13:04:39.677909 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:39.677867 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dczcf" podUID="8a9bc0be-51b3-4a52-83ae-580c3a2f342c" Nov 03 13:04:39.700622 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:39.700594 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzl6r\" (UniqueName: \"kubernetes.io/projected/58a9c629-dff8-4f48-99b2-385d5d13e188-kube-api-access-kzl6r\") pod \"network-check-target-9v5mk\" (UID: \"58a9c629-dff8-4f48-99b2-385d5d13e188\") " pod="openshift-network-diagnostics/network-check-target-9v5mk" Nov 03 13:04:39.700770 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:39.700748 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 13:04:39.700770 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:39.700764 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 13:04:39.700836 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:39.700772 2580 projected.go:196] Error preparing data for projected volume kube-api-access-kzl6r for pod openshift-network-diagnostics/network-check-target-9v5mk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 13:04:39.700836 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:39.700819 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58a9c629-dff8-4f48-99b2-385d5d13e188-kube-api-access-kzl6r podName:58a9c629-dff8-4f48-99b2-385d5d13e188 nodeName:}" failed. No retries permitted until 2026-11-03 13:05:11.70080655 +0000 UTC m=+65.552812716 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-kzl6r" (UniqueName: "kubernetes.io/projected/58a9c629-dff8-4f48-99b2-385d5d13e188-kube-api-access-kzl6r") pod "network-check-target-9v5mk" (UID: "58a9c629-dff8-4f48-99b2-385d5d13e188") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 13:04:41.677776 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:41.677743 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dczcf" Nov 03 13:04:41.677776 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:41.677770 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48vd8" Nov 03 13:04:41.678254 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:41.677846 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dczcf" podUID="8a9bc0be-51b3-4a52-83ae-580c3a2f342c" Nov 03 13:04:41.678254 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:41.677883 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9v5mk" Nov 03 13:04:41.678254 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:41.677944 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9v5mk" podUID="58a9c629-dff8-4f48-99b2-385d5d13e188" Nov 03 13:04:41.678254 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:41.678019 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48vd8" podUID="2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895" Nov 03 13:04:43.677109 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:43.677072 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9v5mk" Nov 03 13:04:43.677109 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:43.677086 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48vd8" Nov 03 13:04:43.677657 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:43.677086 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dczcf" Nov 03 13:04:43.677657 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:43.677183 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9v5mk" podUID="58a9c629-dff8-4f48-99b2-385d5d13e188" Nov 03 13:04:43.677657 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:43.677265 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48vd8" podUID="2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895" Nov 03 13:04:43.677657 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:43.677392 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dczcf" podUID="8a9bc0be-51b3-4a52-83ae-580c3a2f342c" Nov 03 13:04:44.525209 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:44.525171 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-48vd8"] Nov 03 13:04:44.525375 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:44.525352 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48vd8" Nov 03 13:04:44.525514 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:44.525486 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48vd8" podUID="2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895" Nov 03 13:04:44.529178 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:44.529077 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9v5mk"] Nov 03 13:04:44.529280 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:44.529237 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9v5mk" Nov 03 13:04:44.529331 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:44.529314 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9v5mk" podUID="58a9c629-dff8-4f48-99b2-385d5d13e188" Nov 03 13:04:44.529844 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:44.529823 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dczcf"] Nov 03 13:04:44.529947 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:44.529935 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dczcf" Nov 03 13:04:44.530020 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:44.530005 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dczcf" podUID="8a9bc0be-51b3-4a52-83ae-580c3a2f342c" Nov 03 13:04:45.676918 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:45.676890 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48vd8" Nov 03 13:04:45.677311 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:45.676890 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dczcf" Nov 03 13:04:45.677311 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:45.676998 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48vd8" podUID="2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895" Nov 03 13:04:45.677311 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:45.677069 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dczcf" podUID="8a9bc0be-51b3-4a52-83ae-580c3a2f342c" Nov 03 13:04:46.679664 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:46.679504 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9v5mk" Nov 03 13:04:46.680064 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:46.679723 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9v5mk" podUID="58a9c629-dff8-4f48-99b2-385d5d13e188" Nov 03 13:04:47.677416 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:47.677380 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dczcf" Nov 03 13:04:47.677589 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:47.677512 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dczcf" podUID="8a9bc0be-51b3-4a52-83ae-580c3a2f342c" Nov 03 13:04:47.677589 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:47.677537 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48vd8" Nov 03 13:04:47.677664 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:47.677623 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48vd8" podUID="2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895" Nov 03 13:04:48.680130 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:48.680105 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9v5mk" Nov 03 13:04:48.680498 ip-10-0-130-198 kubenswrapper[2580]: E1103 13:04:48.680192 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9v5mk" podUID="58a9c629-dff8-4f48-99b2-385d5d13e188" Nov 03 13:04:49.435405 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.435343 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-198.us-west-2.compute.internal" event="NodeReady" Nov 03 13:04:49.435602 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.435583 2580 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 13:04:49.581549 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.581449 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-87tkk"] Nov 03 13:04:49.604034 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.603997 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-n2nst"] Nov 03 13:04:49.604235 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.604123 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-87tkk" Nov 03 13:04:49.617965 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.617941 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-n2nst" Nov 03 13:04:49.625488 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.625451 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-czwz4"] Nov 03 13:04:49.638055 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.638032 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-czwz4" Nov 03 13:04:49.645872 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.645843 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-n2nst"] Nov 03 13:04:49.648550 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.648515 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-87tkk"] Nov 03 13:04:49.660822 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.660800 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 13:04:49.660943 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.660858 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 13:04:49.660943 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.660923 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-m8wc5\"" Nov 03 13:04:49.667806 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.667785 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9pvzn\"" Nov 03 13:04:49.667806 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.667786 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 13:04:49.670145 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.670127 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57428473-a3f3-4a91-9a28-41619c285147-tmp-dir\") pod \"dns-default-87tkk\" (UID: \"57428473-a3f3-4a91-9a28-41619c285147\") " pod="openshift-dns/dns-default-87tkk" Nov 03 13:04:49.670198 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.670154 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6mtj\" (UniqueName: \"kubernetes.io/projected/57428473-a3f3-4a91-9a28-41619c285147-kube-api-access-h6mtj\") pod \"dns-default-87tkk\" (UID: \"57428473-a3f3-4a91-9a28-41619c285147\") " pod="openshift-dns/dns-default-87tkk" Nov 03 13:04:49.670198 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.670179 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57428473-a3f3-4a91-9a28-41619c285147-metrics-tls\") pod \"dns-default-87tkk\" (UID: \"57428473-a3f3-4a91-9a28-41619c285147\") " pod="openshift-dns/dns-default-87tkk" Nov 03 13:04:49.670284 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.670270 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/57428473-a3f3-4a91-9a28-41619c285147-config-volume\") pod \"dns-default-87tkk\" (UID: \"57428473-a3f3-4a91-9a28-41619c285147\") " pod="openshift-dns/dns-default-87tkk" Nov 03 13:04:49.672055 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.672041 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 13:04:49.672108 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.672042 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 13:04:49.672307 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.672296 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 13:04:49.677492 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.677464 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dczcf" Nov 03 13:04:49.677681 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.677665 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48vd8" Nov 03 13:04:49.692753 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.692733 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-czwz4"] Nov 03 13:04:49.764264 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.764231 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 13:04:49.764506 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.764485 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 13:04:49.764608 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.764572 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pvm6f\"" Nov 03 13:04:49.764666 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.764652 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-9jvp2\"" Nov 03 13:04:49.767720 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.765053 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 13:04:49.767720 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.765212 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mnhgg\"" Nov 03 13:04:49.767720 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.765422 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 13:04:49.767720 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.765572 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 13:04:49.770554 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.770537 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/57428473-a3f3-4a91-9a28-41619c285147-config-volume\") pod \"dns-default-87tkk\" (UID: \"57428473-a3f3-4a91-9a28-41619c285147\") " pod="openshift-dns/dns-default-87tkk" Nov 03 13:04:49.770603 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.770568 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2cn4\" (UniqueName: \"kubernetes.io/projected/62a2ee28-83a7-42bb-8018-7918027b3e18-kube-api-access-k2cn4\") pod \"ingress-canary-czwz4\" (UID: \"62a2ee28-83a7-42bb-8018-7918027b3e18\") " pod="openshift-ingress-canary/ingress-canary-czwz4" Nov 03 13:04:49.770603 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.770587 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/62a2ee28-83a7-42bb-8018-7918027b3e18-cert\") pod \"ingress-canary-czwz4\" (UID: \"62a2ee28-83a7-42bb-8018-7918027b3e18\") " pod="openshift-ingress-canary/ingress-canary-czwz4" Nov 03 13:04:49.770725 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.770709 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsf72\" (UniqueName: \"kubernetes.io/projected/31a398f7-8539-4c36-8386-455ef83b3d2b-kube-api-access-jsf72\") pod \"insights-runtime-extractor-n2nst\" (UID: \"31a398f7-8539-4c36-8386-455ef83b3d2b\") " pod="openshift-insights/insights-runtime-extractor-n2nst" Nov 03 13:04:49.770759 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.770742 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/31a398f7-8539-4c36-8386-455ef83b3d2b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n2nst\" (UID: \"31a398f7-8539-4c36-8386-455ef83b3d2b\") " pod="openshift-insights/insights-runtime-extractor-n2nst" Nov 03 13:04:49.770840 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.770806 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/31a398f7-8539-4c36-8386-455ef83b3d2b-data-volume\") pod \"insights-runtime-extractor-n2nst\" (UID: \"31a398f7-8539-4c36-8386-455ef83b3d2b\") " pod="openshift-insights/insights-runtime-extractor-n2nst" Nov 03 13:04:49.770957 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.770880 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57428473-a3f3-4a91-9a28-41619c285147-tmp-dir\") pod \"dns-default-87tkk\" (UID: \"57428473-a3f3-4a91-9a28-41619c285147\") " pod="openshift-dns/dns-default-87tkk" Nov 03 13:04:49.770957 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.770897 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6mtj\" (UniqueName: \"kubernetes.io/projected/57428473-a3f3-4a91-9a28-41619c285147-kube-api-access-h6mtj\") pod \"dns-default-87tkk\" (UID: \"57428473-a3f3-4a91-9a28-41619c285147\") " pod="openshift-dns/dns-default-87tkk" Nov 03 13:04:49.770957 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.770919 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/31a398f7-8539-4c36-8386-455ef83b3d2b-crio-socket\") pod \"insights-runtime-extractor-n2nst\" (UID: \"31a398f7-8539-4c36-8386-455ef83b3d2b\") " pod="openshift-insights/insights-runtime-extractor-n2nst" Nov 03 13:04:49.770957 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.770936 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/31a398f7-8539-4c36-8386-455ef83b3d2b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n2nst\" (UID: \"31a398f7-8539-4c36-8386-455ef83b3d2b\") " pod="openshift-insights/insights-runtime-extractor-n2nst" Nov 03 13:04:49.770957 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.770961 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57428473-a3f3-4a91-9a28-41619c285147-metrics-tls\") pod \"dns-default-87tkk\" (UID: \"57428473-a3f3-4a91-9a28-41619c285147\") " pod="openshift-dns/dns-default-87tkk" Nov 03 13:04:49.771204 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.771146 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/57428473-a3f3-4a91-9a28-41619c285147-config-volume\") pod \"dns-default-87tkk\" (UID: \"57428473-a3f3-4a91-9a28-41619c285147\") " pod="openshift-dns/dns-default-87tkk" Nov 03 13:04:49.771204 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.771149 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57428473-a3f3-4a91-9a28-41619c285147-tmp-dir\") pod \"dns-default-87tkk\" (UID: \"57428473-a3f3-4a91-9a28-41619c285147\") " pod="openshift-dns/dns-default-87tkk" Nov 03 13:04:49.774661 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.774645 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57428473-a3f3-4a91-9a28-41619c285147-metrics-tls\") pod \"dns-default-87tkk\" (UID: \"57428473-a3f3-4a91-9a28-41619c285147\") " pod="openshift-dns/dns-default-87tkk" Nov 03 13:04:49.825035 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.825000 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6mtj\" (UniqueName: \"kubernetes.io/projected/57428473-a3f3-4a91-9a28-41619c285147-kube-api-access-h6mtj\") pod \"dns-default-87tkk\" (UID: \"57428473-a3f3-4a91-9a28-41619c285147\") " pod="openshift-dns/dns-default-87tkk" Nov 03 13:04:49.871708 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.871603 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/62a2ee28-83a7-42bb-8018-7918027b3e18-cert\") pod \"ingress-canary-czwz4\" (UID: \"62a2ee28-83a7-42bb-8018-7918027b3e18\") " pod="openshift-ingress-canary/ingress-canary-czwz4" Nov 03 13:04:49.871708 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.871659 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsf72\" (UniqueName: \"kubernetes.io/projected/31a398f7-8539-4c36-8386-455ef83b3d2b-kube-api-access-jsf72\") pod \"insights-runtime-extractor-n2nst\" (UID: \"31a398f7-8539-4c36-8386-455ef83b3d2b\") " pod="openshift-insights/insights-runtime-extractor-n2nst" Nov 03 13:04:49.871708 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.871693 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/31a398f7-8539-4c36-8386-455ef83b3d2b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n2nst\" (UID: \"31a398f7-8539-4c36-8386-455ef83b3d2b\") " pod="openshift-insights/insights-runtime-extractor-n2nst" Nov 03 13:04:49.871985 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.871743 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/31a398f7-8539-4c36-8386-455ef83b3d2b-data-volume\") pod \"insights-runtime-extractor-n2nst\" (UID: \"31a398f7-8539-4c36-8386-455ef83b3d2b\") " pod="openshift-insights/insights-runtime-extractor-n2nst" Nov 03 13:04:49.871985 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.871811 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/31a398f7-8539-4c36-8386-455ef83b3d2b-crio-socket\") pod \"insights-runtime-extractor-n2nst\" (UID: \"31a398f7-8539-4c36-8386-455ef83b3d2b\") " pod="openshift-insights/insights-runtime-extractor-n2nst" Nov 03 13:04:49.871985 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.871838 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/31a398f7-8539-4c36-8386-455ef83b3d2b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n2nst\" (UID: \"31a398f7-8539-4c36-8386-455ef83b3d2b\") " pod="openshift-insights/insights-runtime-extractor-n2nst" Nov 03 13:04:49.871985 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.871910 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2cn4\" (UniqueName: \"kubernetes.io/projected/62a2ee28-83a7-42bb-8018-7918027b3e18-kube-api-access-k2cn4\") pod \"ingress-canary-czwz4\" (UID: \"62a2ee28-83a7-42bb-8018-7918027b3e18\") " pod="openshift-ingress-canary/ingress-canary-czwz4" Nov 03 13:04:49.872188 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.872147 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/31a398f7-8539-4c36-8386-455ef83b3d2b-crio-socket\") pod \"insights-runtime-extractor-n2nst\" (UID: \"31a398f7-8539-4c36-8386-455ef83b3d2b\") " pod="openshift-insights/insights-runtime-extractor-n2nst" Nov 03 13:04:49.872554 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.872482 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/31a398f7-8539-4c36-8386-455ef83b3d2b-data-volume\") pod \"insights-runtime-extractor-n2nst\" (UID: \"31a398f7-8539-4c36-8386-455ef83b3d2b\") " pod="openshift-insights/insights-runtime-extractor-n2nst" Nov 03 13:04:49.872848 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.872825 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/31a398f7-8539-4c36-8386-455ef83b3d2b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n2nst\" (UID: \"31a398f7-8539-4c36-8386-455ef83b3d2b\") " pod="openshift-insights/insights-runtime-extractor-n2nst" Nov 03 13:04:49.873976 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.873940 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/62a2ee28-83a7-42bb-8018-7918027b3e18-cert\") pod \"ingress-canary-czwz4\" (UID: \"62a2ee28-83a7-42bb-8018-7918027b3e18\") " pod="openshift-ingress-canary/ingress-canary-czwz4" Nov 03 13:04:49.873976 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.873956 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/31a398f7-8539-4c36-8386-455ef83b3d2b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n2nst\" (UID: \"31a398f7-8539-4c36-8386-455ef83b3d2b\") " pod="openshift-insights/insights-runtime-extractor-n2nst" Nov 03 13:04:49.926244 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.926215 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsf72\" (UniqueName: \"kubernetes.io/projected/31a398f7-8539-4c36-8386-455ef83b3d2b-kube-api-access-jsf72\") pod \"insights-runtime-extractor-n2nst\" (UID: \"31a398f7-8539-4c36-8386-455ef83b3d2b\") " pod="openshift-insights/insights-runtime-extractor-n2nst" Nov 03 13:04:49.926382 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.926302 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2cn4\" (UniqueName: \"kubernetes.io/projected/62a2ee28-83a7-42bb-8018-7918027b3e18-kube-api-access-k2cn4\") pod \"ingress-canary-czwz4\" (UID: \"62a2ee28-83a7-42bb-8018-7918027b3e18\") " pod="openshift-ingress-canary/ingress-canary-czwz4" Nov 03 13:04:49.938529 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.938501 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-87tkk" Nov 03 13:04:49.991313 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:49.991286 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-n2nst" Nov 03 13:04:50.005437 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:50.005412 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-czwz4" Nov 03 13:04:50.448644 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:50.448613 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-czwz4"] Nov 03 13:04:50.451630 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:50.451588 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-n2nst"] Nov 03 13:04:50.452222 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:50.452202 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-87tkk"] Nov 03 13:04:50.456699 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:50.456662 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62a2ee28_83a7_42bb_8018_7918027b3e18.slice/crio-63f68d72e1402d9f5b019be6e003a98b22ba8f0f8c43f2199bdd191bcf34a87c WatchSource:0}: Error finding container 63f68d72e1402d9f5b019be6e003a98b22ba8f0f8c43f2199bdd191bcf34a87c: Status 404 returned error can't find the container with id 63f68d72e1402d9f5b019be6e003a98b22ba8f0f8c43f2199bdd191bcf34a87c Nov 03 13:04:50.458692 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:50.458669 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31a398f7_8539_4c36_8386_455ef83b3d2b.slice/crio-dea167f7f04649f1c8c740ea754a186c7f21aa1229d68d244baf80de748a0b55 WatchSource:0}: Error finding container dea167f7f04649f1c8c740ea754a186c7f21aa1229d68d244baf80de748a0b55: Status 404 returned error can't find the container with id dea167f7f04649f1c8c740ea754a186c7f21aa1229d68d244baf80de748a0b55 Nov 03 13:04:50.459644 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:50.459622 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57428473_a3f3_4a91_9a28_41619c285147.slice/crio-b404676561694e010ac4e528fbf5d92e6d2411bcd1eb9bf7a48d5ffe8f138f6f WatchSource:0}: Error finding container b404676561694e010ac4e528fbf5d92e6d2411bcd1eb9bf7a48d5ffe8f138f6f: Status 404 returned error can't find the container with id b404676561694e010ac4e528fbf5d92e6d2411bcd1eb9bf7a48d5ffe8f138f6f Nov 03 13:04:50.677821 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:50.677791 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9v5mk" Nov 03 13:04:50.727493 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:50.727391 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 13:04:50.728126 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:50.727625 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 13:04:50.728126 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:50.727638 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-x4nbm\"" Nov 03 13:04:50.901912 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:50.901876 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n2nst" event={"ID":"31a398f7-8539-4c36-8386-455ef83b3d2b","Type":"ContainerStarted","Data":"c4513df94a3311eaa4388d2c79994c0195d5617172d4261405cbd19ce6c3d357"} Nov 03 13:04:50.902093 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:50.901938 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n2nst" event={"ID":"31a398f7-8539-4c36-8386-455ef83b3d2b","Type":"ContainerStarted","Data":"dea167f7f04649f1c8c740ea754a186c7f21aa1229d68d244baf80de748a0b55"} Nov 03 13:04:50.902913 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:50.902890 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-czwz4" event={"ID":"62a2ee28-83a7-42bb-8018-7918027b3e18","Type":"ContainerStarted","Data":"63f68d72e1402d9f5b019be6e003a98b22ba8f0f8c43f2199bdd191bcf34a87c"} Nov 03 13:04:50.903925 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:50.903907 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-87tkk" event={"ID":"57428473-a3f3-4a91-9a28-41619c285147","Type":"ContainerStarted","Data":"b404676561694e010ac4e528fbf5d92e6d2411bcd1eb9bf7a48d5ffe8f138f6f"} Nov 03 13:04:52.911739 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:52.911699 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n2nst" event={"ID":"31a398f7-8539-4c36-8386-455ef83b3d2b","Type":"ContainerStarted","Data":"0c2a098fd73534683ed4c097d53eccf4aa51bbe35ac28d3569a1b3f3b7382e9b"} Nov 03 13:04:53.184544 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.184453 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-848bc84498-gxvtr"] Nov 03 13:04:53.198583 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.198554 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-848bc84498-gxvtr" Nov 03 13:04:53.244356 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.244212 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-848bc84498-gxvtr"] Nov 03 13:04:53.290595 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.290565 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 03 13:04:53.294251 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.291051 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 03 13:04:53.294251 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.291133 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 03 13:04:53.294251 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.291248 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 03 13:04:53.294251 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.291262 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 03 13:04:53.295728 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.295704 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-qqlft\"" Nov 03 13:04:53.295728 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.295728 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 03 13:04:53.295909 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.295714 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 03 13:04:53.300043 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.300021 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-console-config\") pod \"console-848bc84498-gxvtr\" (UID: \"58dabb02-fd21-40ee-a4f5-e990b2b9a35a\") " pod="openshift-console/console-848bc84498-gxvtr" Nov 03 13:04:53.300150 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.300050 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-console-serving-cert\") pod \"console-848bc84498-gxvtr\" (UID: \"58dabb02-fd21-40ee-a4f5-e990b2b9a35a\") " pod="openshift-console/console-848bc84498-gxvtr" Nov 03 13:04:53.300150 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.300066 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-service-ca\") pod \"console-848bc84498-gxvtr\" (UID: \"58dabb02-fd21-40ee-a4f5-e990b2b9a35a\") " pod="openshift-console/console-848bc84498-gxvtr" Nov 03 13:04:53.300150 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.300108 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-oauth-serving-cert\") pod \"console-848bc84498-gxvtr\" (UID: \"58dabb02-fd21-40ee-a4f5-e990b2b9a35a\") " pod="openshift-console/console-848bc84498-gxvtr" Nov 03 13:04:53.300261 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.300202 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jmgk\" (UniqueName: \"kubernetes.io/projected/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-kube-api-access-6jmgk\") pod \"console-848bc84498-gxvtr\" (UID: \"58dabb02-fd21-40ee-a4f5-e990b2b9a35a\") " pod="openshift-console/console-848bc84498-gxvtr" Nov 03 13:04:53.300303 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.300256 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-console-oauth-config\") pod \"console-848bc84498-gxvtr\" (UID: \"58dabb02-fd21-40ee-a4f5-e990b2b9a35a\") " pod="openshift-console/console-848bc84498-gxvtr" Nov 03 13:04:53.300348 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.300309 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-tmp\") pod \"console-848bc84498-gxvtr\" (UID: \"58dabb02-fd21-40ee-a4f5-e990b2b9a35a\") " pod="openshift-console/console-848bc84498-gxvtr" Nov 03 13:04:53.400897 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.400858 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-console-config\") pod \"console-848bc84498-gxvtr\" (UID: \"58dabb02-fd21-40ee-a4f5-e990b2b9a35a\") " pod="openshift-console/console-848bc84498-gxvtr" Nov 03 13:04:53.400897 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.400902 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-console-serving-cert\") pod \"console-848bc84498-gxvtr\" (UID: \"58dabb02-fd21-40ee-a4f5-e990b2b9a35a\") " pod="openshift-console/console-848bc84498-gxvtr" Nov 03 13:04:53.401115 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.400924 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-service-ca\") pod \"console-848bc84498-gxvtr\" (UID: \"58dabb02-fd21-40ee-a4f5-e990b2b9a35a\") " pod="openshift-console/console-848bc84498-gxvtr" Nov 03 13:04:53.401115 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.400945 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-oauth-serving-cert\") pod \"console-848bc84498-gxvtr\" (UID: \"58dabb02-fd21-40ee-a4f5-e990b2b9a35a\") " pod="openshift-console/console-848bc84498-gxvtr" Nov 03 13:04:53.401115 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.400971 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jmgk\" (UniqueName: \"kubernetes.io/projected/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-kube-api-access-6jmgk\") pod \"console-848bc84498-gxvtr\" (UID: \"58dabb02-fd21-40ee-a4f5-e990b2b9a35a\") " pod="openshift-console/console-848bc84498-gxvtr" Nov 03 13:04:53.401115 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.401014 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-console-oauth-config\") pod \"console-848bc84498-gxvtr\" (UID: \"58dabb02-fd21-40ee-a4f5-e990b2b9a35a\") " pod="openshift-console/console-848bc84498-gxvtr" Nov 03 13:04:53.401115 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.401043 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-tmp\") pod \"console-848bc84498-gxvtr\" (UID: \"58dabb02-fd21-40ee-a4f5-e990b2b9a35a\") " pod="openshift-console/console-848bc84498-gxvtr" Nov 03 13:04:53.401389 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.401368 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-tmp\") pod \"console-848bc84498-gxvtr\" (UID: \"58dabb02-fd21-40ee-a4f5-e990b2b9a35a\") " pod="openshift-console/console-848bc84498-gxvtr" Nov 03 13:04:53.401796 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.401771 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-console-config\") pod \"console-848bc84498-gxvtr\" (UID: \"58dabb02-fd21-40ee-a4f5-e990b2b9a35a\") " pod="openshift-console/console-848bc84498-gxvtr" Nov 03 13:04:53.401897 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.401811 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-service-ca\") pod \"console-848bc84498-gxvtr\" (UID: \"58dabb02-fd21-40ee-a4f5-e990b2b9a35a\") " pod="openshift-console/console-848bc84498-gxvtr" Nov 03 13:04:53.401942 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.401912 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-oauth-serving-cert\") pod \"console-848bc84498-gxvtr\" (UID: \"58dabb02-fd21-40ee-a4f5-e990b2b9a35a\") " pod="openshift-console/console-848bc84498-gxvtr" Nov 03 13:04:53.404834 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.404810 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-console-oauth-config\") pod \"console-848bc84498-gxvtr\" (UID: \"58dabb02-fd21-40ee-a4f5-e990b2b9a35a\") " pod="openshift-console/console-848bc84498-gxvtr" Nov 03 13:04:53.404913 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.404834 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-console-serving-cert\") pod \"console-848bc84498-gxvtr\" (UID: \"58dabb02-fd21-40ee-a4f5-e990b2b9a35a\") " pod="openshift-console/console-848bc84498-gxvtr" Nov 03 13:04:53.455933 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.455853 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jmgk\" (UniqueName: \"kubernetes.io/projected/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-kube-api-access-6jmgk\") pod \"console-848bc84498-gxvtr\" (UID: \"58dabb02-fd21-40ee-a4f5-e990b2b9a35a\") " pod="openshift-console/console-848bc84498-gxvtr" Nov 03 13:04:53.527984 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.527957 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-848bc84498-gxvtr" Nov 03 13:04:53.700225 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.699315 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-fqskz"] Nov 03 13:04:53.719328 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.718856 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fqskz" Nov 03 13:04:53.756884 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.756860 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-fqskz"] Nov 03 13:04:53.768691 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.768667 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 03 13:04:53.768811 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.768782 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 13:04:53.768981 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.768968 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 13:04:53.769050 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.769040 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-tvxrk\"" Nov 03 13:04:53.769103 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.769086 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 03 13:04:53.769153 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.769139 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 13:04:53.804882 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.804844 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0d0c5822-ab7b-409d-a755-49e918278fa1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-fqskz\" (UID: \"0d0c5822-ab7b-409d-a755-49e918278fa1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fqskz" Nov 03 13:04:53.805056 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.804932 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d0c5822-ab7b-409d-a755-49e918278fa1-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-fqskz\" (UID: \"0d0c5822-ab7b-409d-a755-49e918278fa1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fqskz" Nov 03 13:04:53.805056 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.804968 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkznq\" (UniqueName: \"kubernetes.io/projected/0d0c5822-ab7b-409d-a755-49e918278fa1-kube-api-access-gkznq\") pod \"openshift-state-metrics-7bd7489957-fqskz\" (UID: \"0d0c5822-ab7b-409d-a755-49e918278fa1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fqskz" Nov 03 13:04:53.805056 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.805003 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d0c5822-ab7b-409d-a755-49e918278fa1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-fqskz\" (UID: \"0d0c5822-ab7b-409d-a755-49e918278fa1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fqskz" Nov 03 13:04:53.805269 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.805251 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-krsxs"] Nov 03 13:04:53.827503 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.827412 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-zhl5n"] Nov 03 13:04:53.829246 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.827619 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-krsxs" Nov 03 13:04:53.843436 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.843412 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zhl5n" Nov 03 13:04:53.860502 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.860423 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-krsxs"] Nov 03 13:04:53.878795 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.878765 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 03 13:04:53.878945 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.878923 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-tqrl8\"" Nov 03 13:04:53.878994 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.878982 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 03 13:04:53.881684 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.881656 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 03 13:04:53.905424 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.905398 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0d0c5822-ab7b-409d-a755-49e918278fa1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-fqskz\" (UID: \"0d0c5822-ab7b-409d-a755-49e918278fa1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fqskz" Nov 03 13:04:53.905573 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.905441 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6263f8d-cced-465a-b393-ef51cb902e11-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-krsxs\" (UID: \"b6263f8d-cced-465a-b393-ef51cb902e11\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-krsxs" Nov 03 13:04:53.905573 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.905493 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d0c5822-ab7b-409d-a755-49e918278fa1-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-fqskz\" (UID: \"0d0c5822-ab7b-409d-a755-49e918278fa1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fqskz" Nov 03 13:04:53.905731 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.905706 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6263f8d-cced-465a-b393-ef51cb902e11-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-krsxs\" (UID: \"b6263f8d-cced-465a-b393-ef51cb902e11\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-krsxs" Nov 03 13:04:53.905786 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.905748 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b6263f8d-cced-465a-b393-ef51cb902e11-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-krsxs\" (UID: \"b6263f8d-cced-465a-b393-ef51cb902e11\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-krsxs" Nov 03 13:04:53.905883 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.905858 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk2xl\" (UniqueName: \"kubernetes.io/projected/b6263f8d-cced-465a-b393-ef51cb902e11-kube-api-access-bk2xl\") pod \"kube-state-metrics-5dc8d7d89b-krsxs\" (UID: \"b6263f8d-cced-465a-b393-ef51cb902e11\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-krsxs" Nov 03 13:04:53.905980 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.905956 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkznq\" (UniqueName: \"kubernetes.io/projected/0d0c5822-ab7b-409d-a755-49e918278fa1-kube-api-access-gkznq\") pod \"openshift-state-metrics-7bd7489957-fqskz\" (UID: \"0d0c5822-ab7b-409d-a755-49e918278fa1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fqskz" Nov 03 13:04:53.906052 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.905988 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b6263f8d-cced-465a-b393-ef51cb902e11-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-krsxs\" (UID: \"b6263f8d-cced-465a-b393-ef51cb902e11\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-krsxs" Nov 03 13:04:53.906052 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.906022 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b6263f8d-cced-465a-b393-ef51cb902e11-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-krsxs\" (UID: \"b6263f8d-cced-465a-b393-ef51cb902e11\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-krsxs" Nov 03 13:04:53.906152 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.906056 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d0c5822-ab7b-409d-a755-49e918278fa1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-fqskz\" (UID: \"0d0c5822-ab7b-409d-a755-49e918278fa1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fqskz" Nov 03 13:04:53.906494 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.906447 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d0c5822-ab7b-409d-a755-49e918278fa1-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-fqskz\" (UID: \"0d0c5822-ab7b-409d-a755-49e918278fa1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fqskz" Nov 03 13:04:53.909199 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.909177 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d0c5822-ab7b-409d-a755-49e918278fa1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-fqskz\" (UID: \"0d0c5822-ab7b-409d-a755-49e918278fa1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fqskz" Nov 03 13:04:53.909199 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.909177 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0d0c5822-ab7b-409d-a755-49e918278fa1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-fqskz\" (UID: \"0d0c5822-ab7b-409d-a755-49e918278fa1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fqskz" Nov 03 13:04:53.942493 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.942322 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-87tkk" event={"ID":"57428473-a3f3-4a91-9a28-41619c285147","Type":"ContainerStarted","Data":"b8e02ddb82c19d1126f7068f9c76f468abc438ffdc6ef4f8c40263eb13d0b284"} Nov 03 13:04:53.942493 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.942388 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-87tkk" event={"ID":"57428473-a3f3-4a91-9a28-41619c285147","Type":"ContainerStarted","Data":"21b2b86f9f1c2d4ba18237b5dc48f684614f7fb210f499d34ba5778814100faa"} Nov 03 13:04:53.942493 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.942419 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-87tkk" Nov 03 13:04:53.942493 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.942445 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-848bc84498-gxvtr"] Nov 03 13:04:53.954334 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:53.954307 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58dabb02_fd21_40ee_a4f5_e990b2b9a35a.slice/crio-d722cdc71acb0c1e3a81a70c28ff6ebbba944b71c0fbcad85b5c86e78c8336c4 WatchSource:0}: Error finding container d722cdc71acb0c1e3a81a70c28ff6ebbba944b71c0fbcad85b5c86e78c8336c4: Status 404 returned error can't find the container with id d722cdc71acb0c1e3a81a70c28ff6ebbba944b71c0fbcad85b5c86e78c8336c4 Nov 03 13:04:53.960568 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.960544 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkznq\" (UniqueName: \"kubernetes.io/projected/0d0c5822-ab7b-409d-a755-49e918278fa1-kube-api-access-gkznq\") pod \"openshift-state-metrics-7bd7489957-fqskz\" (UID: \"0d0c5822-ab7b-409d-a755-49e918278fa1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fqskz" Nov 03 13:04:53.969608 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.969502 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 13:04:53.969608 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.969574 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 13:04:53.969608 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.969601 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 13:04:53.969792 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:53.969639 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5rlch\"" Nov 03 13:04:54.007283 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:54.007255 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0fadb078-8983-42df-afde-c1983044492d-sys\") pod \"node-exporter-zhl5n\" (UID: \"0fadb078-8983-42df-afde-c1983044492d\") " pod="openshift-monitoring/node-exporter-zhl5n" Nov 03 13:04:54.007450 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:54.007343 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6263f8d-cced-465a-b393-ef51cb902e11-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-krsxs\" (UID: \"b6263f8d-cced-465a-b393-ef51cb902e11\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-krsxs" Nov 03 13:04:54.007450 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:54.007373 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b6263f8d-cced-465a-b393-ef51cb902e11-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-krsxs\" (UID: \"b6263f8d-cced-465a-b393-ef51cb902e11\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-krsxs" Nov 03 13:04:54.007450 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:54.007400 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b6263f8d-cced-465a-b393-ef51cb902e11-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-krsxs\" (UID: \"b6263f8d-cced-465a-b393-ef51cb902e11\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-krsxs" Nov 03 13:04:54.007450 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:54.007425 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0fadb078-8983-42df-afde-c1983044492d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zhl5n\" (UID: \"0fadb078-8983-42df-afde-c1983044492d\") " pod="openshift-monitoring/node-exporter-zhl5n" Nov 03 13:04:54.007768 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:54.007514 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6263f8d-cced-465a-b393-ef51cb902e11-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-krsxs\" (UID: \"b6263f8d-cced-465a-b393-ef51cb902e11\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-krsxs" Nov 03 13:04:54.007768 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:54.007543 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0fadb078-8983-42df-afde-c1983044492d-root\") pod \"node-exporter-zhl5n\" (UID: \"0fadb078-8983-42df-afde-c1983044492d\") " pod="openshift-monitoring/node-exporter-zhl5n" Nov 03 13:04:54.007922 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:54.007781 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk2xl\" (UniqueName: \"kubernetes.io/projected/b6263f8d-cced-465a-b393-ef51cb902e11-kube-api-access-bk2xl\") pod \"kube-state-metrics-5dc8d7d89b-krsxs\" (UID: \"b6263f8d-cced-465a-b393-ef51cb902e11\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-krsxs" Nov 03 13:04:54.007922 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:54.007820 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0fadb078-8983-42df-afde-c1983044492d-node-exporter-textfile\") pod \"node-exporter-zhl5n\" (UID: \"0fadb078-8983-42df-afde-c1983044492d\") " pod="openshift-monitoring/node-exporter-zhl5n" Nov 03 13:04:54.007922 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:54.007846 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgsql\" (UniqueName: \"kubernetes.io/projected/0fadb078-8983-42df-afde-c1983044492d-kube-api-access-wgsql\") pod \"node-exporter-zhl5n\" (UID: \"0fadb078-8983-42df-afde-c1983044492d\") " pod="openshift-monitoring/node-exporter-zhl5n" Nov 03 13:04:54.007922 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:54.007878 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0fadb078-8983-42df-afde-c1983044492d-metrics-client-ca\") pod \"node-exporter-zhl5n\" (UID: \"0fadb078-8983-42df-afde-c1983044492d\") " pod="openshift-monitoring/node-exporter-zhl5n" Nov 03 13:04:54.008171 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:54.008151 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6263f8d-cced-465a-b393-ef51cb902e11-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-krsxs\" (UID: \"b6263f8d-cced-465a-b393-ef51cb902e11\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-krsxs" Nov 03 13:04:54.008245 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:54.008176 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b6263f8d-cced-465a-b393-ef51cb902e11-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-krsxs\" (UID: \"b6263f8d-cced-465a-b393-ef51cb902e11\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-krsxs" Nov 03 13:04:54.008245 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:54.008185 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0fadb078-8983-42df-afde-c1983044492d-node-exporter-wtmp\") pod \"node-exporter-zhl5n\" (UID: \"0fadb078-8983-42df-afde-c1983044492d\") " pod="openshift-monitoring/node-exporter-zhl5n" Nov 03 13:04:54.008245 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:54.008231 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b6263f8d-cced-465a-b393-ef51cb902e11-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-krsxs\" (UID: \"b6263f8d-cced-465a-b393-ef51cb902e11\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-krsxs" Nov 03 13:04:54.008388 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:54.008264 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0fadb078-8983-42df-afde-c1983044492d-node-exporter-tls\") pod \"node-exporter-zhl5n\" (UID: \"0fadb078-8983-42df-afde-c1983044492d\") " pod="openshift-monitoring/node-exporter-zhl5n" Nov 03 13:04:54.008388 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:54.008293 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0fadb078-8983-42df-afde-c1983044492d-node-exporter-accelerators-collector-config\") pod \"node-exporter-zhl5n\" (UID: \"0fadb078-8983-42df-afde-c1983044492d\") " pod="openshift-monitoring/node-exporter-zhl5n" Nov 03 13:04:54.008569 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:54.008548 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b6263f8d-cced-465a-b393-ef51cb902e11-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-krsxs\" (UID: \"b6263f8d-cced-465a-b393-ef51cb902e11\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-krsxs" Nov 03 13:04:54.010194 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:54.010168 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b6263f8d-cced-465a-b393-ef51cb902e11-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-krsxs\" (UID: \"b6263f8d-cced-465a-b393-ef51cb902e11\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-krsxs" Nov 03 13:04:54.010298 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:54.010274 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6263f8d-cced-465a-b393-ef51cb902e11-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-krsxs\" (UID: \"b6263f8d-cced-465a-b393-ef51cb902e11\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-krsxs" Nov 03 13:04:54.053238 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:54.053191 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-87tkk" podStartSLOduration=2.008856574 podStartE2EDuration="5.053178844s" totalImagesPullingTime="3.04432227s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:04:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:04:54.052767827 +0000 UTC m=+47.904774015" watchObservedRunningTime="2026-11-03 13:04:54.053178844 +0000 UTC m=+47.905185031" Nov 03 13:04:54.065181 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:54.065153 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk2xl\" (UniqueName: \"kubernetes.io/projected/b6263f8d-cced-465a-b393-ef51cb902e11-kube-api-access-bk2xl\") pod \"kube-state-metrics-5dc8d7d89b-krsxs\" (UID: \"b6263f8d-cced-465a-b393-ef51cb902e11\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-krsxs" Nov 03 13:04:54.075968 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:54.075946 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fqskz" Nov 03 13:04:54.109412 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:54.109390 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0fadb078-8983-42df-afde-c1983044492d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zhl5n\" (UID: \"0fadb078-8983-42df-afde-c1983044492d\") " pod="openshift-monitoring/node-exporter-zhl5n" Nov 03 13:04:54.109527 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:54.109454 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0fadb078-8983-42df-afde-c1983044492d-root\") pod \"node-exporter-zhl5n\" (UID: \"0fadb078-8983-42df-afde-c1983044492d\") " pod="openshift-monitoring/node-exporter-zhl5n" Nov 03 13:04:54.109527 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:54.109495 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0fadb078-8983-42df-afde-c1983044492d-node-exporter-textfile\") pod \"node-exporter-zhl5n\" (UID: \"0fadb078-8983-42df-afde-c1983044492d\") " pod="openshift-monitoring/node-exporter-zhl5n" Nov 03 13:04:54.109527 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:54.109514 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgsql\" (UniqueName: \"kubernetes.io/projected/0fadb078-8983-42df-afde-c1983044492d-kube-api-access-wgsql\") pod \"node-exporter-zhl5n\" (UID: \"0fadb078-8983-42df-afde-c1983044492d\") " pod="openshift-monitoring/node-exporter-zhl5n" Nov 03 13:04:54.109672 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:54.109532 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0fadb078-8983-42df-afde-c1983044492d-metrics-client-ca\") pod \"node-exporter-zhl5n\" (UID: \"0fadb078-8983-42df-afde-c1983044492d\") " pod="openshift-monitoring/node-exporter-zhl5n" Nov 03 13:04:54.109672 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:54.109552 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0fadb078-8983-42df-afde-c1983044492d-node-exporter-wtmp\") pod \"node-exporter-zhl5n\" (UID: \"0fadb078-8983-42df-afde-c1983044492d\") " pod="openshift-monitoring/node-exporter-zhl5n" Nov 03 13:04:54.109672 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:54.109580 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0fadb078-8983-42df-afde-c1983044492d-node-exporter-tls\") pod \"node-exporter-zhl5n\" (UID: \"0fadb078-8983-42df-afde-c1983044492d\") " pod="openshift-monitoring/node-exporter-zhl5n" Nov 03 13:04:54.109672 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:54.109606 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0fadb078-8983-42df-afde-c1983044492d-node-exporter-accelerators-collector-config\") pod \"node-exporter-zhl5n\" (UID: \"0fadb078-8983-42df-afde-c1983044492d\") " pod="openshift-monitoring/node-exporter-zhl5n" Nov 03 13:04:54.109672 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:54.109624 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0fadb078-8983-42df-afde-c1983044492d-root\") pod \"node-exporter-zhl5n\" (UID: \"0fadb078-8983-42df-afde-c1983044492d\") " pod="openshift-monitoring/node-exporter-zhl5n" Nov 03 13:04:54.109672 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:54.109672 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0fadb078-8983-42df-afde-c1983044492d-sys\") pod \"node-exporter-zhl5n\" (UID: \"0fadb078-8983-42df-afde-c1983044492d\") " pod="openshift-monitoring/node-exporter-zhl5n" Nov 03 13:04:54.109956 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:54.109751 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0fadb078-8983-42df-afde-c1983044492d-sys\") pod \"node-exporter-zhl5n\" (UID: \"0fadb078-8983-42df-afde-c1983044492d\") " pod="openshift-monitoring/node-exporter-zhl5n" Nov 03 13:04:54.109956 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:54.109928 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0fadb078-8983-42df-afde-c1983044492d-node-exporter-textfile\") pod \"node-exporter-zhl5n\" (UID: \"0fadb078-8983-42df-afde-c1983044492d\") " pod="openshift-monitoring/node-exporter-zhl5n" Nov 03 13:04:54.110068 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:54.109997 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0fadb078-8983-42df-afde-c1983044492d-node-exporter-wtmp\") pod \"node-exporter-zhl5n\" (UID: \"0fadb078-8983-42df-afde-c1983044492d\") " pod="openshift-monitoring/node-exporter-zhl5n" Nov 03 13:04:54.111149 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:54.111103 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0fadb078-8983-42df-afde-c1983044492d-metrics-client-ca\") pod \"node-exporter-zhl5n\" (UID: \"0fadb078-8983-42df-afde-c1983044492d\") " pod="openshift-monitoring/node-exporter-zhl5n" Nov 03 13:04:54.111246 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:54.111181 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0fadb078-8983-42df-afde-c1983044492d-node-exporter-accelerators-collector-config\") pod \"node-exporter-zhl5n\" (UID: \"0fadb078-8983-42df-afde-c1983044492d\") " pod="openshift-monitoring/node-exporter-zhl5n" Nov 03 13:04:54.112341 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:54.112322 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0fadb078-8983-42df-afde-c1983044492d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zhl5n\" (UID: \"0fadb078-8983-42df-afde-c1983044492d\") " pod="openshift-monitoring/node-exporter-zhl5n" Nov 03 13:04:54.112416 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:54.112338 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0fadb078-8983-42df-afde-c1983044492d-node-exporter-tls\") pod \"node-exporter-zhl5n\" (UID: \"0fadb078-8983-42df-afde-c1983044492d\") " pod="openshift-monitoring/node-exporter-zhl5n" Nov 03 13:04:54.167150 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:54.167120 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgsql\" (UniqueName: \"kubernetes.io/projected/0fadb078-8983-42df-afde-c1983044492d-kube-api-access-wgsql\") pod \"node-exporter-zhl5n\" (UID: \"0fadb078-8983-42df-afde-c1983044492d\") " pod="openshift-monitoring/node-exporter-zhl5n" Nov 03 13:04:54.186167 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:54.186141 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-krsxs" Nov 03 13:04:54.243255 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:54.243183 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zhl5n" Nov 03 13:04:54.385040 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:54.384930 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fadb078_8983_42df_afde_c1983044492d.slice/crio-ca280a5e21c2dd513ba61de02f0ae58881b1753b0bdf0b1d15c4e428bb0fbb1f WatchSource:0}: Error finding container ca280a5e21c2dd513ba61de02f0ae58881b1753b0bdf0b1d15c4e428bb0fbb1f: Status 404 returned error can't find the container with id ca280a5e21c2dd513ba61de02f0ae58881b1753b0bdf0b1d15c4e428bb0fbb1f Nov 03 13:04:54.463774 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:54.463613 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-fqskz"] Nov 03 13:04:54.748886 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:54.748850 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-krsxs"] Nov 03 13:04:54.756886 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:54.756857 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6263f8d_cced_465a_b393_ef51cb902e11.slice/crio-3b26249f5bd52d607583665ec09161ee13a1786eaaed3a195ce24774d76f055b WatchSource:0}: Error finding container 3b26249f5bd52d607583665ec09161ee13a1786eaaed3a195ce24774d76f055b: Status 404 returned error can't find the container with id 3b26249f5bd52d607583665ec09161ee13a1786eaaed3a195ce24774d76f055b Nov 03 13:04:54.948773 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:54.948738 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-848bc84498-gxvtr" event={"ID":"58dabb02-fd21-40ee-a4f5-e990b2b9a35a","Type":"ContainerStarted","Data":"d722cdc71acb0c1e3a81a70c28ff6ebbba944b71c0fbcad85b5c86e78c8336c4"} Nov 03 13:04:54.949878 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:54.949855 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zhl5n" event={"ID":"0fadb078-8983-42df-afde-c1983044492d","Type":"ContainerStarted","Data":"ca280a5e21c2dd513ba61de02f0ae58881b1753b0bdf0b1d15c4e428bb0fbb1f"} Nov 03 13:04:54.951456 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:54.951439 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fqskz" event={"ID":"0d0c5822-ab7b-409d-a755-49e918278fa1","Type":"ContainerStarted","Data":"e2161d433d7d6b2e457edc38ddc45223224e5fd9919c7f9a847394e44b8489af"} Nov 03 13:04:54.951553 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:54.951502 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fqskz" event={"ID":"0d0c5822-ab7b-409d-a755-49e918278fa1","Type":"ContainerStarted","Data":"863fa010f7bee4aaeefd2352abbdb903d87a97cec0c7eb4cf7e641a46aecce27"} Nov 03 13:04:54.951553 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:54.951533 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fqskz" event={"ID":"0d0c5822-ab7b-409d-a755-49e918278fa1","Type":"ContainerStarted","Data":"a9d7c3dabde3dda56feb11acc4d301b588e998ab794a8368b9abdca272ec1905"} Nov 03 13:04:54.952816 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:54.952796 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-czwz4" event={"ID":"62a2ee28-83a7-42bb-8018-7918027b3e18","Type":"ContainerStarted","Data":"4e6e48c74335568e8d83899e7d2e0e78842832cbc0e2c50ebe3244cb260a9d47"} Nov 03 13:04:54.953853 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:54.953835 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-krsxs" event={"ID":"b6263f8d-cced-465a-b393-ef51cb902e11","Type":"ContainerStarted","Data":"3b26249f5bd52d607583665ec09161ee13a1786eaaed3a195ce24774d76f055b"} Nov 03 13:04:54.955531 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:54.955510 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n2nst" event={"ID":"31a398f7-8539-4c36-8386-455ef83b3d2b","Type":"ContainerStarted","Data":"d3a154c525bc4446582f9e944b8ce488cfd381607f2eedc3afce01de3ce9de74"} Nov 03 13:04:55.059926 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:55.059820 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-czwz4" podStartSLOduration=2.128148555 podStartE2EDuration="6.059807469s" totalImagesPullingTime="3.931658914s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:04:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:04:55.059728819 +0000 UTC m=+48.911735006" watchObservedRunningTime="2026-11-03 13:04:55.059807469 +0000 UTC m=+48.911813657" Nov 03 13:04:55.168909 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:55.168855 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-n2nst" podStartSLOduration=2.02365516 podStartE2EDuration="6.168837586s" totalImagesPullingTime="4.145182426s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 13:04:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:04:55.16845347 +0000 UTC m=+49.020459659" watchObservedRunningTime="2026-11-03 13:04:55.168837586 +0000 UTC m=+49.020843773" Nov 03 13:04:55.987290 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:55.985388 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 13:04:55.990465 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:55.989514 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:04:56.040494 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.039703 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 03 13:04:56.040494 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.039784 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 03 13:04:56.040494 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.039836 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 03 13:04:56.040867 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.040817 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 03 13:04:56.041503 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.041463 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 03 13:04:56.087070 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.087041 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 03 13:04:56.126933 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.125437 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-config-volume\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:04:56.126933 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.125546 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7361d2a9-d089-4073-bbfb-e7978943fd88-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:04:56.126933 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.125578 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7361d2a9-d089-4073-bbfb-e7978943fd88-config-out\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:04:56.126933 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.125596 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:04:56.126933 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.125647 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7361d2a9-d089-4073-bbfb-e7978943fd88-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:04:56.126933 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.125663 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwz9f\" (UniqueName: \"kubernetes.io/projected/7361d2a9-d089-4073-bbfb-e7978943fd88-kube-api-access-cwz9f\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:04:56.126933 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.125680 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7361d2a9-d089-4073-bbfb-e7978943fd88-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:04:56.126933 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.125694 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:04:56.126933 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.125729 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:04:56.126933 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.125751 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:04:56.126933 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.125784 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-web-config\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:04:56.126933 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.125809 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:04:56.126933 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.125829 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7361d2a9-d089-4073-bbfb-e7978943fd88-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:04:56.136524 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.136496 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 13:04:56.138516 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.138449 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 03 13:04:56.139733 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.139715 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 03 13:04:56.139842 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.139825 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-swq8s\"" Nov 03 13:04:56.139914 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.139866 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 03 13:04:56.227644 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.226342 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7361d2a9-d089-4073-bbfb-e7978943fd88-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:04:56.227644 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.226378 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7361d2a9-d089-4073-bbfb-e7978943fd88-config-out\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:04:56.227644 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.226395 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:04:56.227644 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.226416 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7361d2a9-d089-4073-bbfb-e7978943fd88-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:04:56.227644 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.226434 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwz9f\" (UniqueName: \"kubernetes.io/projected/7361d2a9-d089-4073-bbfb-e7978943fd88-kube-api-access-cwz9f\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:04:56.227644 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.226450 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7361d2a9-d089-4073-bbfb-e7978943fd88-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:04:56.227644 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.226464 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:04:56.227644 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.226502 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:04:56.227644 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.226522 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:04:56.227644 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.226548 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-web-config\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:04:56.227644 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.226569 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:04:56.227644 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.226586 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7361d2a9-d089-4073-bbfb-e7978943fd88-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:04:56.227644 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.226604 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-config-volume\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:04:56.230643 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.229305 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7361d2a9-d089-4073-bbfb-e7978943fd88-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:04:56.230643 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.230141 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7361d2a9-d089-4073-bbfb-e7978943fd88-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:04:56.232853 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.232800 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7361d2a9-d089-4073-bbfb-e7978943fd88-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:04:56.238226 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.238166 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-config-volume\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:04:56.238226 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.238182 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:04:56.238986 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.238503 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-web-config\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:04:56.238986 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.238637 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:04:56.238986 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.238763 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7361d2a9-d089-4073-bbfb-e7978943fd88-config-out\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:04:56.238986 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.238860 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7361d2a9-d089-4073-bbfb-e7978943fd88-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:04:56.238986 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.238947 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:04:56.240667 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.240627 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:04:56.242038 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.242017 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:04:56.286418 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.286383 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwz9f\" (UniqueName: \"kubernetes.io/projected/7361d2a9-d089-4073-bbfb-e7978943fd88-kube-api-access-cwz9f\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:04:56.316872 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.316624 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:04:56.385868 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.385806 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-86b8856f97-b75hg"] Nov 03 13:04:56.389219 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.389196 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-86b8856f97-b75hg" Nov 03 13:04:56.440142 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.440122 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 03 13:04:56.440386 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.440365 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 03 13:04:56.440503 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.440444 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 03 13:04:56.440571 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.440531 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 03 13:04:56.440693 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.440669 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 03 13:04:56.440799 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.440756 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-9ciao1qbg16k6\"" Nov 03 13:04:56.440799 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.440760 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-jjf7n\"" Nov 03 13:04:56.454530 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.454506 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-86b8856f97-b75hg"] Nov 03 13:04:56.528539 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.528503 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-86b8856f97-b75hg\" (UID: \"0fef60a4-2a84-4318-b1bf-feb8f69ef7e9\") " pod="openshift-monitoring/thanos-querier-86b8856f97-b75hg" Nov 03 13:04:56.528709 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.528560 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-86b8856f97-b75hg\" (UID: \"0fef60a4-2a84-4318-b1bf-feb8f69ef7e9\") " pod="openshift-monitoring/thanos-querier-86b8856f97-b75hg" Nov 03 13:04:56.528709 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.528610 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9-tmp\") pod \"thanos-querier-86b8856f97-b75hg\" (UID: \"0fef60a4-2a84-4318-b1bf-feb8f69ef7e9\") " pod="openshift-monitoring/thanos-querier-86b8856f97-b75hg" Nov 03 13:04:56.528709 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.528651 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9-metrics-client-ca\") pod \"thanos-querier-86b8856f97-b75hg\" (UID: \"0fef60a4-2a84-4318-b1bf-feb8f69ef7e9\") " pod="openshift-monitoring/thanos-querier-86b8856f97-b75hg" Nov 03 13:04:56.528709 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.528670 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-86b8856f97-b75hg\" (UID: \"0fef60a4-2a84-4318-b1bf-feb8f69ef7e9\") " pod="openshift-monitoring/thanos-querier-86b8856f97-b75hg" Nov 03 13:04:56.528922 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.528725 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9-secret-grpc-tls\") pod \"thanos-querier-86b8856f97-b75hg\" (UID: \"0fef60a4-2a84-4318-b1bf-feb8f69ef7e9\") " pod="openshift-monitoring/thanos-querier-86b8856f97-b75hg" Nov 03 13:04:56.528922 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.528757 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls676\" (UniqueName: \"kubernetes.io/projected/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9-kube-api-access-ls676\") pod \"thanos-querier-86b8856f97-b75hg\" (UID: \"0fef60a4-2a84-4318-b1bf-feb8f69ef7e9\") " pod="openshift-monitoring/thanos-querier-86b8856f97-b75hg" Nov 03 13:04:56.528922 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.528785 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9-secret-thanos-querier-tls\") pod \"thanos-querier-86b8856f97-b75hg\" (UID: \"0fef60a4-2a84-4318-b1bf-feb8f69ef7e9\") " pod="openshift-monitoring/thanos-querier-86b8856f97-b75hg" Nov 03 13:04:56.528922 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.528803 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-86b8856f97-b75hg\" (UID: \"0fef60a4-2a84-4318-b1bf-feb8f69ef7e9\") " pod="openshift-monitoring/thanos-querier-86b8856f97-b75hg" Nov 03 13:04:56.629288 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.629236 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-86b8856f97-b75hg\" (UID: \"0fef60a4-2a84-4318-b1bf-feb8f69ef7e9\") " pod="openshift-monitoring/thanos-querier-86b8856f97-b75hg" Nov 03 13:04:56.629569 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.629315 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-86b8856f97-b75hg\" (UID: \"0fef60a4-2a84-4318-b1bf-feb8f69ef7e9\") " pod="openshift-monitoring/thanos-querier-86b8856f97-b75hg" Nov 03 13:04:56.629569 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.629341 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9-tmp\") pod \"thanos-querier-86b8856f97-b75hg\" (UID: \"0fef60a4-2a84-4318-b1bf-feb8f69ef7e9\") " pod="openshift-monitoring/thanos-querier-86b8856f97-b75hg" Nov 03 13:04:56.629569 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.629368 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9-metrics-client-ca\") pod \"thanos-querier-86b8856f97-b75hg\" (UID: \"0fef60a4-2a84-4318-b1bf-feb8f69ef7e9\") " pod="openshift-monitoring/thanos-querier-86b8856f97-b75hg" Nov 03 13:04:56.629569 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.629394 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-86b8856f97-b75hg\" (UID: \"0fef60a4-2a84-4318-b1bf-feb8f69ef7e9\") " pod="openshift-monitoring/thanos-querier-86b8856f97-b75hg" Nov 03 13:04:56.629569 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.629462 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9-secret-grpc-tls\") pod \"thanos-querier-86b8856f97-b75hg\" (UID: \"0fef60a4-2a84-4318-b1bf-feb8f69ef7e9\") " pod="openshift-monitoring/thanos-querier-86b8856f97-b75hg" Nov 03 13:04:56.629569 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.629556 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls676\" (UniqueName: \"kubernetes.io/projected/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9-kube-api-access-ls676\") pod \"thanos-querier-86b8856f97-b75hg\" (UID: \"0fef60a4-2a84-4318-b1bf-feb8f69ef7e9\") " pod="openshift-monitoring/thanos-querier-86b8856f97-b75hg" Nov 03 13:04:56.629930 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.629600 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9-secret-thanos-querier-tls\") pod \"thanos-querier-86b8856f97-b75hg\" (UID: \"0fef60a4-2a84-4318-b1bf-feb8f69ef7e9\") " pod="openshift-monitoring/thanos-querier-86b8856f97-b75hg" Nov 03 13:04:56.629930 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.629625 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-86b8856f97-b75hg\" (UID: \"0fef60a4-2a84-4318-b1bf-feb8f69ef7e9\") " pod="openshift-monitoring/thanos-querier-86b8856f97-b75hg" Nov 03 13:04:56.631049 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.630876 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9-tmp\") pod \"thanos-querier-86b8856f97-b75hg\" (UID: \"0fef60a4-2a84-4318-b1bf-feb8f69ef7e9\") " pod="openshift-monitoring/thanos-querier-86b8856f97-b75hg" Nov 03 13:04:56.631212 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.631181 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9-metrics-client-ca\") pod \"thanos-querier-86b8856f97-b75hg\" (UID: \"0fef60a4-2a84-4318-b1bf-feb8f69ef7e9\") " pod="openshift-monitoring/thanos-querier-86b8856f97-b75hg" Nov 03 13:04:56.633073 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.633044 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-86b8856f97-b75hg\" (UID: \"0fef60a4-2a84-4318-b1bf-feb8f69ef7e9\") " pod="openshift-monitoring/thanos-querier-86b8856f97-b75hg" Nov 03 13:04:56.633554 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.633529 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-86b8856f97-b75hg\" (UID: \"0fef60a4-2a84-4318-b1bf-feb8f69ef7e9\") " pod="openshift-monitoring/thanos-querier-86b8856f97-b75hg" Nov 03 13:04:56.634648 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.634607 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9-secret-thanos-querier-tls\") pod \"thanos-querier-86b8856f97-b75hg\" (UID: \"0fef60a4-2a84-4318-b1bf-feb8f69ef7e9\") " pod="openshift-monitoring/thanos-querier-86b8856f97-b75hg" Nov 03 13:04:56.635583 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.635455 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-86b8856f97-b75hg\" (UID: \"0fef60a4-2a84-4318-b1bf-feb8f69ef7e9\") " pod="openshift-monitoring/thanos-querier-86b8856f97-b75hg" Nov 03 13:04:56.636020 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.635998 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9-secret-grpc-tls\") pod \"thanos-querier-86b8856f97-b75hg\" (UID: \"0fef60a4-2a84-4318-b1bf-feb8f69ef7e9\") " pod="openshift-monitoring/thanos-querier-86b8856f97-b75hg" Nov 03 13:04:56.639120 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.639080 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-86b8856f97-b75hg\" (UID: \"0fef60a4-2a84-4318-b1bf-feb8f69ef7e9\") " pod="openshift-monitoring/thanos-querier-86b8856f97-b75hg" Nov 03 13:04:56.684882 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.684853 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls676\" (UniqueName: \"kubernetes.io/projected/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9-kube-api-access-ls676\") pod \"thanos-querier-86b8856f97-b75hg\" (UID: \"0fef60a4-2a84-4318-b1bf-feb8f69ef7e9\") " pod="openshift-monitoring/thanos-querier-86b8856f97-b75hg" Nov 03 13:04:56.717403 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.717374 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-86b8856f97-b75hg" Nov 03 13:04:56.773195 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.773171 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 13:04:56.964161 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.964125 2580 generic.go:356] "Generic (PLEG): container finished" podID="0fadb078-8983-42df-afde-c1983044492d" containerID="a78f039b00668ee59cd1dbce0854bed1ed8fa5479aab6f34f37c830fdbea21c2" exitCode=0 Nov 03 13:04:56.964330 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.964219 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zhl5n" event={"ID":"0fadb078-8983-42df-afde-c1983044492d","Type":"ContainerDied","Data":"a78f039b00668ee59cd1dbce0854bed1ed8fa5479aab6f34f37c830fdbea21c2"} Nov 03 13:04:56.968096 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.966732 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fqskz" event={"ID":"0d0c5822-ab7b-409d-a755-49e918278fa1","Type":"ContainerStarted","Data":"0cc87b03b66e39962a7ba19c3f7e5e6b3876ff87c6837c4012b534c08cf0a8e3"} Nov 03 13:04:56.970753 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.970648 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7361d2a9-d089-4073-bbfb-e7978943fd88","Type":"ContainerStarted","Data":"f56f8782dab339e2e6d83b3ae256a7331821a82dc572428363224d45d35c046a"} Nov 03 13:04:56.972597 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:56.972542 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-krsxs" event={"ID":"b6263f8d-cced-465a-b393-ef51cb902e11","Type":"ContainerStarted","Data":"efb93c8d37a504658a3035b8b10ff032dacad6fa42c2961cc7d0b9ab5ed46cfe"} Nov 03 13:04:57.097172 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:57.097080 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-86b8856f97-b75hg"] Nov 03 13:04:57.109108 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:57.109077 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fef60a4_2a84_4318_b1bf_feb8f69ef7e9.slice/crio-6eac71e8dec9af14956357fa10a6d4595db34ba6f85cf32f4b8efd29d9426efc WatchSource:0}: Error finding container 6eac71e8dec9af14956357fa10a6d4595db34ba6f85cf32f4b8efd29d9426efc: Status 404 returned error can't find the container with id 6eac71e8dec9af14956357fa10a6d4595db34ba6f85cf32f4b8efd29d9426efc Nov 03 13:04:57.189052 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:57.188997 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fqskz" podStartSLOduration=2.427763657 podStartE2EDuration="4.188983049s" totalImagesPullingTime="1.761219392s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:04:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:04:57.188528414 +0000 UTC m=+51.040534602" watchObservedRunningTime="2026-11-03 13:04:57.188983049 +0000 UTC m=+51.040989236" Nov 03 13:04:57.978226 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:57.978184 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-848bc84498-gxvtr" event={"ID":"58dabb02-fd21-40ee-a4f5-e990b2b9a35a","Type":"ContainerStarted","Data":"9dd9a05764c13ae5588888535b4021e9098f621187d7ff9c8681c8a717ed3a9e"} Nov 03 13:04:57.980654 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:57.980625 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zhl5n" event={"ID":"0fadb078-8983-42df-afde-c1983044492d","Type":"ContainerStarted","Data":"ea0582b85047e57ef61163f7a90fb091c35478f6bd29d3be11d04d164fce1f8d"} Nov 03 13:04:57.980777 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:57.980667 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zhl5n" event={"ID":"0fadb078-8983-42df-afde-c1983044492d","Type":"ContainerStarted","Data":"fd033e51cf722b64e722c3978d85f077307c0bcc396880fa150c3cef615fc051"} Nov 03 13:04:57.982381 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:57.982354 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86b8856f97-b75hg" event={"ID":"0fef60a4-2a84-4318-b1bf-feb8f69ef7e9","Type":"ContainerStarted","Data":"6eac71e8dec9af14956357fa10a6d4595db34ba6f85cf32f4b8efd29d9426efc"} Nov 03 13:04:57.984741 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:57.984707 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-krsxs" event={"ID":"b6263f8d-cced-465a-b393-ef51cb902e11","Type":"ContainerStarted","Data":"a34c0f67b85b1aeda6c79681d1d6036dc44b344ea64ec183c9e886c52381b87e"} Nov 03 13:04:57.984831 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:57.984757 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-krsxs" event={"ID":"b6263f8d-cced-465a-b393-ef51cb902e11","Type":"ContainerStarted","Data":"254a47487a7e7a65d1fa6e4f57c89c606bdd2064e941edb4f822aef555a3484b"} Nov 03 13:04:58.087979 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:58.087878 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-848bc84498-gxvtr" podStartSLOduration=1.282094688 podStartE2EDuration="5.087863666s" totalImagesPullingTime="3.805768978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:04:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:04:58.086842713 +0000 UTC m=+51.938848901" watchObservedRunningTime="2026-11-03 13:04:58.087863666 +0000 UTC m=+51.939869853" Nov 03 13:04:58.196792 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:58.196702 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-zhl5n" podStartSLOduration=3.377670817 podStartE2EDuration="5.196688596s" totalImagesPullingTime="1.722989436s" totalInitContainerRuntime="96.028343ms" isStatefulPod=true podCreationTimestamp="2026-11-03 13:04:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:04:58.195916223 +0000 UTC m=+52.047922413" watchObservedRunningTime="2026-11-03 13:04:58.196688596 +0000 UTC m=+52.048694783" Nov 03 13:04:58.311135 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:58.311082 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-krsxs" podStartSLOduration=3.190439866 podStartE2EDuration="5.311067263s" totalImagesPullingTime="2.120627397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:04:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:04:58.310035514 +0000 UTC m=+52.162041702" watchObservedRunningTime="2026-11-03 13:04:58.311067263 +0000 UTC m=+52.163073448" Nov 03 13:04:58.542910 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:58.542874 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-cxrp7"] Nov 03 13:04:58.545225 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:58.545204 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-cxrp7" Nov 03 13:04:58.603010 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:58.602939 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-cxrp7"] Nov 03 13:04:58.619096 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:58.619066 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 03 13:04:58.619223 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:58.619156 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-8vx8q\"" Nov 03 13:04:58.649693 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:58.649661 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7bmn\" (UniqueName: \"kubernetes.io/projected/2c1f3b2f-1d7f-44b5-bcba-f9310e7bdb96-kube-api-access-z7bmn\") pod \"monitoring-plugin-77f4f76594-cxrp7\" (UID: \"2c1f3b2f-1d7f-44b5-bcba-f9310e7bdb96\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-cxrp7" Nov 03 13:04:58.649851 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:58.649703 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2c1f3b2f-1d7f-44b5-bcba-f9310e7bdb96-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-cxrp7\" (UID: \"2c1f3b2f-1d7f-44b5-bcba-f9310e7bdb96\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-cxrp7" Nov 03 13:04:58.750950 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:58.750914 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7bmn\" (UniqueName: \"kubernetes.io/projected/2c1f3b2f-1d7f-44b5-bcba-f9310e7bdb96-kube-api-access-z7bmn\") pod \"monitoring-plugin-77f4f76594-cxrp7\" (UID: \"2c1f3b2f-1d7f-44b5-bcba-f9310e7bdb96\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-cxrp7" Nov 03 13:04:58.751120 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:58.750966 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2c1f3b2f-1d7f-44b5-bcba-f9310e7bdb96-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-cxrp7\" (UID: \"2c1f3b2f-1d7f-44b5-bcba-f9310e7bdb96\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-cxrp7" Nov 03 13:04:58.754128 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:58.753092 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6488577f47-wwpwm"] Nov 03 13:04:58.754128 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:58.753909 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2c1f3b2f-1d7f-44b5-bcba-f9310e7bdb96-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-cxrp7\" (UID: \"2c1f3b2f-1d7f-44b5-bcba-f9310e7bdb96\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-cxrp7" Nov 03 13:04:58.755583 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:58.755563 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6488577f47-wwpwm" Nov 03 13:04:58.805456 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:58.805424 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 03 13:04:58.805678 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:58.805637 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 13:04:58.805678 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:58.805641 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 03 13:04:58.805831 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:58.805684 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-798zc\"" Nov 03 13:04:58.805831 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:58.805695 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 03 13:04:58.805831 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:58.805766 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-7adkcnd7vtot1\"" Nov 03 13:04:58.805984 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:58.805956 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7bmn\" (UniqueName: \"kubernetes.io/projected/2c1f3b2f-1d7f-44b5-bcba-f9310e7bdb96-kube-api-access-z7bmn\") pod \"monitoring-plugin-77f4f76594-cxrp7\" (UID: \"2c1f3b2f-1d7f-44b5-bcba-f9310e7bdb96\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-cxrp7" Nov 03 13:04:58.814636 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:58.813533 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6488577f47-wwpwm"] Nov 03 13:04:58.852142 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:58.852114 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5bfbbeb5-621b-43b0-bad1-0420e8b07b62-metrics-server-audit-profiles\") pod \"metrics-server-6488577f47-wwpwm\" (UID: \"5bfbbeb5-621b-43b0-bad1-0420e8b07b62\") " pod="openshift-monitoring/metrics-server-6488577f47-wwpwm" Nov 03 13:04:58.852350 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:58.852156 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgv82\" (UniqueName: \"kubernetes.io/projected/5bfbbeb5-621b-43b0-bad1-0420e8b07b62-kube-api-access-lgv82\") pod \"metrics-server-6488577f47-wwpwm\" (UID: \"5bfbbeb5-621b-43b0-bad1-0420e8b07b62\") " pod="openshift-monitoring/metrics-server-6488577f47-wwpwm" Nov 03 13:04:58.852350 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:58.852186 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5bfbbeb5-621b-43b0-bad1-0420e8b07b62-audit-log\") pod \"metrics-server-6488577f47-wwpwm\" (UID: \"5bfbbeb5-621b-43b0-bad1-0420e8b07b62\") " pod="openshift-monitoring/metrics-server-6488577f47-wwpwm" Nov 03 13:04:58.852461 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:58.852347 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5bfbbeb5-621b-43b0-bad1-0420e8b07b62-client-ca-bundle\") pod \"metrics-server-6488577f47-wwpwm\" (UID: \"5bfbbeb5-621b-43b0-bad1-0420e8b07b62\") " pod="openshift-monitoring/metrics-server-6488577f47-wwpwm" Nov 03 13:04:58.852461 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:58.852385 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bfbbeb5-621b-43b0-bad1-0420e8b07b62-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6488577f47-wwpwm\" (UID: \"5bfbbeb5-621b-43b0-bad1-0420e8b07b62\") " pod="openshift-monitoring/metrics-server-6488577f47-wwpwm" Nov 03 13:04:58.852461 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:58.852437 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5bfbbeb5-621b-43b0-bad1-0420e8b07b62-secret-metrics-server-client-certs\") pod \"metrics-server-6488577f47-wwpwm\" (UID: \"5bfbbeb5-621b-43b0-bad1-0420e8b07b62\") " pod="openshift-monitoring/metrics-server-6488577f47-wwpwm" Nov 03 13:04:58.852644 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:58.852461 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5bfbbeb5-621b-43b0-bad1-0420e8b07b62-secret-metrics-server-tls\") pod \"metrics-server-6488577f47-wwpwm\" (UID: \"5bfbbeb5-621b-43b0-bad1-0420e8b07b62\") " pod="openshift-monitoring/metrics-server-6488577f47-wwpwm" Nov 03 13:04:58.859078 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:58.859023 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-cxrp7" Nov 03 13:04:58.953068 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:58.953033 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bfbbeb5-621b-43b0-bad1-0420e8b07b62-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6488577f47-wwpwm\" (UID: \"5bfbbeb5-621b-43b0-bad1-0420e8b07b62\") " pod="openshift-monitoring/metrics-server-6488577f47-wwpwm" Nov 03 13:04:58.953223 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:58.953076 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5bfbbeb5-621b-43b0-bad1-0420e8b07b62-secret-metrics-server-client-certs\") pod \"metrics-server-6488577f47-wwpwm\" (UID: \"5bfbbeb5-621b-43b0-bad1-0420e8b07b62\") " pod="openshift-monitoring/metrics-server-6488577f47-wwpwm" Nov 03 13:04:58.953223 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:58.953109 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5bfbbeb5-621b-43b0-bad1-0420e8b07b62-secret-metrics-server-tls\") pod \"metrics-server-6488577f47-wwpwm\" (UID: \"5bfbbeb5-621b-43b0-bad1-0420e8b07b62\") " pod="openshift-monitoring/metrics-server-6488577f47-wwpwm" Nov 03 13:04:58.953223 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:58.953170 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5bfbbeb5-621b-43b0-bad1-0420e8b07b62-metrics-server-audit-profiles\") pod \"metrics-server-6488577f47-wwpwm\" (UID: \"5bfbbeb5-621b-43b0-bad1-0420e8b07b62\") " pod="openshift-monitoring/metrics-server-6488577f47-wwpwm" Nov 03 13:04:58.953223 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:58.953213 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgv82\" (UniqueName: \"kubernetes.io/projected/5bfbbeb5-621b-43b0-bad1-0420e8b07b62-kube-api-access-lgv82\") pod \"metrics-server-6488577f47-wwpwm\" (UID: \"5bfbbeb5-621b-43b0-bad1-0420e8b07b62\") " pod="openshift-monitoring/metrics-server-6488577f47-wwpwm" Nov 03 13:04:58.953436 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:58.953250 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5bfbbeb5-621b-43b0-bad1-0420e8b07b62-audit-log\") pod \"metrics-server-6488577f47-wwpwm\" (UID: \"5bfbbeb5-621b-43b0-bad1-0420e8b07b62\") " pod="openshift-monitoring/metrics-server-6488577f47-wwpwm" Nov 03 13:04:58.953436 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:58.953319 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5bfbbeb5-621b-43b0-bad1-0420e8b07b62-client-ca-bundle\") pod \"metrics-server-6488577f47-wwpwm\" (UID: \"5bfbbeb5-621b-43b0-bad1-0420e8b07b62\") " pod="openshift-monitoring/metrics-server-6488577f47-wwpwm" Nov 03 13:04:58.953868 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:58.953842 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bfbbeb5-621b-43b0-bad1-0420e8b07b62-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6488577f47-wwpwm\" (UID: \"5bfbbeb5-621b-43b0-bad1-0420e8b07b62\") " pod="openshift-monitoring/metrics-server-6488577f47-wwpwm" Nov 03 13:04:58.954441 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:58.954415 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5bfbbeb5-621b-43b0-bad1-0420e8b07b62-audit-log\") pod \"metrics-server-6488577f47-wwpwm\" (UID: \"5bfbbeb5-621b-43b0-bad1-0420e8b07b62\") " pod="openshift-monitoring/metrics-server-6488577f47-wwpwm" Nov 03 13:04:58.954637 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:58.954614 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5bfbbeb5-621b-43b0-bad1-0420e8b07b62-metrics-server-audit-profiles\") pod \"metrics-server-6488577f47-wwpwm\" (UID: \"5bfbbeb5-621b-43b0-bad1-0420e8b07b62\") " pod="openshift-monitoring/metrics-server-6488577f47-wwpwm" Nov 03 13:04:58.955984 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:58.955961 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5bfbbeb5-621b-43b0-bad1-0420e8b07b62-client-ca-bundle\") pod \"metrics-server-6488577f47-wwpwm\" (UID: \"5bfbbeb5-621b-43b0-bad1-0420e8b07b62\") " pod="openshift-monitoring/metrics-server-6488577f47-wwpwm" Nov 03 13:04:58.956489 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:58.956451 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5bfbbeb5-621b-43b0-bad1-0420e8b07b62-secret-metrics-server-client-certs\") pod \"metrics-server-6488577f47-wwpwm\" (UID: \"5bfbbeb5-621b-43b0-bad1-0420e8b07b62\") " pod="openshift-monitoring/metrics-server-6488577f47-wwpwm" Nov 03 13:04:58.956569 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:58.956462 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5bfbbeb5-621b-43b0-bad1-0420e8b07b62-secret-metrics-server-tls\") pod \"metrics-server-6488577f47-wwpwm\" (UID: \"5bfbbeb5-621b-43b0-bad1-0420e8b07b62\") " pod="openshift-monitoring/metrics-server-6488577f47-wwpwm" Nov 03 13:04:58.990747 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:58.990536 2580 generic.go:356] "Generic (PLEG): container finished" podID="7361d2a9-d089-4073-bbfb-e7978943fd88" containerID="aaea79a6899ef0f795f4cbb0ee4424a703035e5af1aeed5baf0609f417f64e5c" exitCode=0 Nov 03 13:04:58.990747 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:58.990631 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7361d2a9-d089-4073-bbfb-e7978943fd88","Type":"ContainerDied","Data":"aaea79a6899ef0f795f4cbb0ee4424a703035e5af1aeed5baf0609f417f64e5c"} Nov 03 13:04:59.007919 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:59.007897 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgv82\" (UniqueName: \"kubernetes.io/projected/5bfbbeb5-621b-43b0-bad1-0420e8b07b62-kube-api-access-lgv82\") pod \"metrics-server-6488577f47-wwpwm\" (UID: \"5bfbbeb5-621b-43b0-bad1-0420e8b07b62\") " pod="openshift-monitoring/metrics-server-6488577f47-wwpwm" Nov 03 13:04:59.072174 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:59.071931 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6488577f47-wwpwm" Nov 03 13:04:59.225906 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:59.225875 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-cxrp7"] Nov 03 13:04:59.232896 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:59.232858 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c1f3b2f_1d7f_44b5_bcba_f9310e7bdb96.slice/crio-49b66c76afba64193c0a47b9415b935a5553d5caae03c6370b8574979bb86404 WatchSource:0}: Error finding container 49b66c76afba64193c0a47b9415b935a5553d5caae03c6370b8574979bb86404: Status 404 returned error can't find the container with id 49b66c76afba64193c0a47b9415b935a5553d5caae03c6370b8574979bb86404 Nov 03 13:04:59.443044 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:59.443007 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6488577f47-wwpwm"] Nov 03 13:04:59.450233 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:04:59.450192 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bfbbeb5_621b_43b0_bad1_0420e8b07b62.slice/crio-a9e9596490c552decf61398ebee95872f8e6ece435e16fd80d0daea5fc85f19e WatchSource:0}: Error finding container a9e9596490c552decf61398ebee95872f8e6ece435e16fd80d0daea5fc85f19e: Status 404 returned error can't find the container with id a9e9596490c552decf61398ebee95872f8e6ece435e16fd80d0daea5fc85f19e Nov 03 13:04:59.996593 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:59.996502 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86b8856f97-b75hg" event={"ID":"0fef60a4-2a84-4318-b1bf-feb8f69ef7e9","Type":"ContainerStarted","Data":"3565c5b83032631fb865dfae6e77ec5b16b98b268ee2c1b8b5b0a0a28dd6188a"} Nov 03 13:04:59.996593 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:59.996582 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86b8856f97-b75hg" event={"ID":"0fef60a4-2a84-4318-b1bf-feb8f69ef7e9","Type":"ContainerStarted","Data":"72481b8827d18d71da99a05be403c50f7c84c660e1b544d254a0fdcf3a72fd2a"} Nov 03 13:04:59.996779 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:59.996618 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86b8856f97-b75hg" event={"ID":"0fef60a4-2a84-4318-b1bf-feb8f69ef7e9","Type":"ContainerStarted","Data":"6aa572c67cb6d1073c354587ee12c0332d7bf6fa3636ae243e8c213e9498853d"} Nov 03 13:04:59.997733 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:59.997701 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-cxrp7" event={"ID":"2c1f3b2f-1d7f-44b5-bcba-f9310e7bdb96","Type":"ContainerStarted","Data":"49b66c76afba64193c0a47b9415b935a5553d5caae03c6370b8574979bb86404"} Nov 03 13:04:59.998841 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:04:59.998818 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6488577f47-wwpwm" event={"ID":"5bfbbeb5-621b-43b0-bad1-0420e8b07b62","Type":"ContainerStarted","Data":"a9e9596490c552decf61398ebee95872f8e6ece435e16fd80d0daea5fc85f19e"} Nov 03 13:05:01.874857 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:01.874793 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:05:02.010805 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.010766 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86b8856f97-b75hg" event={"ID":"0fef60a4-2a84-4318-b1bf-feb8f69ef7e9","Type":"ContainerStarted","Data":"6d0be7c2d97f4ab26efaeaab0ebcfa0e84eb30445a27c1ae040671890aa0bee1"} Nov 03 13:05:02.010987 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.010842 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86b8856f97-b75hg" event={"ID":"0fef60a4-2a84-4318-b1bf-feb8f69ef7e9","Type":"ContainerStarted","Data":"f4827bf90b1debdbe9d97b3689dc2664112ce7e8cb821dab6ef64faf260549ba"} Nov 03 13:05:02.010987 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.010882 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86b8856f97-b75hg" event={"ID":"0fef60a4-2a84-4318-b1bf-feb8f69ef7e9","Type":"ContainerStarted","Data":"e2c356ea0abb7283b99c9a070a2124ecea4ef336362609279704242fb408c2a7"} Nov 03 13:05:02.010987 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.010930 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-86b8856f97-b75hg" Nov 03 13:05:02.012808 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.012778 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-cxrp7" event={"ID":"2c1f3b2f-1d7f-44b5-bcba-f9310e7bdb96","Type":"ContainerStarted","Data":"68203a2436d0a3a26d405a23b32ab0659945bdc5ee3d4a7aae092ee6aa3ac134"} Nov 03 13:05:02.013104 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.013087 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-cxrp7" Nov 03 13:05:02.014460 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.014413 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6488577f47-wwpwm" event={"ID":"5bfbbeb5-621b-43b0-bad1-0420e8b07b62","Type":"ContainerStarted","Data":"94cd22826c7d7403ce2b8e139bc9364fb887f75cb89bdaea9c09869471229c9b"} Nov 03 13:05:02.018733 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.018714 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-cxrp7" Nov 03 13:05:02.153888 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.153835 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-cxrp7" podStartSLOduration=2.22825776 podStartE2EDuration="4.153819233s" totalImagesPullingTime="1.925561473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:04:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:05:02.153104689 +0000 UTC m=+56.005110888" watchObservedRunningTime="2026-11-03 13:05:02.153819233 +0000 UTC m=+56.005825420" Nov 03 13:05:02.269606 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.269563 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-86b8856f97-b75hg" podStartSLOduration=2.46579754 podStartE2EDuration="6.269549563s" totalImagesPullingTime="3.803752023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:04:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:05:02.268193488 +0000 UTC m=+56.120199678" watchObservedRunningTime="2026-11-03 13:05:02.269549563 +0000 UTC m=+56.121555750" Nov 03 13:05:02.380308 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.380243 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6488577f47-wwpwm" podStartSLOduration=2.393013223 podStartE2EDuration="4.38021857s" totalImagesPullingTime="1.987205347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:04:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:05:02.377740015 +0000 UTC m=+56.229746205" watchObservedRunningTime="2026-11-03 13:05:02.38021857 +0000 UTC m=+56.232224758" Nov 03 13:05:02.763092 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.763015 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6f44499665-ff6gp"] Nov 03 13:05:02.765344 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.765324 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f44499665-ff6gp" Nov 03 13:05:02.862930 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.862891 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 03 13:05:02.864178 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.864156 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f44499665-ff6gp"] Nov 03 13:05:02.869073 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.869051 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 13:05:02.871891 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.871875 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:02.892640 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.892613 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2dcde035-9b90-4961-8cfc-3f37bfb67581-console-config\") pod \"console-6f44499665-ff6gp\" (UID: \"2dcde035-9b90-4961-8cfc-3f37bfb67581\") " pod="openshift-console/console-6f44499665-ff6gp" Nov 03 13:05:02.893018 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.892667 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2dcde035-9b90-4961-8cfc-3f37bfb67581-console-oauth-config\") pod \"console-6f44499665-ff6gp\" (UID: \"2dcde035-9b90-4961-8cfc-3f37bfb67581\") " pod="openshift-console/console-6f44499665-ff6gp" Nov 03 13:05:02.893018 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.892733 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dcde035-9b90-4961-8cfc-3f37bfb67581-trusted-ca-bundle\") pod \"console-6f44499665-ff6gp\" (UID: \"2dcde035-9b90-4961-8cfc-3f37bfb67581\") " pod="openshift-console/console-6f44499665-ff6gp" Nov 03 13:05:02.893018 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.892782 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gb4j\" (UniqueName: \"kubernetes.io/projected/2dcde035-9b90-4961-8cfc-3f37bfb67581-kube-api-access-7gb4j\") pod \"console-6f44499665-ff6gp\" (UID: \"2dcde035-9b90-4961-8cfc-3f37bfb67581\") " pod="openshift-console/console-6f44499665-ff6gp" Nov 03 13:05:02.893018 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.892851 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2dcde035-9b90-4961-8cfc-3f37bfb67581-tmp\") pod \"console-6f44499665-ff6gp\" (UID: \"2dcde035-9b90-4961-8cfc-3f37bfb67581\") " pod="openshift-console/console-6f44499665-ff6gp" Nov 03 13:05:02.893018 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.892904 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2dcde035-9b90-4961-8cfc-3f37bfb67581-service-ca\") pod \"console-6f44499665-ff6gp\" (UID: \"2dcde035-9b90-4961-8cfc-3f37bfb67581\") " pod="openshift-console/console-6f44499665-ff6gp" Nov 03 13:05:02.893018 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.892960 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2dcde035-9b90-4961-8cfc-3f37bfb67581-console-serving-cert\") pod \"console-6f44499665-ff6gp\" (UID: \"2dcde035-9b90-4961-8cfc-3f37bfb67581\") " pod="openshift-console/console-6f44499665-ff6gp" Nov 03 13:05:02.893018 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.892976 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2dcde035-9b90-4961-8cfc-3f37bfb67581-oauth-serving-cert\") pod \"console-6f44499665-ff6gp\" (UID: \"2dcde035-9b90-4961-8cfc-3f37bfb67581\") " pod="openshift-console/console-6f44499665-ff6gp" Nov 03 13:05:02.922331 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.922148 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 03 13:05:02.922331 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.922170 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 03 13:05:02.922331 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.922159 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 03 13:05:02.922331 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.922249 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 03 13:05:02.922331 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.922267 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 03 13:05:02.922331 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.922269 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 03 13:05:02.922331 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.922291 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 03 13:05:02.922331 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.922303 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 03 13:05:02.922331 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.922302 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 03 13:05:02.922331 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.922314 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4ia0ld9dido9e\"" Nov 03 13:05:02.922331 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.922344 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 03 13:05:02.923544 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.922563 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 03 13:05:02.923544 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.922572 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 03 13:05:02.923544 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.922631 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-cgshq\"" Nov 03 13:05:02.970041 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.970004 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 03 13:05:02.970670 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.970652 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 03 13:05:02.971678 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.971661 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 13:05:02.993627 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.993594 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:02.993749 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.993631 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:02.993749 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.993659 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-web-config\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:02.993854 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.993747 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:02.993854 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.993775 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:02.993854 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.993844 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:02.993982 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.993900 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2dcde035-9b90-4961-8cfc-3f37bfb67581-console-serving-cert\") pod \"console-6f44499665-ff6gp\" (UID: \"2dcde035-9b90-4961-8cfc-3f37bfb67581\") " pod="openshift-console/console-6f44499665-ff6gp" Nov 03 13:05:02.993982 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.993922 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2dcde035-9b90-4961-8cfc-3f37bfb67581-oauth-serving-cert\") pod \"console-6f44499665-ff6gp\" (UID: \"2dcde035-9b90-4961-8cfc-3f37bfb67581\") " pod="openshift-console/console-6f44499665-ff6gp" Nov 03 13:05:02.993982 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.993948 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:02.993982 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.993964 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdvh2\" (UniqueName: \"kubernetes.io/projected/30d608b6-d82e-4fcf-ab44-b8e064c463d0-kube-api-access-jdvh2\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:02.994165 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.993990 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:02.994165 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.994025 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:02.994250 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.994176 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2dcde035-9b90-4961-8cfc-3f37bfb67581-console-config\") pod \"console-6f44499665-ff6gp\" (UID: \"2dcde035-9b90-4961-8cfc-3f37bfb67581\") " pod="openshift-console/console-6f44499665-ff6gp" Nov 03 13:05:02.994250 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.994217 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2dcde035-9b90-4961-8cfc-3f37bfb67581-console-oauth-config\") pod \"console-6f44499665-ff6gp\" (UID: \"2dcde035-9b90-4961-8cfc-3f37bfb67581\") " pod="openshift-console/console-6f44499665-ff6gp" Nov 03 13:05:02.994250 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.994237 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dcde035-9b90-4961-8cfc-3f37bfb67581-trusted-ca-bundle\") pod \"console-6f44499665-ff6gp\" (UID: \"2dcde035-9b90-4961-8cfc-3f37bfb67581\") " pod="openshift-console/console-6f44499665-ff6gp" Nov 03 13:05:02.994386 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.994253 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:02.994386 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.994288 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:02.994386 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.994354 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/30d608b6-d82e-4fcf-ab44-b8e064c463d0-config-out\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:02.994545 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.994394 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/30d608b6-d82e-4fcf-ab44-b8e064c463d0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:02.994545 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.994427 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gb4j\" (UniqueName: \"kubernetes.io/projected/2dcde035-9b90-4961-8cfc-3f37bfb67581-kube-api-access-7gb4j\") pod \"console-6f44499665-ff6gp\" (UID: \"2dcde035-9b90-4961-8cfc-3f37bfb67581\") " pod="openshift-console/console-6f44499665-ff6gp" Nov 03 13:05:02.994545 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.994485 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/30d608b6-d82e-4fcf-ab44-b8e064c463d0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:02.994689 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.994634 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2dcde035-9b90-4961-8cfc-3f37bfb67581-tmp\") pod \"console-6f44499665-ff6gp\" (UID: \"2dcde035-9b90-4961-8cfc-3f37bfb67581\") " pod="openshift-console/console-6f44499665-ff6gp" Nov 03 13:05:02.994689 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.994682 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-config\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:02.994792 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.994709 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:02.994792 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.994735 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:02.994792 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.994766 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2dcde035-9b90-4961-8cfc-3f37bfb67581-service-ca\") pod \"console-6f44499665-ff6gp\" (UID: \"2dcde035-9b90-4961-8cfc-3f37bfb67581\") " pod="openshift-console/console-6f44499665-ff6gp" Nov 03 13:05:02.994945 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.994793 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:02.994945 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.994809 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2dcde035-9b90-4961-8cfc-3f37bfb67581-oauth-serving-cert\") pod \"console-6f44499665-ff6gp\" (UID: \"2dcde035-9b90-4961-8cfc-3f37bfb67581\") " pod="openshift-console/console-6f44499665-ff6gp" Nov 03 13:05:02.994945 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.994836 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:02.994945 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.994926 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2dcde035-9b90-4961-8cfc-3f37bfb67581-console-config\") pod \"console-6f44499665-ff6gp\" (UID: \"2dcde035-9b90-4961-8cfc-3f37bfb67581\") " pod="openshift-console/console-6f44499665-ff6gp" Nov 03 13:05:02.995151 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.995013 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2dcde035-9b90-4961-8cfc-3f37bfb67581-tmp\") pod \"console-6f44499665-ff6gp\" (UID: \"2dcde035-9b90-4961-8cfc-3f37bfb67581\") " pod="openshift-console/console-6f44499665-ff6gp" Nov 03 13:05:02.995211 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.995172 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dcde035-9b90-4961-8cfc-3f37bfb67581-trusted-ca-bundle\") pod \"console-6f44499665-ff6gp\" (UID: \"2dcde035-9b90-4961-8cfc-3f37bfb67581\") " pod="openshift-console/console-6f44499665-ff6gp" Nov 03 13:05:02.995436 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.995416 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2dcde035-9b90-4961-8cfc-3f37bfb67581-service-ca\") pod \"console-6f44499665-ff6gp\" (UID: \"2dcde035-9b90-4961-8cfc-3f37bfb67581\") " pod="openshift-console/console-6f44499665-ff6gp" Nov 03 13:05:02.996462 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.996442 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2dcde035-9b90-4961-8cfc-3f37bfb67581-console-serving-cert\") pod \"console-6f44499665-ff6gp\" (UID: \"2dcde035-9b90-4961-8cfc-3f37bfb67581\") " pod="openshift-console/console-6f44499665-ff6gp" Nov 03 13:05:02.996562 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:02.996511 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2dcde035-9b90-4961-8cfc-3f37bfb67581-console-oauth-config\") pod \"console-6f44499665-ff6gp\" (UID: \"2dcde035-9b90-4961-8cfc-3f37bfb67581\") " pod="openshift-console/console-6f44499665-ff6gp" Nov 03 13:05:03.020706 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:03.020636 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7361d2a9-d089-4073-bbfb-e7978943fd88","Type":"ContainerStarted","Data":"05a184bc5f5cf4f5a33ef8586c6a32a7f5ad9641153bddc9a56ae94ccbe70d22"} Nov 03 13:05:03.020706 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:03.020701 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7361d2a9-d089-4073-bbfb-e7978943fd88","Type":"ContainerStarted","Data":"12e48d688f061a1abfcc0dc66deb4cec44009b7f2fda1d12dea0d3e7a5d49a12"} Nov 03 13:05:03.020861 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:03.020731 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7361d2a9-d089-4073-bbfb-e7978943fd88","Type":"ContainerStarted","Data":"f79932046af38f76c865b7d005c526fc9e7e5523481bcd2f827dd953ca44aa88"} Nov 03 13:05:03.020861 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:03.020757 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7361d2a9-d089-4073-bbfb-e7978943fd88","Type":"ContainerStarted","Data":"aa9e6c8cb1885d86a9fced9f978440068e3b11ef943accc278c67b2da19f39ba"} Nov 03 13:05:03.020861 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:03.020792 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7361d2a9-d089-4073-bbfb-e7978943fd88","Type":"ContainerStarted","Data":"e56f4e26a0db91adbfd5a8af6e1988a5d70051fc325afc248091a808c3d0121b"} Nov 03 13:05:03.020861 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:03.020829 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7361d2a9-d089-4073-bbfb-e7978943fd88","Type":"ContainerStarted","Data":"75e30b1841c2fb988dd3a2203d48ab05c7189a12cb64c10e3a29c35cede9ecaf"} Nov 03 13:05:03.048452 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:03.048423 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gb4j\" (UniqueName: \"kubernetes.io/projected/2dcde035-9b90-4961-8cfc-3f37bfb67581-kube-api-access-7gb4j\") pod \"console-6f44499665-ff6gp\" (UID: \"2dcde035-9b90-4961-8cfc-3f37bfb67581\") " pod="openshift-console/console-6f44499665-ff6gp" Nov 03 13:05:03.075729 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:03.075700 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f44499665-ff6gp" Nov 03 13:05:03.095399 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:03.095372 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-config\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:03.095530 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:03.095420 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:03.095530 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:03.095445 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:03.095530 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:03.095493 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:03.096505 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:03.095938 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:03.096505 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:03.095994 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:03.096505 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:03.096017 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:03.096505 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:03.096040 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-web-config\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:03.096505 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:03.096063 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:03.096505 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:03.096089 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:03.096505 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:03.096125 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:03.096505 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:03.096171 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:03.096505 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:03.096198 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdvh2\" (UniqueName: \"kubernetes.io/projected/30d608b6-d82e-4fcf-ab44-b8e064c463d0-kube-api-access-jdvh2\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:03.096505 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:03.096239 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:03.096505 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:03.096264 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:03.096505 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:03.096334 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:03.096505 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:03.096380 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:03.097221 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:03.096410 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/30d608b6-d82e-4fcf-ab44-b8e064c463d0-config-out\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:03.097221 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:03.096822 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/30d608b6-d82e-4fcf-ab44-b8e064c463d0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:03.097221 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:03.096907 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/30d608b6-d82e-4fcf-ab44-b8e064c463d0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:03.097399 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:03.097375 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:03.097687 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:03.097666 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:03.098824 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:03.098799 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:03.098963 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:03.098923 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:03.099430 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:03.099405 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:03.100376 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:03.100351 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:03.100943 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:03.100916 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/30d608b6-d82e-4fcf-ab44-b8e064c463d0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:03.102506 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:03.102256 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:03.102892 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:03.102866 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-config\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:03.103066 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:03.103045 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:03.103828 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:03.103730 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-web-config\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:03.104249 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:03.104229 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/30d608b6-d82e-4fcf-ab44-b8e064c463d0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:03.104951 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:03.104923 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:03.105367 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:03.105342 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/30d608b6-d82e-4fcf-ab44-b8e064c463d0-config-out\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:03.105564 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:03.105530 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:03.105905 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:03.105881 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:03.107818 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:03.107797 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:03.108116 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:03.108095 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:03.108332 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:03.108310 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:03.153549 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:03.153463 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.357236162 podStartE2EDuration="8.153449992s" totalImagesPullingTime="4.782835889s" totalInitContainerRuntime="13.377941ms" isStatefulPod=false podCreationTimestamp="2026-11-03 13:04:55 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:05:03.152209214 +0000 UTC m=+57.004215439" watchObservedRunningTime="2026-11-03 13:05:03.153449992 +0000 UTC m=+57.005456180" Nov 03 13:05:03.155567 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:03.155545 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdvh2\" (UniqueName: \"kubernetes.io/projected/30d608b6-d82e-4fcf-ab44-b8e064c463d0-kube-api-access-jdvh2\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:03.190055 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:03.190024 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:03.442829 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:03.442794 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f44499665-ff6gp"] Nov 03 13:05:03.448577 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:05:03.448550 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dcde035_9b90_4961_8cfc_3f37bfb67581.slice/crio-021415e37c70ce8f7864ca9d751fb5be0f42c656f462e392f6bbcfb72e3be2ad WatchSource:0}: Error finding container 021415e37c70ce8f7864ca9d751fb5be0f42c656f462e392f6bbcfb72e3be2ad: Status 404 returned error can't find the container with id 021415e37c70ce8f7864ca9d751fb5be0f42c656f462e392f6bbcfb72e3be2ad Nov 03 13:05:03.528636 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:03.528600 2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-848bc84498-gxvtr" Nov 03 13:05:03.528792 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:03.528685 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-848bc84498-gxvtr" Nov 03 13:05:03.533857 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:03.533833 2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-848bc84498-gxvtr" Nov 03 13:05:03.614856 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:03.614826 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 13:05:03.621588 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:05:03.621553 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30d608b6_d82e_4fcf_ab44_b8e064c463d0.slice/crio-cfe8a5916ef9fe15a5e0a8f79b48c137087a707357efd660e29b358bd865bb40 WatchSource:0}: Error finding container cfe8a5916ef9fe15a5e0a8f79b48c137087a707357efd660e29b358bd865bb40: Status 404 returned error can't find the container with id cfe8a5916ef9fe15a5e0a8f79b48c137087a707357efd660e29b358bd865bb40 Nov 03 13:05:03.957817 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:03.957748 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-87tkk" Nov 03 13:05:04.026000 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:04.025969 2580 generic.go:356] "Generic (PLEG): container finished" podID="30d608b6-d82e-4fcf-ab44-b8e064c463d0" containerID="1caee93a0441c7c697a1ab2aa46e921bc1aca35c3270332e260a2665db7555a6" exitCode=0 Nov 03 13:05:04.026157 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:04.026059 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"30d608b6-d82e-4fcf-ab44-b8e064c463d0","Type":"ContainerDied","Data":"1caee93a0441c7c697a1ab2aa46e921bc1aca35c3270332e260a2665db7555a6"} Nov 03 13:05:04.026157 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:04.026140 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"30d608b6-d82e-4fcf-ab44-b8e064c463d0","Type":"ContainerStarted","Data":"cfe8a5916ef9fe15a5e0a8f79b48c137087a707357efd660e29b358bd865bb40"} Nov 03 13:05:04.027665 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:04.027644 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f44499665-ff6gp" event={"ID":"2dcde035-9b90-4961-8cfc-3f37bfb67581","Type":"ContainerStarted","Data":"8e111aef53dca254384b00b073010d66493f2c2fd5e216ce5bacfa845f43c72a"} Nov 03 13:05:04.027767 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:04.027689 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f44499665-ff6gp" event={"ID":"2dcde035-9b90-4961-8cfc-3f37bfb67581","Type":"ContainerStarted","Data":"021415e37c70ce8f7864ca9d751fb5be0f42c656f462e392f6bbcfb72e3be2ad"} Nov 03 13:05:04.031669 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:04.031652 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-848bc84498-gxvtr" Nov 03 13:05:04.286407 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:04.286309 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6f44499665-ff6gp" podStartSLOduration=2.286295506 podStartE2EDuration="2.286295506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:05:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:05:04.285575048 +0000 UTC m=+58.137581260" watchObservedRunningTime="2026-11-03 13:05:04.286295506 +0000 UTC m=+58.138301693" Nov 03 13:05:08.026835 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:08.026754 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-86b8856f97-b75hg" Nov 03 13:05:08.051632 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:08.051602 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"30d608b6-d82e-4fcf-ab44-b8e064c463d0","Type":"ContainerStarted","Data":"3bb5b563b33e764f2cacb6395268c2a267d365ce2eb2be0658dbef1abe48d34a"} Nov 03 13:05:08.051782 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:08.051667 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"30d608b6-d82e-4fcf-ab44-b8e064c463d0","Type":"ContainerStarted","Data":"be6f49e9d541c310aac05d244ec6ee360d277537e656c034e6099e963b47149f"} Nov 03 13:05:08.051782 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:08.051696 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"30d608b6-d82e-4fcf-ab44-b8e064c463d0","Type":"ContainerStarted","Data":"040ae9aec0f27bf15e42a48919412f565f4fa698b28cf15787eabe7b4fabe2f9"} Nov 03 13:05:08.051782 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:08.051725 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"30d608b6-d82e-4fcf-ab44-b8e064c463d0","Type":"ContainerStarted","Data":"9861343ab263f81c1dbee8304feff9c17d8de8b8d176c47e5196c2b17e6287ce"} Nov 03 13:05:08.051782 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:08.051750 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"30d608b6-d82e-4fcf-ab44-b8e064c463d0","Type":"ContainerStarted","Data":"0acd1240c69ae69c18d842f9defe65913faf7126e2d9c432a6199e0da68b4570"} Nov 03 13:05:08.051782 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:08.051777 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"30d608b6-d82e-4fcf-ab44-b8e064c463d0","Type":"ContainerStarted","Data":"e797acde2976802df9507cb8adea62190a05660ef0fced2b2afe7cb4075fdc72"} Nov 03 13:05:08.190405 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:08.190374 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:08.390954 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:08.390897 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.968971758 podStartE2EDuration="6.390883344s" totalImagesPullingTime="3.406399302s" totalInitContainerRuntime="15.512284ms" isStatefulPod=false podCreationTimestamp="2026-11-03 13:05:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:05:08.390429841 +0000 UTC m=+62.242436028" watchObservedRunningTime="2026-11-03 13:05:08.390883344 +0000 UTC m=+62.242889561" Nov 03 13:05:11.670022 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:11.669985 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895-original-pull-secret\") pod \"global-pull-secret-syncer-48vd8\" (UID: \"2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895\") " pod="kube-system/global-pull-secret-syncer-48vd8" Nov 03 13:05:11.670417 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:11.670130 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a9bc0be-51b3-4a52-83ae-580c3a2f342c-metrics-certs\") pod \"network-metrics-daemon-dczcf\" (UID: \"8a9bc0be-51b3-4a52-83ae-580c3a2f342c\") " pod="openshift-multus/network-metrics-daemon-dczcf" Nov 03 13:05:11.766041 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:11.766009 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 13:05:11.766041 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:11.766028 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 13:05:11.770540 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:11.770515 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzl6r\" (UniqueName: \"kubernetes.io/projected/58a9c629-dff8-4f48-99b2-385d5d13e188-kube-api-access-kzl6r\") pod \"network-check-target-9v5mk\" (UID: \"58a9c629-dff8-4f48-99b2-385d5d13e188\") " pod="openshift-network-diagnostics/network-check-target-9v5mk" Nov 03 13:05:11.772342 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:11.772314 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895-original-pull-secret\") pod \"global-pull-secret-syncer-48vd8\" (UID: \"2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895\") " pod="kube-system/global-pull-secret-syncer-48vd8" Nov 03 13:05:11.772592 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:11.772574 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a9bc0be-51b3-4a52-83ae-580c3a2f342c-metrics-certs\") pod \"network-metrics-daemon-dczcf\" (UID: \"8a9bc0be-51b3-4a52-83ae-580c3a2f342c\") " pod="openshift-multus/network-metrics-daemon-dczcf" Nov 03 13:05:11.819360 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:11.819330 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 13:05:11.869872 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:11.869841 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 13:05:11.874329 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:11.874299 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzl6r\" (UniqueName: \"kubernetes.io/projected/58a9c629-dff8-4f48-99b2-385d5d13e188-kube-api-access-kzl6r\") pod \"network-check-target-9v5mk\" (UID: \"58a9c629-dff8-4f48-99b2-385d5d13e188\") " pod="openshift-network-diagnostics/network-check-target-9v5mk" Nov 03 13:05:11.967674 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:11.967598 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mnhgg\"" Nov 03 13:05:11.969623 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:11.969604 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48vd8" Nov 03 13:05:11.994944 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:11.994917 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pvm6f\"" Nov 03 13:05:11.996926 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:11.996909 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dczcf" Nov 03 13:05:12.056344 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:12.056323 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-x4nbm\"" Nov 03 13:05:12.058891 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:12.058867 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9v5mk" Nov 03 13:05:12.339592 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:12.339516 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-48vd8"] Nov 03 13:05:12.369895 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:05:12.369871 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ca5d5b3_4d10_46f6_8f5f_22d0d43e9895.slice/crio-6c3945e8883e0a259fd5f007da8c7429799c94f6107a8e45f05f5e2948efe6ed WatchSource:0}: Error finding container 6c3945e8883e0a259fd5f007da8c7429799c94f6107a8e45f05f5e2948efe6ed: Status 404 returned error can't find the container with id 6c3945e8883e0a259fd5f007da8c7429799c94f6107a8e45f05f5e2948efe6ed Nov 03 13:05:12.392408 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:12.392379 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dczcf"] Nov 03 13:05:12.398302 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:05:12.398275 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a9bc0be_51b3_4a52_83ae_580c3a2f342c.slice/crio-c1369db2442df97ea5391783faba8be0a5326524580dac31a65caaa88066fb82 WatchSource:0}: Error finding container c1369db2442df97ea5391783faba8be0a5326524580dac31a65caaa88066fb82: Status 404 returned error can't find the container with id c1369db2442df97ea5391783faba8be0a5326524580dac31a65caaa88066fb82 Nov 03 13:05:12.441981 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:12.441952 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9v5mk"] Nov 03 13:05:12.447808 ip-10-0-130-198 kubenswrapper[2580]: W1103 13:05:12.447784 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a9c629_dff8_4f48_99b2_385d5d13e188.slice/crio-b32d70e0c0296b39175111442498979160d7e804c4d7b1876b6e227343cfc7ab WatchSource:0}: Error finding container b32d70e0c0296b39175111442498979160d7e804c4d7b1876b6e227343cfc7ab: Status 404 returned error can't find the container with id b32d70e0c0296b39175111442498979160d7e804c4d7b1876b6e227343cfc7ab Nov 03 13:05:13.075840 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:13.075811 2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6f44499665-ff6gp" Nov 03 13:05:13.076316 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:13.075873 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6f44499665-ff6gp" Nov 03 13:05:13.076316 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:13.076005 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9v5mk" event={"ID":"58a9c629-dff8-4f48-99b2-385d5d13e188","Type":"ContainerStarted","Data":"b32d70e0c0296b39175111442498979160d7e804c4d7b1876b6e227343cfc7ab"} Nov 03 13:05:13.077387 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:13.077343 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dczcf" event={"ID":"8a9bc0be-51b3-4a52-83ae-580c3a2f342c","Type":"ContainerStarted","Data":"c1369db2442df97ea5391783faba8be0a5326524580dac31a65caaa88066fb82"} Nov 03 13:05:13.078554 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:13.078533 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-48vd8" event={"ID":"2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895","Type":"ContainerStarted","Data":"6c3945e8883e0a259fd5f007da8c7429799c94f6107a8e45f05f5e2948efe6ed"} Nov 03 13:05:13.080826 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:13.080806 2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6f44499665-ff6gp" Nov 03 13:05:14.089693 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:14.089665 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6f44499665-ff6gp" Nov 03 13:05:14.248455 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:14.248425 2580 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-848bc84498-gxvtr"] Nov 03 13:05:15.089835 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:15.089789 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dczcf" event={"ID":"8a9bc0be-51b3-4a52-83ae-580c3a2f342c","Type":"ContainerStarted","Data":"0cff19dbdd49893dacaa9876bb752ee0060f3e772cccc73951212e26cc9c4f0e"} Nov 03 13:05:15.090328 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:15.089865 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dczcf" event={"ID":"8a9bc0be-51b3-4a52-83ae-580c3a2f342c","Type":"ContainerStarted","Data":"642a35856c1b730481197a131c5fed4ecf3dcac975ecdb6f975d717fe15c0118"} Nov 03 13:05:17.104110 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:17.104076 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9v5mk" event={"ID":"58a9c629-dff8-4f48-99b2-385d5d13e188","Type":"ContainerStarted","Data":"679f2b09cda0d51925e44de0d21c1d986f5a000b11f28e0689b174fa2e4739f7"} Nov 03 13:05:17.104606 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:17.104238 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9v5mk" Nov 03 13:05:17.211796 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:17.211734 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-dczcf" podStartSLOduration=68.583186592 podStartE2EDuration="1m10.211718772s" totalImagesPullingTime="1.62853218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:04:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:05:15.206918753 +0000 UTC m=+69.058924938" watchObservedRunningTime="2026-11-03 13:05:17.211718772 +0000 UTC m=+71.063725021" Nov 03 13:05:17.211984 ip-10-0-130-198 kubenswrapper[2580]: I1103 13:05:17.211840 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9v5mk" podStartSLOduration=66.349559829 podStartE2EDuration="1m10.211831713s" totalImagesPullingTime="3.862271884s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:04:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:05:17.21116522 +0000 UTC m=+71.063171410" watchObservedRunningTime="2026-11-03 13:05:17.211831713 +0000 UTC m=+71.063837931" Nov 03 13:05:17.916307 ip-10-0-130-198 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 13:05:17.930842 ip-10-0-130-198 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 13:05:17.931015 ip-10-0-130-198 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 13:05:17.931185 ip-10-0-130-198 systemd[1]: kubelet.service: Consumed 4.548s CPU time. Nov 03 13:05:17.948120 ip-10-0-130-198 systemd[1]: Starting Kubernetes Kubelet... Nov 03 13:05:18.020452 ip-10-0-130-198 kubenswrapper[5979]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 13:05:18.020452 ip-10-0-130-198 kubenswrapper[5979]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 13:05:18.020452 ip-10-0-130-198 kubenswrapper[5979]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 13:05:18.020452 ip-10-0-130-198 kubenswrapper[5979]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 13:05:18.022538 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022438 5979 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:05:18.022538 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022535 5979 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:05:18.022538 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022539 5979 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:05:18.022538 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022542 5979 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:05:18.022538 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022545 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:05:18.022538 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022548 5979 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:05:18.022858 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022551 5979 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:05:18.022858 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022554 5979 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:05:18.022858 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022558 5979 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:05:18.022858 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022560 5979 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:05:18.022858 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022563 5979 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:05:18.022858 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022566 5979 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:05:18.022858 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022569 5979 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:05:18.022858 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022571 5979 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:05:18.022858 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022574 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:05:18.022858 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022577 5979 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:05:18.022858 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022579 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:05:18.022858 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022582 5979 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:05:18.022858 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022585 5979 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:05:18.022858 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022587 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:05:18.022858 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022590 5979 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:05:18.022858 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022593 5979 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:05:18.022858 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022595 5979 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:05:18.022858 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022598 5979 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:05:18.022858 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022600 5979 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:05:18.023692 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022603 5979 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:05:18.023692 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022609 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:05:18.023692 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022612 5979 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:05:18.023692 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022615 5979 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:05:18.023692 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022617 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:05:18.023692 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022621 5979 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:05:18.023692 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022624 5979 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:05:18.023692 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022626 5979 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:05:18.023692 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022629 5979 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:05:18.023692 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022632 5979 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:05:18.023692 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022634 5979 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:05:18.023692 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022638 5979 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:05:18.023692 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022640 5979 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:05:18.023692 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022643 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:05:18.023692 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022646 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:05:18.023692 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022649 5979 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:05:18.023692 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022651 5979 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:05:18.023692 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022654 5979 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:05:18.023692 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022657 5979 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:05:18.023692 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022659 5979 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:05:18.024632 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022662 5979 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:05:18.024632 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022665 5979 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:05:18.024632 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022668 5979 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:05:18.024632 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022671 5979 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:05:18.024632 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022673 5979 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:05:18.024632 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022676 5979 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:05:18.024632 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022678 5979 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:05:18.024632 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022681 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:05:18.024632 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022705 5979 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:05:18.024632 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022710 5979 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:05:18.024632 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022714 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:05:18.024632 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022716 5979 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:05:18.024632 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022720 5979 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:05:18.024632 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022723 5979 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:05:18.024632 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022725 5979 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:05:18.024632 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022728 5979 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:05:18.024632 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022731 5979 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:05:18.024632 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022733 5979 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:05:18.024632 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022737 5979 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:05:18.024632 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022740 5979 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:05:18.025614 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022743 5979 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:05:18.025614 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022745 5979 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:05:18.025614 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022748 5979 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:05:18.025614 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022751 5979 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:05:18.025614 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022755 5979 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:05:18.025614 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022759 5979 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:05:18.025614 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022762 5979 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:05:18.025614 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022765 5979 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:05:18.025614 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022768 5979 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:05:18.025614 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022771 5979 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:05:18.025614 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022773 5979 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:05:18.025614 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022776 5979 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:05:18.025614 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022778 5979 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:05:18.025614 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022781 5979 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:05:18.025614 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022785 5979 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:05:18.025614 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022788 5979 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:05:18.025614 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022790 5979 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:05:18.025614 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022793 5979 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:05:18.025614 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022796 5979 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:05:18.026515 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022798 5979 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:05:18.026515 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022801 5979 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:05:18.026515 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022803 5979 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:05:18.026515 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022806 5979 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:05:18.026515 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022809 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:05:18.026515 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022811 5979 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:05:18.026515 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022815 5979 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:05:18.026515 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022819 5979 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:05:18.026515 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022822 5979 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:05:18.026515 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022826 5979 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:05:18.026515 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022829 5979 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:05:18.026515 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022832 5979 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:05:18.026515 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022835 5979 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:05:18.026515 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022838 5979 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:05:18.026515 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022841 5979 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:05:18.026515 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022845 5979 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:05:18.026515 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022849 5979 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:05:18.026515 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022851 5979 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:05:18.027298 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022854 5979 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:05:18.027298 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022856 5979 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:05:18.027298 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022859 5979 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:05:18.027298 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.022862 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:05:18.027298 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023715 5979 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:05:18.027298 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023726 5979 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:05:18.027298 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023730 5979 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:05:18.027298 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023736 5979 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:05:18.027298 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023740 5979 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:05:18.027298 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023743 5979 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:05:18.027298 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023747 5979 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:05:18.027298 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023750 5979 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:05:18.027298 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023753 5979 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:05:18.027298 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023755 5979 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:05:18.027298 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023758 5979 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:05:18.027298 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023760 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:05:18.027298 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023763 5979 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:05:18.027298 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023765 5979 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:05:18.027298 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023768 5979 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:05:18.027298 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023771 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:05:18.028194 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023774 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:05:18.028194 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023777 5979 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:05:18.028194 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023779 5979 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:05:18.028194 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023782 5979 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:05:18.028194 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023784 5979 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:05:18.028194 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023787 5979 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:05:18.028194 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023790 5979 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:05:18.028194 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023793 5979 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:05:18.028194 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023796 5979 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:05:18.028194 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023799 5979 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:05:18.028194 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023803 5979 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:05:18.028194 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023807 5979 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:05:18.028194 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023812 5979 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:05:18.028194 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023816 5979 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:05:18.028194 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023820 5979 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:05:18.028194 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023822 5979 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:05:18.028194 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023825 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:05:18.028194 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023828 5979 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:05:18.028194 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023830 5979 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:05:18.029149 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023833 5979 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:05:18.029149 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023835 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:05:18.029149 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023838 5979 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:05:18.029149 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023841 5979 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:05:18.029149 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023844 5979 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:05:18.029149 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023846 5979 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:05:18.029149 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023849 5979 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:05:18.029149 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023851 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:05:18.029149 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023854 5979 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:05:18.029149 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023856 5979 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:05:18.029149 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023859 5979 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:05:18.029149 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023862 5979 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:05:18.029149 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023865 5979 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:05:18.029149 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023867 5979 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:05:18.029149 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023872 5979 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:05:18.029149 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023877 5979 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:05:18.029149 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023881 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:05:18.029149 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023885 5979 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:05:18.029149 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023890 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:05:18.030048 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023895 5979 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:05:18.030048 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023899 5979 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:05:18.030048 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023902 5979 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:05:18.030048 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023904 5979 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:05:18.030048 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023914 5979 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:05:18.030048 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023917 5979 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:05:18.030048 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023920 5979 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:05:18.030048 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023922 5979 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:05:18.030048 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023925 5979 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:05:18.030048 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023927 5979 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:05:18.030048 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023930 5979 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:05:18.030048 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023933 5979 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:05:18.030048 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023936 5979 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:05:18.030048 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023938 5979 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:05:18.030048 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023941 5979 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:05:18.030048 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023943 5979 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:05:18.030048 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023946 5979 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:05:18.030048 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023948 5979 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:05:18.030048 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023952 5979 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:05:18.030048 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023954 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:05:18.030938 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023957 5979 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:05:18.030938 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023961 5979 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:05:18.030938 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023965 5979 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:05:18.030938 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023969 5979 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:05:18.030938 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023974 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:05:18.030938 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023979 5979 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:05:18.030938 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023982 5979 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:05:18.030938 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023984 5979 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:05:18.030938 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023987 5979 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:05:18.030938 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023990 5979 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:05:18.030938 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023992 5979 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:05:18.030938 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023996 5979 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:05:18.030938 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.023998 5979 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:05:18.030938 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024001 5979 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:05:18.030938 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024004 5979 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:05:18.030938 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024006 5979 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:05:18.030938 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024008 5979 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:05:18.030938 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024011 5979 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:05:18.030938 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024014 5979 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:05:18.030938 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024016 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:05:18.031822 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024019 5979 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:05:18.031822 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024021 5979 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:05:18.031822 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024024 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:05:18.031822 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024026 5979 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:05:18.031822 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024031 5979 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:05:18.031822 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024034 5979 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:05:18.031822 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024037 5979 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:05:18.031822 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024040 5979 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:05:18.031822 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024046 5979 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:05:18.031822 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024050 5979 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:05:18.031822 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024054 5979 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:05:18.031822 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024059 5979 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:05:18.031822 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024570 5979 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:05:18.031822 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024576 5979 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:05:18.031822 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024579 5979 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:05:18.031822 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024583 5979 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:05:18.031822 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024586 5979 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:05:18.031822 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024588 5979 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:05:18.031822 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024591 5979 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:05:18.032708 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024594 5979 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:05:18.032708 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024597 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:05:18.032708 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024599 5979 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:05:18.032708 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024602 5979 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:05:18.032708 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024605 5979 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:05:18.032708 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024607 5979 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:05:18.032708 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024610 5979 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:05:18.032708 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024613 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:05:18.032708 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024615 5979 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:05:18.032708 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024618 5979 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:05:18.032708 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024620 5979 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:05:18.032708 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024623 5979 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:05:18.032708 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024626 5979 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:05:18.032708 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024628 5979 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:05:18.032708 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024631 5979 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:05:18.032708 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024633 5979 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:05:18.032708 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024636 5979 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:05:18.032708 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024638 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:05:18.032708 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024643 5979 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:05:18.033535 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024647 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:05:18.033535 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024650 5979 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:05:18.033535 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024653 5979 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:05:18.033535 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024656 5979 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:05:18.033535 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024659 5979 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:05:18.033535 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024663 5979 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:05:18.033535 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024666 5979 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:05:18.033535 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024669 5979 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:05:18.033535 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024672 5979 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:05:18.033535 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024675 5979 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:05:18.033535 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024678 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:05:18.033535 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024680 5979 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:05:18.033535 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024683 5979 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:05:18.033535 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024686 5979 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:05:18.033535 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024688 5979 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:05:18.033535 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024691 5979 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:05:18.033535 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024693 5979 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:05:18.033535 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024695 5979 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:05:18.033535 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024698 5979 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:05:18.034398 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024700 5979 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:05:18.034398 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024703 5979 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:05:18.034398 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024705 5979 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:05:18.034398 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024708 5979 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:05:18.034398 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024711 5979 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:05:18.034398 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024713 5979 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:05:18.034398 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024715 5979 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:05:18.034398 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024718 5979 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:05:18.034398 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024720 5979 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:05:18.034398 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024723 5979 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:05:18.034398 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024726 5979 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:05:18.034398 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024728 5979 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:05:18.034398 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024731 5979 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:05:18.034398 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024734 5979 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:05:18.034398 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024737 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:05:18.034398 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024739 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:05:18.034398 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024742 5979 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:05:18.034398 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024744 5979 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:05:18.034398 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024747 5979 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:05:18.034398 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024750 5979 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:05:18.035348 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024753 5979 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:05:18.035348 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024755 5979 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:05:18.035348 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024758 5979 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:05:18.035348 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024761 5979 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:05:18.035348 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024763 5979 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:05:18.035348 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024767 5979 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:05:18.035348 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024769 5979 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:05:18.035348 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024772 5979 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:05:18.035348 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024775 5979 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:05:18.035348 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024777 5979 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:05:18.035348 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024780 5979 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:05:18.035348 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024782 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:05:18.035348 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024785 5979 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:05:18.035348 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024787 5979 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:05:18.035348 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024790 5979 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:05:18.035348 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024792 5979 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:05:18.035348 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024795 5979 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:05:18.035348 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024797 5979 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:05:18.035348 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024800 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:05:18.035348 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024803 5979 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:05:18.036284 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024806 5979 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:05:18.036284 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024809 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:05:18.036284 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024811 5979 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:05:18.036284 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024814 5979 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:05:18.036284 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024816 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:05:18.036284 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024819 5979 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:05:18.036284 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024823 5979 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:05:18.036284 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024826 5979 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:05:18.036284 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024829 5979 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:05:18.036284 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024831 5979 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:05:18.036284 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024834 5979 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:05:18.036284 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024836 5979 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:05:18.036284 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024839 5979 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:05:18.036284 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024842 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:05:18.036284 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024847 5979 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:05:18.036284 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024850 5979 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:05:18.036284 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024852 5979 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:05:18.036284 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024855 5979 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:05:18.036284 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024858 5979 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:05:18.037105 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024861 5979 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:05:18.037105 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.024864 5979 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:05:18.037105 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.024953 5979 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 13:05:18.037105 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.024968 5979 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 13:05:18.037105 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.024974 5979 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 13:05:18.037105 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.024979 5979 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 13:05:18.037105 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.024984 5979 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 13:05:18.037105 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.024987 5979 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 13:05:18.037105 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.024994 5979 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 13:05:18.037105 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.024998 5979 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 13:05:18.037105 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025002 5979 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 13:05:18.037105 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025005 5979 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 13:05:18.037105 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025008 5979 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 13:05:18.037105 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025012 5979 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 13:05:18.037105 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025015 5979 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 13:05:18.037105 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025018 5979 flags.go:64] FLAG: --cgroup-root="" Nov 03 13:05:18.037105 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025021 5979 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 13:05:18.037105 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025024 5979 flags.go:64] FLAG: --client-ca-file="" Nov 03 13:05:18.037105 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025027 5979 flags.go:64] FLAG: --cloud-provider="external" Nov 03 13:05:18.037105 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025030 5979 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 13:05:18.037105 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025034 5979 flags.go:64] FLAG: --cluster-domain="" Nov 03 13:05:18.037105 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025037 5979 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 13:05:18.037105 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025040 5979 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 13:05:18.038094 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025043 5979 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 13:05:18.038094 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025047 5979 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 13:05:18.038094 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025051 5979 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 13:05:18.038094 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025054 5979 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 13:05:18.038094 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025059 5979 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 13:05:18.038094 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025062 5979 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 13:05:18.038094 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025065 5979 flags.go:64] FLAG: --contention-profiling="false" Nov 03 13:05:18.038094 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025068 5979 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 13:05:18.038094 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025071 5979 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 13:05:18.038094 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025074 5979 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 13:05:18.038094 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025077 5979 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 13:05:18.038094 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025082 5979 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 13:05:18.038094 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025085 5979 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 13:05:18.038094 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025088 5979 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 13:05:18.038094 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025091 5979 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 13:05:18.038094 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025094 5979 flags.go:64] FLAG: --enable-server="true" Nov 03 13:05:18.038094 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025097 5979 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 13:05:18.038094 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025104 5979 flags.go:64] FLAG: --event-burst="100" Nov 03 13:05:18.038094 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025107 5979 flags.go:64] FLAG: --event-qps="50" Nov 03 13:05:18.038094 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025110 5979 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 13:05:18.038094 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025113 5979 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 13:05:18.038094 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025117 5979 flags.go:64] FLAG: --eviction-hard="" Nov 03 13:05:18.038094 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025121 5979 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 13:05:18.038094 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025124 5979 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 13:05:18.039227 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025127 5979 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 13:05:18.039227 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025130 5979 flags.go:64] FLAG: --eviction-soft="" Nov 03 13:05:18.039227 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025133 5979 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 13:05:18.039227 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025136 5979 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 13:05:18.039227 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025139 5979 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 13:05:18.039227 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025142 5979 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 13:05:18.039227 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025145 5979 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 13:05:18.039227 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025148 5979 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 13:05:18.039227 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025151 5979 flags.go:64] FLAG: --feature-gates="" Nov 03 13:05:18.039227 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025155 5979 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 13:05:18.039227 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025158 5979 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 13:05:18.039227 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025162 5979 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 13:05:18.039227 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025167 5979 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 13:05:18.039227 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025171 5979 flags.go:64] FLAG: --healthz-port="10248" Nov 03 13:05:18.039227 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025174 5979 flags.go:64] FLAG: --help="false" Nov 03 13:05:18.039227 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025177 5979 flags.go:64] FLAG: --hostname-override="ip-10-0-130-198.us-west-2.compute.internal" Nov 03 13:05:18.039227 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025180 5979 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 13:05:18.039227 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025183 5979 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 13:05:18.039227 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025186 5979 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 13:05:18.039227 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025190 5979 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 13:05:18.039227 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025193 5979 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 13:05:18.039227 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025197 5979 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 13:05:18.039227 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025200 5979 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 13:05:18.039863 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025203 5979 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 13:05:18.039863 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025206 5979 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 13:05:18.039863 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025210 5979 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 13:05:18.039863 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025214 5979 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 13:05:18.039863 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025216 5979 flags.go:64] FLAG: --kube-reserved="" Nov 03 13:05:18.039863 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025220 5979 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 13:05:18.039863 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025223 5979 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 13:05:18.039863 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025226 5979 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 13:05:18.039863 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025229 5979 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 13:05:18.039863 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025232 5979 flags.go:64] FLAG: --lock-file="" Nov 03 13:05:18.039863 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025235 5979 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 13:05:18.039863 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025238 5979 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 13:05:18.039863 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025241 5979 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 13:05:18.039863 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025247 5979 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 13:05:18.039863 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025249 5979 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 13:05:18.039863 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025252 5979 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 13:05:18.039863 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025256 5979 flags.go:64] FLAG: --logging-format="text" Nov 03 13:05:18.039863 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025258 5979 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 13:05:18.039863 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025262 5979 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 13:05:18.039863 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025264 5979 flags.go:64] FLAG: --manifest-url="" Nov 03 13:05:18.039863 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025267 5979 flags.go:64] FLAG: --manifest-url-header="" Nov 03 13:05:18.039863 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025273 5979 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 13:05:18.039863 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025277 5979 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 13:05:18.039863 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025281 5979 flags.go:64] FLAG: --max-pods="110" Nov 03 13:05:18.039863 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025284 5979 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 13:05:18.040603 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025287 5979 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 13:05:18.040603 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025290 5979 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 13:05:18.040603 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025293 5979 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 13:05:18.040603 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025296 5979 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 13:05:18.040603 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025299 5979 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 13:05:18.040603 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025302 5979 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 13:05:18.040603 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025309 5979 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 13:05:18.040603 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025312 5979 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 13:05:18.040603 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025315 5979 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 13:05:18.040603 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025319 5979 flags.go:64] FLAG: --pod-cidr="" Nov 03 13:05:18.040603 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025322 5979 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 13:05:18.040603 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025325 5979 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 13:05:18.040603 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025329 5979 flags.go:64] FLAG: --pods-per-core="0" Nov 03 13:05:18.040603 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025332 5979 flags.go:64] FLAG: --port="10250" Nov 03 13:05:18.040603 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025335 5979 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 13:05:18.040603 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025338 5979 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-072d2c3726ae77995" Nov 03 13:05:18.040603 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025342 5979 flags.go:64] FLAG: --qos-reserved="" Nov 03 13:05:18.040603 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025345 5979 flags.go:64] FLAG: --read-only-port="10255" Nov 03 13:05:18.040603 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025348 5979 flags.go:64] FLAG: --register-node="true" Nov 03 13:05:18.040603 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025351 5979 flags.go:64] FLAG: --register-with-taints="" Nov 03 13:05:18.040603 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025354 5979 flags.go:64] FLAG: --registry-burst="10" Nov 03 13:05:18.040603 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025357 5979 flags.go:64] FLAG: --registry-qps="5" Nov 03 13:05:18.040603 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025360 5979 flags.go:64] FLAG: --reserved-cpus="" Nov 03 13:05:18.040603 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025363 5979 flags.go:64] FLAG: --reserved-memory="" Nov 03 13:05:18.040603 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025368 5979 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 13:05:18.040603 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025371 5979 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 13:05:18.041232 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025374 5979 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 13:05:18.041232 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025377 5979 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 13:05:18.041232 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025382 5979 flags.go:64] FLAG: --runonce="false" Nov 03 13:05:18.041232 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025385 5979 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 13:05:18.041232 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025388 5979 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 13:05:18.041232 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025391 5979 flags.go:64] FLAG: --seccomp-default="false" Nov 03 13:05:18.041232 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025394 5979 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 13:05:18.041232 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025397 5979 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 13:05:18.041232 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025400 5979 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 13:05:18.041232 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025403 5979 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 13:05:18.041232 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025406 5979 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 13:05:18.041232 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025409 5979 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 13:05:18.041232 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025413 5979 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 13:05:18.041232 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025415 5979 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 13:05:18.041232 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025418 5979 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 13:05:18.041232 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025423 5979 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 13:05:18.041232 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025426 5979 flags.go:64] FLAG: --system-cgroups="" Nov 03 13:05:18.041232 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025430 5979 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 13:05:18.041232 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025435 5979 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 13:05:18.041232 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025438 5979 flags.go:64] FLAG: --tls-cert-file="" Nov 03 13:05:18.041232 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025441 5979 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 13:05:18.041232 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025445 5979 flags.go:64] FLAG: --tls-min-version="" Nov 03 13:05:18.041232 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025448 5979 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 13:05:18.041232 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025451 5979 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 13:05:18.041232 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025454 5979 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 13:05:18.041891 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025457 5979 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 13:05:18.041891 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025460 5979 flags.go:64] FLAG: --v="2" Nov 03 13:05:18.041891 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025465 5979 flags.go:64] FLAG: --version="false" Nov 03 13:05:18.041891 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025481 5979 flags.go:64] FLAG: --vmodule="" Nov 03 13:05:18.041891 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025485 5979 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 13:05:18.041891 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025488 5979 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 13:05:18.041891 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025618 5979 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:05:18.041891 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025621 5979 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:05:18.041891 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025624 5979 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:05:18.041891 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025629 5979 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:05:18.041891 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025631 5979 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:05:18.041891 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025634 5979 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:05:18.041891 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025638 5979 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:05:18.041891 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025641 5979 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:05:18.041891 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025643 5979 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:05:18.041891 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025646 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:05:18.041891 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025648 5979 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:05:18.041891 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025651 5979 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:05:18.041891 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025654 5979 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:05:18.041891 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025656 5979 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:05:18.041891 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025659 5979 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:05:18.041891 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025662 5979 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:05:18.042434 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025665 5979 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:05:18.042434 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025668 5979 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:05:18.042434 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025671 5979 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:05:18.042434 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025674 5979 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:05:18.042434 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025676 5979 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:05:18.042434 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025679 5979 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:05:18.042434 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025682 5979 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:05:18.042434 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025684 5979 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:05:18.042434 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025687 5979 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:05:18.042434 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025690 5979 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:05:18.042434 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025692 5979 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:05:18.042434 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025695 5979 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:05:18.042434 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025698 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:05:18.042434 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025700 5979 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:05:18.042434 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025703 5979 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:05:18.042434 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025705 5979 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:05:18.042434 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025708 5979 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:05:18.042434 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025710 5979 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:05:18.042434 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025713 5979 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:05:18.042434 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025717 5979 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:05:18.042972 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025719 5979 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:05:18.042972 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025722 5979 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:05:18.042972 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025725 5979 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:05:18.042972 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025731 5979 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:05:18.042972 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025734 5979 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:05:18.042972 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025736 5979 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:05:18.042972 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025739 5979 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:05:18.042972 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025741 5979 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:05:18.042972 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025745 5979 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:05:18.042972 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025749 5979 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:05:18.042972 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025752 5979 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:05:18.042972 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025754 5979 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:05:18.042972 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025759 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:05:18.042972 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025761 5979 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:05:18.042972 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025764 5979 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:05:18.042972 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025766 5979 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:05:18.042972 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025769 5979 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:05:18.042972 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025772 5979 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:05:18.042972 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025774 5979 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:05:18.043431 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025777 5979 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:05:18.043431 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025779 5979 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:05:18.043431 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025782 5979 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:05:18.043431 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025785 5979 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:05:18.043431 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025787 5979 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:05:18.043431 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025790 5979 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:05:18.043431 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025793 5979 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:05:18.043431 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025795 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:05:18.043431 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025798 5979 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:05:18.043431 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025800 5979 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:05:18.043431 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025803 5979 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:05:18.043431 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025805 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:05:18.043431 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025809 5979 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:05:18.043431 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025812 5979 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:05:18.043431 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025815 5979 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:05:18.043431 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025818 5979 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:05:18.043431 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025820 5979 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:05:18.043431 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025823 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:05:18.043431 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025826 5979 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:05:18.043431 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025828 5979 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:05:18.043957 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025830 5979 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:05:18.043957 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025833 5979 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:05:18.043957 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025835 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:05:18.043957 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025838 5979 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:05:18.043957 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025840 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:05:18.043957 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025844 5979 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:05:18.043957 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025847 5979 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:05:18.043957 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025849 5979 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:05:18.043957 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025852 5979 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:05:18.043957 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025854 5979 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:05:18.043957 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025857 5979 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:05:18.043957 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025860 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:05:18.043957 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025862 5979 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:05:18.043957 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025865 5979 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:05:18.043957 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025868 5979 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:05:18.043957 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025872 5979 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:05:18.043957 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025876 5979 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:05:18.043957 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025879 5979 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:05:18.043957 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025882 5979 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:05:18.043957 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025884 5979 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:05:18.044442 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025887 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:05:18.044442 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025890 5979 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:05:18.044442 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025893 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:05:18.044442 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025896 5979 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:05:18.044442 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025900 5979 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:05:18.044442 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025902 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:05:18.044442 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025905 5979 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:05:18.044442 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025908 5979 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:05:18.044442 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025911 5979 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:05:18.044442 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025913 5979 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:05:18.044442 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025917 5979 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:05:18.044442 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.025942 5979 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 13:05:18.044442 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025977 5979 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:05:18.044442 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025981 5979 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:05:18.044442 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025985 5979 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:05:18.044442 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025987 5979 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:05:18.044846 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025990 5979 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:05:18.044846 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025993 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:05:18.044846 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025997 5979 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:05:18.044846 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.025999 5979 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:05:18.044846 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026002 5979 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:05:18.044846 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026005 5979 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:05:18.044846 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026007 5979 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:05:18.044846 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026010 5979 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:05:18.044846 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026013 5979 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:05:18.044846 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026015 5979 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:05:18.044846 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026018 5979 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:05:18.044846 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026021 5979 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:05:18.044846 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026024 5979 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:05:18.044846 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026027 5979 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:05:18.044846 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026029 5979 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:05:18.044846 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026032 5979 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:05:18.044846 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026035 5979 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:05:18.044846 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026039 5979 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:05:18.045276 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026043 5979 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:05:18.045276 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026046 5979 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:05:18.045276 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026050 5979 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:05:18.045276 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026054 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:05:18.045276 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026056 5979 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:05:18.045276 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026059 5979 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:05:18.045276 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026062 5979 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:05:18.045276 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026065 5979 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:05:18.045276 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026068 5979 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:05:18.045276 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026071 5979 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:05:18.045276 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026073 5979 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:05:18.045276 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026076 5979 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:05:18.045276 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026078 5979 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:05:18.045276 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026081 5979 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:05:18.045276 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026084 5979 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:05:18.045276 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026086 5979 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:05:18.045276 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026091 5979 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:05:18.045276 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026094 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:05:18.045276 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026097 5979 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:05:18.045276 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026099 5979 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:05:18.045780 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026102 5979 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:05:18.045780 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026105 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:05:18.045780 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026107 5979 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:05:18.045780 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026110 5979 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:05:18.045780 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026113 5979 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:05:18.045780 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026115 5979 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:05:18.045780 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026118 5979 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:05:18.045780 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026120 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:05:18.045780 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026123 5979 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:05:18.045780 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026125 5979 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:05:18.045780 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026128 5979 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:05:18.045780 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026130 5979 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:05:18.045780 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026133 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:05:18.045780 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026135 5979 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:05:18.045780 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026140 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:05:18.045780 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026142 5979 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:05:18.045780 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026145 5979 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:05:18.045780 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026147 5979 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:05:18.045780 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026150 5979 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:05:18.045780 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026153 5979 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:05:18.046279 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026155 5979 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:05:18.046279 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026158 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:05:18.046279 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026161 5979 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:05:18.046279 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026164 5979 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:05:18.046279 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026166 5979 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:05:18.046279 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026170 5979 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:05:18.046279 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026174 5979 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:05:18.046279 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026177 5979 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:05:18.046279 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026182 5979 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:05:18.046279 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026185 5979 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:05:18.046279 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026187 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:05:18.046279 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026190 5979 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:05:18.046279 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026193 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:05:18.046279 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026195 5979 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:05:18.046279 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026198 5979 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:05:18.046279 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026200 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:05:18.046279 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026203 5979 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:05:18.046279 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026206 5979 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:05:18.046279 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026209 5979 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:05:18.046749 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026211 5979 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:05:18.046749 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026214 5979 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:05:18.046749 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026217 5979 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:05:18.046749 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026219 5979 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:05:18.046749 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026222 5979 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:05:18.046749 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026224 5979 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:05:18.046749 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026227 5979 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:05:18.046749 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026231 5979 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:05:18.046749 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026234 5979 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:05:18.046749 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026236 5979 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:05:18.046749 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026239 5979 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:05:18.046749 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026242 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:05:18.046749 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026244 5979 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:05:18.046749 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026247 5979 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:05:18.046749 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026249 5979 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:05:18.046749 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026252 5979 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:05:18.046749 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026255 5979 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:05:18.046749 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026257 5979 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:05:18.046749 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026260 5979 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:05:18.046749 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026262 5979 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:05:18.047270 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026265 5979 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:05:18.047270 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026269 5979 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:05:18.047270 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026272 5979 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:05:18.047270 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026275 5979 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:05:18.047270 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.026278 5979 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:05:18.047270 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.030990 5979 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 13:05:18.047270 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.031004 5979 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 13:05:18.047270 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.031023 5979 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 13:05:18.047270 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.031028 5979 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 13:05:18.047270 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031084 5979 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:05:18.047270 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031090 5979 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:05:18.047270 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031095 5979 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:05:18.047270 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031099 5979 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:05:18.047270 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031103 5979 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:05:18.047270 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031106 5979 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:05:18.047270 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031109 5979 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:05:18.047270 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031112 5979 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:05:18.047270 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031116 5979 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:05:18.047270 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031118 5979 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:05:18.047784 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031122 5979 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:05:18.047784 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031125 5979 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:05:18.047784 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031128 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:05:18.047784 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031131 5979 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:05:18.047784 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031134 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:05:18.047784 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031136 5979 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:05:18.047784 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031139 5979 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:05:18.047784 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031142 5979 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:05:18.047784 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031145 5979 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:05:18.047784 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031147 5979 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:05:18.047784 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031150 5979 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:05:18.047784 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031152 5979 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:05:18.047784 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031155 5979 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:05:18.047784 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031158 5979 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:05:18.047784 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031161 5979 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:05:18.047784 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031164 5979 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:05:18.047784 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031166 5979 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:05:18.047784 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031169 5979 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:05:18.047784 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031177 5979 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:05:18.048378 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031180 5979 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:05:18.048378 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031182 5979 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:05:18.048378 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031185 5979 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:05:18.048378 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031187 5979 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:05:18.048378 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031190 5979 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:05:18.048378 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031193 5979 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:05:18.048378 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031195 5979 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:05:18.048378 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031198 5979 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:05:18.048378 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031200 5979 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:05:18.048378 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031203 5979 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:05:18.048378 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031205 5979 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:05:18.048378 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031208 5979 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:05:18.048378 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031210 5979 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:05:18.048378 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031213 5979 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:05:18.048378 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031216 5979 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:05:18.048378 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031219 5979 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:05:18.048378 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031222 5979 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:05:18.048378 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031224 5979 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:05:18.048378 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031227 5979 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:05:18.048378 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031229 5979 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:05:18.049177 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031232 5979 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:05:18.049177 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031235 5979 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:05:18.049177 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031237 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:05:18.049177 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031240 5979 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:05:18.049177 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031243 5979 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:05:18.049177 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031245 5979 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:05:18.049177 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031248 5979 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:05:18.049177 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031251 5979 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:05:18.049177 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031254 5979 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:05:18.049177 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031256 5979 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:05:18.049177 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031259 5979 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:05:18.049177 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031262 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:05:18.049177 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031265 5979 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:05:18.049177 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031267 5979 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:05:18.049177 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031269 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:05:18.049177 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031272 5979 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:05:18.049177 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031275 5979 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:05:18.049177 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031277 5979 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:05:18.049177 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031280 5979 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:05:18.049177 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031282 5979 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:05:18.049987 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031285 5979 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:05:18.049987 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031287 5979 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:05:18.049987 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031290 5979 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:05:18.049987 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031292 5979 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:05:18.049987 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031295 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:05:18.049987 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031298 5979 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:05:18.049987 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031301 5979 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:05:18.049987 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031303 5979 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:05:18.049987 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031306 5979 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:05:18.049987 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031308 5979 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:05:18.049987 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031311 5979 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:05:18.049987 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031314 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:05:18.049987 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031317 5979 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:05:18.049987 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031319 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:05:18.049987 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031322 5979 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:05:18.049987 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031324 5979 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:05:18.049987 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031327 5979 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:05:18.049987 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031330 5979 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:05:18.049987 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031333 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:05:18.050584 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031335 5979 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:05:18.050584 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031337 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:05:18.050584 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031340 5979 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:05:18.050584 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031343 5979 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:05:18.050584 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031347 5979 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:05:18.050584 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031350 5979 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:05:18.050584 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031353 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:05:18.050584 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031356 5979 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:05:18.050584 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031358 5979 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:05:18.050584 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031361 5979 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:05:18.050584 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031364 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:05:18.050584 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031366 5979 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:05:18.050584 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031369 5979 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:05:18.050584 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031372 5979 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:05:18.050584 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031374 5979 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:05:18.050584 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031377 5979 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:05:18.050584 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031379 5979 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:05:18.050584 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031381 5979 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:05:18.051049 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.031411 5979 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 13:05:18.051049 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031577 5979 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:05:18.051049 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031582 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:05:18.051049 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031586 5979 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:05:18.051049 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031589 5979 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:05:18.051049 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031592 5979 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:05:18.051049 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031595 5979 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:05:18.051049 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031598 5979 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:05:18.051049 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031601 5979 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:05:18.051049 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031604 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:05:18.051049 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031607 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:05:18.051049 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031609 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:05:18.051049 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031612 5979 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:05:18.051049 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031618 5979 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:05:18.051049 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031621 5979 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:05:18.051049 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031624 5979 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:05:18.051448 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031628 5979 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:05:18.051448 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031632 5979 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:05:18.051448 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031635 5979 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:05:18.051448 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031638 5979 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:05:18.051448 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031641 5979 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:05:18.051448 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031643 5979 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:05:18.051448 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031646 5979 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:05:18.051448 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031649 5979 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:05:18.051448 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031651 5979 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:05:18.051448 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031655 5979 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:05:18.051448 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031657 5979 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:05:18.051448 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031660 5979 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:05:18.051448 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031663 5979 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:05:18.051448 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031665 5979 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:05:18.051448 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031668 5979 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:05:18.051448 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031670 5979 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:05:18.051448 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031674 5979 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:05:18.051448 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031677 5979 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:05:18.051448 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031680 5979 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:05:18.052020 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031682 5979 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:05:18.052020 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031685 5979 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:05:18.052020 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031688 5979 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:05:18.052020 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031690 5979 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:05:18.052020 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031694 5979 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:05:18.052020 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031696 5979 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:05:18.052020 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031699 5979 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:05:18.052020 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031701 5979 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:05:18.052020 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031704 5979 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:05:18.052020 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031707 5979 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:05:18.052020 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031711 5979 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:05:18.052020 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031714 5979 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:05:18.052020 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031716 5979 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:05:18.052020 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031719 5979 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:05:18.052020 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031721 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:05:18.052020 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031724 5979 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:05:18.052020 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031727 5979 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:05:18.052020 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031730 5979 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:05:18.052020 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031732 5979 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:05:18.052830 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031735 5979 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:05:18.052830 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031738 5979 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:05:18.052830 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031740 5979 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:05:18.052830 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031743 5979 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:05:18.052830 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031746 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:05:18.052830 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031748 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:05:18.052830 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031751 5979 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:05:18.052830 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031754 5979 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:05:18.052830 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031757 5979 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:05:18.052830 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031759 5979 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:05:18.052830 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031763 5979 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:05:18.052830 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031766 5979 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:05:18.052830 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031769 5979 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:05:18.052830 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031771 5979 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:05:18.052830 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031774 5979 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:05:18.052830 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031777 5979 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:05:18.052830 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031779 5979 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:05:18.052830 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031782 5979 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:05:18.052830 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031785 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:05:18.052830 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031787 5979 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:05:18.053644 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031790 5979 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:05:18.053644 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031792 5979 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:05:18.053644 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031795 5979 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:05:18.053644 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031800 5979 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:05:18.053644 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031802 5979 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:05:18.053644 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031805 5979 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:05:18.053644 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031807 5979 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:05:18.053644 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031810 5979 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:05:18.053644 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031813 5979 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:05:18.053644 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031815 5979 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:05:18.053644 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031818 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:05:18.053644 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031820 5979 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:05:18.053644 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031823 5979 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:05:18.053644 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031826 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:05:18.053644 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031829 5979 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:05:18.053644 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031832 5979 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:05:18.053644 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031835 5979 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:05:18.053644 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031838 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:05:18.053644 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031840 5979 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:05:18.054519 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031843 5979 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:05:18.054519 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031846 5979 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:05:18.054519 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031848 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:05:18.054519 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031852 5979 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:05:18.054519 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031855 5979 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:05:18.054519 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031858 5979 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:05:18.054519 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031860 5979 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:05:18.054519 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031863 5979 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:05:18.054519 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031865 5979 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:05:18.054519 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031868 5979 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:05:18.054519 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031870 5979 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:05:18.054519 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031874 5979 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:05:18.054519 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031879 5979 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:05:18.054519 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031882 5979 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:05:18.054519 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.031913 5979 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 13:05:18.055110 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031949 5979 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 13:05:18.055110 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031953 5979 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 13:05:18.055110 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031958 5979 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 13:05:18.055110 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031961 5979 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 13:05:18.055110 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031963 5979 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 13:05:18.055110 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031966 5979 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 13:05:18.055110 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031969 5979 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 13:05:18.055110 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031972 5979 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 13:05:18.055110 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031974 5979 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 13:05:18.055110 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031977 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 13:05:18.055110 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031979 5979 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 13:05:18.055110 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031982 5979 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 13:05:18.055110 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031985 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 13:05:18.055110 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031988 5979 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 13:05:18.055110 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031990 5979 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 13:05:18.055110 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031993 5979 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 13:05:18.055110 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031996 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 13:05:18.055110 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.031999 5979 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 13:05:18.055110 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032001 5979 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 13:05:18.055110 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032004 5979 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 13:05:18.055746 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032008 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 13:05:18.055746 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032010 5979 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 13:05:18.055746 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032013 5979 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 13:05:18.055746 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032016 5979 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 13:05:18.055746 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032018 5979 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 13:05:18.055746 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032021 5979 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 13:05:18.055746 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032023 5979 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 13:05:18.055746 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032026 5979 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 13:05:18.055746 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032029 5979 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 13:05:18.055746 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032033 5979 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 13:05:18.055746 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032041 5979 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 13:05:18.055746 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032044 5979 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 13:05:18.055746 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032047 5979 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 13:05:18.055746 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032050 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 13:05:18.055746 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032054 5979 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 13:05:18.055746 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032056 5979 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 13:05:18.055746 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032059 5979 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 13:05:18.055746 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032061 5979 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 13:05:18.055746 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032064 5979 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 13:05:18.056255 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032067 5979 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 13:05:18.056255 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032069 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 13:05:18.056255 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032072 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 13:05:18.056255 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032075 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 13:05:18.056255 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032077 5979 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 13:05:18.056255 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032080 5979 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 13:05:18.056255 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032083 5979 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 13:05:18.056255 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032085 5979 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 13:05:18.056255 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032089 5979 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 13:05:18.056255 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032093 5979 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 13:05:18.056255 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032095 5979 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 13:05:18.056255 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032098 5979 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 13:05:18.056255 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032101 5979 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 13:05:18.056255 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032105 5979 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 13:05:18.056255 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032107 5979 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 13:05:18.056255 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032110 5979 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 13:05:18.056255 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032113 5979 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 13:05:18.056255 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032116 5979 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 13:05:18.056255 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032119 5979 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 13:05:18.056738 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032122 5979 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 13:05:18.056738 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032124 5979 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 13:05:18.056738 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032126 5979 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 13:05:18.056738 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032129 5979 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 13:05:18.056738 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032132 5979 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 13:05:18.056738 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032135 5979 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 13:05:18.056738 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032138 5979 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 13:05:18.056738 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032140 5979 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 13:05:18.056738 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032144 5979 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 13:05:18.056738 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032147 5979 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 13:05:18.056738 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032150 5979 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 13:05:18.056738 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032152 5979 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 13:05:18.056738 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032155 5979 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 13:05:18.056738 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032158 5979 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 13:05:18.056738 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032160 5979 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 13:05:18.056738 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032163 5979 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 13:05:18.056738 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032165 5979 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 13:05:18.056738 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032168 5979 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 13:05:18.056738 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032170 5979 feature_gate.go:386] unrecognized feature gate: Example Nov 03 13:05:18.057313 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032173 5979 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 13:05:18.057313 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032176 5979 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 13:05:18.057313 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032178 5979 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 13:05:18.057313 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032181 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 13:05:18.057313 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032184 5979 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 13:05:18.057313 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032186 5979 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 13:05:18.057313 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032189 5979 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 13:05:18.057313 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032193 5979 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 13:05:18.057313 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032196 5979 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 13:05:18.057313 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032198 5979 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 13:05:18.057313 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032201 5979 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 13:05:18.057313 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032204 5979 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 13:05:18.057313 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032206 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 13:05:18.057313 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032209 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 13:05:18.057313 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032212 5979 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 13:05:18.057313 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032214 5979 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 13:05:18.057313 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032217 5979 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 13:05:18.057313 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032220 5979 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 13:05:18.057313 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032223 5979 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 13:05:18.057313 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032226 5979 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 13:05:18.057817 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032228 5979 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 13:05:18.057817 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032232 5979 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 13:05:18.057817 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032235 5979 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 13:05:18.057817 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032238 5979 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 13:05:18.057817 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032240 5979 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 13:05:18.057817 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032243 5979 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 13:05:18.057817 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032246 5979 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 13:05:18.057817 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032248 5979 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 13:05:18.057817 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:18.032251 5979 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 13:05:18.057817 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.032300 5979 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 13:05:18.057817 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.033521 5979 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 13:05:18.057817 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.033576 5979 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 13:05:18.057817 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.033786 5979 server.go:1014] "Starting client certificate rotation" Nov 03 13:05:18.057817 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.033925 5979 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 13:05:18.057817 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.033994 5979 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 12:59:06 +0000 UTC" deadline="2028-05-20 07:53:15.923106308 +0000 UTC" Nov 03 13:05:18.057817 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.034033 5979 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13530h47m57.889076768s" Nov 03 13:05:18.058248 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.034289 5979 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 13:05:18.058248 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.035698 5979 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 13:05:18.058248 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.036910 5979 log.go:25] "Validated CRI v1 runtime API" Nov 03 13:05:18.058248 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.038829 5979 log.go:25] "Validated CRI v1 image API" Nov 03 13:05:18.058248 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.039560 5979 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 13:05:18.058248 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.041926 5979 fs.go:135] Filesystem UUIDs: map[1724f46f-a6a0-4519-9880-204f9dc2cbfc:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 9ac2c955-d6f5-4042-b768-daf11305ec3d:/dev/nvme0n1p4] Nov 03 13:05:18.058446 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.041946 5979 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/021415e37c70ce8f7864ca9d751fb5be0f42c656f462e392f6bbcfb72e3be2ad/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/021415e37c70ce8f7864ca9d751fb5be0f42c656f462e392f6bbcfb72e3be2ad/userdata/shm major:0 minor:442 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/022573a872e57bdae8e145e998cc17c3d120a445311e823f943e50e7c76cc596/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/022573a872e57bdae8e145e998cc17c3d120a445311e823f943e50e7c76cc596/userdata/shm major:0 minor:87 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/04419437a7c583e8767facf7b6dd8ac270e24b70dd856e7d9a1bcd0e8b71bb22/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/04419437a7c583e8767facf7b6dd8ac270e24b70dd856e7d9a1bcd0e8b71bb22/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1ba727bf8ab0e0dbb6df93b3ac18c1f3f0af05d767a6a66729782a995c797fc0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1ba727bf8ab0e0dbb6df93b3ac18c1f3f0af05d767a6a66729782a995c797fc0/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3b26249f5bd52d607583665ec09161ee13a1786eaaed3a195ce24774d76f055b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3b26249f5bd52d607583665ec09161ee13a1786eaaed3a195ce24774d76f055b/userdata/shm major:0 minor:258 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/400ccad898a899b91604c1a67253b559529031211d7f42dbae40a19cc322a55d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/400ccad898a899b91604c1a67253b559529031211d7f42dbae40a19cc322a55d/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/49b66c76afba64193c0a47b9415b935a5553d5caae03c6370b8574979bb86404/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/49b66c76afba64193c0a47b9415b935a5553d5caae03c6370b8574979bb86404/userdata/shm major:0 minor:353 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4c54177f6ce7c0b369791011f7103908e9f3a45261bef440845bddd12613fbab/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4c54177f6ce7c0b369791011f7103908e9f3a45261bef440845bddd12613fbab/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/63f68d72e1402d9f5b019be6e003a98b22ba8f0f8c43f2199bdd191bcf34a87c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/63f68d72e1402d9f5b019be6e003a98b22ba8f0f8c43f2199bdd191bcf34a87c/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6c3945e8883e0a259fd5f007da8c7429799c94f6107a8e45f05f5e2948efe6ed/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6c3945e8883e0a259fd5f007da8c7429799c94f6107a8e45f05f5e2948efe6ed/userdata/shm major:0 minor:502 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6de1e4a21fe31cf1358127b395a713134fadd5a4cf986e8b1b3e4fe8cea83e27/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6de1e4a21fe31cf1358127b395a713134fadd5a4cf986e8b1b3e4fe8cea83e27/userdata/shm major:0 minor:64 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6eac71e8dec9af14956357fa10a6d4595db34ba6f85cf32f4b8efd29d9426efc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6eac71e8dec9af14956357fa10a6d4595db34ba6f85cf32f4b8efd29d9426efc/userdata/shm major:0 minor:313 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/97e7b232bd9bde5b04704bff068be4e42c0c8b238cbda87d3cf80eea9248717a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/97e7b232bd9bde5b04704bff068be4e42c0c8b238cbda87d3cf80eea9248717a/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9a91b9c57cf303820186cee25a3e09aa472d0822dbae53e22020ed5505ea4fa9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9a91b9c57cf303820186cee25a3e09aa472d0822dbae53e22020ed5505ea4fa9/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a9d7c3dabde3dda56feb11acc4d301b588e998ab794a8368b9abdca272ec1905/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a9d7c3dabde3dda56feb11acc4d301b588e998ab794a8368b9abdca272ec1905/userdata/shm major:0 minor:252 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a9e9596490c552decf61398ebee95872f8e6ece435e16fd80d0daea5fc85f19e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a9e9596490c552decf61398ebee95872f8e6ece435e16fd80d0daea5fc85f19e/userdata/shm major:0 minor:359 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b32d70e0c0296b39175111442498979160d7e804c4d7b1876b6e227343cfc7ab/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b32d70e0c0296b39175111442498979160d7e804c4d7b1876b6e227343cfc7ab/userdata/shm major:0 minor:506 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b404676561694e010ac4e528fbf5d92e6d2411bcd1eb9bf7a48d5ffe8f138f6f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b404676561694e010ac4e528fbf5d92e6d2411bcd1eb9bf7a48d5ffe8f138f6f/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c1369db2442df97ea5391783faba8be0a5326524580dac31a65caaa88066fb82/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c1369db2442df97ea5391783faba8be0a5326524580dac31a65caaa88066fb82/userdata/shm major:0 minor:504 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ca280a5e21c2dd513ba61de02f0ae58881b1753b0bdf0b1d15c4e428bb0fbb1f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ca280a5e21c2dd513ba61de02f0ae58881b1753b0bdf0b1d15c4e428bb0fbb1f/userdata/shm major:0 minor:257 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cfe8a5916ef9fe15a5e0a8f79b48c137087a707357efd660e29b358bd865bb40/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cfe8a5916ef9fe15a5e0a8f79b48c137087a707357efd660e29b358bd865bb40/userdata/shm major:0 minor:456 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d665fd1da21957030f49e490695e1a57f3438cc4f5b6592c77dca06f03345d8a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d665fd1da21957030f49e490695e1a57f3438cc4f5b6592c77dca06f03345d8a/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d722cdc71acb0c1e3a81a70c28ff6ebbba944b71c0fbcad85b5c86e78c8336c4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d722cdc71acb0c1e3a81a70c28ff6ebbba944b71c0fbcad85b5c86e78c8336c4/userdata/shm major:0 minor:233 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dea167f7f04649f1c8c740ea754a186c7f21aa1229d68d244baf80de748a0b55/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dea167f7f04649f1c8c740ea754a186c7f21aa1229d68d244baf80de748a0b55/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f56f8782dab339e2e6d83b3ae256a7331821a82dc572428363224d45d35c046a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f56f8782dab339e2e6d83b3ae256a7331821a82dc572428363224d45d35c046a/userdata/shm major:0 minor:286 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ff1e220fab575099d63cac650b412e7123bbdba006089d74e309a9535e7600d1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ff1e220fab575099d63cac650b412e7123bbdba006089d74e309a9535e7600d1/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1/volumes/kubernetes.io~projected/kube-api-access-mj6m5:{mountpoint:/var/lib/kubelet/pods/06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1/volumes/kubernetes.io~projected/kube-api-access-mj6m5 major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0/volumes/kubernetes.io~projected/kube-api-access-nd649:{mountpoint:/var/lib/kubelet/pods/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0/volumes/kubernetes.io~projected/kube-api-access-nd649 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d0c5822-ab7b-409d-a755-49e918278fa1/volumes/kubernetes.io~projected/kube-api-access-gkznq:{mountpoint:/var/lib/kubelet/pods/0d0c5822-ab7b-409d-a755-49e918278fa1/volumes/kubernetes.io~projected/kube-api-access-gkznq major:0 minor:248 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d0c5822-ab7b-409d-a755-49e918278fa1/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/0d0c5822-ab7b-409d-a755-49e918278fa1/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d0c5822-ab7b-409d-a755-49e918278fa1/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/0d0c5822-ab7b-409d-a755-49e918278fa1/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:245 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0ec0b81c-6052-45f2-bd5b-6096ebb72223/volumes/kubernetes.io~projected/kube-api-access-sgwqj:{mountpoint:/var/lib/kubelet/pods/0ec0b81c-6052-45f2-bd5b-6096ebb72223/volumes/kubernetes.io~projected/kube-api-access-sgwqj major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f7bd8dd-6293-414d-8b9d-064f04678e61/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/0f7bd8dd-6293-414d-8b9d-064f04678e61/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f7bd8dd-6293-414d-8b9d-064f04678e61/volumes/kubernetes.io~projected/kube-api-access-z298l:{mountpoint:/var/lib/kubelet/pods/0f7bd8dd-6293-414d-8b9d-064f04678e61/volumes/kubernetes.io~projected/kube-api-access-z298l major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f7bd8dd-6293-414d-8b9d-064f04678e61/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/0f7bd8dd-6293-414d-8b9d-064f04678e61/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fadb078-8983-42df-afde-c1983044492d/volumes/kubernetes.io~projected/kube-api-access-wgsql:{mountpoint:/var/lib/kubelet/pods/0fadb078-8983-42df-afde-c1983044492d/volumes/kubernetes.io~projected/kube-api-access-wgsql major:0 minor:256 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fadb078-8983-42df-afde-c1983044492d/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/0fadb078-8983-42df-afde-c1983044492d/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:255 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fadb078-8983-42df-afde-c1983044492d/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/0fadb078-8983-42df-afde-c1983044492d/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:254 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9/volumes/kubernetes.io~projected/kube-api-access-ls676:{mountpoint:/var/lib/kubelet/pods/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9/volumes/kubernetes.io~projected/kube-api-access-ls676 major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:311 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:294 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:307 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:308 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:309 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2c1f3b2f-1d7f-44b5-bcba-f9310e7bdb96/volumes/kubernetes.io~projected/kube-api-access-z7bmn:{mountpoint:/var/lib/kubelet/pods/2c1f3b2f-1d7f-44b5-bcba-f9310e7bdb96/volumes/kubernetes.io~projected/kube-api-access-z7bmn major:0 minor:352 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2c1f3b2f-1d7f-44b5-bcba-f9310e7bdb96/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/2c1f3b2f-1d7f-44b5-bcba-f9310e7bdb96/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:347 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:496 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2dcde035-9b90-4961-8cfc-3f37bfb67581/volumes/kubernetes.io~projected/kube-api-access-7gb4j:{mountpoint:/var/lib/kubelet/pods/2dcde035-9b90-4961-8cfc-3f37bfb67581/volumes/kubernetes.io~projected/kube-api-access-7gb4j major:0 minor:441 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2dcde035-9b90-4961-8cfc-3f37bfb67581/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/2dcde035-9b90-4961-8cfc-3f37bfb67581/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:440 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2dcde035-9b90-4961-8cfc-3f37bfb67581/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/2dcde035-9b90-4961-8cfc-3f37bfb67581/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:436 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3083cbfe-7833-4b37-8325-cb18c20a7bea/volumes/kubernetes.io~projected/kube-api-access-9vdxd:{mountpoint:/var/lib/kubelet/pods/3083cbfe-7833-4b37-8325-cb18c20a7bea/volumes/kubernetes.io~projected/kube-api-access-9vdxd major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/30d608b6-d82e-4fcf-ab44-b8e064c463d0/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/30d608b6-d82e-4fcf-ab44-b8e064c463d0/volume-subpaths/web-config/config-reloader/2 major:0 minor:449 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/30d608b6-d82e-4fcf-ab44-b8e064c463d0/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/30d608b6-d82e-4fcf-ab44-b8e064c463d0/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:449 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/30d608b6-d82e-4fcf-ab44-b8e064c463d0/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/30d608b6-d82e-4fcf-ab44-b8e064c463d0/volume-subpaths/web-config/prometheus/15 major:0 minor:449 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/30d608b6-d82e-4fcf-ab44-b8e064c463d0/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/30d608b6-d82e-4fcf-ab44-b8e064c463d0/volumes/kubernetes.io~empty-dir/config-out major:0 minor:451 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/30d608b6-d82e-4fcf-ab44-b8e064c463d0/volumes/kubernetes.io~projected/kube-api-access-jdvh2:{mountpoint:/var/lib/kubelet/pods/30d608b6-d82e-4fcf-ab44-b8e064c463d0/volumes/kubernetes.io~projected/kube-api-access-jdvh2 major:0 minor:455 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/30d608b6-d82e-4fcf-ab44-b8e064c463d0/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/30d608b6-d82e-4fcf-ab44-b8e064c463d0/volumes/kubernetes.io~projected/tls-assets major:0 minor:447 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/30d608b6-d82e-4fcf-ab44-b8e064c463d0/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/30d608b6-d82e-4fcf-ab44-b8e064c463d0/volumes/kubernetes.io~secret/config major:0 minor:444 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/30d608b6-d82e-4fcf-ab44-b8e064c463d0/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/30d608b6-d82e-4fcf-ab44-b8e064c463d0/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:454 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/30d608b6-d82e-4fcf-ab44-b8e064c463d0/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/30d608b6-d82e-4fcf-ab44-b8e064c463d0/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:445 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/30d608b6-d82e-4fcf-ab44-b8e064c463d0/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/30d608b6-d82e-4fcf-ab44-b8e064c463d0/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:450 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/30d608b6-d82e-4fcf-ab44-b8e064c463d0/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/30d608b6-d82e-4fcf-ab44-b8e064c463d0/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:453 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/30d608b6-d82e-4fcf-ab44-b8e064c463d0/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/30d608b6-d82e-4fcf-ab44-b8e064c463d0/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:452 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/30d608b6-d82e-4fcf-ab44-b8e064c463d0/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/30d608b6-d82e-4fcf-ab44-b8e064c463d0/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:446 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/30d608b6-d82e-4fcf-ab44-b8e064c463d0/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/30d608b6-d82e-4fcf-ab44-b8e064c463d0/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:448 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/30d608b6-d82e-4fcf-ab44-b8e064c463d0/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/30d608b6-d82e-4fcf-ab44-b8e064c463d0/volumes/kubernetes.io~secret/web-config major:0 minor:449 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31a398f7-8539-4c36-8386-455ef83b3d2b/volumes/kubernetes.io~projected/kube-api-access-jsf72:{mountpoint:/var/lib/kubelet/pods/31a398f7-8539-4c36-8386-455ef83b3d2b/volumes/kubernetes.io~projected/kube-api-access-jsf72 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31a398f7-8539-4c36-8386-455ef83b3d2b/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/31a398f7-8539-4c36-8386-455ef83b3d2b/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/57428473-a3f3-4a91-9a28-41619c285147/volumes/kubernetes.io~projected/kube-api-access-h6mtj:{mountpoint:/var/lib/kubelet/pods/57428473-a3f3-4a91-9a28-41619c285147/volumes/kubernetes.io~projected/kube-api-access-h6mtj major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/57428473-a3f3-4a91-9a28-41619c285147/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/57428473-a3f3-4a91-9a28-41619c285147/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58a9c629-dff8-4f48-99b2-385d5d13e188/volumes/kubernetes.io~projected/kube-api-access-kzl6r:{mountpoint:/var/lib/kubelet/pods/58a9c629-dff8-4f48-99b2-385d5d13e188/volumes/kubernetes.io~projected/kube-api-access-kzl6r major:0 minor:501 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58dabb02-fd21-40ee-a4f5-e990b2b9a35a/volumes/kubernetes.io~projected/kube-api-access-6jmgk:{mountpoint:/var/lib/kubelet/pods/58dabb02-fd21-40ee-a4f5-e990b2b9a35a/volumes/kubernetes.io~projected/kube-api-access-6jmgk major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58dabb02-fd21-40ee-a4f5-e990b2b9a35a/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/58dabb02-fd21-40ee-a4f5-e990b2b9a35a/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58dabb02-fd21-40ee-a4f5-e990b2b9a35a/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/58dabb02-fd21-40ee-a4f5-e990b2b9a35a/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5bfbbeb5-621b-43b0-bad1-0420e8b07b62/volumes/kubernetes.io~projected/kube-api-access-lgv82:{mountpoint:/var/lib/kubelet/pods/5bfbbeb5-621b-43b0-bad1-0420e8b07b62/volumes/kubernetes.io~projected/kube-api-access-lgv82 major:0 minor:358 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5bfbbeb5-621b-43b0-bad1-0420e8b07b62/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/5bfbbeb5-621b-43b0-bad1-0420e8b07b62/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:355 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5bfbbeb5-621b-43b0-bad1-0420e8b07b62/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/5bfbbeb5-621b-43b0-bad1-0420e8b07b62/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:356 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5bfbbeb5-621b-43b0-bad1-0420e8b07b62/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/5bfbbeb5-621b-43b0-bad1-0420e8b07b62/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:357 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/62a2ee28-83a7-42bb-8018-7918027b3e18/volumes/kubernetes.io~projected/kube-api-access-k2cn4:{mountpoint:/var/lib/kubelet/pods/62a2ee28-83a7-42bb-8018-7918027b3e18/volumes/kubernetes.io~projected/kube-api-access-k2cn4 major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/62a2ee28-83a7-42bb-8018-7918027b3e18/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/62a2ee28-83a7-42bb-8018-7918027b3e18/volumes/kubernetes.io~secret/cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7361d2a9-d089-4073-bbfb-e7978943fd88/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/7361d2a9-d089-4073-bbfb-e7978943fd88/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7361d2a9-d089-4073-bbfb-e7978943fd88/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/7361d2a9-d089-4073-bbfb-e7978943fd88/volume-subpaths/web-config/alertmanager/9 major:0 minor:300 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7361d2a9-d089-4073-bbfb-e7978943fd88/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/7361d2a9-d089-4073-bbfb-e7978943fd88/volume-subpaths/web-config/config-reloader/6 major:0 minor:300 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7361d2a9-d089-4073-bbfb-e7978943fd88/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/7361d2a9-d089-4073-bbfb-e7978943fd88/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:300 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7361d2a9-d089-4073-bbfb-e7978943fd88/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/7361d2a9-d089-4073-bbfb-e7978943fd88/volumes/kubernetes.io~empty-dir/config-out major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7361d2a9-d089-4073-bbfb-e7978943fd88/volumes/kubernetes.io~projected/kube-api-access-cwz9f:{mountpoint:/var/lib/kubelet/pods/7361d2a9-d089-4073-bbfb-e7978943fd88/volumes/kubernetes.io~projected/kube-api-access-cwz9f major:0 minor:303 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7361d2a9-d089-4073-bbfb-e7978943fd88/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/7361d2a9-d089-4073-bbfb-e7978943fd88/volumes/kubernetes.io~projected/tls-assets major:0 minor:298 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7361d2a9-d089-4073-bbfb-e7978943fd88/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/7361d2a9-d089-4073-bbfb-e7978943fd88/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7361d2a9-d089-4073-bbfb-e7978943fd88/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/7361d2a9-d089-4073-bbfb-e7978943fd88/volumes/kubernetes.io~secret/config-volume major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7361d2a9-d089-4073-bbfb-e7978943fd88/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/7361d2a9-d089-4073-bbfb-e7978943fd88/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:302 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7361d2a9-d089-4073-bbfb-e7978943fd88/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/7361d2a9-d089-4073-bbfb-e7978943fd88/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7361d2a9-d089-4073-bbfb-e7978943fd88/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/7361d2a9-d089-4073-bbfb-e7978943fd88/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:301 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7361d2a9-d089-4073-bbfb-e7978943fd88/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/7361d2a9-d089-4073-bbfb-e7978943fd88/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7361d2a9-d089-4073-bbfb-e7978943fd88/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/7361d2a9-d089-4073-bbfb-e7978943fd88/volumes/kubernetes.io~secret/web-config major:0 minor:300 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a9bc0be-51b3-4a52-83ae-580c3a2f342c/volumes/kubernetes.io~projected/kube-api-access-xjhfh:{mountpoint:/var/lib/kubelet/pods/8a9bc0be-51b3-4a52-83ae-580c3a2f342c/volumes/kubernetes.io~projected/kube-api-access-xjhfh major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a9bc0be-51b3-4a52-83ae-580c3a2f342c/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/8a9bc0be-51b3-4a52-83ae-580c3a2f342c/volumes/kubernetes.io~secret/metrics-certs major:0 minor:500 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a83707c3-8143-4aad-9131-59986a79968b/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/a83707c3-8143-4aad-9131-59986a79968b/volumes/kubernetes.io~secret/agent-certs major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6263f8d-cced-465a-b393-ef51cb902e11/volumes/kubernetes.io~projected/kube-api-access-bk2xl:{mountpoint:/var/lib/kubelet/pods/b6263f8d-cced-465a-b393-ef51cb902e11/volumes/kubernetes.io~projected/kube-api-access-bk2xl major:0 minor:251 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6263f8d-cced-465a-b393-ef51cb902e11/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/b6263f8d-cced-465a-b393-ef51cb902e11/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:249 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6263f8d-cced-465a-b393-ef51cb902e11/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/b6263f8d-cced-465a-b393-ef51cb902e11/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:250 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c663a29c-713d-4184-9105-e4a442a894c6/volumes/kubernetes.io~projected/kube-api-access-566w9:{mountpoint:/var/lib/kubelet/pods/c663a29c-713d-4184-9105-e4a442a894c6/volumes/kubernetes.io~projected/kube-api-access-566w9 major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c7868eea-4b46-49e6-afb8-7fb4bb472fca/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/c7868eea-4b46-49e6-afb8-7fb4bb472fca/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c7868eea-4b46-49e6-afb8-7fb4bb472fca/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/c7868eea-4b46-49e6-afb8-7fb4bb472fca/volumes/kubernetes.io~empty-dir/tmp major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c7868eea-4b46-49e6-afb8-7fb4bb472fca/volumes/kubernetes.io~projected/kube-api-access-8df59:{mountpoint:/var/lib/kubelet/pods/c7868eea-4b46-49e6-afb8-7fb4bb472fca/volumes/kubernetes.io~projected/kube-api-access-8df59 major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7/volumes/kubernetes.io~projected/kube-api-access-qtq2f:{mountpoint:/var/lib/kubelet/pods/d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7/volumes/kubernetes.io~projected/kube-api-access-qtq2f major:0 minor:59 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/f9291491178db641dfde4639573721803be739fdac1b77a80943fd80978f0755/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/b11ae75b8e70cccfbfec653af056821d69936c372a59c9aed2cf2bb2e3b24570/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/9224e6bea290b21ab187183614a0ac606ebfcc6828e9b9629d4da55562b3ee58/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-112:{mountpoint:/var/lib/containers/storage/overlay/e4575a998c44da47250969fb4499912eaf5892f7efbf09dccbdd6b7a2fb54ad0/merged major:0 minor:112 fsType:overlay blockSize:0} overlay_0-113:{mountpoint:/var/lib/containers/storage/overlay/d9388ca434145db951214495c5cd3c296484ad439fe3c4a36fd130c8bfe53eec/merged major:0 minor:113 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/cca17802c65f5f20772f24996fa391b476d42c0c7526d4dfaf247f652eedd8ca/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/1d1f45b2c069b8133799c6301c4357e86d4e739b64d922f01be4a9bb74371cf6/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/6eb2aa92ef5626faf29aa6087f86a156d7720658b7fc748706685d4daec222a9/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/8b1fbe2f04bcbb556624f5047aedb131f9f660d981348ed17ae1005ea930b38a/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/bd2fcb99cf7789076c4a353a3149f35f17f8c40eb5d073e444853230eab9a132/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/897357d40f12cbc5c28f24e86475d3142732e487230bbabe2de8ab8679a828f5/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/7eda38f0a54275f33f489733ac7e0b887c3b2509491093ef5ac22308c65089ac/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/c740f3947b114fc8f6000b5a287c422b0c0d511f89b9a6ce7b6680019fc50a5f/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/cd81eaadf53291edabf50c0c97c223f853a3cc713125eee0dda79d2a3145a848/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/aca92b77a4e42f244e4fa94abeaa0b435dbb15aea1b0479391b8dc56792642b0/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/2c595c2cb257f89a8bf1d3e8c77468f9877187f4ef584cb84635dd4528119268/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/4321599d0c53cb3c9ebec14d8dbd49229380dbbb8b4f549748b3b8f0023e8d0f/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/149a8a7ee25e35f9c511099c6e25597ab4261c81503c167cfc6062c1defa7046/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/213ed9f517ee1da20f2ad893c443bd3243961d3dd1fbbd7ac619db594df3357c/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/6caf5abee23397d16d81290f716019015c709290cde6228eff2992af2b247b6b/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/3df79e014835b1fb2a6f1cf70f7510246a77ef17844a4c9b405a042418b40db8/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/29ed21839268eca5ff7ef20a43a3fb4db3c42eb8130ad4b5ff3a0141619e573f/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/fbc447b144af57b0fb8121f30ab045966c9f79fdbc54577df7c23506b912b670/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/c09183d7f35bd38bd4f2f1c3625111bc61ebb22895b3688b5e241efb469baef1/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/3967e3c6427665585e86b1d0e294a89ef3f84ca8d8a489a1fb7720e294ce50db/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/b4c4e327e67975ddebc527688489547e6dae538cc52dd93894724e22222141ac/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/ab4706278e5cb205cb0ba399d07a16e7073c12b850315362a445b4dc4b1c3e47/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/5e597074736aee1f8ddc831d6fcf8dd8b9f562963e3dc37a971ae6d1d4de1353/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-246:{mountpoint:/var/lib/containers/storage/overlay/018b74be879e63e65d586a4e0dcb8d2022585a8c254ba9b2d9a0dacba3bdb8d4/merged major:0 minor:246 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/8500ad6782e7b22c39bc44011108a783fcfb576f804a3408a681647ca5481ebb/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-263:{mountpoint:/var/lib/containers/storage/overlay/f955dfd9fe4835d487b38f0769c6e69f9307467a1b426af1cf611dfff539f6fe/merged major:0 minor:263 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/5e1226985bda551a92b3b658a0b19e89a698a1093b5e906fe8af9c3c98a6512e/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/5ce60eb274b756afaff342fd7efeeae8fb02003c802a3bef580fc378393f53b5/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/6cbbcdd6adcfa65b1b8d72a0ad360fdaad380da6ab3d8a932babd8443d33ca89/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/b85c675140e722a1fba8e841c16ade29cfa89a98cf374056305424c5c012b947/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/c59a15e85637983e38b395f8ba9814944400d5b618f1b7f85064dd0b49de15bd/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/323b2906d79b7ee7dafcea2b175b2704c1896ffa5af1fe559c3cc163b63fa03f/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-315:{mountpoint:/var/lib/containers/storage/overlay/298f2dd2328b92d64f628713cbd95b34b7b38b18ba124adc4d64e88bba6b4773/merged major:0 minor:315 fsType:overlay blockSize:0} overlay_0-317:{mountpoint:/var/lib/containers/storage/overlay/dcd7be1bbf6d53af23d5b0e215d1824bd382874a6569214eeb5b4a8db77a66b1/merged major:0 minor:317 fsType:overlay blockSize:0} overlay_0-319:{mountpoint:/var/lib/containers/storage/overlay/c05ce2a3f71019c248fa4a2defd7e0e6150efab9b112d5b30d8d8ccb3a30dad3/merged major:0 minor:319 fsType:overlay blockSize:0} overlay_0-325:{mountpoint:/var/lib/containers/storage/overlay/c82f78f4e46f319d83a2d879170de3900979e1984f79776f2993fb2e6af99c36/merged major:0 minor:325 fsType:overlay blockSize:0} overlay_0-326:{mountpoint:/var/lib/containers/storage/overlay/ea95954910d4f0730504fc2d4dfa57ff2c8abb6afe5a7c86296398f23469f989/merged major:0 minor:326 fsType:overlay blockSize:0} overlay_0-335:{mountpoint:/var/lib/containers/storage/overlay/0e27e9cdf9e325c6cf5722848818b9630035e78da88985b74cbfb97c39856aca/merged major:0 minor:335 fsType:overlay blockSize:0} overlay_0-337:{mountpoint:/var/lib/containers/storage/overlay/b3453fa8075c9748e432df4430b91b63ad058500c80019a19e68bc6ae3e7625a/merged major:0 minor:337 fsType:overlay blockSize:0} overlay_0-345:{mountpoint:/var/lib/containers/storage/overlay/ef98733ad53f37efd91076dd89fbc2d2ca95268fa70810273715f91d44bf395d/merged major:0 minor:345 fsType:overlay blockSize:0} overlay_0-361:{mountpoint:/var/lib/containers/storage/overlay/a597b2a6df36111104ef8a0dce4960fa91e062a09745b8ca78556267bdf88e54/merged major:0 minor:361 fsType:overlay blockSize:0} overlay_0-363:{mountpoint:/var/lib/containers/storage/overlay/ab5634dda1ba336abb535477b02e5f46811857e374f8c049a6e802eccf56342a/merged major:0 minor:363 fsType:overlay blockSize:0} overlay_0-365:{mountpoint:/var/lib/containers/storage/overlay/e4b905cd7c209c040671e2593e347b985c72ecf45e47df5231933fd91be82e00/merged major:0 minor:365 fsType:overlay blockSize:0} overlay_0-367:{mountpoint:/var/lib/containers/storage/overlay/46ccb09cf2bd755c3c252f32241daf353baa9402acf21979747143f202a51ec1/merged major:0 minor:367 fsType:overlay blockSize:0} overlay_0-373:{mountpoint:/var/lib/containers/storage/overlay/3f331bf7a81be157b2ff96793b3a4b343915bfd41243b0545d6087a84647926c/merged major:0 minor:373 fsType:overlay blockSize:0} overlay_0-378:{mountpoint:/var/lib/containers/storage/overlay/0c33c590f0308059774ca878d25e52d1be5c4f090df899fbf33a39ae9c8bff28/merged major:0 minor:378 fsType:overlay blockSize:0} overlay_0-383:{mountpoint:/var/lib/containers/storage/overlay/6ece984d2fc6ced5354cded35e35b427c28fa86a8ff2f8a2f92ff90b910af83b/merged major:0 minor:383 fsType:overlay blockSize:0} overlay_0-388:{mountpoint:/var/lib/containers/storage/overlay/48ed28728e5d005f291ff4af1354ec9282e92cc5426a5e5890195e9b24737000/merged major:0 minor:388 fsType:overlay blockSize:0} overlay_0-390:{mountpoint:/var/lib/containers/storage/overlay/2e8a299138653e3221ba0d8472103bd9ef804bcae704a64d48d071b6a6ecd686/merged major:0 minor:390 fsType:overlay blockSize:0} overlay_0-399:{mountpoint:/var/lib/containers/storage/overlay/426ec27c9bafd02786c8ca2da9e3052845d8ac4e0e43b4abd9281199d8f15c40/merged major:0 minor:399 fsType:overlay blockSize:0} overlay_0-404:{mountpoint:/var/lib/containers/storage/overlay/95115ac3e7d9b6277335ee4c55a2b2d87ff36a699f8f4d16141e077acf06ee55/merged major:0 minor:404 fsType:overlay blockSize:0} overlay_0-410:{mountpoint:/var/lib/containers/storage/overlay/58f6d175cc1544cd8f7a35418d40dc1e8a7a3d45714f14e5f604a755f1f84ff6/merged major:0 minor:410 fsType:overlay blockSize:0} overlay_0-416:{mountpoint:/var/lib/containers/storage/overlay/f92f292a95a1162074e3b99abefb8841248ccfa3f81a693e062948a6576d4624/merged major:0 minor:416 fsType:overlay blockSize:0} overlay_0-421:{mountpoint:/var/lib/containers/storage/overlay/6c9d448316572686cfc9ebf721852253fa19afe54467e79a10710d6101524a68/merged major:0 minor:421 fsType:overlay blockSize:0} overlay_0-426:{mountpoint:/var/lib/containers/storage/overlay/be2b77bfb8db45f2ea00ef62f29b73ffa75de1ec885e0289a0575ffab11efd1b/merged major:0 minor:426 fsType:overlay blockSize:0} overlay_0-431:{mountpoint:/var/lib/containers/storage/overlay/789e9f9ca4fad13d4ffc0cf52f5490098b704344d898d4f2aef083e6b771b0e9/merged major:0 minor:431 fsType:overlay blockSize:0} overlay_0-458:{mountpoint:/var/lib/containers/storage/overlay/5873c794d7228f8d6939abd298b8d1dd61b28529bee2c944aff7c47d2705a5b8/merged major:0 minor:458 fsType:overlay blockSize:0} overlay_0-460:{mountpoint:/var/lib/containers/storage/overlay/7d2d5029c087ded6bbae4be3f672590c225dd777e1f47429039f798ec7c5024e/merged major:0 minor:460 fsType:overlay blockSize:0} overlay_0-462:{mountpoint:/var/lib/containers/storage/overlay/141f45d838f48f90d1cb39871a3ccb3853fa6a59d5d3290028cb38e3840bc6a9/merged major:0 minor:462 fsType:overlay blockSize:0} overlay_0-468:{mountpoint:/var/lib/containers/storage/overlay/798a60a588e6704577950bae5fa78eb997c8da2458abf069d15f1e4fa51071e1/merged major:0 minor:468 fsType:overlay blockSize:0} overlay_0-470:{mountpoint:/var/lib/containers/storage/overlay/69f2b1ed9e51efc087807065458e4657fcca41e77ae572472c6c33a11c391a11/merged major:0 minor:470 fsType:overlay blockSize:0} overlay_0-476:{mountpoint:/var/lib/containers/storage/overlay/1812dda7a30c3635ca141a0be77112cfcf4cf1e76917510fd8624a71bf6b626b/merged major:0 minor:476 fsType:overlay blockSize:0} overlay_0-481:{mountpoint:/var/lib/containers/storage/overlay/52063c6d91ea3cdfebd90ec24ecdd9cdce49a9bf92507df6bb41c8af2c2a6812/merged major:0 minor:481 fsType:overlay blockSize:0} overlay_0-486:{mountpoint:/var/lib/containers/storage/overlay/01492aaab8bbe425d55e7e5f6f563b4f6f58cd272958d5f9620d5801fb6db956/merged major:0 minor:486 fsType:overlay blockSize:0} overlay_0-491:{mountpoint:/var/lib/containers/storage/overlay/ff95969294d97ec4468e5e447139d03e76e1e7fb7ce1a3ce720a6e681ceb657d/merged major:0 minor:491 fsType:overlay blockSize:0} overlay_0-508:{mountpoint:/var/lib/containers/storage/overlay/1f08f5d15579105561dc33b76e6d76d358478eae0ed1ea7b8da4cef893f1f9d6/merged major:0 minor:508 fsType:overlay blockSize:0} overlay_0-510:{mountpoint:/var/lib/containers/storage/overlay/4445e5a2857f0a56e719885538eec539e8af809a9ce5df5a906cc5258da768db/merged major:0 minor:510 fsType:overlay blockSize:0} overlay_0-512:{mountpoint:/var/lib/containers/storage/overlay/2912742c17441a45603ae4ef52aea6d6d9e9bb1b4ef61f12bfd7987bca1b0484/merged major:0 minor:512 fsType:overlay blockSize:0} overlay_0-514:{mountpoint:/var/lib/containers/storage/overlay/d6ae35a96cf3dcb50fd9c2beff8e9640160109178fb74bda554a9b3f99439445/merged major:0 minor:514 fsType:overlay blockSize:0} overlay_0-516:{mountpoint:/var/lib/containers/storage/overlay/4bfa4f802d4a6ff5ea05a76a258ae6658604cca513ff225df2a857210c0355b0/merged major:0 minor:516 fsType:overlay blockSize:0} overlay_0-522:{mountpoint:/var/lib/containers/storage/overlay/0c5a22cb9fc5f78bbf8fde5883903ab5f9e237a8f1d1360c6ad84d3a34ab8ca6/merged major:0 minor:522 fsType:overlay blockSize:0} overlay_0-527:{mountpoint:/var/lib/containers/storage/overlay/17970e8e1d21a63ddcc3f938652903559e7aba7414a63497f924ad5cc7085fb6/merged major:0 minor:527 fsType:overlay blockSize:0} overlay_0-62:{mountpoint:/var/lib/containers/storage/overlay/850883dd166fea9880bea778bcb74a259d5673261ee1dd56629c531eebadb6b9/merged major:0 minor:62 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/e6642916b793153cd910bba01eb6812ed7877df8b48dda6e34e671de480f4015/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/2eaefbeb29d7928625d8d9292a1d720b5672e280c6e999ea77d042b874a86c1a/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/266f54f54cf85df54c610c152843b15badd3acf262449415db3e50b0c59295ec/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/40f1639f1bb02178e2d33ecf730a5540415b15fa2d7abbf9fb50832875071300/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/df6a28c40959996d0f3b36d35a8bf99573c5224c5adb13036a2f04364e89d259/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/a58a257b9083369ca1bc955b05749340a7145a6b108731eec0c5e1ee5598010b/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/54ae5042ef6291e4f5ed19c4ce30e7b11ba6529c60b2b97f8b541037af843c4f/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/104768ee6df29a77f7d7e2f0683880e9f0bdf73d504ef63192bb1043a4426048/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/1cf65b11601f9dfc569f3abe3edc147cb7bdb09a0b01c058114d0dbec8bde8d6/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 03 13:05:18.058825 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.051626 5979 manager.go:217] Machine: {Timestamp:2026-11-03 13:05:18.051134941 +0000 UTC m=+0.061823173 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3115195 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25f2ad0e51ce23d76bf1e6f6fe8c75 SystemUUID:ec25f2ad-0e51-ce23-d76b-f1e6f6fe8c75 BootID:57b3a02b-23d5-4098-84e1-f37f4f8f675f Filesystems:[{Device:overlay_0-510 DeviceMajor:0 DeviceMinor:510 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0f7bd8dd-6293-414d-8b9d-064f04678e61/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-416 DeviceMajor:0 DeviceMinor:416 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/30d608b6-d82e-4fcf-ab44-b8e064c463d0/volumes/kubernetes.io~projected/kube-api-access-jdvh2 DeviceMajor:0 DeviceMinor:455 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:311 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-399 DeviceMajor:0 DeviceMinor:399 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/30d608b6-d82e-4fcf-ab44-b8e064c463d0/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:449 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/62a2ee28-83a7-42bb-8018-7918027b3e18/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/58dabb02-fd21-40ee-a4f5-e990b2b9a35a/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:229 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2dcde035-9b90-4961-8cfc-3f37bfb67581/volumes/kubernetes.io~projected/kube-api-access-7gb4j DeviceMajor:0 DeviceMinor:441 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8a9bc0be-51b3-4a52-83ae-580c3a2f342c/volumes/kubernetes.io~projected/kube-api-access-xjhfh DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0d0c5822-ab7b-409d-a755-49e918278fa1/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:241 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7361d2a9-d089-4073-bbfb-e7978943fd88/volumes/kubernetes.io~projected/kube-api-access-cwz9f DeviceMajor:0 DeviceMinor:303 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-383 DeviceMajor:0 DeviceMinor:383 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2dcde035-9b90-4961-8cfc-3f37bfb67581/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:440 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c7868eea-4b46-49e6-afb8-7fb4bb472fca/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:47 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/30d608b6-d82e-4fcf-ab44-b8e064c463d0/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:445 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/30d608b6-d82e-4fcf-ab44-b8e064c463d0/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:450 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-470 DeviceMajor:0 DeviceMinor:470 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c663a29c-713d-4184-9105-e4a442a894c6/volumes/kubernetes.io~projected/kube-api-access-566w9 DeviceMajor:0 DeviceMinor:54 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a9d7c3dabde3dda56feb11acc4d301b588e998ab794a8368b9abdca272ec1905/userdata/shm DeviceMajor:0 DeviceMinor:252 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c1369db2442df97ea5391783faba8be0a5326524580dac31a65caaa88066fb82/userdata/shm DeviceMajor:0 DeviceMinor:504 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4c54177f6ce7c0b369791011f7103908e9f3a45261bef440845bddd12613fbab/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:308 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:307 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-326 DeviceMajor:0 DeviceMinor:326 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2c1f3b2f-1d7f-44b5-bcba-f9310e7bdb96/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:347 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/30d608b6-d82e-4fcf-ab44-b8e064c463d0/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:449 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/0f7bd8dd-6293-414d-8b9d-064f04678e61/volumes/kubernetes.io~projected/kube-api-access-z298l DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/31a398f7-8539-4c36-8386-455ef83b3d2b/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7/volumes/kubernetes.io~projected/kube-api-access-qtq2f DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-113 DeviceMajor:0 DeviceMinor:113 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/49b66c76afba64193c0a47b9415b935a5553d5caae03c6370b8574979bb86404/userdata/shm DeviceMajor:0 DeviceMinor:353 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/30d608b6-d82e-4fcf-ab44-b8e064c463d0/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:449 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c7868eea-4b46-49e6-afb8-7fb4bb472fca/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:49 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-431 DeviceMajor:0 DeviceMinor:431 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-462 DeviceMajor:0 DeviceMinor:462 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0ec0b81c-6052-45f2-bd5b-6096ebb72223/volumes/kubernetes.io~projected/kube-api-access-sgwqj DeviceMajor:0 DeviceMinor:53 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-246 DeviceMajor:0 DeviceMinor:246 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:294 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6eac71e8dec9af14956357fa10a6d4595db34ba6f85cf32f4b8efd29d9426efc/userdata/shm DeviceMajor:0 DeviceMinor:313 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a9e9596490c552decf61398ebee95872f8e6ece435e16fd80d0daea5fc85f19e/userdata/shm DeviceMajor:0 DeviceMinor:359 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-390 DeviceMajor:0 DeviceMinor:390 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-491 DeviceMajor:0 DeviceMinor:491 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/31a398f7-8539-4c36-8386-455ef83b3d2b/volumes/kubernetes.io~projected/kube-api-access-jsf72 DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0d0c5822-ab7b-409d-a755-49e918278fa1/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:245 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-325 DeviceMajor:0 DeviceMinor:325 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/30d608b6-d82e-4fcf-ab44-b8e064c463d0/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:447 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-315 DeviceMajor:0 DeviceMinor:315 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-319 DeviceMajor:0 DeviceMinor:319 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-361 DeviceMajor:0 DeviceMinor:361 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3083cbfe-7833-4b37-8325-cb18c20a7bea/volumes/kubernetes.io~projected/kube-api-access-9vdxd DeviceMajor:0 DeviceMinor:51 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0/volumes/kubernetes.io~projected/kube-api-access-nd649 DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b6263f8d-cced-465a-b393-ef51cb902e11/volumes/kubernetes.io~projected/kube-api-access-bk2xl DeviceMajor:0 DeviceMinor:251 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3b26249f5bd52d607583665ec09161ee13a1786eaaed3a195ce24774d76f055b/userdata/shm DeviceMajor:0 DeviceMinor:258 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7361d2a9-d089-4073-bbfb-e7978943fd88/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:298 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f56f8782dab339e2e6d83b3ae256a7331821a82dc572428363224d45d35c046a/userdata/shm DeviceMajor:0 DeviceMinor:286 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-335 DeviceMajor:0 DeviceMinor:335 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-404 DeviceMajor:0 DeviceMinor:404 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-421 DeviceMajor:0 DeviceMinor:421 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1ba727bf8ab0e0dbb6df93b3ac18c1f3f0af05d767a6a66729782a995c797fc0/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5bfbbeb5-621b-43b0-bad1-0420e8b07b62/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:357 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-426 DeviceMajor:0 DeviceMinor:426 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/30d608b6-d82e-4fcf-ab44-b8e064c463d0/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:444 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cfe8a5916ef9fe15a5e0a8f79b48c137087a707357efd660e29b358bd865bb40/userdata/shm DeviceMajor:0 DeviceMinor:456 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/30d608b6-d82e-4fcf-ab44-b8e064c463d0/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:449 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7361d2a9-d089-4073-bbfb-e7978943fd88/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:295 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7361d2a9-d089-4073-bbfb-e7978943fd88/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:300 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:309 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7361d2a9-d089-4073-bbfb-e7978943fd88/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:300 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/30d608b6-d82e-4fcf-ab44-b8e064c463d0/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:454 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8a9bc0be-51b3-4a52-83ae-580c3a2f342c/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:500 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/58dabb02-fd21-40ee-a4f5-e990b2b9a35a/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:225 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-508 DeviceMajor:0 DeviceMinor:508 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/30d608b6-d82e-4fcf-ab44-b8e064c463d0/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:452 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-476 DeviceMajor:0 DeviceMinor:476 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-527 DeviceMajor:0 DeviceMinor:527 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/57428473-a3f3-4a91-9a28-41619c285147/volumes/kubernetes.io~projected/kube-api-access-h6mtj DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ca280a5e21c2dd513ba61de02f0ae58881b1753b0bdf0b1d15c4e428bb0fbb1f/userdata/shm DeviceMajor:0 DeviceMinor:257 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-410 DeviceMajor:0 DeviceMinor:410 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/30d608b6-d82e-4fcf-ab44-b8e064c463d0/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:451 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-458 DeviceMajor:0 DeviceMinor:458 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-468 DeviceMajor:0 DeviceMinor:468 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-486 DeviceMajor:0 DeviceMinor:486 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1/volumes/kubernetes.io~projected/kube-api-access-mj6m5 DeviceMajor:0 DeviceMinor:52 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/04419437a7c583e8767facf7b6dd8ac270e24b70dd856e7d9a1bcd0e8b71bb22/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dea167f7f04649f1c8c740ea754a186c7f21aa1229d68d244baf80de748a0b55/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0fadb078-8983-42df-afde-c1983044492d/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:254 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:310 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2dcde035-9b90-4961-8cfc-3f37bfb67581/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:436 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/30d608b6-d82e-4fcf-ab44-b8e064c463d0/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:448 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0fadb078-8983-42df-afde-c1983044492d/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:255 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7361d2a9-d089-4073-bbfb-e7978943fd88/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:291 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5bfbbeb5-621b-43b0-bad1-0420e8b07b62/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:355 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-365 DeviceMajor:0 DeviceMinor:365 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/30d608b6-d82e-4fcf-ab44-b8e064c463d0/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:453 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-514 DeviceMajor:0 DeviceMinor:514 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/57428473-a3f3-4a91-9a28-41619c285147/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0d0c5822-ab7b-409d-a755-49e918278fa1/volumes/kubernetes.io~projected/kube-api-access-gkznq DeviceMajor:0 DeviceMinor:248 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-345 DeviceMajor:0 DeviceMinor:345 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-481 DeviceMajor:0 DeviceMinor:481 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-512 DeviceMajor:0 DeviceMinor:512 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-516 DeviceMajor:0 DeviceMinor:516 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9a91b9c57cf303820186cee25a3e09aa472d0822dbae53e22020ed5505ea4fa9/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/400ccad898a899b91604c1a67253b559529031211d7f42dbae40a19cc322a55d/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-263 DeviceMajor:0 DeviceMinor:263 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7361d2a9-d089-4073-bbfb-e7978943fd88/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:299 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b32d70e0c0296b39175111442498979160d7e804c4d7b1876b6e227343cfc7ab/userdata/shm DeviceMajor:0 DeviceMinor:506 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-522 DeviceMajor:0 DeviceMinor:522 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c7868eea-4b46-49e6-afb8-7fb4bb472fca/volumes/kubernetes.io~projected/kube-api-access-8df59 DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/97e7b232bd9bde5b04704bff068be4e42c0c8b238cbda87d3cf80eea9248717a/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7361d2a9-d089-4073-bbfb-e7978943fd88/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:301 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-367 DeviceMajor:0 DeviceMinor:367 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d665fd1da21957030f49e490695e1a57f3438cc4f5b6592c77dca06f03345d8a/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d722cdc71acb0c1e3a81a70c28ff6ebbba944b71c0fbcad85b5c86e78c8336c4/userdata/shm DeviceMajor:0 DeviceMinor:233 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7361d2a9-d089-4073-bbfb-e7978943fd88/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:300 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5bfbbeb5-621b-43b0-bad1-0420e8b07b62/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:356 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/30d608b6-d82e-4fcf-ab44-b8e064c463d0/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:446 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6c3945e8883e0a259fd5f007da8c7429799c94f6107a8e45f05f5e2948efe6ed/userdata/shm DeviceMajor:0 DeviceMinor:502 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0f7bd8dd-6293-414d-8b9d-064f04678e61/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:48 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-62 DeviceMajor:0 DeviceMinor:62 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6de1e4a21fe31cf1358127b395a713134fadd5a4cf986e8b1b3e4fe8cea83e27/userdata/shm DeviceMajor:0 DeviceMinor:64 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/022573a872e57bdae8e145e998cc17c3d120a445311e823f943e50e7c76cc596/userdata/shm DeviceMajor:0 DeviceMinor:87 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-112 DeviceMajor:0 DeviceMinor:112 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b404676561694e010ac4e528fbf5d92e6d2411bcd1eb9bf7a48d5ffe8f138f6f/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/63f68d72e1402d9f5b019be6e003a98b22ba8f0f8c43f2199bdd191bcf34a87c/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7361d2a9-d089-4073-bbfb-e7978943fd88/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:297 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0fadb078-8983-42df-afde-c1983044492d/volumes/kubernetes.io~projected/kube-api-access-wgsql DeviceMajor:0 DeviceMinor:256 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-317 DeviceMajor:0 DeviceMinor:317 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-337 DeviceMajor:0 DeviceMinor:337 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2c1f3b2f-1d7f-44b5-bcba-f9310e7bdb96/volumes/kubernetes.io~projected/kube-api-access-z7bmn DeviceMajor:0 DeviceMinor:352 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-373 DeviceMajor:0 DeviceMinor:373 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-388 DeviceMajor:0 DeviceMinor:388 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/021415e37c70ce8f7864ca9d751fb5be0f42c656f462e392f6bbcfb72e3be2ad/userdata/shm DeviceMajor:0 DeviceMinor:442 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7361d2a9-d089-4073-bbfb-e7978943fd88/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:299 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7361d2a9-d089-4073-bbfb-e7978943fd88/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:302 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-378 DeviceMajor:0 DeviceMinor:378 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/58dabb02-fd21-40ee-a4f5-e990b2b9a35a/volumes/kubernetes.io~projected/kube-api-access-6jmgk DeviceMajor:0 DeviceMinor:230 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b6263f8d-cced-465a-b393-ef51cb902e11/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:249 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7361d2a9-d089-4073-bbfb-e7978943fd88/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:296 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9/volumes/kubernetes.io~projected/kube-api-access-ls676 DeviceMajor:0 DeviceMinor:312 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5bfbbeb5-621b-43b0-bad1-0420e8b07b62/volumes/kubernetes.io~projected/kube-api-access-lgv82 DeviceMajor:0 DeviceMinor:358 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7361d2a9-d089-4073-bbfb-e7978943fd88/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:300 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a83707c3-8143-4aad-9131-59986a79968b/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:50 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ff1e220fab575099d63cac650b412e7123bbdba006089d74e309a9535e7600d1/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:496 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/58a9c629-dff8-4f48-99b2-385d5d13e188/volumes/kubernetes.io~projected/kube-api-access-kzl6r DeviceMajor:0 DeviceMinor:501 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/62a2ee28-83a7-42bb-8018-7918027b3e18/volumes/kubernetes.io~projected/kube-api-access-k2cn4 DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b6263f8d-cced-465a-b393-ef51cb902e11/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:250 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-363 DeviceMajor:0 DeviceMinor:363 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-460 DeviceMajor:0 DeviceMinor:460 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:021415e37c70ce8 MacAddress:c6:c2:de:7c:9e:aa Speed:10000 Mtu:8901} {Name:3b26249f5bd52d6 MacAddress:96:6e:d7:36:3f:23 Speed:10000 Mtu:8901} {Name:49b66c76afba641 MacAddress:aa:d6:29:44:99:8a Speed:10000 Mtu:8901} {Name:63f68d72e1402d9 MacAddress:5a:06:b5:13:62:83 Speed:10000 Mtu:8901} {Name:6c3945e8883e0a2 MacAddress:ca:69:0f:4c:b9:eb Speed:10000 Mtu:8901} {Name:6eac71e8dec9af1 MacAddress:66:2c:db:62:9e:f9 Speed:10000 Mtu:8901} {Name:a9d7c3dabde3dda MacAddress:b2:40:d5:20:9b:a5 Speed:10000 Mtu:8901} {Name:a9e9596490c552d MacAddress:d6:f0:87:14:4d:2f Speed:10000 Mtu:8901} {Name:b32d70e0c0296b3 MacAddress:a6:6d:93:82:83:83 Speed:10000 Mtu:8901} {Name:b404676561694e0 MacAddress:92:b3:68:e6:86:f9 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:d7:16:ec:3f Speed:0 Mtu:9001} {Name:br-int MacAddress:3e:32:0e:cd:fb:24 Speed:0 Mtu:8901} {Name:c1369db2442df97 MacAddress:c6:8e:d1:aa:58:97 Speed:10000 Mtu:8901} {Name:cfe8a5916ef9fe1 MacAddress:3a:66:92:d5:96:b6 Speed:10000 Mtu:8901} {Name:d722cdc71acb0c1 MacAddress:1e:d7:15:85:e2:5c Speed:10000 Mtu:8901} {Name:dea167f7f04649f MacAddress:b6:6c:f1:10:be:f2 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:d7:16:ec:3f Speed:0 Mtu:9001} {Name:f56f8782dab339e MacAddress:5a:4e:6a:a3:ad:16 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:8a:a3:23:ae:04:7c Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:ce:d5:f3:32:f4:d3 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 13:05:18.058825 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.052115 5979 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 13:05:18.058825 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.052163 5979 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 13:05:18.058825 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.052523 5979 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 13:05:18.058825 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.052561 5979 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-198.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 13:05:18.058825 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.052754 5979 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 13:05:18.058825 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.052767 5979 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 13:05:18.058825 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.052782 5979 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 13:05:18.058825 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.052811 5979 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 13:05:18.058825 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.052942 5979 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 13:05:18.058825 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.053168 5979 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 13:05:18.058825 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.053316 5979 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 13:05:18.058825 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.053361 5979 kubelet.go:493] "Attempting to sync node with API server" Nov 03 13:05:18.058825 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.053376 5979 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 13:05:18.058825 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.053392 5979 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 13:05:18.058825 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.053406 5979 kubelet.go:405] "Adding apiserver pod source" Nov 03 13:05:18.058825 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.053417 5979 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 13:05:18.058825 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.054388 5979 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 13:05:18.058825 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.054402 5979 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 13:05:18.058825 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.055084 5979 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 13:05:18.058825 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.057765 5979 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 13:05:18.058825 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.057775 5979 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 13:05:18.058825 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.057872 5979 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 13:05:18.058825 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.058054 5979 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 13:05:18.058825 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.058081 5979 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 13:05:18.058825 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.058170 5979 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 13:05:18.058825 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.058182 5979 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 13:05:18.058825 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.058188 5979 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 13:05:18.058825 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.058194 5979 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 13:05:18.058825 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.058201 5979 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 13:05:18.058825 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.058207 5979 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 13:05:18.058825 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.058213 5979 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 13:05:18.058825 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.058218 5979 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 13:05:18.058825 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.058225 5979 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 13:05:18.058825 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.058230 5979 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 13:05:18.058825 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.058244 5979 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 13:05:18.058825 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.058257 5979 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 13:05:18.058825 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.058287 5979 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 13:05:18.058825 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.058293 5979 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 13:05:18.061719 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.061706 5979 server.go:1263] "Started kubelet" Nov 03 13:05:18.061835 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.061801 5979 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 13:05:18.061975 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.061934 5979 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 13:05:18.062032 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.061995 5979 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 13:05:18.062215 ip-10-0-130-198 systemd[1]: Started Kubernetes Kubelet. Nov 03 13:05:18.064686 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.063146 5979 server.go:317] "Adding debug handlers to kubelet server" Nov 03 13:05:18.065842 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.065726 5979 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 13:05:18.069279 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.067634 5979 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 13:05:18.069279 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.067710 5979 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 13:05:18.069279 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.067781 5979 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 12:59:07 +0000 UTC" deadline="2028-06-19 13:26:22.866422873 +0000 UTC" Nov 03 13:05:18.069279 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.067818 5979 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14256h21m4.798608314s" Nov 03 13:05:18.069279 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.067884 5979 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 13:05:18.069279 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.067895 5979 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 13:05:18.069279 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.068414 5979 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 13:05:18.069279 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:05:18.068503 5979 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-198.us-west-2.compute.internal\" not found" Nov 03 13:05:18.072561 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.072541 5979 factory.go:55] Registering systemd factory Nov 03 13:05:18.072561 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.072564 5979 factory.go:223] Registration of the systemd container factory successfully Nov 03 13:05:18.072716 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.072677 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2dcde035-9b90-4961-8cfc-3f37bfb67581" volumeName="kubernetes.io/configmap/2dcde035-9b90-4961-8cfc-3f37bfb67581-trusted-ca-bundle" seLinuxMountContext="" Nov 03 13:05:18.072751 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.072723 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d0c5822-ab7b-409d-a755-49e918278fa1" volumeName="kubernetes.io/projected/0d0c5822-ab7b-409d-a755-49e918278fa1-kube-api-access-gkznq" seLinuxMountContext="" Nov 03 13:05:18.072751 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.072738 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fef60a4-2a84-4318-b1bf-feb8f69ef7e9" volumeName="kubernetes.io/secret/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Nov 03 13:05:18.072806 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.072752 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30d608b6-d82e-4fcf-ab44-b8e064c463d0" volumeName="kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Nov 03 13:05:18.072806 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.072763 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6263f8d-cced-465a-b393-ef51cb902e11" volumeName="kubernetes.io/secret/b6263f8d-cced-465a-b393-ef51cb902e11-kube-state-metrics-tls" seLinuxMountContext="" Nov 03 13:05:18.072806 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.072775 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7868eea-4b46-49e6-afb8-7fb4bb472fca" volumeName="kubernetes.io/empty-dir/c7868eea-4b46-49e6-afb8-7fb4bb472fca-tmp" seLinuxMountContext="" Nov 03 13:05:18.072806 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.072786 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fadb078-8983-42df-afde-c1983044492d" volumeName="kubernetes.io/configmap/0fadb078-8983-42df-afde-c1983044492d-node-exporter-accelerators-collector-config" seLinuxMountContext="" Nov 03 13:05:18.072806 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.072798 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2dcde035-9b90-4961-8cfc-3f37bfb67581" volumeName="kubernetes.io/secret/2dcde035-9b90-4961-8cfc-3f37bfb67581-console-serving-cert" seLinuxMountContext="" Nov 03 13:05:18.072936 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.072812 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3083cbfe-7833-4b37-8325-cb18c20a7bea" volumeName="kubernetes.io/projected/3083cbfe-7833-4b37-8325-cb18c20a7bea-kube-api-access-9vdxd" seLinuxMountContext="" Nov 03 13:05:18.072936 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.072823 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5bfbbeb5-621b-43b0-bad1-0420e8b07b62" volumeName="kubernetes.io/secret/5bfbbeb5-621b-43b0-bad1-0420e8b07b62-secret-metrics-server-client-certs" seLinuxMountContext="" Nov 03 13:05:18.072936 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.072834 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30d608b6-d82e-4fcf-ab44-b8e064c463d0" volumeName="kubernetes.io/empty-dir/30d608b6-d82e-4fcf-ab44-b8e064c463d0-prometheus-k8s-db" seLinuxMountContext="" Nov 03 13:05:18.072936 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.072849 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30d608b6-d82e-4fcf-ab44-b8e064c463d0" volumeName="kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-thanos-prometheus-http-client-file" seLinuxMountContext="" Nov 03 13:05:18.072936 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.072860 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6263f8d-cced-465a-b393-ef51cb902e11" volumeName="kubernetes.io/configmap/b6263f8d-cced-465a-b393-ef51cb902e11-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Nov 03 13:05:18.072936 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.072871 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0" volumeName="kubernetes.io/configmap/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-multus-daemon-config" seLinuxMountContext="" Nov 03 13:05:18.072936 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.072885 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fef60a4-2a84-4318-b1bf-feb8f69ef7e9" volumeName="kubernetes.io/secret/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Nov 03 13:05:18.072936 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.072896 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31a398f7-8539-4c36-8386-455ef83b3d2b" volumeName="kubernetes.io/projected/31a398f7-8539-4c36-8386-455ef83b3d2b-kube-api-access-jsf72" seLinuxMountContext="" Nov 03 13:05:18.072936 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.072908 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7361d2a9-d089-4073-bbfb-e7978943fd88" volumeName="kubernetes.io/configmap/7361d2a9-d089-4073-bbfb-e7978943fd88-metrics-client-ca" seLinuxMountContext="" Nov 03 13:05:18.072936 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.072921 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6263f8d-cced-465a-b393-ef51cb902e11" volumeName="kubernetes.io/secret/b6263f8d-cced-465a-b393-ef51cb902e11-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 13:05:18.072936 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.072931 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1" volumeName="kubernetes.io/configmap/06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1-cni-binary-copy" seLinuxMountContext="" Nov 03 13:05:18.073410 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.072942 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2dcde035-9b90-4961-8cfc-3f37bfb67581" volumeName="kubernetes.io/configmap/2dcde035-9b90-4961-8cfc-3f37bfb67581-console-config" seLinuxMountContext="" Nov 03 13:05:18.073410 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.072953 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30d608b6-d82e-4fcf-ab44-b8e064c463d0" volumeName="kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Nov 03 13:05:18.073410 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.072964 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30d608b6-d82e-4fcf-ab44-b8e064c463d0" volumeName="kubernetes.io/empty-dir/30d608b6-d82e-4fcf-ab44-b8e064c463d0-config-out" seLinuxMountContext="" Nov 03 13:05:18.073410 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.072974 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30d608b6-d82e-4fcf-ab44-b8e064c463d0" volumeName="kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-secret-kube-rbac-proxy" seLinuxMountContext="" Nov 03 13:05:18.073410 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.072986 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5bfbbeb5-621b-43b0-bad1-0420e8b07b62" volumeName="kubernetes.io/projected/5bfbbeb5-621b-43b0-bad1-0420e8b07b62-kube-api-access-lgv82" seLinuxMountContext="" Nov 03 13:05:18.073410 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.072997 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6263f8d-cced-465a-b393-ef51cb902e11" volumeName="kubernetes.io/configmap/b6263f8d-cced-465a-b393-ef51cb902e11-metrics-client-ca" seLinuxMountContext="" Nov 03 13:05:18.073410 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073007 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fef60a4-2a84-4318-b1bf-feb8f69ef7e9" volumeName="kubernetes.io/empty-dir/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9-tmp" seLinuxMountContext="" Nov 03 13:05:18.073410 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073018 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2dcde035-9b90-4961-8cfc-3f37bfb67581" volumeName="kubernetes.io/projected/2dcde035-9b90-4961-8cfc-3f37bfb67581-kube-api-access-7gb4j" seLinuxMountContext="" Nov 03 13:05:18.073410 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073030 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31a398f7-8539-4c36-8386-455ef83b3d2b" volumeName="kubernetes.io/configmap/31a398f7-8539-4c36-8386-455ef83b3d2b-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 03 13:05:18.073410 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073050 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31a398f7-8539-4c36-8386-455ef83b3d2b" volumeName="kubernetes.io/empty-dir/31a398f7-8539-4c36-8386-455ef83b3d2b-data-volume" seLinuxMountContext="" Nov 03 13:05:18.073410 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073062 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1" volumeName="kubernetes.io/configmap/06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 13:05:18.073410 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073073 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fef60a4-2a84-4318-b1bf-feb8f69ef7e9" volumeName="kubernetes.io/configmap/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9-metrics-client-ca" seLinuxMountContext="" Nov 03 13:05:18.073410 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073085 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fef60a4-2a84-4318-b1bf-feb8f69ef7e9" volumeName="kubernetes.io/secret/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Nov 03 13:05:18.073410 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073096 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c1f3b2f-1d7f-44b5-bcba-f9310e7bdb96" volumeName="kubernetes.io/projected/2c1f3b2f-1d7f-44b5-bcba-f9310e7bdb96-kube-api-access-z7bmn" seLinuxMountContext="" Nov 03 13:05:18.073410 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073108 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f7bd8dd-6293-414d-8b9d-064f04678e61" volumeName="kubernetes.io/projected/0f7bd8dd-6293-414d-8b9d-064f04678e61-kube-api-access-z298l" seLinuxMountContext="" Nov 03 13:05:18.073410 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073120 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fadb078-8983-42df-afde-c1983044492d" volumeName="kubernetes.io/configmap/0fadb078-8983-42df-afde-c1983044492d-metrics-client-ca" seLinuxMountContext="" Nov 03 13:05:18.073410 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073132 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30d608b6-d82e-4fcf-ab44-b8e064c463d0" volumeName="kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-configmap-metrics-client-ca" seLinuxMountContext="" Nov 03 13:05:18.073410 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073137 5979 factory.go:153] Registering CRI-O factory Nov 03 13:05:18.073410 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073144 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31a398f7-8539-4c36-8386-455ef83b3d2b" volumeName="kubernetes.io/secret/31a398f7-8539-4c36-8386-455ef83b3d2b-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 03 13:05:18.073410 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073150 5979 factory.go:223] Registration of the crio container factory successfully Nov 03 13:05:18.073410 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073157 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5bfbbeb5-621b-43b0-bad1-0420e8b07b62" volumeName="kubernetes.io/configmap/5bfbbeb5-621b-43b0-bad1-0420e8b07b62-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Nov 03 13:05:18.073410 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073168 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6263f8d-cced-465a-b393-ef51cb902e11" volumeName="kubernetes.io/empty-dir/b6263f8d-cced-465a-b393-ef51cb902e11-volume-directive-shadow" seLinuxMountContext="" Nov 03 13:05:18.073410 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073178 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6263f8d-cced-465a-b393-ef51cb902e11" volumeName="kubernetes.io/projected/b6263f8d-cced-465a-b393-ef51cb902e11-kube-api-access-bk2xl" seLinuxMountContext="" Nov 03 13:05:18.073410 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073190 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2dcde035-9b90-4961-8cfc-3f37bfb67581" volumeName="kubernetes.io/configmap/2dcde035-9b90-4961-8cfc-3f37bfb67581-service-ca" seLinuxMountContext="" Nov 03 13:05:18.073410 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073202 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30d608b6-d82e-4fcf-ab44-b8e064c463d0" volumeName="kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-prometheus-trusted-ca-bundle" seLinuxMountContext="" Nov 03 13:05:18.073410 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073216 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30d608b6-d82e-4fcf-ab44-b8e064c463d0" volumeName="kubernetes.io/projected/30d608b6-d82e-4fcf-ab44-b8e064c463d0-kube-api-access-jdvh2" seLinuxMountContext="" Nov 03 13:05:18.073410 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073230 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f7bd8dd-6293-414d-8b9d-064f04678e61" volumeName="kubernetes.io/configmap/0f7bd8dd-6293-414d-8b9d-064f04678e61-env-overrides" seLinuxMountContext="" Nov 03 13:05:18.073410 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073246 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57428473-a3f3-4a91-9a28-41619c285147" volumeName="kubernetes.io/empty-dir/57428473-a3f3-4a91-9a28-41619c285147-tmp-dir" seLinuxMountContext="" Nov 03 13:05:18.073410 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073259 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7361d2a9-d089-4073-bbfb-e7978943fd88" volumeName="kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-cluster-tls-config" seLinuxMountContext="" Nov 03 13:05:18.073410 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073272 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a83707c3-8143-4aad-9131-59986a79968b" volumeName="kubernetes.io/configmap/a83707c3-8143-4aad-9131-59986a79968b-konnectivity-ca" seLinuxMountContext="" Nov 03 13:05:18.073410 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073205 5979 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 13:05:18.073410 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073308 5979 factory.go:103] Registering Raw factory Nov 03 13:05:18.073410 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073285 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2dcde035-9b90-4961-8cfc-3f37bfb67581" volumeName="kubernetes.io/secret/2dcde035-9b90-4961-8cfc-3f37bfb67581-console-oauth-config" seLinuxMountContext="" Nov 03 13:05:18.073410 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073323 5979 manager.go:1199] Started watching for new ooms in manager Nov 03 13:05:18.073410 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073321 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30d608b6-d82e-4fcf-ab44-b8e064c463d0" volumeName="kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-config" seLinuxMountContext="" Nov 03 13:05:18.073410 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073334 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7361d2a9-d089-4073-bbfb-e7978943fd88" volumeName="kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-secret-alertmanager-main-tls" seLinuxMountContext="" Nov 03 13:05:18.073410 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073342 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895" volumeName="kubernetes.io/secret/2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895-original-pull-secret" seLinuxMountContext="" Nov 03 13:05:18.073410 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073350 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7361d2a9-d089-4073-bbfb-e7978943fd88" volumeName="kubernetes.io/configmap/7361d2a9-d089-4073-bbfb-e7978943fd88-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Nov 03 13:05:18.073410 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073358 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d0c5822-ab7b-409d-a755-49e918278fa1" volumeName="kubernetes.io/configmap/0d0c5822-ab7b-409d-a755-49e918278fa1-metrics-client-ca" seLinuxMountContext="" Nov 03 13:05:18.073410 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073367 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f7bd8dd-6293-414d-8b9d-064f04678e61" volumeName="kubernetes.io/configmap/0f7bd8dd-6293-414d-8b9d-064f04678e61-ovnkube-config" seLinuxMountContext="" Nov 03 13:05:18.073410 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073374 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30d608b6-d82e-4fcf-ab44-b8e064c463d0" volumeName="kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Nov 03 13:05:18.073410 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073382 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30d608b6-d82e-4fcf-ab44-b8e064c463d0" volumeName="kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-secret-metrics-client-certs" seLinuxMountContext="" Nov 03 13:05:18.073410 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073396 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7361d2a9-d089-4073-bbfb-e7978943fd88" volumeName="kubernetes.io/projected/7361d2a9-d089-4073-bbfb-e7978943fd88-tls-assets" seLinuxMountContext="" Nov 03 13:05:18.073410 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073404 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7361d2a9-d089-4073-bbfb-e7978943fd88" volumeName="kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-config-volume" seLinuxMountContext="" Nov 03 13:05:18.073410 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073411 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f7bd8dd-6293-414d-8b9d-064f04678e61" volumeName="kubernetes.io/configmap/0f7bd8dd-6293-414d-8b9d-064f04678e61-ovnkube-script-lib" seLinuxMountContext="" Nov 03 13:05:18.073410 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073421 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fef60a4-2a84-4318-b1bf-feb8f69ef7e9" volumeName="kubernetes.io/secret/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Nov 03 13:05:18.073410 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073432 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5bfbbeb5-621b-43b0-bad1-0420e8b07b62" volumeName="kubernetes.io/secret/5bfbbeb5-621b-43b0-bad1-0420e8b07b62-client-ca-bundle" seLinuxMountContext="" Nov 03 13:05:18.073410 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073441 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7361d2a9-d089-4073-bbfb-e7978943fd88" volumeName="kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-web-config" seLinuxMountContext="" Nov 03 13:05:18.073410 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073449 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30d608b6-d82e-4fcf-ab44-b8e064c463d0" volumeName="kubernetes.io/projected/30d608b6-d82e-4fcf-ab44-b8e064c463d0-tls-assets" seLinuxMountContext="" Nov 03 13:05:18.075811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073456 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30d608b6-d82e-4fcf-ab44-b8e064c463d0" volumeName="kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-secret-prometheus-k8s-tls" seLinuxMountContext="" Nov 03 13:05:18.075811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073463 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57428473-a3f3-4a91-9a28-41619c285147" volumeName="kubernetes.io/projected/57428473-a3f3-4a91-9a28-41619c285147-kube-api-access-h6mtj" seLinuxMountContext="" Nov 03 13:05:18.075811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073490 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58a9c629-dff8-4f48-99b2-385d5d13e188" volumeName="kubernetes.io/projected/58a9c629-dff8-4f48-99b2-385d5d13e188-kube-api-access-kzl6r" seLinuxMountContext="" Nov 03 13:05:18.075811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073498 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5bfbbeb5-621b-43b0-bad1-0420e8b07b62" volumeName="kubernetes.io/empty-dir/5bfbbeb5-621b-43b0-bad1-0420e8b07b62-audit-log" seLinuxMountContext="" Nov 03 13:05:18.075811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073510 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0" volumeName="kubernetes.io/configmap/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-cni-binary-copy" seLinuxMountContext="" Nov 03 13:05:18.075811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073522 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fef60a4-2a84-4318-b1bf-feb8f69ef7e9" volumeName="kubernetes.io/secret/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9-secret-thanos-querier-tls" seLinuxMountContext="" Nov 03 13:05:18.075811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073539 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fef60a4-2a84-4318-b1bf-feb8f69ef7e9" volumeName="kubernetes.io/secret/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9-secret-grpc-tls" seLinuxMountContext="" Nov 03 13:05:18.075811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073559 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57428473-a3f3-4a91-9a28-41619c285147" volumeName="kubernetes.io/secret/57428473-a3f3-4a91-9a28-41619c285147-metrics-tls" seLinuxMountContext="" Nov 03 13:05:18.075811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073567 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fef60a4-2a84-4318-b1bf-feb8f69ef7e9" volumeName="kubernetes.io/projected/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9-kube-api-access-ls676" seLinuxMountContext="" Nov 03 13:05:18.075811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073574 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2dcde035-9b90-4961-8cfc-3f37bfb67581" volumeName="kubernetes.io/configmap/2dcde035-9b90-4961-8cfc-3f37bfb67581-oauth-serving-cert" seLinuxMountContext="" Nov 03 13:05:18.075811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073581 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7361d2a9-d089-4073-bbfb-e7978943fd88" volumeName="kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Nov 03 13:05:18.075811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073591 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f7bd8dd-6293-414d-8b9d-064f04678e61" volumeName="kubernetes.io/secret/0f7bd8dd-6293-414d-8b9d-064f04678e61-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 13:05:18.075811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073603 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30d608b6-d82e-4fcf-ab44-b8e064c463d0" volumeName="kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Nov 03 13:05:18.075811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073614 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30d608b6-d82e-4fcf-ab44-b8e064c463d0" volumeName="kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-web-config" seLinuxMountContext="" Nov 03 13:05:18.075811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073626 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58dabb02-fd21-40ee-a4f5-e990b2b9a35a" volumeName="kubernetes.io/configmap/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-service-ca" seLinuxMountContext="" Nov 03 13:05:18.075811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073639 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3083cbfe-7833-4b37-8325-cb18c20a7bea" volumeName="kubernetes.io/empty-dir/3083cbfe-7833-4b37-8325-cb18c20a7bea-tmp-dir" seLinuxMountContext="" Nov 03 13:05:18.075811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073651 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30d608b6-d82e-4fcf-ab44-b8e064c463d0" volumeName="kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Nov 03 13:05:18.075811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073662 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7868eea-4b46-49e6-afb8-7fb4bb472fca" volumeName="kubernetes.io/projected/c7868eea-4b46-49e6-afb8-7fb4bb472fca-kube-api-access-8df59" seLinuxMountContext="" Nov 03 13:05:18.075811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073673 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ec0b81c-6052-45f2-bd5b-6096ebb72223" volumeName="kubernetes.io/projected/0ec0b81c-6052-45f2-bd5b-6096ebb72223-kube-api-access-sgwqj" seLinuxMountContext="" Nov 03 13:05:18.075811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073684 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58dabb02-fd21-40ee-a4f5-e990b2b9a35a" volumeName="kubernetes.io/secret/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-console-oauth-config" seLinuxMountContext="" Nov 03 13:05:18.075811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073696 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7361d2a9-d089-4073-bbfb-e7978943fd88" volumeName="kubernetes.io/empty-dir/7361d2a9-d089-4073-bbfb-e7978943fd88-alertmanager-main-db" seLinuxMountContext="" Nov 03 13:05:18.075811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073707 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fadb078-8983-42df-afde-c1983044492d" volumeName="kubernetes.io/secret/0fadb078-8983-42df-afde-c1983044492d-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 13:05:18.075811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073719 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c663a29c-713d-4184-9105-e4a442a894c6" volumeName="kubernetes.io/projected/c663a29c-713d-4184-9105-e4a442a894c6-kube-api-access-566w9" seLinuxMountContext="" Nov 03 13:05:18.075811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073731 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0" volumeName="kubernetes.io/projected/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-kube-api-access-nd649" seLinuxMountContext="" Nov 03 13:05:18.075811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073742 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fadb078-8983-42df-afde-c1983044492d" volumeName="kubernetes.io/projected/0fadb078-8983-42df-afde-c1983044492d-kube-api-access-wgsql" seLinuxMountContext="" Nov 03 13:05:18.075811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073754 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c1f3b2f-1d7f-44b5-bcba-f9310e7bdb96" volumeName="kubernetes.io/secret/2c1f3b2f-1d7f-44b5-bcba-f9310e7bdb96-monitoring-plugin-cert" seLinuxMountContext="" Nov 03 13:05:18.075811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073768 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58dabb02-fd21-40ee-a4f5-e990b2b9a35a" volumeName="kubernetes.io/empty-dir/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-tmp" seLinuxMountContext="" Nov 03 13:05:18.075811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073780 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58dabb02-fd21-40ee-a4f5-e990b2b9a35a" volumeName="kubernetes.io/projected/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-kube-api-access-6jmgk" seLinuxMountContext="" Nov 03 13:05:18.075811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073792 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5bfbbeb5-621b-43b0-bad1-0420e8b07b62" volumeName="kubernetes.io/configmap/5bfbbeb5-621b-43b0-bad1-0420e8b07b62-metrics-server-audit-profiles" seLinuxMountContext="" Nov 03 13:05:18.075811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073804 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7361d2a9-d089-4073-bbfb-e7978943fd88" volumeName="kubernetes.io/empty-dir/7361d2a9-d089-4073-bbfb-e7978943fd88-config-out" seLinuxMountContext="" Nov 03 13:05:18.075811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073816 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a9bc0be-51b3-4a52-83ae-580c3a2f342c" volumeName="kubernetes.io/projected/8a9bc0be-51b3-4a52-83ae-580c3a2f342c-kube-api-access-xjhfh" seLinuxMountContext="" Nov 03 13:05:18.075811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073829 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c663a29c-713d-4184-9105-e4a442a894c6" volumeName="kubernetes.io/configmap/c663a29c-713d-4184-9105-e4a442a894c6-iptables-alerter-script" seLinuxMountContext="" Nov 03 13:05:18.075811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073842 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7868eea-4b46-49e6-afb8-7fb4bb472fca" volumeName="kubernetes.io/empty-dir/c7868eea-4b46-49e6-afb8-7fb4bb472fca-etc-tuned" seLinuxMountContext="" Nov 03 13:05:18.075811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073851 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d0c5822-ab7b-409d-a755-49e918278fa1" volumeName="kubernetes.io/secret/0d0c5822-ab7b-409d-a755-49e918278fa1-openshift-state-metrics-tls" seLinuxMountContext="" Nov 03 13:05:18.075811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073877 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fadb078-8983-42df-afde-c1983044492d" volumeName="kubernetes.io/empty-dir/0fadb078-8983-42df-afde-c1983044492d-node-exporter-textfile" seLinuxMountContext="" Nov 03 13:05:18.075811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073889 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30d608b6-d82e-4fcf-ab44-b8e064c463d0" volumeName="kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Nov 03 13:05:18.075811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073898 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58dabb02-fd21-40ee-a4f5-e990b2b9a35a" volumeName="kubernetes.io/configmap/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-console-config" seLinuxMountContext="" Nov 03 13:05:18.075811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073907 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7" volumeName="kubernetes.io/projected/d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7-kube-api-access-qtq2f" seLinuxMountContext="" Nov 03 13:05:18.075811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073917 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0fadb078-8983-42df-afde-c1983044492d" volumeName="kubernetes.io/secret/0fadb078-8983-42df-afde-c1983044492d-node-exporter-tls" seLinuxMountContext="" Nov 03 13:05:18.075811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073928 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30d608b6-d82e-4fcf-ab44-b8e064c463d0" volumeName="kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Nov 03 13:05:18.075811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073940 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58dabb02-fd21-40ee-a4f5-e990b2b9a35a" volumeName="kubernetes.io/secret/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-console-serving-cert" seLinuxMountContext="" Nov 03 13:05:18.075811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073948 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5bfbbeb5-621b-43b0-bad1-0420e8b07b62" volumeName="kubernetes.io/secret/5bfbbeb5-621b-43b0-bad1-0420e8b07b62-secret-metrics-server-tls" seLinuxMountContext="" Nov 03 13:05:18.075811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073955 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7361d2a9-d089-4073-bbfb-e7978943fd88" volumeName="kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Nov 03 13:05:18.075811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073963 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7361d2a9-d089-4073-bbfb-e7978943fd88" volumeName="kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Nov 03 13:05:18.075811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073971 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a83707c3-8143-4aad-9131-59986a79968b" volumeName="kubernetes.io/secret/a83707c3-8143-4aad-9131-59986a79968b-agent-certs" seLinuxMountContext="" Nov 03 13:05:18.075811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073978 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="30d608b6-d82e-4fcf-ab44-b8e064c463d0" volumeName="kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-secret-grpc-tls" seLinuxMountContext="" Nov 03 13:05:18.075811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073986 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57428473-a3f3-4a91-9a28-41619c285147" volumeName="kubernetes.io/configmap/57428473-a3f3-4a91-9a28-41619c285147-config-volume" seLinuxMountContext="" Nov 03 13:05:18.075811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.073995 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1" volumeName="kubernetes.io/configmap/06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 13:05:18.075811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.074008 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1" volumeName="kubernetes.io/projected/06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1-kube-api-access-mj6m5" seLinuxMountContext="" Nov 03 13:05:18.075811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.074059 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ec0b81c-6052-45f2-bd5b-6096ebb72223" volumeName="kubernetes.io/configmap/0ec0b81c-6052-45f2-bd5b-6096ebb72223-serviceca" seLinuxMountContext="" Nov 03 13:05:18.075811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.074074 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2dcde035-9b90-4961-8cfc-3f37bfb67581" volumeName="kubernetes.io/empty-dir/2dcde035-9b90-4961-8cfc-3f37bfb67581-tmp" seLinuxMountContext="" Nov 03 13:05:18.075811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.074086 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58dabb02-fd21-40ee-a4f5-e990b2b9a35a" volumeName="kubernetes.io/configmap/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-oauth-serving-cert" seLinuxMountContext="" Nov 03 13:05:18.075811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.074096 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62a2ee28-83a7-42bb-8018-7918027b3e18" volumeName="kubernetes.io/projected/62a2ee28-83a7-42bb-8018-7918027b3e18-kube-api-access-k2cn4" seLinuxMountContext="" Nov 03 13:05:18.075811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.074105 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62a2ee28-83a7-42bb-8018-7918027b3e18" volumeName="kubernetes.io/secret/62a2ee28-83a7-42bb-8018-7918027b3e18-cert" seLinuxMountContext="" Nov 03 13:05:18.075811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.074119 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d0c5822-ab7b-409d-a755-49e918278fa1" volumeName="kubernetes.io/secret/0d0c5822-ab7b-409d-a755-49e918278fa1-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 03 13:05:18.075811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.074129 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7361d2a9-d089-4073-bbfb-e7978943fd88" volumeName="kubernetes.io/projected/7361d2a9-d089-4073-bbfb-e7978943fd88-kube-api-access-cwz9f" seLinuxMountContext="" Nov 03 13:05:18.075811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.074139 5979 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a9bc0be-51b3-4a52-83ae-580c3a2f342c" volumeName="kubernetes.io/secret/8a9bc0be-51b3-4a52-83ae-580c3a2f342c-metrics-certs" seLinuxMountContext="" Nov 03 13:05:18.075811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.074160 5979 reconstruct.go:97] "Volume reconstruction finished" Nov 03 13:05:18.075811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.074167 5979 reconciler.go:29] "Reconciler: start to sync state" Nov 03 13:05:18.075811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.075130 5979 manager.go:319] Starting recovery of all containers Nov 03 13:05:18.104954 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.104901 5979 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 13:05:18.107283 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.107242 5979 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 13:05:18.107283 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.107273 5979 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 13:05:18.107605 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.107304 5979 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 13:05:18.107605 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:05:18.107357 5979 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 13:05:18.127419 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.127362 5979 generic.go:356] "Generic (PLEG): container finished" podID="30d608b6-d82e-4fcf-ab44-b8e064c463d0" containerID="1caee93a0441c7c697a1ab2aa46e921bc1aca35c3270332e260a2665db7555a6" exitCode=0 Nov 03 13:05:18.168415 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.168321 5979 generic.go:356] "Generic (PLEG): container finished" podID="7361d2a9-d089-4073-bbfb-e7978943fd88" containerID="aaea79a6899ef0f795f4cbb0ee4424a703035e5af1aeed5baf0609f417f64e5c" exitCode=0 Nov 03 13:05:18.168675 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:05:18.168638 5979 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-198.us-west-2.compute.internal\" not found" Nov 03 13:05:18.180555 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.180527 5979 generic.go:356] "Generic (PLEG): container finished" podID="f76315e331bcfab33ea20c406e740706" containerID="fd188fb652c965ce60014fab5c8b8f58ee6de26cfd2185c7c01b9e6e39660839" exitCode=0 Nov 03 13:05:18.188877 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.188848 5979 generic.go:356] "Generic (PLEG): container finished" podID="0fadb078-8983-42df-afde-c1983044492d" containerID="a78f039b00668ee59cd1dbce0854bed1ed8fa5479aab6f34f37c830fdbea21c2" exitCode=0 Nov 03 13:05:18.203460 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.203434 5979 generic.go:356] "Generic (PLEG): container finished" podID="06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1" containerID="4ae8b45c1c72ab262def8c944455f637b7a80a201fc6b07a9e1a8ecb06bd9541" exitCode=0 Nov 03 13:05:18.203460 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.203456 5979 generic.go:356] "Generic (PLEG): container finished" podID="06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1" containerID="a3532f7d52d5b86e95969d5e54d622b946b8fee936c164924abfd55520ccfc3d" exitCode=0 Nov 03 13:05:18.203658 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.203464 5979 generic.go:356] "Generic (PLEG): container finished" podID="06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1" containerID="ad81f969e0651c5fae5288d9b5e8b5bb83c666efb3d8a19cf99e91bb830a6605" exitCode=0 Nov 03 13:05:18.203658 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.203509 5979 generic.go:356] "Generic (PLEG): container finished" podID="06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1" containerID="ced9e3a3b5ed9dbbb36de460971869c91ea88503569f70e4147e98c99cb277f3" exitCode=0 Nov 03 13:05:18.203658 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.203515 5979 generic.go:356] "Generic (PLEG): container finished" podID="06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1" containerID="706a116018ce9b1048168c4649d08853a7aa36dbc181cf7a6861de402fce6f41" exitCode=0 Nov 03 13:05:18.203658 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.203522 5979 generic.go:356] "Generic (PLEG): container finished" podID="06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1" containerID="0f4eb43afa1d166d0ce1d07fe458aea5ffe18ce10220af393eec86b984a5783a" exitCode=0 Nov 03 13:05:18.207490 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:05:18.207451 5979 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 03 13:05:18.229699 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.229675 5979 manager.go:324] Recovery completed Nov 03 13:05:18.249422 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.249391 5979 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 13:05:18.249422 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.249391 5979 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 13:05:18.249659 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.249562 5979 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 13:05:18.249659 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.249625 5979 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 13:05:18.256535 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.256507 5979 cpu_manager.go:226] "Starting" policy="none" Nov 03 13:05:18.256535 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.256528 5979 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 13:05:18.256728 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.256554 5979 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 13:05:18.256826 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.256781 5979 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 13:05:18.257034 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.257003 5979 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 13:05:18.257068 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.257034 5979 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 13:05:18.257068 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.257040 5979 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 13:05:18.257068 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.257046 5979 policy_none.go:50] "Start" Nov 03 13:05:18.257068 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.257055 5979 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 13:05:18.257068 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.257069 5979 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 13:05:18.257227 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.257218 5979 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 13:05:18.257262 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.257228 5979 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 13:05:18.257262 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.257234 5979 policy_none.go:44] "Start" Nov 03 13:05:18.261971 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.261943 5979 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 13:05:18.264491 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.264455 5979 manager.go:341] "Starting Device Plugin manager" Nov 03 13:05:18.264945 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:05:18.264503 5979 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 13:05:18.264945 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.264943 5979 server.go:87] "Starting device plugin registration server" Nov 03 13:05:18.265214 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.265199 5979 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 13:05:18.265313 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.265212 5979 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 13:05:18.265368 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.265304 5979 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 13:05:18.265450 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.265417 5979 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 13:05:18.265529 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.265459 5979 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 13:05:18.265529 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.265466 5979 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 13:05:18.265622 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.265532 5979 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T13:05:18.265448399Z","UUID":"0f575e21-edca-4afc-bd9c-9cc3ef61aa69","Handler":null,"Name":"","Endpoint":""} Nov 03 13:05:18.267029 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.266998 5979 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 13:05:18.267131 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.267032 5979 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 13:05:18.267388 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:05:18.267358 5979 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 13:05:18.341066 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.341032 5979 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 13:05:18.365582 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.365555 5979 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 13:05:18.368846 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.368826 5979 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-198.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 13:05:18.368976 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.368854 5979 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-198.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 13:05:18.368976 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.368865 5979 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-198.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 03 13:05:18.368976 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.368912 5979 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-198.us-west-2.compute.internal" Nov 03 13:05:18.408322 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.408268 5979 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-198.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-198.us-west-2.compute.internal"] Nov 03 13:05:18.408998 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.408960 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-198.us-west-2.compute.internal" event={"ID":"f76315e331bcfab33ea20c406e740706","Type":"ContainerStarted","Data":"8aa2894f700864fbc3427299e2cfb8e67e78daa6839ac1dc5ff467281a75861c"} Nov 03 13:05:18.409100 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.409002 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-198.us-west-2.compute.internal" event={"ID":"f76315e331bcfab33ea20c406e740706","Type":"ContainerDied","Data":"fd188fb652c965ce60014fab5c8b8f58ee6de26cfd2185c7c01b9e6e39660839"} Nov 03 13:05:18.409100 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.409016 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-198.us-west-2.compute.internal" event={"ID":"f76315e331bcfab33ea20c406e740706","Type":"ContainerStarted","Data":"04419437a7c583e8767facf7b6dd8ac270e24b70dd856e7d9a1bcd0e8b71bb22"} Nov 03 13:05:18.409100 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.409060 5979 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-198.us-west-2.compute.internal" Nov 03 13:05:18.409100 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.409073 5979 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-198.us-west-2.compute.internal" Nov 03 13:05:18.409100 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.409087 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-198.us-west-2.compute.internal" event={"ID":"069487ba3512639ba31fb8a59a1ce2dd","Type":"ContainerStarted","Data":"7c197bda48c555b29f8a01d33782f981e659a0e4c91acdbcaf266d0f5c32bb86"} Nov 03 13:05:18.409100 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.409099 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-198.us-west-2.compute.internal" event={"ID":"069487ba3512639ba31fb8a59a1ce2dd","Type":"ContainerStarted","Data":"6de1e4a21fe31cf1358127b395a713134fadd5a4cf986e8b1b3e4fe8cea83e27"} Nov 03 13:05:18.468355 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.468284 5979 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 13:05:18.468355 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:05:18.468330 5979 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-198.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-198.us-west-2.compute.internal" Nov 03 13:05:18.468581 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.468562 5979 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 13:05:18.468625 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:05:18.468606 5979 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-198.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-198.us-west-2.compute.internal" Nov 03 13:05:18.469781 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.469766 5979 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-130-198.us-west-2.compute.internal" Nov 03 13:05:18.469851 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.469820 5979 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-198.us-west-2.compute.internal" Nov 03 13:05:18.551875 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.551848 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f76315e331bcfab33ea20c406e740706-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-198.us-west-2.compute.internal\" (UID: \"f76315e331bcfab33ea20c406e740706\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-198.us-west-2.compute.internal" Nov 03 13:05:18.551875 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.551878 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f76315e331bcfab33ea20c406e740706-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-198.us-west-2.compute.internal\" (UID: \"f76315e331bcfab33ea20c406e740706\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-198.us-west-2.compute.internal" Nov 03 13:05:18.552074 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.551897 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/069487ba3512639ba31fb8a59a1ce2dd-config\") pod \"kube-apiserver-proxy-ip-10-0-130-198.us-west-2.compute.internal\" (UID: \"069487ba3512639ba31fb8a59a1ce2dd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-198.us-west-2.compute.internal" Nov 03 13:05:18.652316 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.652281 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/069487ba3512639ba31fb8a59a1ce2dd-config\") pod \"kube-apiserver-proxy-ip-10-0-130-198.us-west-2.compute.internal\" (UID: \"069487ba3512639ba31fb8a59a1ce2dd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-198.us-west-2.compute.internal" Nov 03 13:05:18.652316 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.652318 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f76315e331bcfab33ea20c406e740706-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-198.us-west-2.compute.internal\" (UID: \"f76315e331bcfab33ea20c406e740706\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-198.us-west-2.compute.internal" Nov 03 13:05:18.652546 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.652343 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f76315e331bcfab33ea20c406e740706-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-198.us-west-2.compute.internal\" (UID: \"f76315e331bcfab33ea20c406e740706\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-198.us-west-2.compute.internal" Nov 03 13:05:18.652546 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.652394 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f76315e331bcfab33ea20c406e740706-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-198.us-west-2.compute.internal\" (UID: \"f76315e331bcfab33ea20c406e740706\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-198.us-west-2.compute.internal" Nov 03 13:05:18.652546 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.652395 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f76315e331bcfab33ea20c406e740706-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-198.us-west-2.compute.internal\" (UID: \"f76315e331bcfab33ea20c406e740706\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-198.us-west-2.compute.internal" Nov 03 13:05:18.652546 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:18.652394 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/069487ba3512639ba31fb8a59a1ce2dd-config\") pod \"kube-apiserver-proxy-ip-10-0-130-198.us-west-2.compute.internal\" (UID: \"069487ba3512639ba31fb8a59a1ce2dd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-198.us-west-2.compute.internal" Nov 03 13:05:19.053868 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.053835 5979 apiserver.go:52] "Watching apiserver" Nov 03 13:05:19.215016 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.214983 5979 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-198.us-west-2.compute.internal" Nov 03 13:05:19.215179 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.215065 5979 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-198.us-west-2.compute.internal" Nov 03 13:05:19.244276 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.244249 5979 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 13:05:19.244929 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.244905 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-krsxs","kube-system/kube-apiserver-proxy-ip-10-0-130-198.us-west-2.compute.internal","openshift-console/console-848bc84498-gxvtr","openshift-monitoring/metrics-server-6488577f47-wwpwm","openshift-monitoring/openshift-state-metrics-7bd7489957-fqskz","openshift-multus/multus-additional-cni-plugins-zzbqs","openshift-network-operator/iptables-alerter-999mg","openshift-network-diagnostics/network-check-target-9v5mk","openshift-console/console-6f44499665-ff6gp","openshift-insights/insights-runtime-extractor-n2nst","openshift-monitoring/monitoring-plugin-77f4f76594-cxrp7","openshift-dns/dns-default-87tkk","openshift-ingress-canary/ingress-canary-czwz4","openshift-monitoring/alertmanager-main-0","openshift-image-registry/node-ca-zrwrz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-198.us-west-2.compute.internal","openshift-multus/multus-nkdmz","openshift-ovn-kubernetes/ovnkube-node-frppp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4qjz","kube-system/konnectivity-agent-ghgsf","openshift-dns/node-resolver-zzbpd","openshift-monitoring/node-exporter-zhl5n","openshift-monitoring/prometheus-k8s-0","openshift-multus/network-metrics-daemon-dczcf","kube-system/global-pull-secret-syncer-48vd8","openshift-cluster-node-tuning-operator/tuned-k78zw","openshift-monitoring/thanos-querier-86b8856f97-b75hg"] Nov 03 13:05:19.271236 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.271204 5979 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 13:05:19.272369 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.272350 5979 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 13:05:19.272447 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:05:19.272414 5979 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-198.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-198.us-west-2.compute.internal" Nov 03 13:05:19.272678 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.272655 5979 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 13:05:19.272768 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:05:19.272705 5979 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-198.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-198.us-west-2.compute.internal" Nov 03 13:05:19.295126 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.295109 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 13:05:19.295282 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.295254 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 13:05:19.295382 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.295335 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 13:05:19.295382 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.295357 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 13:05:19.296667 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.296649 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 13:05:19.296760 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.296663 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 13:05:19.296760 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.296698 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 13:05:19.296760 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.296712 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 13:05:19.296760 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.296719 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 13:05:19.355108 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.355039 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-86b8856f97-b75hg\" (UID: \"0fef60a4-2a84-4318-b1bf-feb8f69ef7e9\") " pod="openshift-monitoring/thanos-querier-86b8856f97-b75hg" Nov 03 13:05:19.355108 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.355069 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdvh2\" (UniqueName: \"kubernetes.io/projected/30d608b6-d82e-4fcf-ab44-b8e064c463d0-kube-api-access-jdvh2\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:19.355108 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.355086 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a83707c3-8143-4aad-9131-59986a79968b-konnectivity-ca\") pod \"konnectivity-agent-ghgsf\" (UID: \"a83707c3-8143-4aad-9131-59986a79968b\") " pod="kube-system/konnectivity-agent-ghgsf" Nov 03 13:05:19.355313 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.355223 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c7868eea-4b46-49e6-afb8-7fb4bb472fca-etc-tuned\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:05:19.355313 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.355254 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-var-lib-openvswitch\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:05:19.355313 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.355272 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-host-run-ovn-kubernetes\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:05:19.355313 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.355289 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:05:19.355536 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.355346 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-system-cni-dir\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:05:19.355536 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.355504 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gb4j\" (UniqueName: \"kubernetes.io/projected/2dcde035-9b90-4961-8cfc-3f37bfb67581-kube-api-access-7gb4j\") pod \"console-6f44499665-ff6gp\" (UID: \"2dcde035-9b90-4961-8cfc-3f37bfb67581\") " pod="openshift-console/console-6f44499665-ff6gp" Nov 03 13:05:19.355813 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.355796 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c7868eea-4b46-49e6-afb8-7fb4bb472fca-etc-tuned\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:05:19.355868 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.355851 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/30d608b6-d82e-4fcf-ab44-b8e064c463d0-config-out\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:19.355946 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.355929 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-etc-kubernetes\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:05:19.355991 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.355947 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/30d608b6-d82e-4fcf-ab44-b8e064c463d0-config-out\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:19.355991 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.355958 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vdxd\" (UniqueName: \"kubernetes.io/projected/3083cbfe-7833-4b37-8325-cb18c20a7bea-kube-api-access-9vdxd\") pod \"node-resolver-zzbpd\" (UID: \"3083cbfe-7833-4b37-8325-cb18c20a7bea\") " pod="openshift-dns/node-resolver-zzbpd" Nov 03 13:05:19.356067 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.355989 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/31a398f7-8539-4c36-8386-455ef83b3d2b-data-volume\") pod \"insights-runtime-extractor-n2nst\" (UID: \"31a398f7-8539-4c36-8386-455ef83b3d2b\") " pod="openshift-insights/insights-runtime-extractor-n2nst" Nov 03 13:05:19.356120 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.356066 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7-device-dir\") pod \"aws-ebs-csi-driver-node-q4qjz\" (UID: \"d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4qjz" Nov 03 13:05:19.356120 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.356109 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd649\" (UniqueName: \"kubernetes.io/projected/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-kube-api-access-nd649\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:05:19.356221 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.356115 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/31a398f7-8539-4c36-8386-455ef83b3d2b-data-volume\") pod \"insights-runtime-extractor-n2nst\" (UID: \"31a398f7-8539-4c36-8386-455ef83b3d2b\") " pod="openshift-insights/insights-runtime-extractor-n2nst" Nov 03 13:05:19.356221 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.356203 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7361d2a9-d089-4073-bbfb-e7978943fd88-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:19.356329 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.356249 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dcde035-9b90-4961-8cfc-3f37bfb67581-trusted-ca-bundle\") pod \"console-6f44499665-ff6gp\" (UID: \"2dcde035-9b90-4961-8cfc-3f37bfb67581\") " pod="openshift-console/console-6f44499665-ff6gp" Nov 03 13:05:19.356382 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.356324 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/31a398f7-8539-4c36-8386-455ef83b3d2b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n2nst\" (UID: \"31a398f7-8539-4c36-8386-455ef83b3d2b\") " pod="openshift-insights/insights-runtime-extractor-n2nst" Nov 03 13:05:19.356382 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.356350 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-var-lib-kubelet\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:05:19.356382 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.356369 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d0c5822-ab7b-409d-a755-49e918278fa1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-fqskz\" (UID: \"0d0c5822-ab7b-409d-a755-49e918278fa1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fqskz" Nov 03 13:05:19.356538 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.356385 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0f7bd8dd-6293-414d-8b9d-064f04678e61-ovn-node-metrics-cert\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:05:19.356538 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.356427 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:19.356538 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.356528 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:19.356692 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.356553 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0fadb078-8983-42df-afde-c1983044492d-metrics-client-ca\") pod \"node-exporter-zhl5n\" (UID: \"0fadb078-8983-42df-afde-c1983044492d\") " pod="openshift-monitoring/node-exporter-zhl5n" Nov 03 13:05:19.356692 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.356573 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-run-ovn\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:05:19.356692 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.356591 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57428473-a3f3-4a91-9a28-41619c285147-metrics-tls\") pod \"dns-default-87tkk\" (UID: \"57428473-a3f3-4a91-9a28-41619c285147\") " pod="openshift-dns/dns-default-87tkk" Nov 03 13:05:19.356692 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.356606 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jmgk\" (UniqueName: \"kubernetes.io/projected/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-kube-api-access-6jmgk\") pod \"console-848bc84498-gxvtr\" (UID: \"58dabb02-fd21-40ee-a4f5-e990b2b9a35a\") " pod="openshift-console/console-848bc84498-gxvtr" Nov 03 13:05:19.356692 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.356644 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-host-var-lib-cni-bin\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:05:19.356692 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.356668 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-multus-conf-dir\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:05:19.356692 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.356688 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-web-config\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:19.357012 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.356769 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2dcde035-9b90-4961-8cfc-3f37bfb67581-console-config\") pod \"console-6f44499665-ff6gp\" (UID: \"2dcde035-9b90-4961-8cfc-3f37bfb67581\") " pod="openshift-console/console-6f44499665-ff6gp" Nov 03 13:05:19.357012 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.356812 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b6263f8d-cced-465a-b393-ef51cb902e11-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-krsxs\" (UID: \"b6263f8d-cced-465a-b393-ef51cb902e11\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-krsxs" Nov 03 13:05:19.357012 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.356836 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzl6r\" (UniqueName: \"kubernetes.io/projected/58a9c629-dff8-4f48-99b2-385d5d13e188-kube-api-access-kzl6r\") pod \"network-check-target-9v5mk\" (UID: \"58a9c629-dff8-4f48-99b2-385d5d13e188\") " pod="openshift-network-diagnostics/network-check-target-9v5mk" Nov 03 13:05:19.357012 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.356868 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgv82\" (UniqueName: \"kubernetes.io/projected/5bfbbeb5-621b-43b0-bad1-0420e8b07b62-kube-api-access-lgv82\") pod \"metrics-server-6488577f47-wwpwm\" (UID: \"5bfbbeb5-621b-43b0-bad1-0420e8b07b62\") " pod="openshift-monitoring/metrics-server-6488577f47-wwpwm" Nov 03 13:05:19.357012 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.356877 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b6263f8d-cced-465a-b393-ef51cb902e11-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-krsxs\" (UID: \"b6263f8d-cced-465a-b393-ef51cb902e11\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-krsxs" Nov 03 13:05:19.357012 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.356897 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2dcde035-9b90-4961-8cfc-3f37bfb67581-oauth-serving-cert\") pod \"console-6f44499665-ff6gp\" (UID: \"2dcde035-9b90-4961-8cfc-3f37bfb67581\") " pod="openshift-console/console-6f44499665-ff6gp" Nov 03 13:05:19.357012 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.356914 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3083cbfe-7833-4b37-8325-cb18c20a7bea-tmp-dir\") pod \"node-resolver-zzbpd\" (UID: \"3083cbfe-7833-4b37-8325-cb18c20a7bea\") " pod="openshift-dns/node-resolver-zzbpd" Nov 03 13:05:19.357012 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.356930 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2cn4\" (UniqueName: \"kubernetes.io/projected/62a2ee28-83a7-42bb-8018-7918027b3e18-kube-api-access-k2cn4\") pod \"ingress-canary-czwz4\" (UID: \"62a2ee28-83a7-42bb-8018-7918027b3e18\") " pod="openshift-ingress-canary/ingress-canary-czwz4" Nov 03 13:05:19.357331 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.357029 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3083cbfe-7833-4b37-8325-cb18c20a7bea-tmp-dir\") pod \"node-resolver-zzbpd\" (UID: \"3083cbfe-7833-4b37-8325-cb18c20a7bea\") " pod="openshift-dns/node-resolver-zzbpd" Nov 03 13:05:19.357331 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.357030 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-run-openvswitch\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:05:19.357331 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.357079 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:19.357331 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.357142 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9-secret-thanos-querier-tls\") pod \"thanos-querier-86b8856f97-b75hg\" (UID: \"0fef60a4-2a84-4318-b1bf-feb8f69ef7e9\") " pod="openshift-monitoring/thanos-querier-86b8856f97-b75hg" Nov 03 13:05:19.357331 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.357203 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-etc-systemd\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:05:19.357331 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.357235 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-host\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:05:19.357331 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.357269 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895-original-pull-secret\") pod \"global-pull-secret-syncer-48vd8\" (UID: \"2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895\") " pod="kube-system/global-pull-secret-syncer-48vd8" Nov 03 13:05:19.357331 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.357311 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7361d2a9-d089-4073-bbfb-e7978943fd88-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:19.357739 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.357344 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/62a2ee28-83a7-42bb-8018-7918027b3e18-cert\") pod \"ingress-canary-czwz4\" (UID: \"62a2ee28-83a7-42bb-8018-7918027b3e18\") " pod="openshift-ingress-canary/ingress-canary-czwz4" Nov 03 13:05:19.357739 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.357379 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk2xl\" (UniqueName: \"kubernetes.io/projected/b6263f8d-cced-465a-b393-ef51cb902e11-kube-api-access-bk2xl\") pod \"kube-state-metrics-5dc8d7d89b-krsxs\" (UID: \"b6263f8d-cced-465a-b393-ef51cb902e11\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-krsxs" Nov 03 13:05:19.357739 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.357414 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0fadb078-8983-42df-afde-c1983044492d-node-exporter-textfile\") pod \"node-exporter-zhl5n\" (UID: \"0fadb078-8983-42df-afde-c1983044492d\") " pod="openshift-monitoring/node-exporter-zhl5n" Nov 03 13:05:19.357739 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.357431 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-run-systemd\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:05:19.357739 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.357449 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-node-log\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:05:19.357739 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.357523 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6mtj\" (UniqueName: \"kubernetes.io/projected/57428473-a3f3-4a91-9a28-41619c285147-kube-api-access-h6mtj\") pod \"dns-default-87tkk\" (UID: \"57428473-a3f3-4a91-9a28-41619c285147\") " pod="openshift-dns/dns-default-87tkk" Nov 03 13:05:19.357739 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.357535 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0fadb078-8983-42df-afde-c1983044492d-node-exporter-textfile\") pod \"node-exporter-zhl5n\" (UID: \"0fadb078-8983-42df-afde-c1983044492d\") " pod="openshift-monitoring/node-exporter-zhl5n" Nov 03 13:05:19.357739 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.357551 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwz9f\" (UniqueName: \"kubernetes.io/projected/7361d2a9-d089-4073-bbfb-e7978943fd88-kube-api-access-cwz9f\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:19.357739 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.357639 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5bfbbeb5-621b-43b0-bad1-0420e8b07b62-client-ca-bundle\") pod \"metrics-server-6488577f47-wwpwm\" (UID: \"5bfbbeb5-621b-43b0-bad1-0420e8b07b62\") " pod="openshift-monitoring/metrics-server-6488577f47-wwpwm" Nov 03 13:05:19.357739 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.357667 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-etc-kubernetes\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:05:19.357739 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.357689 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-hostroot\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:05:19.358165 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.357751 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-host-run-multus-certs\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:05:19.358165 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.357844 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:19.358165 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.357894 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-86b8856f97-b75hg\" (UID: \"0fef60a4-2a84-4318-b1bf-feb8f69ef7e9\") " pod="openshift-monitoring/thanos-querier-86b8856f97-b75hg" Nov 03 13:05:19.358165 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.357918 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/30d608b6-d82e-4fcf-ab44-b8e064c463d0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:19.358165 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.357937 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3083cbfe-7833-4b37-8325-cb18c20a7bea-hosts-file\") pod \"node-resolver-zzbpd\" (UID: \"3083cbfe-7833-4b37-8325-cb18c20a7bea\") " pod="openshift-dns/node-resolver-zzbpd" Nov 03 13:05:19.358165 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.357960 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6263f8d-cced-465a-b393-ef51cb902e11-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-krsxs\" (UID: \"b6263f8d-cced-465a-b393-ef51cb902e11\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-krsxs" Nov 03 13:05:19.358165 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.357976 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7-registration-dir\") pod \"aws-ebs-csi-driver-node-q4qjz\" (UID: \"d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4qjz" Nov 03 13:05:19.358165 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.357991 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-host-kubelet\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:05:19.358165 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.358007 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/30d608b6-d82e-4fcf-ab44-b8e064c463d0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:19.358165 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.358032 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkznq\" (UniqueName: \"kubernetes.io/projected/0d0c5822-ab7b-409d-a755-49e918278fa1-kube-api-access-gkznq\") pod \"openshift-state-metrics-7bd7489957-fqskz\" (UID: \"0d0c5822-ab7b-409d-a755-49e918278fa1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fqskz" Nov 03 13:05:19.358165 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.358059 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:19.358165 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.358076 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:19.358165 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.358134 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtq2f\" (UniqueName: \"kubernetes.io/projected/d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7-kube-api-access-qtq2f\") pod \"aws-ebs-csi-driver-node-q4qjz\" (UID: \"d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4qjz" Nov 03 13:05:19.358764 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.358243 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-systemd-units\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:05:19.358764 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.358266 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0f7bd8dd-6293-414d-8b9d-064f04678e61-ovnkube-script-lib\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:05:19.358764 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.358346 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5bfbbeb5-621b-43b0-bad1-0420e8b07b62-metrics-server-audit-profiles\") pod \"metrics-server-6488577f47-wwpwm\" (UID: \"5bfbbeb5-621b-43b0-bad1-0420e8b07b62\") " pod="openshift-monitoring/metrics-server-6488577f47-wwpwm" Nov 03 13:05:19.358764 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.358379 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/31a398f7-8539-4c36-8386-455ef83b3d2b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n2nst\" (UID: \"31a398f7-8539-4c36-8386-455ef83b3d2b\") " pod="openshift-insights/insights-runtime-extractor-n2nst" Nov 03 13:05:19.358764 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.358403 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsf72\" (UniqueName: \"kubernetes.io/projected/31a398f7-8539-4c36-8386-455ef83b3d2b-kube-api-access-jsf72\") pod \"insights-runtime-extractor-n2nst\" (UID: \"31a398f7-8539-4c36-8386-455ef83b3d2b\") " pod="openshift-insights/insights-runtime-extractor-n2nst" Nov 03 13:05:19.358764 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.358502 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6263f8d-cced-465a-b393-ef51cb902e11-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-krsxs\" (UID: \"b6263f8d-cced-465a-b393-ef51cb902e11\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-krsxs" Nov 03 13:05:19.358764 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.358545 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2c1f3b2f-1d7f-44b5-bcba-f9310e7bdb96-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-cxrp7\" (UID: \"2c1f3b2f-1d7f-44b5-bcba-f9310e7bdb96\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-cxrp7" Nov 03 13:05:19.358764 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.358575 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7-etc-selinux\") pod \"aws-ebs-csi-driver-node-q4qjz\" (UID: \"d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4qjz" Nov 03 13:05:19.358764 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.358605 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c663a29c-713d-4184-9105-e4a442a894c6-host-slash\") pod \"iptables-alerter-999mg\" (UID: \"c663a29c-713d-4184-9105-e4a442a894c6\") " pod="openshift-network-operator/iptables-alerter-999mg" Nov 03 13:05:19.358764 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.358717 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-multus-socket-dir-parent\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:05:19.358764 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.358749 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2dcde035-9b90-4961-8cfc-3f37bfb67581-service-ca\") pod \"console-6f44499665-ff6gp\" (UID: \"2dcde035-9b90-4961-8cfc-3f37bfb67581\") " pod="openshift-console/console-6f44499665-ff6gp" Nov 03 13:05:19.359268 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.358823 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b6263f8d-cced-465a-b393-ef51cb902e11-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-krsxs\" (UID: \"b6263f8d-cced-465a-b393-ef51cb902e11\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-krsxs" Nov 03 13:05:19.359268 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.358910 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57428473-a3f3-4a91-9a28-41619c285147-tmp-dir\") pod \"dns-default-87tkk\" (UID: \"57428473-a3f3-4a91-9a28-41619c285147\") " pod="openshift-dns/dns-default-87tkk" Nov 03 13:05:19.359268 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.358945 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-console-serving-cert\") pod \"console-848bc84498-gxvtr\" (UID: \"58dabb02-fd21-40ee-a4f5-e990b2b9a35a\") " pod="openshift-console/console-848bc84498-gxvtr" Nov 03 13:05:19.359268 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.358972 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-console-config\") pod \"console-848bc84498-gxvtr\" (UID: \"58dabb02-fd21-40ee-a4f5-e990b2b9a35a\") " pod="openshift-console/console-848bc84498-gxvtr" Nov 03 13:05:19.359268 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.358981 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57428473-a3f3-4a91-9a28-41619c285147-tmp-dir\") pod \"dns-default-87tkk\" (UID: \"57428473-a3f3-4a91-9a28-41619c285147\") " pod="openshift-dns/dns-default-87tkk" Nov 03 13:05:19.359268 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.359006 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-host-var-lib-cni-multus\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:05:19.359268 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.359042 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9-tmp\") pod \"thanos-querier-86b8856f97-b75hg\" (UID: \"0fef60a4-2a84-4318-b1bf-feb8f69ef7e9\") " pod="openshift-monitoring/thanos-querier-86b8856f97-b75hg" Nov 03 13:05:19.359268 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.359072 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5bfbbeb5-621b-43b0-bad1-0420e8b07b62-audit-log\") pod \"metrics-server-6488577f47-wwpwm\" (UID: \"5bfbbeb5-621b-43b0-bad1-0420e8b07b62\") " pod="openshift-monitoring/metrics-server-6488577f47-wwpwm" Nov 03 13:05:19.359268 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.359097 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2dcde035-9b90-4961-8cfc-3f37bfb67581-console-serving-cert\") pod \"console-6f44499665-ff6gp\" (UID: \"2dcde035-9b90-4961-8cfc-3f37bfb67581\") " pod="openshift-console/console-6f44499665-ff6gp" Nov 03 13:05:19.359268 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.359105 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9-tmp\") pod \"thanos-querier-86b8856f97-b75hg\" (UID: \"0fef60a4-2a84-4318-b1bf-feb8f69ef7e9\") " pod="openshift-monitoring/thanos-querier-86b8856f97-b75hg" Nov 03 13:05:19.359268 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.359124 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-log-socket\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:05:19.359268 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.359158 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-multus-daemon-config\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:05:19.359268 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.359173 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5bfbbeb5-621b-43b0-bad1-0420e8b07b62-audit-log\") pod \"metrics-server-6488577f47-wwpwm\" (UID: \"5bfbbeb5-621b-43b0-bad1-0420e8b07b62\") " pod="openshift-monitoring/metrics-server-6488577f47-wwpwm" Nov 03 13:05:19.359268 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.359185 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7361d2a9-d089-4073-bbfb-e7978943fd88-config-out\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:19.359268 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.359230 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7361d2a9-d089-4073-bbfb-e7978943fd88-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:19.359268 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.359268 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7361d2a9-d089-4073-bbfb-e7978943fd88-config-out\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:19.360069 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.359268 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-86b8856f97-b75hg\" (UID: \"0fef60a4-2a84-4318-b1bf-feb8f69ef7e9\") " pod="openshift-monitoring/thanos-querier-86b8856f97-b75hg" Nov 03 13:05:19.360069 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.359310 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a83707c3-8143-4aad-9131-59986a79968b-agent-certs\") pod \"konnectivity-agent-ghgsf\" (UID: \"a83707c3-8143-4aad-9131-59986a79968b\") " pod="kube-system/konnectivity-agent-ghgsf" Nov 03 13:05:19.360069 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.359341 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b6263f8d-cced-465a-b393-ef51cb902e11-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-krsxs\" (UID: \"b6263f8d-cced-465a-b393-ef51cb902e11\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-krsxs" Nov 03 13:05:19.360069 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.359371 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8df59\" (UniqueName: \"kubernetes.io/projected/c7868eea-4b46-49e6-afb8-7fb4bb472fca-kube-api-access-8df59\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:05:19.360069 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.359427 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-566w9\" (UniqueName: \"kubernetes.io/projected/c663a29c-713d-4184-9105-e4a442a894c6-kube-api-access-566w9\") pod \"iptables-alerter-999mg\" (UID: \"c663a29c-713d-4184-9105-e4a442a894c6\") " pod="openshift-network-operator/iptables-alerter-999mg" Nov 03 13:05:19.360069 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.359465 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-cni-binary-copy\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:05:19.360069 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.359523 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0fadb078-8983-42df-afde-c1983044492d-node-exporter-wtmp\") pod \"node-exporter-zhl5n\" (UID: \"0fadb078-8983-42df-afde-c1983044492d\") " pod="openshift-monitoring/node-exporter-zhl5n" Nov 03 13:05:19.360069 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.359561 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:19.360069 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.359599 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-86b8856f97-b75hg\" (UID: \"0fef60a4-2a84-4318-b1bf-feb8f69ef7e9\") " pod="openshift-monitoring/thanos-querier-86b8856f97-b75hg" Nov 03 13:05:19.360069 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.359635 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bfbbeb5-621b-43b0-bad1-0420e8b07b62-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6488577f47-wwpwm\" (UID: \"5bfbbeb5-621b-43b0-bad1-0420e8b07b62\") " pod="openshift-monitoring/metrics-server-6488577f47-wwpwm" Nov 03 13:05:19.360069 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.359665 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:19.360069 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.359691 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0fadb078-8983-42df-afde-c1983044492d-sys\") pod \"node-exporter-zhl5n\" (UID: \"0fadb078-8983-42df-afde-c1983044492d\") " pod="openshift-monitoring/node-exporter-zhl5n" Nov 03 13:05:19.360069 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.359720 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7-socket-dir\") pod \"aws-ebs-csi-driver-node-q4qjz\" (UID: \"d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4qjz" Nov 03 13:05:19.360069 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.359744 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-cni-binary-copy\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:05:19.360069 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.359774 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-multus-cni-dir\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:05:19.360069 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.359804 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9-metrics-client-ca\") pod \"thanos-querier-86b8856f97-b75hg\" (UID: \"0fef60a4-2a84-4318-b1bf-feb8f69ef7e9\") " pod="openshift-monitoring/thanos-querier-86b8856f97-b75hg" Nov 03 13:05:19.360069 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.359836 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:19.360069 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.359862 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:19.360069 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.359889 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:19.360069 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.359925 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7-sys-fs\") pod \"aws-ebs-csi-driver-node-q4qjz\" (UID: \"d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4qjz" Nov 03 13:05:19.360069 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.359952 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0f7bd8dd-6293-414d-8b9d-064f04678e61-ovnkube-config\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:05:19.360069 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.359985 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-oauth-serving-cert\") pod \"console-848bc84498-gxvtr\" (UID: \"58dabb02-fd21-40ee-a4f5-e990b2b9a35a\") " pod="openshift-console/console-848bc84498-gxvtr" Nov 03 13:05:19.360069 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.360009 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0ec0b81c-6052-45f2-bd5b-6096ebb72223-serviceca\") pod \"node-ca-zrwrz\" (UID: \"0ec0b81c-6052-45f2-bd5b-6096ebb72223\") " pod="openshift-image-registry/node-ca-zrwrz" Nov 03 13:05:19.360069 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.360032 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-etc-sysctl-d\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:05:19.360069 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.360049 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-host-run-netns\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:05:19.360069 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.360066 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z298l\" (UniqueName: \"kubernetes.io/projected/0f7bd8dd-6293-414d-8b9d-064f04678e61-kube-api-access-z298l\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:05:19.361276 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.360089 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:19.361276 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.360205 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5bfbbeb5-621b-43b0-bad1-0420e8b07b62-secret-metrics-server-client-certs\") pod \"metrics-server-6488577f47-wwpwm\" (UID: \"5bfbbeb5-621b-43b0-bad1-0420e8b07b62\") " pod="openshift-monitoring/metrics-server-6488577f47-wwpwm" Nov 03 13:05:19.361276 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.360241 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgwqj\" (UniqueName: \"kubernetes.io/projected/0ec0b81c-6052-45f2-bd5b-6096ebb72223-kube-api-access-sgwqj\") pod \"node-ca-zrwrz\" (UID: \"0ec0b81c-6052-45f2-bd5b-6096ebb72223\") " pod="openshift-image-registry/node-ca-zrwrz" Nov 03 13:05:19.361276 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.360269 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0fadb078-8983-42df-afde-c1983044492d-root\") pod \"node-exporter-zhl5n\" (UID: \"0fadb078-8983-42df-afde-c1983044492d\") " pod="openshift-monitoring/node-exporter-zhl5n" Nov 03 13:05:19.361276 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.360302 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgsql\" (UniqueName: \"kubernetes.io/projected/0fadb078-8983-42df-afde-c1983044492d-kube-api-access-wgsql\") pod \"node-exporter-zhl5n\" (UID: \"0fadb078-8983-42df-afde-c1983044492d\") " pod="openshift-monitoring/node-exporter-zhl5n" Nov 03 13:05:19.361276 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.360342 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d0c5822-ab7b-409d-a755-49e918278fa1-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-fqskz\" (UID: \"0d0c5822-ab7b-409d-a755-49e918278fa1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fqskz" Nov 03 13:05:19.361276 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.360371 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1-cnibin\") pod \"multus-additional-cni-plugins-zzbqs\" (UID: \"06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1\") " pod="openshift-multus/multus-additional-cni-plugins-zzbqs" Nov 03 13:05:19.361276 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.360405 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c663a29c-713d-4184-9105-e4a442a894c6-iptables-alerter-script\") pod \"iptables-alerter-999mg\" (UID: \"c663a29c-713d-4184-9105-e4a442a894c6\") " pod="openshift-network-operator/iptables-alerter-999mg" Nov 03 13:05:19.361276 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.360447 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2dcde035-9b90-4961-8cfc-3f37bfb67581-tmp\") pod \"console-6f44499665-ff6gp\" (UID: \"2dcde035-9b90-4961-8cfc-3f37bfb67581\") " pod="openshift-console/console-6f44499665-ff6gp" Nov 03 13:05:19.361276 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.360512 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-config\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:19.361276 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.360525 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2dcde035-9b90-4961-8cfc-3f37bfb67581-tmp\") pod \"console-6f44499665-ff6gp\" (UID: \"2dcde035-9b90-4961-8cfc-3f37bfb67581\") " pod="openshift-console/console-6f44499665-ff6gp" Nov 03 13:05:19.361276 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.360531 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0fadb078-8983-42df-afde-c1983044492d-node-exporter-tls\") pod \"node-exporter-zhl5n\" (UID: \"0fadb078-8983-42df-afde-c1983044492d\") " pod="openshift-monitoring/node-exporter-zhl5n" Nov 03 13:05:19.361276 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.360552 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-run\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:05:19.361276 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.360590 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1-cni-binary-copy\") pod \"multus-additional-cni-plugins-zzbqs\" (UID: \"06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1\") " pod="openshift-multus/multus-additional-cni-plugins-zzbqs" Nov 03 13:05:19.361276 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.360608 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj6m5\" (UniqueName: \"kubernetes.io/projected/06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1-kube-api-access-mj6m5\") pod \"multus-additional-cni-plugins-zzbqs\" (UID: \"06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1\") " pod="openshift-multus/multus-additional-cni-plugins-zzbqs" Nov 03 13:05:19.361276 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.360646 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-host-cni-bin\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:05:19.361276 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.360784 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-tmp\") pod \"console-848bc84498-gxvtr\" (UID: \"58dabb02-fd21-40ee-a4f5-e990b2b9a35a\") " pod="openshift-console/console-848bc84498-gxvtr" Nov 03 13:05:19.361276 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.360807 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5bfbbeb5-621b-43b0-bad1-0420e8b07b62-secret-metrics-server-tls\") pod \"metrics-server-6488577f47-wwpwm\" (UID: \"5bfbbeb5-621b-43b0-bad1-0420e8b07b62\") " pod="openshift-monitoring/metrics-server-6488577f47-wwpwm" Nov 03 13:05:19.361276 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.360825 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7361d2a9-d089-4073-bbfb-e7978943fd88-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:19.361276 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.360829 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-tmp\") pod \"console-848bc84498-gxvtr\" (UID: \"58dabb02-fd21-40ee-a4f5-e990b2b9a35a\") " pod="openshift-console/console-848bc84498-gxvtr" Nov 03 13:05:19.361276 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.360841 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:19.361276 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.360896 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7361d2a9-d089-4073-bbfb-e7978943fd88-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:19.361276 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.360968 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:19.361276 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.360992 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0fadb078-8983-42df-afde-c1983044492d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zhl5n\" (UID: \"0fadb078-8983-42df-afde-c1983044492d\") " pod="openshift-monitoring/node-exporter-zhl5n" Nov 03 13:05:19.361276 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.361024 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-lib-modules\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:05:19.361276 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.361054 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-cnibin\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:05:19.361276 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.361105 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9-secret-grpc-tls\") pod \"thanos-querier-86b8856f97-b75hg\" (UID: \"0fef60a4-2a84-4318-b1bf-feb8f69ef7e9\") " pod="openshift-monitoring/thanos-querier-86b8856f97-b75hg" Nov 03 13:05:19.361276 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.361152 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zzbqs\" (UID: \"06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1\") " pod="openshift-multus/multus-additional-cni-plugins-zzbqs" Nov 03 13:05:19.361276 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.361193 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-host-var-lib-kubelet\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:05:19.361276 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.361216 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1-cni-binary-copy\") pod \"multus-additional-cni-plugins-zzbqs\" (UID: \"06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1\") " pod="openshift-multus/multus-additional-cni-plugins-zzbqs" Nov 03 13:05:19.361276 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.361230 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-etc-sysctl-conf\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:05:19.362556 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.361298 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c7868eea-4b46-49e6-afb8-7fb4bb472fca-tmp\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:05:19.362556 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.361322 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2dcde035-9b90-4961-8cfc-3f37bfb67581-console-oauth-config\") pod \"console-6f44499665-ff6gp\" (UID: \"2dcde035-9b90-4961-8cfc-3f37bfb67581\") " pod="openshift-console/console-6f44499665-ff6gp" Nov 03 13:05:19.362556 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.361340 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:19.362556 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.361362 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0ec0b81c-6052-45f2-bd5b-6096ebb72223-host\") pod \"node-ca-zrwrz\" (UID: \"0ec0b81c-6052-45f2-bd5b-6096ebb72223\") " pod="openshift-image-registry/node-ca-zrwrz" Nov 03 13:05:19.362556 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.361395 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/31a398f7-8539-4c36-8386-455ef83b3d2b-crio-socket\") pod \"insights-runtime-extractor-n2nst\" (UID: \"31a398f7-8539-4c36-8386-455ef83b3d2b\") " pod="openshift-insights/insights-runtime-extractor-n2nst" Nov 03 13:05:19.362556 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.361428 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-etc-sysconfig\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:05:19.362556 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.361454 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-host-cni-netd\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:05:19.362556 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.361516 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:19.362556 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.361542 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-etc-modprobe-d\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:05:19.362556 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.361558 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a9bc0be-51b3-4a52-83ae-580c3a2f342c-metrics-certs\") pod \"network-metrics-daemon-dczcf\" (UID: \"8a9bc0be-51b3-4a52-83ae-580c3a2f342c\") " pod="openshift-multus/network-metrics-daemon-dczcf" Nov 03 13:05:19.362556 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.361601 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjhfh\" (UniqueName: \"kubernetes.io/projected/8a9bc0be-51b3-4a52-83ae-580c3a2f342c-kube-api-access-xjhfh\") pod \"network-metrics-daemon-dczcf\" (UID: \"8a9bc0be-51b3-4a52-83ae-580c3a2f342c\") " pod="openshift-multus/network-metrics-daemon-dczcf" Nov 03 13:05:19.362556 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.361606 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c7868eea-4b46-49e6-afb8-7fb4bb472fca-tmp\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:05:19.362556 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.361648 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zzbqs\" (UID: \"06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1\") " pod="openshift-multus/multus-additional-cni-plugins-zzbqs" Nov 03 13:05:19.362556 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.361705 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-host-slash\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:05:19.362556 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.361746 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0f7bd8dd-6293-414d-8b9d-064f04678e61-env-overrides\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:05:19.362556 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.361780 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-config-volume\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:19.362556 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.361807 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:19.362556 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.361832 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-web-config\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:19.362556 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.361857 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0fadb078-8983-42df-afde-c1983044492d-node-exporter-accelerators-collector-config\") pod \"node-exporter-zhl5n\" (UID: \"0fadb078-8983-42df-afde-c1983044492d\") " pod="openshift-monitoring/node-exporter-zhl5n" Nov 03 13:05:19.362556 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.361877 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7bmn\" (UniqueName: \"kubernetes.io/projected/2c1f3b2f-1d7f-44b5-bcba-f9310e7bdb96-kube-api-access-z7bmn\") pod \"monitoring-plugin-77f4f76594-cxrp7\" (UID: \"2c1f3b2f-1d7f-44b5-bcba-f9310e7bdb96\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-cxrp7" Nov 03 13:05:19.362556 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.361904 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/57428473-a3f3-4a91-9a28-41619c285147-config-volume\") pod \"dns-default-87tkk\" (UID: \"57428473-a3f3-4a91-9a28-41619c285147\") " pod="openshift-dns/dns-default-87tkk" Nov 03 13:05:19.362556 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.361932 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-console-oauth-config\") pod \"console-848bc84498-gxvtr\" (UID: \"58dabb02-fd21-40ee-a4f5-e990b2b9a35a\") " pod="openshift-console/console-848bc84498-gxvtr" Nov 03 13:05:19.362556 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.362018 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:19.362556 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.362047 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls676\" (UniqueName: \"kubernetes.io/projected/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9-kube-api-access-ls676\") pod \"thanos-querier-86b8856f97-b75hg\" (UID: \"0fef60a4-2a84-4318-b1bf-feb8f69ef7e9\") " pod="openshift-monitoring/thanos-querier-86b8856f97-b75hg" Nov 03 13:05:19.362556 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.362065 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:19.362556 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.362087 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q4qjz\" (UID: \"d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4qjz" Nov 03 13:05:19.362556 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.362115 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-host-run-netns\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:05:19.362556 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.362163 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/30d608b6-d82e-4fcf-ab44-b8e064c463d0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:19.362556 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.362178 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-sys\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:05:19.362556 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.362209 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0d0c5822-ab7b-409d-a755-49e918278fa1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-fqskz\" (UID: \"0d0c5822-ab7b-409d-a755-49e918278fa1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fqskz" Nov 03 13:05:19.362556 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.362285 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895-dbus\") pod \"global-pull-secret-syncer-48vd8\" (UID: \"2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895\") " pod="kube-system/global-pull-secret-syncer-48vd8" Nov 03 13:05:19.362556 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.362319 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1-system-cni-dir\") pod \"multus-additional-cni-plugins-zzbqs\" (UID: \"06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1\") " pod="openshift-multus/multus-additional-cni-plugins-zzbqs" Nov 03 13:05:19.362556 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.362355 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zzbqs\" (UID: \"06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1\") " pod="openshift-multus/multus-additional-cni-plugins-zzbqs" Nov 03 13:05:19.362556 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.362415 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-service-ca\") pod \"console-848bc84498-gxvtr\" (UID: \"58dabb02-fd21-40ee-a4f5-e990b2b9a35a\") " pod="openshift-console/console-848bc84498-gxvtr" Nov 03 13:05:19.362556 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.362444 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-host-run-k8s-cni-cncf-io\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:05:19.362556 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.362491 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895-kubelet-config\") pod \"global-pull-secret-syncer-48vd8\" (UID: \"2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895\") " pod="kube-system/global-pull-secret-syncer-48vd8" Nov 03 13:05:19.362556 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.362520 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-etc-openvswitch\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:05:19.387557 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.387535 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 03 13:05:19.387557 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.387535 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 03 13:05:19.387750 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.387646 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Nov 03 13:05:19.387750 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.387712 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 03 13:05:19.387838 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.387760 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 03 13:05:19.387838 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.387764 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 03 13:05:19.388202 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.388175 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 03 13:05:19.388278 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.388262 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 03 13:05:19.388506 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.388351 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:19.388506 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.388414 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pvm6f\"" Nov 03 13:05:19.389653 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.388398 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 13:05:19.389653 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.389237 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 13:05:19.389653 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.389284 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 13:05:19.389653 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.389332 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 13:05:19.389653 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.389370 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-x4nbm\"" Nov 03 13:05:19.389653 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.389601 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b6263f8d-cced-465a-b393-ef51cb902e11-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-krsxs\" (UID: \"b6263f8d-cced-465a-b393-ef51cb902e11\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-krsxs" Nov 03 13:05:19.389999 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.389626 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-86b8856f97-b75hg\" (UID: \"0fef60a4-2a84-4318-b1bf-feb8f69ef7e9\") " pod="openshift-monitoring/thanos-querier-86b8856f97-b75hg" Nov 03 13:05:19.389999 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.389925 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-86b8856f97-b75hg\" (UID: \"0fef60a4-2a84-4318-b1bf-feb8f69ef7e9\") " pod="openshift-monitoring/thanos-querier-86b8856f97-b75hg" Nov 03 13:05:19.389999 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.389947 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:19.391117 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.391097 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-m8wc5\"" Nov 03 13:05:19.391202 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.391164 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 13:05:19.391447 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.391332 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 03 13:05:19.391447 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.391379 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 03 13:05:19.392145 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.392124 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-tvxrk\"" Nov 03 13:05:19.392253 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.392239 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 03 13:05:19.392328 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.392317 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-swq8s\"" Nov 03 13:05:19.392411 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.392399 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 03 13:05:19.392520 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.392508 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 03 13:05:19.392577 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.392554 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 03 13:05:19.392634 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.392623 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 03 13:05:19.392730 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.392719 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 03 13:05:19.392806 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.392795 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 03 13:05:19.392858 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.392837 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 03 13:05:19.392912 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.392901 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-jjf7n\"" Nov 03 13:05:19.393048 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.393035 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-9ciao1qbg16k6\"" Nov 03 13:05:19.393147 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.393136 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 03 13:05:19.393221 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.393210 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-8vx8q\"" Nov 03 13:05:19.393321 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.393309 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 03 13:05:19.393459 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.393442 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/30d608b6-d82e-4fcf-ab44-b8e064c463d0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:19.393553 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.393537 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 03 13:05:19.393820 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.393798 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-798zc\"" Nov 03 13:05:19.393891 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.393798 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9-secret-grpc-tls\") pod \"thanos-querier-86b8856f97-b75hg\" (UID: \"0fef60a4-2a84-4318-b1bf-feb8f69ef7e9\") " pod="openshift-monitoring/thanos-querier-86b8856f97-b75hg" Nov 03 13:05:19.393891 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.393831 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 03 13:05:19.393891 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.393869 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 03 13:05:19.394049 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.393883 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0d0c5822-ab7b-409d-a755-49e918278fa1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-fqskz\" (UID: \"0d0c5822-ab7b-409d-a755-49e918278fa1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fqskz" Nov 03 13:05:19.394049 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.393924 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 03 13:05:19.394049 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.393932 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-7adkcnd7vtot1\"" Nov 03 13:05:19.394049 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.394045 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-qqlft\"" Nov 03 13:05:19.394245 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.394164 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:19.394245 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.394188 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-s6pph\"" Nov 03 13:05:19.394245 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.394199 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 13:05:19.394245 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.394233 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-config-volume\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:19.394435 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.394253 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9pvzn\"" Nov 03 13:05:19.394435 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.394266 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 13:05:19.394435 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.394280 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-tqrl8\"" Nov 03 13:05:19.394435 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.394295 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 03 13:05:19.394435 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.394256 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-cgshq\"" Nov 03 13:05:19.394435 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.394348 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Nov 03 13:05:19.394435 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.394357 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 13:05:19.394435 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.394361 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 03 13:05:19.394435 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.394378 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5rlch\"" Nov 03 13:05:19.394435 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.394386 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-9jvp2\"" Nov 03 13:05:19.394435 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.394193 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lbs9x\"" Nov 03 13:05:19.394435 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.394397 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 03 13:05:19.394435 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.394424 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 13:05:19.395092 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.394461 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 13:05:19.395092 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.394488 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 03 13:05:19.395092 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.394503 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-l9fl5\"" Nov 03 13:05:19.395092 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.394555 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 03 13:05:19.395092 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.394579 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 03 13:05:19.395092 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.394596 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4ia0ld9dido9e\"" Nov 03 13:05:19.395092 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.394605 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-8v28q\"" Nov 03 13:05:19.395092 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.394646 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-5ntlc\"" Nov 03 13:05:19.395092 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.394645 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-85kzz\"" Nov 03 13:05:19.395092 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.394696 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mnhgg\"" Nov 03 13:05:19.395092 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.394738 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-g4lr6\"" Nov 03 13:05:19.395092 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.394735 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a83707c3-8143-4aad-9131-59986a79968b-agent-certs\") pod \"konnectivity-agent-ghgsf\" (UID: \"a83707c3-8143-4aad-9131-59986a79968b\") " pod="kube-system/konnectivity-agent-ghgsf" Nov 03 13:05:19.395092 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.394841 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5wxnl\"" Nov 03 13:05:19.395092 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.394891 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:19.395092 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.394908 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-sctm8\"" Nov 03 13:05:19.395756 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.395675 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-86b8856f97-b75hg\" (UID: \"0fef60a4-2a84-4318-b1bf-feb8f69ef7e9\") " pod="openshift-monitoring/thanos-querier-86b8856f97-b75hg" Nov 03 13:05:19.395756 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.395705 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 13:05:19.395756 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.395730 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 13:05:19.395756 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.395735 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 13:05:19.396431 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.396415 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7361d2a9-d089-4073-bbfb-e7978943fd88-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:19.397000 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.396985 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:19.397053 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.397016 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0d0c5822-ab7b-409d-a755-49e918278fa1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-fqskz\" (UID: \"0d0c5822-ab7b-409d-a755-49e918278fa1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fqskz" Nov 03 13:05:19.397053 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.397026 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-web-config\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:19.397053 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.397018 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/57428473-a3f3-4a91-9a28-41619c285147-metrics-tls\") pod \"dns-default-87tkk\" (UID: \"57428473-a3f3-4a91-9a28-41619c285147\") " pod="openshift-dns/dns-default-87tkk" Nov 03 13:05:19.397053 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.397048 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0f7bd8dd-6293-414d-8b9d-064f04678e61-ovn-node-metrics-cert\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:05:19.398166 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.398144 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:19.398458 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.398248 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-86b8856f97-b75hg\" (UID: \"0fef60a4-2a84-4318-b1bf-feb8f69ef7e9\") " pod="openshift-monitoring/thanos-querier-86b8856f97-b75hg" Nov 03 13:05:19.398458 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.398255 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5bfbbeb5-621b-43b0-bad1-0420e8b07b62-client-ca-bundle\") pod \"metrics-server-6488577f47-wwpwm\" (UID: \"5bfbbeb5-621b-43b0-bad1-0420e8b07b62\") " pod="openshift-monitoring/metrics-server-6488577f47-wwpwm" Nov 03 13:05:19.398458 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.398268 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/62a2ee28-83a7-42bb-8018-7918027b3e18-cert\") pod \"ingress-canary-czwz4\" (UID: \"62a2ee28-83a7-42bb-8018-7918027b3e18\") " pod="openshift-ingress-canary/ingress-canary-czwz4" Nov 03 13:05:19.398458 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.398285 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:19.398458 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.398315 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9-secret-thanos-querier-tls\") pod \"thanos-querier-86b8856f97-b75hg\" (UID: \"0fef60a4-2a84-4318-b1bf-feb8f69ef7e9\") " pod="openshift-monitoring/thanos-querier-86b8856f97-b75hg" Nov 03 13:05:19.398458 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.398368 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895-original-pull-secret\") pod \"global-pull-secret-syncer-48vd8\" (UID: \"2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895\") " pod="kube-system/global-pull-secret-syncer-48vd8" Nov 03 13:05:19.398458 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.398411 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b6263f8d-cced-465a-b393-ef51cb902e11-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-krsxs\" (UID: \"b6263f8d-cced-465a-b393-ef51cb902e11\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-krsxs" Nov 03 13:05:19.398458 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.398432 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:19.399647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.399627 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/31a398f7-8539-4c36-8386-455ef83b3d2b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n2nst\" (UID: \"31a398f7-8539-4c36-8386-455ef83b3d2b\") " pod="openshift-insights/insights-runtime-extractor-n2nst" Nov 03 13:05:19.399647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.399643 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2c1f3b2f-1d7f-44b5-bcba-f9310e7bdb96-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-cxrp7\" (UID: \"2c1f3b2f-1d7f-44b5-bcba-f9310e7bdb96\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-cxrp7" Nov 03 13:05:19.399779 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.399626 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-console-serving-cert\") pod \"console-848bc84498-gxvtr\" (UID: \"58dabb02-fd21-40ee-a4f5-e990b2b9a35a\") " pod="openshift-console/console-848bc84498-gxvtr" Nov 03 13:05:19.399779 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.399643 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2dcde035-9b90-4961-8cfc-3f37bfb67581-console-serving-cert\") pod \"console-6f44499665-ff6gp\" (UID: \"2dcde035-9b90-4961-8cfc-3f37bfb67581\") " pod="openshift-console/console-6f44499665-ff6gp" Nov 03 13:05:19.400769 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.400751 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5bfbbeb5-621b-43b0-bad1-0420e8b07b62-secret-metrics-server-client-certs\") pod \"metrics-server-6488577f47-wwpwm\" (UID: \"5bfbbeb5-621b-43b0-bad1-0420e8b07b62\") " pod="openshift-monitoring/metrics-server-6488577f47-wwpwm" Nov 03 13:05:19.400814 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.400768 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:19.402007 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.401986 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0fadb078-8983-42df-afde-c1983044492d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zhl5n\" (UID: \"0fadb078-8983-42df-afde-c1983044492d\") " pod="openshift-monitoring/node-exporter-zhl5n" Nov 03 13:05:19.402097 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.402003 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2dcde035-9b90-4961-8cfc-3f37bfb67581-console-oauth-config\") pod \"console-6f44499665-ff6gp\" (UID: \"2dcde035-9b90-4961-8cfc-3f37bfb67581\") " pod="openshift-console/console-6f44499665-ff6gp" Nov 03 13:05:19.402097 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.402006 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0fadb078-8983-42df-afde-c1983044492d-node-exporter-tls\") pod \"node-exporter-zhl5n\" (UID: \"0fadb078-8983-42df-afde-c1983044492d\") " pod="openshift-monitoring/node-exporter-zhl5n" Nov 03 13:05:19.402097 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.402043 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a9bc0be-51b3-4a52-83ae-580c3a2f342c-metrics-certs\") pod \"network-metrics-daemon-dczcf\" (UID: \"8a9bc0be-51b3-4a52-83ae-580c3a2f342c\") " pod="openshift-multus/network-metrics-daemon-dczcf" Nov 03 13:05:19.402097 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.402060 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-config\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:19.402097 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.402044 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:19.402097 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.402082 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5bfbbeb5-621b-43b0-bad1-0420e8b07b62-secret-metrics-server-tls\") pod \"metrics-server-6488577f47-wwpwm\" (UID: \"5bfbbeb5-621b-43b0-bad1-0420e8b07b62\") " pod="openshift-monitoring/metrics-server-6488577f47-wwpwm" Nov 03 13:05:19.402306 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.402130 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-console-oauth-config\") pod \"console-848bc84498-gxvtr\" (UID: \"58dabb02-fd21-40ee-a4f5-e990b2b9a35a\") " pod="openshift-console/console-848bc84498-gxvtr" Nov 03 13:05:19.402306 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.402146 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-web-config\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:19.402306 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.402189 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/57428473-a3f3-4a91-9a28-41619c285147-config-volume\") pod \"dns-default-87tkk\" (UID: \"57428473-a3f3-4a91-9a28-41619c285147\") " pod="openshift-dns/dns-default-87tkk" Nov 03 13:05:19.402306 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.402236 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:19.402306 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.402246 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:19.433749 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.433726 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 13:05:19.433846 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.433773 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 13:05:19.434136 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.434062 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 13:05:19.434136 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.434069 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 13:05:19.434136 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.434100 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 13:05:19.434136 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.434108 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 13:05:19.434317 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.434150 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 13:05:19.434414 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.434378 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 03 13:05:19.434414 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.434396 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 03 13:05:19.434414 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.434395 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 03 13:05:19.434605 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.434416 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 03 13:05:19.434605 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.434428 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 03 13:05:19.434605 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.434381 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 13:05:19.434605 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.434432 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 03 13:05:19.434605 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.434519 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 13:05:19.434605 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.434541 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 13:05:19.434605 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.434564 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 13:05:19.434605 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.434595 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 03 13:05:19.434993 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.434792 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 13:05:19.434993 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.434819 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 13:05:19.434993 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.434819 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 13:05:19.434993 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.434823 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 03 13:05:19.434993 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.434837 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 13:05:19.434993 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.434891 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 13:05:19.435158 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.435047 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 13:05:19.435238 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.435222 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 13:05:19.435284 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.435250 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 13:05:19.435284 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.435261 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 03 13:05:19.435380 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.435346 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 13:05:19.435380 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.435360 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a83707c3-8143-4aad-9131-59986a79968b-konnectivity-ca\") pod \"konnectivity-agent-ghgsf\" (UID: \"a83707c3-8143-4aad-9131-59986a79968b\") " pod="kube-system/konnectivity-agent-ghgsf" Nov 03 13:05:19.435556 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.435543 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 03 13:05:19.435556 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.435549 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 13:05:19.435672 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.435660 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Nov 03 13:05:19.435745 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.435730 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 13:05:19.435816 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.435804 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 13:05:19.435896 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.435876 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 13:05:19.437253 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.437237 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2dcde035-9b90-4961-8cfc-3f37bfb67581-console-config\") pod \"console-6f44499665-ff6gp\" (UID: \"2dcde035-9b90-4961-8cfc-3f37bfb67581\") " pod="openshift-console/console-6f44499665-ff6gp" Nov 03 13:05:19.437253 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.437244 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2dcde035-9b90-4961-8cfc-3f37bfb67581-oauth-serving-cert\") pod \"console-6f44499665-ff6gp\" (UID: \"2dcde035-9b90-4961-8cfc-3f37bfb67581\") " pod="openshift-console/console-6f44499665-ff6gp" Nov 03 13:05:19.437360 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.437266 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:19.437360 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.437274 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0fadb078-8983-42df-afde-c1983044492d-metrics-client-ca\") pod \"node-exporter-zhl5n\" (UID: \"0fadb078-8983-42df-afde-c1983044492d\") " pod="openshift-monitoring/node-exporter-zhl5n" Nov 03 13:05:19.437360 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.437298 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/31a398f7-8539-4c36-8386-455ef83b3d2b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n2nst\" (UID: \"31a398f7-8539-4c36-8386-455ef83b3d2b\") " pod="openshift-insights/insights-runtime-extractor-n2nst" Nov 03 13:05:19.437702 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.437636 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7361d2a9-d089-4073-bbfb-e7978943fd88-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:19.438241 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.438222 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd649\" (UniqueName: \"kubernetes.io/projected/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-kube-api-access-nd649\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:05:19.438379 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.438356 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgsql\" (UniqueName: \"kubernetes.io/projected/0fadb078-8983-42df-afde-c1983044492d-kube-api-access-wgsql\") pod \"node-exporter-zhl5n\" (UID: \"0fadb078-8983-42df-afde-c1983044492d\") " pod="openshift-monitoring/node-exporter-zhl5n" Nov 03 13:05:19.438379 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.438371 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-566w9\" (UniqueName: \"kubernetes.io/projected/c663a29c-713d-4184-9105-e4a442a894c6-kube-api-access-566w9\") pod \"iptables-alerter-999mg\" (UID: \"c663a29c-713d-4184-9105-e4a442a894c6\") " pod="openshift-network-operator/iptables-alerter-999mg" Nov 03 13:05:19.438533 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.438377 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6mtj\" (UniqueName: \"kubernetes.io/projected/57428473-a3f3-4a91-9a28-41619c285147-kube-api-access-h6mtj\") pod \"dns-default-87tkk\" (UID: \"57428473-a3f3-4a91-9a28-41619c285147\") " pod="openshift-dns/dns-default-87tkk" Nov 03 13:05:19.438533 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.438415 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdvh2\" (UniqueName: \"kubernetes.io/projected/30d608b6-d82e-4fcf-ab44-b8e064c463d0-kube-api-access-jdvh2\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:19.438647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.438551 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gb4j\" (UniqueName: \"kubernetes.io/projected/2dcde035-9b90-4961-8cfc-3f37bfb67581-kube-api-access-7gb4j\") pod \"console-6f44499665-ff6gp\" (UID: \"2dcde035-9b90-4961-8cfc-3f37bfb67581\") " pod="openshift-console/console-6f44499665-ff6gp" Nov 03 13:05:19.438647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.438628 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0f7bd8dd-6293-414d-8b9d-064f04678e61-ovnkube-script-lib\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:05:19.438814 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.438792 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7bmn\" (UniqueName: \"kubernetes.io/projected/2c1f3b2f-1d7f-44b5-bcba-f9310e7bdb96-kube-api-access-z7bmn\") pod \"monitoring-plugin-77f4f76594-cxrp7\" (UID: \"2c1f3b2f-1d7f-44b5-bcba-f9310e7bdb96\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-cxrp7" Nov 03 13:05:19.438883 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.438810 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6263f8d-cced-465a-b393-ef51cb902e11-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-krsxs\" (UID: \"b6263f8d-cced-465a-b393-ef51cb902e11\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-krsxs" Nov 03 13:05:19.438945 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.438886 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjhfh\" (UniqueName: \"kubernetes.io/projected/8a9bc0be-51b3-4a52-83ae-580c3a2f342c-kube-api-access-xjhfh\") pod \"network-metrics-daemon-dczcf\" (UID: \"8a9bc0be-51b3-4a52-83ae-580c3a2f342c\") " pod="openshift-multus/network-metrics-daemon-dczcf" Nov 03 13:05:19.438945 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.438925 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzl6r\" (UniqueName: \"kubernetes.io/projected/58a9c629-dff8-4f48-99b2-385d5d13e188-kube-api-access-kzl6r\") pod \"network-check-target-9v5mk\" (UID: \"58a9c629-dff8-4f48-99b2-385d5d13e188\") " pod="openshift-network-diagnostics/network-check-target-9v5mk" Nov 03 13:05:19.439044 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.439013 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5bfbbeb5-621b-43b0-bad1-0420e8b07b62-metrics-server-audit-profiles\") pod \"metrics-server-6488577f47-wwpwm\" (UID: \"5bfbbeb5-621b-43b0-bad1-0420e8b07b62\") " pod="openshift-monitoring/metrics-server-6488577f47-wwpwm" Nov 03 13:05:19.439147 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.439126 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2dcde035-9b90-4961-8cfc-3f37bfb67581-service-ca\") pod \"console-6f44499665-ff6gp\" (UID: \"2dcde035-9b90-4961-8cfc-3f37bfb67581\") " pod="openshift-console/console-6f44499665-ff6gp" Nov 03 13:05:19.439408 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.439236 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgv82\" (UniqueName: \"kubernetes.io/projected/5bfbbeb5-621b-43b0-bad1-0420e8b07b62-kube-api-access-lgv82\") pod \"metrics-server-6488577f47-wwpwm\" (UID: \"5bfbbeb5-621b-43b0-bad1-0420e8b07b62\") " pod="openshift-monitoring/metrics-server-6488577f47-wwpwm" Nov 03 13:05:19.439529 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.439420 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwz9f\" (UniqueName: \"kubernetes.io/projected/7361d2a9-d089-4073-bbfb-e7978943fd88-kube-api-access-cwz9f\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:19.439628 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.439610 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk2xl\" (UniqueName: \"kubernetes.io/projected/b6263f8d-cced-465a-b393-ef51cb902e11-kube-api-access-bk2xl\") pod \"kube-state-metrics-5dc8d7d89b-krsxs\" (UID: \"b6263f8d-cced-465a-b393-ef51cb902e11\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-krsxs" Nov 03 13:05:19.439712 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.439674 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls676\" (UniqueName: \"kubernetes.io/projected/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9-kube-api-access-ls676\") pod \"thanos-querier-86b8856f97-b75hg\" (UID: \"0fef60a4-2a84-4318-b1bf-feb8f69ef7e9\") " pod="openshift-monitoring/thanos-querier-86b8856f97-b75hg" Nov 03 13:05:19.439826 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.439772 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jmgk\" (UniqueName: \"kubernetes.io/projected/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-kube-api-access-6jmgk\") pod \"console-848bc84498-gxvtr\" (UID: \"58dabb02-fd21-40ee-a4f5-e990b2b9a35a\") " pod="openshift-console/console-848bc84498-gxvtr" Nov 03 13:05:19.439942 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.439876 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj6m5\" (UniqueName: \"kubernetes.io/projected/06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1-kube-api-access-mj6m5\") pod \"multus-additional-cni-plugins-zzbqs\" (UID: \"06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1\") " pod="openshift-multus/multus-additional-cni-plugins-zzbqs" Nov 03 13:05:19.439942 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.439899 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsf72\" (UniqueName: \"kubernetes.io/projected/31a398f7-8539-4c36-8386-455ef83b3d2b-kube-api-access-jsf72\") pod \"insights-runtime-extractor-n2nst\" (UID: \"31a398f7-8539-4c36-8386-455ef83b3d2b\") " pod="openshift-insights/insights-runtime-extractor-n2nst" Nov 03 13:05:19.440103 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.439975 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vdxd\" (UniqueName: \"kubernetes.io/projected/3083cbfe-7833-4b37-8325-cb18c20a7bea-kube-api-access-9vdxd\") pod \"node-resolver-zzbpd\" (UID: \"3083cbfe-7833-4b37-8325-cb18c20a7bea\") " pod="openshift-dns/node-resolver-zzbpd" Nov 03 13:05:19.440103 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.439989 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgwqj\" (UniqueName: \"kubernetes.io/projected/0ec0b81c-6052-45f2-bd5b-6096ebb72223-kube-api-access-sgwqj\") pod \"node-ca-zrwrz\" (UID: \"0ec0b81c-6052-45f2-bd5b-6096ebb72223\") " pod="openshift-image-registry/node-ca-zrwrz" Nov 03 13:05:19.440337 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.440162 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z298l\" (UniqueName: \"kubernetes.io/projected/0f7bd8dd-6293-414d-8b9d-064f04678e61-kube-api-access-z298l\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:05:19.440337 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.440158 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0fef60a4-2a84-4318-b1bf-feb8f69ef7e9-metrics-client-ca\") pod \"thanos-querier-86b8856f97-b75hg\" (UID: \"0fef60a4-2a84-4318-b1bf-feb8f69ef7e9\") " pod="openshift-monitoring/thanos-querier-86b8856f97-b75hg" Nov 03 13:05:19.440337 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.440241 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-multus-daemon-config\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:05:19.440337 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.440316 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b6263f8d-cced-465a-b393-ef51cb902e11-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-krsxs\" (UID: \"b6263f8d-cced-465a-b393-ef51cb902e11\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-krsxs" Nov 03 13:05:19.440631 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.440330 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0f7bd8dd-6293-414d-8b9d-064f04678e61-ovnkube-config\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:05:19.440631 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.440336 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2cn4\" (UniqueName: \"kubernetes.io/projected/62a2ee28-83a7-42bb-8018-7918027b3e18-kube-api-access-k2cn4\") pod \"ingress-canary-czwz4\" (UID: \"62a2ee28-83a7-42bb-8018-7918027b3e18\") " pod="openshift-ingress-canary/ingress-canary-czwz4" Nov 03 13:05:19.440631 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.440373 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtq2f\" (UniqueName: \"kubernetes.io/projected/d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7-kube-api-access-qtq2f\") pod \"aws-ebs-csi-driver-node-q4qjz\" (UID: \"d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4qjz" Nov 03 13:05:19.440631 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.440391 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-oauth-serving-cert\") pod \"console-848bc84498-gxvtr\" (UID: \"58dabb02-fd21-40ee-a4f5-e990b2b9a35a\") " pod="openshift-console/console-848bc84498-gxvtr" Nov 03 13:05:19.440631 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.440435 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 03 13:05:19.440631 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.440563 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:19.440631 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.440572 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5bfbbeb5-621b-43b0-bad1-0420e8b07b62-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6488577f47-wwpwm\" (UID: \"5bfbbeb5-621b-43b0-bad1-0420e8b07b62\") " pod="openshift-monitoring/metrics-server-6488577f47-wwpwm" Nov 03 13:05:19.440631 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.440577 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-console-config\") pod \"console-848bc84498-gxvtr\" (UID: \"58dabb02-fd21-40ee-a4f5-e990b2b9a35a\") " pod="openshift-console/console-848bc84498-gxvtr" Nov 03 13:05:19.440898 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.440703 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d0c5822-ab7b-409d-a755-49e918278fa1-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-fqskz\" (UID: \"0d0c5822-ab7b-409d-a755-49e918278fa1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fqskz" Nov 03 13:05:19.440898 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.440706 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c663a29c-713d-4184-9105-e4a442a894c6-iptables-alerter-script\") pod \"iptables-alerter-999mg\" (UID: \"c663a29c-713d-4184-9105-e4a442a894c6\") " pod="openshift-network-operator/iptables-alerter-999mg" Nov 03 13:05:19.440898 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.440815 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkznq\" (UniqueName: \"kubernetes.io/projected/0d0c5822-ab7b-409d-a755-49e918278fa1-kube-api-access-gkznq\") pod \"openshift-state-metrics-7bd7489957-fqskz\" (UID: \"0d0c5822-ab7b-409d-a755-49e918278fa1\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fqskz" Nov 03 13:05:19.440898 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.440841 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8df59\" (UniqueName: \"kubernetes.io/projected/c7868eea-4b46-49e6-afb8-7fb4bb472fca-kube-api-access-8df59\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:05:19.441086 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.441008 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0ec0b81c-6052-45f2-bd5b-6096ebb72223-serviceca\") pod \"node-ca-zrwrz\" (UID: \"0ec0b81c-6052-45f2-bd5b-6096ebb72223\") " pod="openshift-image-registry/node-ca-zrwrz" Nov 03 13:05:19.441144 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.441129 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:19.441893 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.441871 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:19.441945 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.441901 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zzbqs\" (UID: \"06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1\") " pod="openshift-multus/multus-additional-cni-plugins-zzbqs" Nov 03 13:05:19.441979 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.441964 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0f7bd8dd-6293-414d-8b9d-064f04678e61-env-overrides\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:05:19.442083 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.442068 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0fadb078-8983-42df-afde-c1983044492d-node-exporter-accelerators-collector-config\") pod \"node-exporter-zhl5n\" (UID: \"0fadb078-8983-42df-afde-c1983044492d\") " pod="openshift-monitoring/node-exporter-zhl5n" Nov 03 13:05:19.442144 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.442129 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:19.443523 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.443506 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-service-ca\") pod \"console-848bc84498-gxvtr\" (UID: \"58dabb02-fd21-40ee-a4f5-e990b2b9a35a\") " pod="openshift-console/console-848bc84498-gxvtr" Nov 03 13:05:19.443589 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.443534 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zzbqs\" (UID: \"06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1\") " pod="openshift-multus/multus-additional-cni-plugins-zzbqs" Nov 03 13:05:19.454123 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.454100 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:19.463608 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.463586 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-run\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:05:19.463677 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.463614 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-host-cni-bin\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:05:19.463677 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.463632 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-lib-modules\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:05:19.463677 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.463646 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-cnibin\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:05:19.463677 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.463662 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-host-var-lib-kubelet\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:05:19.463677 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.463676 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-etc-sysctl-conf\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:05:19.463942 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.463684 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-host-cni-bin\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:05:19.463942 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.463710 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0ec0b81c-6052-45f2-bd5b-6096ebb72223-host\") pod \"node-ca-zrwrz\" (UID: \"0ec0b81c-6052-45f2-bd5b-6096ebb72223\") " pod="openshift-image-registry/node-ca-zrwrz" Nov 03 13:05:19.463942 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.463730 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-run\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:05:19.463942 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.463739 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0ec0b81c-6052-45f2-bd5b-6096ebb72223-host\") pod \"node-ca-zrwrz\" (UID: \"0ec0b81c-6052-45f2-bd5b-6096ebb72223\") " pod="openshift-image-registry/node-ca-zrwrz" Nov 03 13:05:19.463942 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.463717 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-host-var-lib-kubelet\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:05:19.463942 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.463748 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/31a398f7-8539-4c36-8386-455ef83b3d2b-crio-socket\") pod \"insights-runtime-extractor-n2nst\" (UID: \"31a398f7-8539-4c36-8386-455ef83b3d2b\") " pod="openshift-insights/insights-runtime-extractor-n2nst" Nov 03 13:05:19.463942 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.463751 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-cnibin\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:05:19.463942 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.463757 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-lib-modules\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:05:19.463942 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.463776 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-etc-sysconfig\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:05:19.463942 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.463801 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-host-cni-netd\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:05:19.463942 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.463801 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/31a398f7-8539-4c36-8386-455ef83b3d2b-crio-socket\") pod \"insights-runtime-extractor-n2nst\" (UID: \"31a398f7-8539-4c36-8386-455ef83b3d2b\") " pod="openshift-insights/insights-runtime-extractor-n2nst" Nov 03 13:05:19.463942 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.463827 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-etc-modprobe-d\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:05:19.463942 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.463842 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-etc-sysconfig\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:05:19.463942 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.463847 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-host-cni-netd\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:05:19.463942 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.463852 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-etc-sysctl-conf\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:05:19.463942 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.463852 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zzbqs\" (UID: \"06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1\") " pod="openshift-multus/multus-additional-cni-plugins-zzbqs" Nov 03 13:05:19.463942 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.463884 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-host-slash\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:05:19.463942 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.463906 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-etc-modprobe-d\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:05:19.463942 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.463925 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q4qjz\" (UID: \"d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4qjz" Nov 03 13:05:19.463942 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.463949 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-host-slash\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:05:19.464647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.463949 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zzbqs\" (UID: \"06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1\") " pod="openshift-multus/multus-additional-cni-plugins-zzbqs" Nov 03 13:05:19.464647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.463952 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-host-run-netns\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:05:19.464647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.463997 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q4qjz\" (UID: \"d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4qjz" Nov 03 13:05:19.464647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464003 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-sys\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:05:19.464647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464024 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895-dbus\") pod \"global-pull-secret-syncer-48vd8\" (UID: \"2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895\") " pod="kube-system/global-pull-secret-syncer-48vd8" Nov 03 13:05:19.464647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464029 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-sys\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:05:19.464647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.463998 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-host-run-netns\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:05:19.464647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464041 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1-system-cni-dir\") pod \"multus-additional-cni-plugins-zzbqs\" (UID: \"06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1\") " pod="openshift-multus/multus-additional-cni-plugins-zzbqs" Nov 03 13:05:19.464647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464060 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-host-run-k8s-cni-cncf-io\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:05:19.464647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464083 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895-kubelet-config\") pod \"global-pull-secret-syncer-48vd8\" (UID: \"2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895\") " pod="kube-system/global-pull-secret-syncer-48vd8" Nov 03 13:05:19.464647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464106 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-etc-openvswitch\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:05:19.464647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464115 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895-dbus\") pod \"global-pull-secret-syncer-48vd8\" (UID: \"2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895\") " pod="kube-system/global-pull-secret-syncer-48vd8" Nov 03 13:05:19.464647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464131 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-host-run-k8s-cni-cncf-io\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:05:19.464647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464133 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-var-lib-openvswitch\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:05:19.464647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464137 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1-system-cni-dir\") pod \"multus-additional-cni-plugins-zzbqs\" (UID: \"06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1\") " pod="openshift-multus/multus-additional-cni-plugins-zzbqs" Nov 03 13:05:19.464647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464154 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-host-run-ovn-kubernetes\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:05:19.464647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464147 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895-kubelet-config\") pod \"global-pull-secret-syncer-48vd8\" (UID: \"2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895\") " pod="kube-system/global-pull-secret-syncer-48vd8" Nov 03 13:05:19.464647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464183 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-var-lib-openvswitch\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:05:19.464647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464188 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-host-run-ovn-kubernetes\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:05:19.464647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464157 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-etc-openvswitch\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:05:19.464647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464206 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:05:19.464647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464224 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-system-cni-dir\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:05:19.464647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464239 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-etc-kubernetes\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:05:19.464647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464271 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7-device-dir\") pod \"aws-ebs-csi-driver-node-q4qjz\" (UID: \"d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4qjz" Nov 03 13:05:19.464647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464223 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:05:19.464647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464296 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-var-lib-kubelet\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:05:19.464647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464316 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-etc-kubernetes\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:05:19.464647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464323 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-run-ovn\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:05:19.464647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464334 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-var-lib-kubelet\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:05:19.464647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464344 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-run-ovn\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:05:19.464647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464327 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7-device-dir\") pod \"aws-ebs-csi-driver-node-q4qjz\" (UID: \"d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4qjz" Nov 03 13:05:19.464647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464297 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-system-cni-dir\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:05:19.464647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464351 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-host-var-lib-cni-bin\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:05:19.464647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464376 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-multus-conf-dir\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:05:19.464647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464380 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-host-var-lib-cni-bin\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:05:19.464647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464396 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-run-openvswitch\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:05:19.464647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464405 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-multus-conf-dir\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:05:19.464647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464412 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-etc-systemd\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:05:19.464647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464425 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-host\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:05:19.464647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464441 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-run-systemd\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:05:19.464647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464448 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-etc-systemd\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:05:19.464647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464426 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-run-openvswitch\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:05:19.464647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464454 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-node-log\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:05:19.464647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464491 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-node-log\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:05:19.464647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464498 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-etc-kubernetes\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:05:19.464647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464497 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-host\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:05:19.464647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464518 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-hostroot\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:05:19.464647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464521 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-run-systemd\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:05:19.464647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464534 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-host-run-multus-certs\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:05:19.464647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464539 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-etc-kubernetes\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:05:19.464647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464543 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-hostroot\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:05:19.464647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464555 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3083cbfe-7833-4b37-8325-cb18c20a7bea-hosts-file\") pod \"node-resolver-zzbpd\" (UID: \"3083cbfe-7833-4b37-8325-cb18c20a7bea\") " pod="openshift-dns/node-resolver-zzbpd" Nov 03 13:05:19.464647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464580 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7-registration-dir\") pod \"aws-ebs-csi-driver-node-q4qjz\" (UID: \"d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4qjz" Nov 03 13:05:19.464647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464582 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-host-run-multus-certs\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:05:19.464647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464589 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3083cbfe-7833-4b37-8325-cb18c20a7bea-hosts-file\") pod \"node-resolver-zzbpd\" (UID: \"3083cbfe-7833-4b37-8325-cb18c20a7bea\") " pod="openshift-dns/node-resolver-zzbpd" Nov 03 13:05:19.464647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464602 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-host-kubelet\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:05:19.464647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464624 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-host-kubelet\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:05:19.464647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464629 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-systemd-units\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:05:19.464647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464635 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7-registration-dir\") pod \"aws-ebs-csi-driver-node-q4qjz\" (UID: \"d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4qjz" Nov 03 13:05:19.464647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464648 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-systemd-units\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:05:19.464647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464656 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7-etc-selinux\") pod \"aws-ebs-csi-driver-node-q4qjz\" (UID: \"d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4qjz" Nov 03 13:05:19.464647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464674 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c663a29c-713d-4184-9105-e4a442a894c6-host-slash\") pod \"iptables-alerter-999mg\" (UID: \"c663a29c-713d-4184-9105-e4a442a894c6\") " pod="openshift-network-operator/iptables-alerter-999mg" Nov 03 13:05:19.466340 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464689 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-multus-socket-dir-parent\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:05:19.466340 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464706 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-host-var-lib-cni-multus\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:05:19.466340 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464721 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-log-socket\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:05:19.466340 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464732 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c663a29c-713d-4184-9105-e4a442a894c6-host-slash\") pod \"iptables-alerter-999mg\" (UID: \"c663a29c-713d-4184-9105-e4a442a894c6\") " pod="openshift-network-operator/iptables-alerter-999mg" Nov 03 13:05:19.466340 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464751 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0fadb078-8983-42df-afde-c1983044492d-node-exporter-wtmp\") pod \"node-exporter-zhl5n\" (UID: \"0fadb078-8983-42df-afde-c1983044492d\") " pod="openshift-monitoring/node-exporter-zhl5n" Nov 03 13:05:19.466340 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464762 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-host-var-lib-cni-multus\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:05:19.466340 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464760 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-multus-socket-dir-parent\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:05:19.466340 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464769 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7-etc-selinux\") pod \"aws-ebs-csi-driver-node-q4qjz\" (UID: \"d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4qjz" Nov 03 13:05:19.466340 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464773 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-log-socket\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:05:19.466340 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464777 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0fadb078-8983-42df-afde-c1983044492d-sys\") pod \"node-exporter-zhl5n\" (UID: \"0fadb078-8983-42df-afde-c1983044492d\") " pod="openshift-monitoring/node-exporter-zhl5n" Nov 03 13:05:19.466340 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464799 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7-socket-dir\") pod \"aws-ebs-csi-driver-node-q4qjz\" (UID: \"d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4qjz" Nov 03 13:05:19.466340 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464813 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-multus-cni-dir\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:05:19.466340 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464804 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0fadb078-8983-42df-afde-c1983044492d-sys\") pod \"node-exporter-zhl5n\" (UID: \"0fadb078-8983-42df-afde-c1983044492d\") " pod="openshift-monitoring/node-exporter-zhl5n" Nov 03 13:05:19.466340 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464834 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0fadb078-8983-42df-afde-c1983044492d-node-exporter-wtmp\") pod \"node-exporter-zhl5n\" (UID: \"0fadb078-8983-42df-afde-c1983044492d\") " pod="openshift-monitoring/node-exporter-zhl5n" Nov 03 13:05:19.466340 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464835 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7-sys-fs\") pod \"aws-ebs-csi-driver-node-q4qjz\" (UID: \"d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4qjz" Nov 03 13:05:19.466340 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464864 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7-socket-dir\") pod \"aws-ebs-csi-driver-node-q4qjz\" (UID: \"d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4qjz" Nov 03 13:05:19.466340 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464865 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7-sys-fs\") pod \"aws-ebs-csi-driver-node-q4qjz\" (UID: \"d8eeb7e5-6595-4f5d-9e5c-ce6348551ca7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q4qjz" Nov 03 13:05:19.466340 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464872 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0-multus-cni-dir\") pod \"multus-nkdmz\" (UID: \"0cd62d34-ebc1-45cb-8a9c-6936d52bc7f0\") " pod="openshift-multus/multus-nkdmz" Nov 03 13:05:19.466340 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464864 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-etc-sysctl-d\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:05:19.466340 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464907 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-host-run-netns\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:05:19.466340 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464930 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0fadb078-8983-42df-afde-c1983044492d-root\") pod \"node-exporter-zhl5n\" (UID: \"0fadb078-8983-42df-afde-c1983044492d\") " pod="openshift-monitoring/node-exporter-zhl5n" Nov 03 13:05:19.466340 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464950 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c7868eea-4b46-49e6-afb8-7fb4bb472fca-etc-sysctl-d\") pod \"tuned-k78zw\" (UID: \"c7868eea-4b46-49e6-afb8-7fb4bb472fca\") " pod="openshift-cluster-node-tuning-operator/tuned-k78zw" Nov 03 13:05:19.466340 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464956 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0f7bd8dd-6293-414d-8b9d-064f04678e61-host-run-netns\") pod \"ovnkube-node-frppp\" (UID: \"0f7bd8dd-6293-414d-8b9d-064f04678e61\") " pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:05:19.466340 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464955 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1-cnibin\") pod \"multus-additional-cni-plugins-zzbqs\" (UID: \"06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1\") " pod="openshift-multus/multus-additional-cni-plugins-zzbqs" Nov 03 13:05:19.466340 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464985 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1-cnibin\") pod \"multus-additional-cni-plugins-zzbqs\" (UID: \"06e0a591-ec3f-4cdc-bb51-13cc7d0f88e1\") " pod="openshift-multus/multus-additional-cni-plugins-zzbqs" Nov 03 13:05:19.466340 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.464988 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0fadb078-8983-42df-afde-c1983044492d-root\") pod \"node-exporter-zhl5n\" (UID: \"0fadb078-8983-42df-afde-c1983044492d\") " pod="openshift-monitoring/node-exporter-zhl5n" Nov 03 13:05:19.481354 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.481329 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 03 13:05:19.481562 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.481541 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 03 13:05:19.482099 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.482082 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 03 13:05:19.487145 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.487126 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dcde035-9b90-4961-8cfc-3f37bfb67581-trusted-ca-bundle\") pod \"console-6f44499665-ff6gp\" (UID: \"2dcde035-9b90-4961-8cfc-3f37bfb67581\") " pod="openshift-console/console-6f44499665-ff6gp" Nov 03 13:05:19.490376 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.490355 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:19.490376 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:19.490365 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7361d2a9-d089-4073-bbfb-e7978943fd88-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:21.046860 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:21.046828 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-87tkk" Nov 03 13:05:22.103392 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:22.103364 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-86b8856f97-b75hg" Nov 03 13:05:22.240424 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:22.240397 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9v5mk" Nov 03 13:05:23.547168 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:23.547138 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:05:23.751401 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:23.751371 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:24.568428 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:24.568394 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-cxrp7" Nov 03 13:05:27.015764 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:27.015737 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:05:27.016247 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:27.015959 5979 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 13:05:27.116400 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:27.116369 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6f44499665-ff6gp" Nov 03 13:05:37.233142 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:05:37.233105 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:05:38.118569 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:38.118538 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6488577f47-wwpwm" Nov 03 13:05:39.191151 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:39.191115 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ghgsf" Nov 03 13:05:40.468971 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:40.468942 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-frppp" Nov 03 13:05:44.274728 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:44.274686 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-848bc84498-gxvtr" podUID="58dabb02-fd21-40ee-a4f5-e990b2b9a35a" containerName="console" containerID="cri-o://9dd9a05764c13ae5588888535b4021e9098f621187d7ff9c8681c8a717ed3a9e" gracePeriod=15 Nov 03 13:05:44.526454 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:44.526402 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-848bc84498-gxvtr" Nov 03 13:05:44.689542 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:44.689465 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-tmp\" (UniqueName: \"kubernetes.io/empty-dir/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-tmp\") pod \"58dabb02-fd21-40ee-a4f5-e990b2b9a35a\" (UID: \"58dabb02-fd21-40ee-a4f5-e990b2b9a35a\") " Nov 03 13:05:44.689713 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:44.689561 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-console-oauth-config\") pod \"58dabb02-fd21-40ee-a4f5-e990b2b9a35a\" (UID: \"58dabb02-fd21-40ee-a4f5-e990b2b9a35a\") " Nov 03 13:05:44.689713 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:44.689590 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-kube-api-access-6jmgk\" (UniqueName: \"kubernetes.io/projected/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-kube-api-access-6jmgk\") pod \"58dabb02-fd21-40ee-a4f5-e990b2b9a35a\" (UID: \"58dabb02-fd21-40ee-a4f5-e990b2b9a35a\") " Nov 03 13:05:44.689713 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:44.689612 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-oauth-serving-cert\") pod \"58dabb02-fd21-40ee-a4f5-e990b2b9a35a\" (UID: \"58dabb02-fd21-40ee-a4f5-e990b2b9a35a\") " Nov 03 13:05:44.689713 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:44.689653 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-service-ca\" (UniqueName: \"kubernetes.io/configmap/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-service-ca\") pod \"58dabb02-fd21-40ee-a4f5-e990b2b9a35a\" (UID: \"58dabb02-fd21-40ee-a4f5-e990b2b9a35a\") " Nov 03 13:05:44.689713 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:44.689699 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-console-config\" (UniqueName: \"kubernetes.io/configmap/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-console-config\") pod \"58dabb02-fd21-40ee-a4f5-e990b2b9a35a\" (UID: \"58dabb02-fd21-40ee-a4f5-e990b2b9a35a\") " Nov 03 13:05:44.689975 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:44.689734 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-console-serving-cert\") pod \"58dabb02-fd21-40ee-a4f5-e990b2b9a35a\" (UID: \"58dabb02-fd21-40ee-a4f5-e990b2b9a35a\") " Nov 03 13:05:44.689975 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:44.689751 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-tmp" pod "58dabb02-fd21-40ee-a4f5-e990b2b9a35a" (UID: "58dabb02-fd21-40ee-a4f5-e990b2b9a35a"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:05:44.690094 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:44.689990 5979 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-tmp\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:05:44.690094 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:44.690048 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-oauth-serving-cert" pod "58dabb02-fd21-40ee-a4f5-e990b2b9a35a" (UID: "58dabb02-fd21-40ee-a4f5-e990b2b9a35a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:05:44.690184 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:44.690158 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-console-config" pod "58dabb02-fd21-40ee-a4f5-e990b2b9a35a" (UID: "58dabb02-fd21-40ee-a4f5-e990b2b9a35a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:05:44.690227 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:44.690135 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-service-ca" pod "58dabb02-fd21-40ee-a4f5-e990b2b9a35a" (UID: "58dabb02-fd21-40ee-a4f5-e990b2b9a35a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:05:44.692070 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:44.692042 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-console-oauth-config" pod "58dabb02-fd21-40ee-a4f5-e990b2b9a35a" (UID: "58dabb02-fd21-40ee-a4f5-e990b2b9a35a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:05:44.692378 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:44.692355 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-console-serving-cert" pod "58dabb02-fd21-40ee-a4f5-e990b2b9a35a" (UID: "58dabb02-fd21-40ee-a4f5-e990b2b9a35a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:05:44.692378 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:44.692355 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-kube-api-access-6jmgk" pod "58dabb02-fd21-40ee-a4f5-e990b2b9a35a" (UID: "58dabb02-fd21-40ee-a4f5-e990b2b9a35a"). InnerVolumeSpecName "kube-api-access-6jmgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:05:44.790878 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:44.790804 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jmgk\" (UniqueName: \"kubernetes.io/projected/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-kube-api-access-6jmgk\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:05:44.790878 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:44.790835 5979 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-oauth-serving-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:05:44.790878 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:44.790847 5979 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-service-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:05:44.790878 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:44.790855 5979 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-console-config\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:05:44.790878 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:44.790864 5979 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-console-serving-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:05:44.790878 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:44.790872 5979 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/58dabb02-fd21-40ee-a4f5-e990b2b9a35a-console-oauth-config\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:05:45.314280 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:45.314246 5979 generic.go:356] "Generic (PLEG): container finished" podID="58dabb02-fd21-40ee-a4f5-e990b2b9a35a" containerID="9dd9a05764c13ae5588888535b4021e9098f621187d7ff9c8681c8a717ed3a9e" exitCode=0 Nov 03 13:05:45.314762 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:45.314295 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-848bc84498-gxvtr" event={"ID":"58dabb02-fd21-40ee-a4f5-e990b2b9a35a","Type":"ContainerDied","Data":"9dd9a05764c13ae5588888535b4021e9098f621187d7ff9c8681c8a717ed3a9e"} Nov 03 13:05:45.314762 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:45.314303 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-848bc84498-gxvtr" Nov 03 13:05:45.314762 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:45.314332 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-848bc84498-gxvtr" event={"ID":"58dabb02-fd21-40ee-a4f5-e990b2b9a35a","Type":"ContainerDied","Data":"d722cdc71acb0c1e3a81a70c28ff6ebbba944b71c0fbcad85b5c86e78c8336c4"} Nov 03 13:05:45.314762 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:45.314352 5979 scope.go:122] "RemoveContainer" containerID="9dd9a05764c13ae5588888535b4021e9098f621187d7ff9c8681c8a717ed3a9e" Nov 03 13:05:45.323023 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:45.323007 5979 scope.go:122] "RemoveContainer" containerID="9dd9a05764c13ae5588888535b4021e9098f621187d7ff9c8681c8a717ed3a9e" Nov 03 13:05:45.323276 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:05:45.323252 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dd9a05764c13ae5588888535b4021e9098f621187d7ff9c8681c8a717ed3a9e\": container with ID starting with 9dd9a05764c13ae5588888535b4021e9098f621187d7ff9c8681c8a717ed3a9e not found: ID does not exist" containerID="9dd9a05764c13ae5588888535b4021e9098f621187d7ff9c8681c8a717ed3a9e" Nov 03 13:05:45.323320 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:45.323285 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dd9a05764c13ae5588888535b4021e9098f621187d7ff9c8681c8a717ed3a9e"} err="failed to get container status \"9dd9a05764c13ae5588888535b4021e9098f621187d7ff9c8681c8a717ed3a9e\": rpc error: code = NotFound desc = could not find container \"9dd9a05764c13ae5588888535b4021e9098f621187d7ff9c8681c8a717ed3a9e\": container with ID starting with 9dd9a05764c13ae5588888535b4021e9098f621187d7ff9c8681c8a717ed3a9e not found: ID does not exist" Nov 03 13:05:45.465451 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:45.465420 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-848bc84498-gxvtr"] Nov 03 13:05:45.511792 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:45.511727 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-848bc84498-gxvtr"] Nov 03 13:05:46.112575 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:46.112544 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58dabb02-fd21-40ee-a4f5-e990b2b9a35a" path="/var/lib/kubelet/pods/58dabb02-fd21-40ee-a4f5-e990b2b9a35a/volumes" Nov 03 13:05:46.469898 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:46.469862 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 13:05:46.470282 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:46.470133 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7361d2a9-d089-4073-bbfb-e7978943fd88" containerName="alertmanager" containerID="cri-o://75e30b1841c2fb988dd3a2203d48ab05c7189a12cb64c10e3a29c35cede9ecaf" gracePeriod=120 Nov 03 13:05:46.470282 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:46.470237 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7361d2a9-d089-4073-bbfb-e7978943fd88" containerName="kube-rbac-proxy-metric" containerID="cri-o://12e48d688f061a1abfcc0dc66deb4cec44009b7f2fda1d12dea0d3e7a5d49a12" gracePeriod=120 Nov 03 13:05:46.470282 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:46.470257 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7361d2a9-d089-4073-bbfb-e7978943fd88" containerName="kube-rbac-proxy-web" containerID="cri-o://aa9e6c8cb1885d86a9fced9f978440068e3b11ef943accc278c67b2da19f39ba" gracePeriod=120 Nov 03 13:05:46.470446 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:46.470237 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7361d2a9-d089-4073-bbfb-e7978943fd88" containerName="prom-label-proxy" containerID="cri-o://05a184bc5f5cf4f5a33ef8586c6a32a7f5ad9641153bddc9a56ae94ccbe70d22" gracePeriod=120 Nov 03 13:05:46.470446 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:46.470294 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7361d2a9-d089-4073-bbfb-e7978943fd88" containerName="kube-rbac-proxy" containerID="cri-o://f79932046af38f76c865b7d005c526fc9e7e5523481bcd2f827dd953ca44aa88" gracePeriod=120 Nov 03 13:05:46.470446 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:46.470427 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7361d2a9-d089-4073-bbfb-e7978943fd88" containerName="config-reloader" containerID="cri-o://e56f4e26a0db91adbfd5a8af6e1988a5d70051fc325afc248091a808c3d0121b" gracePeriod=120 Nov 03 13:05:47.328021 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:47.327985 5979 generic.go:356] "Generic (PLEG): container finished" podID="7361d2a9-d089-4073-bbfb-e7978943fd88" containerID="05a184bc5f5cf4f5a33ef8586c6a32a7f5ad9641153bddc9a56ae94ccbe70d22" exitCode=0 Nov 03 13:05:47.328021 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:47.328011 5979 generic.go:356] "Generic (PLEG): container finished" podID="7361d2a9-d089-4073-bbfb-e7978943fd88" containerID="12e48d688f061a1abfcc0dc66deb4cec44009b7f2fda1d12dea0d3e7a5d49a12" exitCode=0 Nov 03 13:05:47.328021 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:47.328025 5979 generic.go:356] "Generic (PLEG): container finished" podID="7361d2a9-d089-4073-bbfb-e7978943fd88" containerID="f79932046af38f76c865b7d005c526fc9e7e5523481bcd2f827dd953ca44aa88" exitCode=0 Nov 03 13:05:47.328021 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:47.328032 5979 generic.go:356] "Generic (PLEG): container finished" podID="7361d2a9-d089-4073-bbfb-e7978943fd88" containerID="e56f4e26a0db91adbfd5a8af6e1988a5d70051fc325afc248091a808c3d0121b" exitCode=0 Nov 03 13:05:47.328021 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:47.328037 5979 generic.go:356] "Generic (PLEG): container finished" podID="7361d2a9-d089-4073-bbfb-e7978943fd88" containerID="75e30b1841c2fb988dd3a2203d48ab05c7189a12cb64c10e3a29c35cede9ecaf" exitCode=0 Nov 03 13:05:47.328322 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:47.328060 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7361d2a9-d089-4073-bbfb-e7978943fd88","Type":"ContainerDied","Data":"05a184bc5f5cf4f5a33ef8586c6a32a7f5ad9641153bddc9a56ae94ccbe70d22"} Nov 03 13:05:47.328322 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:47.328125 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7361d2a9-d089-4073-bbfb-e7978943fd88","Type":"ContainerDied","Data":"12e48d688f061a1abfcc0dc66deb4cec44009b7f2fda1d12dea0d3e7a5d49a12"} Nov 03 13:05:47.328322 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:47.328153 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7361d2a9-d089-4073-bbfb-e7978943fd88","Type":"ContainerDied","Data":"f79932046af38f76c865b7d005c526fc9e7e5523481bcd2f827dd953ca44aa88"} Nov 03 13:05:47.328322 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:47.328178 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7361d2a9-d089-4073-bbfb-e7978943fd88","Type":"ContainerDied","Data":"e56f4e26a0db91adbfd5a8af6e1988a5d70051fc325afc248091a808c3d0121b"} Nov 03 13:05:47.328322 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:47.328204 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7361d2a9-d089-4073-bbfb-e7978943fd88","Type":"ContainerDied","Data":"75e30b1841c2fb988dd3a2203d48ab05c7189a12cb64c10e3a29c35cede9ecaf"} Nov 03 13:05:47.711773 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:47.711748 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:47.817001 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:47.816968 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-web-config\" (UniqueName: \"kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-web-config\") pod \"7361d2a9-d089-4073-bbfb-e7978943fd88\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " Nov 03 13:05:47.817001 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:47.817004 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7361d2a9-d089-4073-bbfb-e7978943fd88-kube-api-access-cwz9f\" (UniqueName: \"kubernetes.io/projected/7361d2a9-d089-4073-bbfb-e7978943fd88-kube-api-access-cwz9f\") pod \"7361d2a9-d089-4073-bbfb-e7978943fd88\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " Nov 03 13:05:47.817326 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:47.817026 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-secret-alertmanager-main-tls\") pod \"7361d2a9-d089-4073-bbfb-e7978943fd88\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " Nov 03 13:05:47.817326 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:47.817054 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7361d2a9-d089-4073-bbfb-e7978943fd88-tls-assets\" (UniqueName: \"kubernetes.io/projected/7361d2a9-d089-4073-bbfb-e7978943fd88-tls-assets\") pod \"7361d2a9-d089-4073-bbfb-e7978943fd88\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " Nov 03 13:05:47.817326 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:47.817076 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7361d2a9-d089-4073-bbfb-e7978943fd88-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7361d2a9-d089-4073-bbfb-e7978943fd88-alertmanager-main-db\") pod \"7361d2a9-d089-4073-bbfb-e7978943fd88\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " Nov 03 13:05:47.817326 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:47.817121 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-secret-alertmanager-kube-rbac-proxy-metric\") pod \"7361d2a9-d089-4073-bbfb-e7978943fd88\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " Nov 03 13:05:47.817326 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:47.817180 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7361d2a9-d089-4073-bbfb-e7978943fd88-config-out\" (UniqueName: \"kubernetes.io/empty-dir/7361d2a9-d089-4073-bbfb-e7978943fd88-config-out\") pod \"7361d2a9-d089-4073-bbfb-e7978943fd88\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " Nov 03 13:05:47.817736 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:47.817701 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7361d2a9-d089-4073-bbfb-e7978943fd88-alertmanager-main-db" pod "7361d2a9-d089-4073-bbfb-e7978943fd88" (UID: "7361d2a9-d089-4073-bbfb-e7978943fd88"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:05:47.817970 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:47.817940 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-cluster-tls-config\") pod \"7361d2a9-d089-4073-bbfb-e7978943fd88\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " Nov 03 13:05:47.818055 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:47.817995 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7361d2a9-d089-4073-bbfb-e7978943fd88-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7361d2a9-d089-4073-bbfb-e7978943fd88-metrics-client-ca\") pod \"7361d2a9-d089-4073-bbfb-e7978943fd88\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " Nov 03 13:05:47.818462 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:47.818435 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7361d2a9-d089-4073-bbfb-e7978943fd88-metrics-client-ca" pod "7361d2a9-d089-4073-bbfb-e7978943fd88" (UID: "7361d2a9-d089-4073-bbfb-e7978943fd88"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:05:47.818955 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:47.818931 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-secret-alertmanager-kube-rbac-proxy\") pod \"7361d2a9-d089-4073-bbfb-e7978943fd88\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " Nov 03 13:05:47.819046 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:47.819010 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-secret-alertmanager-kube-rbac-proxy-web\") pod \"7361d2a9-d089-4073-bbfb-e7978943fd88\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " Nov 03 13:05:47.819108 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:47.819056 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-config-volume\" (UniqueName: \"kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-config-volume\") pod \"7361d2a9-d089-4073-bbfb-e7978943fd88\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " Nov 03 13:05:47.819164 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:47.819109 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7361d2a9-d089-4073-bbfb-e7978943fd88-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7361d2a9-d089-4073-bbfb-e7978943fd88-alertmanager-trusted-ca-bundle\") pod \"7361d2a9-d089-4073-bbfb-e7978943fd88\" (UID: \"7361d2a9-d089-4073-bbfb-e7978943fd88\") " Nov 03 13:05:47.819533 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:47.819508 5979 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7361d2a9-d089-4073-bbfb-e7978943fd88-alertmanager-main-db\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:05:47.819649 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:47.819548 5979 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7361d2a9-d089-4073-bbfb-e7978943fd88-metrics-client-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:05:47.820870 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:47.820820 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7361d2a9-d089-4073-bbfb-e7978943fd88-alertmanager-trusted-ca-bundle" pod "7361d2a9-d089-4073-bbfb-e7978943fd88" (UID: "7361d2a9-d089-4073-bbfb-e7978943fd88"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:05:47.821139 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:47.821113 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7361d2a9-d089-4073-bbfb-e7978943fd88-kube-api-access-cwz9f" pod "7361d2a9-d089-4073-bbfb-e7978943fd88" (UID: "7361d2a9-d089-4073-bbfb-e7978943fd88"). InnerVolumeSpecName "kube-api-access-cwz9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:05:47.821208 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:47.821155 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7361d2a9-d089-4073-bbfb-e7978943fd88-config-out" pod "7361d2a9-d089-4073-bbfb-e7978943fd88" (UID: "7361d2a9-d089-4073-bbfb-e7978943fd88"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:05:47.825294 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:47.823128 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7361d2a9-d089-4073-bbfb-e7978943fd88-tls-assets" pod "7361d2a9-d089-4073-bbfb-e7978943fd88" (UID: "7361d2a9-d089-4073-bbfb-e7978943fd88"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:05:47.826303 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:47.825753 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-secret-alertmanager-main-tls" pod "7361d2a9-d089-4073-bbfb-e7978943fd88" (UID: "7361d2a9-d089-4073-bbfb-e7978943fd88"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:05:47.826303 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:47.826175 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-config-volume" pod "7361d2a9-d089-4073-bbfb-e7978943fd88" (UID: "7361d2a9-d089-4073-bbfb-e7978943fd88"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:05:47.826615 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:47.826431 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-secret-alertmanager-kube-rbac-proxy" pod "7361d2a9-d089-4073-bbfb-e7978943fd88" (UID: "7361d2a9-d089-4073-bbfb-e7978943fd88"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:05:47.826615 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:47.826565 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-secret-alertmanager-kube-rbac-proxy-metric" pod "7361d2a9-d089-4073-bbfb-e7978943fd88" (UID: "7361d2a9-d089-4073-bbfb-e7978943fd88"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:05:47.826729 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:47.826708 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-secret-alertmanager-kube-rbac-proxy-web" pod "7361d2a9-d089-4073-bbfb-e7978943fd88" (UID: "7361d2a9-d089-4073-bbfb-e7978943fd88"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:05:47.829079 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:47.829055 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-cluster-tls-config" pod "7361d2a9-d089-4073-bbfb-e7978943fd88" (UID: "7361d2a9-d089-4073-bbfb-e7978943fd88"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:05:47.835225 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:47.835204 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-web-config" pod "7361d2a9-d089-4073-bbfb-e7978943fd88" (UID: "7361d2a9-d089-4073-bbfb-e7978943fd88"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:05:47.920910 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:47.920880 5979 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7361d2a9-d089-4073-bbfb-e7978943fd88-tls-assets\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:05:47.920910 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:47.920908 5979 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:05:47.921099 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:47.920919 5979 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7361d2a9-d089-4073-bbfb-e7978943fd88-config-out\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:05:47.921099 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:47.920929 5979 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-cluster-tls-config\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:05:47.921099 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:47.920939 5979 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:05:47.921099 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:47.920948 5979 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:05:47.921099 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:47.920957 5979 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:05:47.921099 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:47.920966 5979 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7361d2a9-d089-4073-bbfb-e7978943fd88-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:05:47.921099 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:47.920976 5979 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-web-config\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:05:47.921099 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:47.920984 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwz9f\" (UniqueName: \"kubernetes.io/projected/7361d2a9-d089-4073-bbfb-e7978943fd88-kube-api-access-cwz9f\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:05:47.921099 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:47.920993 5979 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7361d2a9-d089-4073-bbfb-e7978943fd88-secret-alertmanager-main-tls\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:05:48.334952 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.334873 5979 generic.go:356] "Generic (PLEG): container finished" podID="7361d2a9-d089-4073-bbfb-e7978943fd88" containerID="aa9e6c8cb1885d86a9fced9f978440068e3b11ef943accc278c67b2da19f39ba" exitCode=0 Nov 03 13:05:48.334952 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.334937 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7361d2a9-d089-4073-bbfb-e7978943fd88","Type":"ContainerDied","Data":"aa9e6c8cb1885d86a9fced9f978440068e3b11ef943accc278c67b2da19f39ba"} Nov 03 13:05:48.335135 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.334965 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:48.335135 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.335004 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7361d2a9-d089-4073-bbfb-e7978943fd88","Type":"ContainerDied","Data":"f56f8782dab339e2e6d83b3ae256a7331821a82dc572428363224d45d35c046a"} Nov 03 13:05:48.335135 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.335035 5979 scope.go:122] "RemoveContainer" containerID="05a184bc5f5cf4f5a33ef8586c6a32a7f5ad9641153bddc9a56ae94ccbe70d22" Nov 03 13:05:48.343536 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.343515 5979 scope.go:122] "RemoveContainer" containerID="12e48d688f061a1abfcc0dc66deb4cec44009b7f2fda1d12dea0d3e7a5d49a12" Nov 03 13:05:48.352261 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.352242 5979 scope.go:122] "RemoveContainer" containerID="f79932046af38f76c865b7d005c526fc9e7e5523481bcd2f827dd953ca44aa88" Nov 03 13:05:48.359059 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.359038 5979 scope.go:122] "RemoveContainer" containerID="aa9e6c8cb1885d86a9fced9f978440068e3b11ef943accc278c67b2da19f39ba" Nov 03 13:05:48.365514 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.365491 5979 scope.go:122] "RemoveContainer" containerID="e56f4e26a0db91adbfd5a8af6e1988a5d70051fc325afc248091a808c3d0121b" Nov 03 13:05:48.372522 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.372498 5979 scope.go:122] "RemoveContainer" containerID="75e30b1841c2fb988dd3a2203d48ab05c7189a12cb64c10e3a29c35cede9ecaf" Nov 03 13:05:48.379167 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.379153 5979 scope.go:122] "RemoveContainer" containerID="aaea79a6899ef0f795f4cbb0ee4424a703035e5af1aeed5baf0609f417f64e5c" Nov 03 13:05:48.386087 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.386072 5979 scope.go:122] "RemoveContainer" containerID="05a184bc5f5cf4f5a33ef8586c6a32a7f5ad9641153bddc9a56ae94ccbe70d22" Nov 03 13:05:48.386365 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:05:48.386346 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05a184bc5f5cf4f5a33ef8586c6a32a7f5ad9641153bddc9a56ae94ccbe70d22\": container with ID starting with 05a184bc5f5cf4f5a33ef8586c6a32a7f5ad9641153bddc9a56ae94ccbe70d22 not found: ID does not exist" containerID="05a184bc5f5cf4f5a33ef8586c6a32a7f5ad9641153bddc9a56ae94ccbe70d22" Nov 03 13:05:48.386430 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.386374 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05a184bc5f5cf4f5a33ef8586c6a32a7f5ad9641153bddc9a56ae94ccbe70d22"} err="failed to get container status \"05a184bc5f5cf4f5a33ef8586c6a32a7f5ad9641153bddc9a56ae94ccbe70d22\": rpc error: code = NotFound desc = could not find container \"05a184bc5f5cf4f5a33ef8586c6a32a7f5ad9641153bddc9a56ae94ccbe70d22\": container with ID starting with 05a184bc5f5cf4f5a33ef8586c6a32a7f5ad9641153bddc9a56ae94ccbe70d22 not found: ID does not exist" Nov 03 13:05:48.386430 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.386392 5979 scope.go:122] "RemoveContainer" containerID="12e48d688f061a1abfcc0dc66deb4cec44009b7f2fda1d12dea0d3e7a5d49a12" Nov 03 13:05:48.386660 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:05:48.386640 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12e48d688f061a1abfcc0dc66deb4cec44009b7f2fda1d12dea0d3e7a5d49a12\": container with ID starting with 12e48d688f061a1abfcc0dc66deb4cec44009b7f2fda1d12dea0d3e7a5d49a12 not found: ID does not exist" containerID="12e48d688f061a1abfcc0dc66deb4cec44009b7f2fda1d12dea0d3e7a5d49a12" Nov 03 13:05:48.386725 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.386667 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12e48d688f061a1abfcc0dc66deb4cec44009b7f2fda1d12dea0d3e7a5d49a12"} err="failed to get container status \"12e48d688f061a1abfcc0dc66deb4cec44009b7f2fda1d12dea0d3e7a5d49a12\": rpc error: code = NotFound desc = could not find container \"12e48d688f061a1abfcc0dc66deb4cec44009b7f2fda1d12dea0d3e7a5d49a12\": container with ID starting with 12e48d688f061a1abfcc0dc66deb4cec44009b7f2fda1d12dea0d3e7a5d49a12 not found: ID does not exist" Nov 03 13:05:48.386725 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.386684 5979 scope.go:122] "RemoveContainer" containerID="f79932046af38f76c865b7d005c526fc9e7e5523481bcd2f827dd953ca44aa88" Nov 03 13:05:48.386914 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:05:48.386897 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f79932046af38f76c865b7d005c526fc9e7e5523481bcd2f827dd953ca44aa88\": container with ID starting with f79932046af38f76c865b7d005c526fc9e7e5523481bcd2f827dd953ca44aa88 not found: ID does not exist" containerID="f79932046af38f76c865b7d005c526fc9e7e5523481bcd2f827dd953ca44aa88" Nov 03 13:05:48.386978 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.386919 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f79932046af38f76c865b7d005c526fc9e7e5523481bcd2f827dd953ca44aa88"} err="failed to get container status \"f79932046af38f76c865b7d005c526fc9e7e5523481bcd2f827dd953ca44aa88\": rpc error: code = NotFound desc = could not find container \"f79932046af38f76c865b7d005c526fc9e7e5523481bcd2f827dd953ca44aa88\": container with ID starting with f79932046af38f76c865b7d005c526fc9e7e5523481bcd2f827dd953ca44aa88 not found: ID does not exist" Nov 03 13:05:48.386978 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.386932 5979 scope.go:122] "RemoveContainer" containerID="aa9e6c8cb1885d86a9fced9f978440068e3b11ef943accc278c67b2da19f39ba" Nov 03 13:05:48.387153 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:05:48.387131 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa9e6c8cb1885d86a9fced9f978440068e3b11ef943accc278c67b2da19f39ba\": container with ID starting with aa9e6c8cb1885d86a9fced9f978440068e3b11ef943accc278c67b2da19f39ba not found: ID does not exist" containerID="aa9e6c8cb1885d86a9fced9f978440068e3b11ef943accc278c67b2da19f39ba" Nov 03 13:05:48.387239 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.387156 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa9e6c8cb1885d86a9fced9f978440068e3b11ef943accc278c67b2da19f39ba"} err="failed to get container status \"aa9e6c8cb1885d86a9fced9f978440068e3b11ef943accc278c67b2da19f39ba\": rpc error: code = NotFound desc = could not find container \"aa9e6c8cb1885d86a9fced9f978440068e3b11ef943accc278c67b2da19f39ba\": container with ID starting with aa9e6c8cb1885d86a9fced9f978440068e3b11ef943accc278c67b2da19f39ba not found: ID does not exist" Nov 03 13:05:48.387239 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.387169 5979 scope.go:122] "RemoveContainer" containerID="e56f4e26a0db91adbfd5a8af6e1988a5d70051fc325afc248091a808c3d0121b" Nov 03 13:05:48.387435 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:05:48.387419 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e56f4e26a0db91adbfd5a8af6e1988a5d70051fc325afc248091a808c3d0121b\": container with ID starting with e56f4e26a0db91adbfd5a8af6e1988a5d70051fc325afc248091a808c3d0121b not found: ID does not exist" containerID="e56f4e26a0db91adbfd5a8af6e1988a5d70051fc325afc248091a808c3d0121b" Nov 03 13:05:48.387543 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.387438 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e56f4e26a0db91adbfd5a8af6e1988a5d70051fc325afc248091a808c3d0121b"} err="failed to get container status \"e56f4e26a0db91adbfd5a8af6e1988a5d70051fc325afc248091a808c3d0121b\": rpc error: code = NotFound desc = could not find container \"e56f4e26a0db91adbfd5a8af6e1988a5d70051fc325afc248091a808c3d0121b\": container with ID starting with e56f4e26a0db91adbfd5a8af6e1988a5d70051fc325afc248091a808c3d0121b not found: ID does not exist" Nov 03 13:05:48.387543 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.387453 5979 scope.go:122] "RemoveContainer" containerID="75e30b1841c2fb988dd3a2203d48ab05c7189a12cb64c10e3a29c35cede9ecaf" Nov 03 13:05:48.387732 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:05:48.387717 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75e30b1841c2fb988dd3a2203d48ab05c7189a12cb64c10e3a29c35cede9ecaf\": container with ID starting with 75e30b1841c2fb988dd3a2203d48ab05c7189a12cb64c10e3a29c35cede9ecaf not found: ID does not exist" containerID="75e30b1841c2fb988dd3a2203d48ab05c7189a12cb64c10e3a29c35cede9ecaf" Nov 03 13:05:48.387775 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.387736 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75e30b1841c2fb988dd3a2203d48ab05c7189a12cb64c10e3a29c35cede9ecaf"} err="failed to get container status \"75e30b1841c2fb988dd3a2203d48ab05c7189a12cb64c10e3a29c35cede9ecaf\": rpc error: code = NotFound desc = could not find container \"75e30b1841c2fb988dd3a2203d48ab05c7189a12cb64c10e3a29c35cede9ecaf\": container with ID starting with 75e30b1841c2fb988dd3a2203d48ab05c7189a12cb64c10e3a29c35cede9ecaf not found: ID does not exist" Nov 03 13:05:48.387775 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.387749 5979 scope.go:122] "RemoveContainer" containerID="aaea79a6899ef0f795f4cbb0ee4424a703035e5af1aeed5baf0609f417f64e5c" Nov 03 13:05:48.387985 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:05:48.387965 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaea79a6899ef0f795f4cbb0ee4424a703035e5af1aeed5baf0609f417f64e5c\": container with ID starting with aaea79a6899ef0f795f4cbb0ee4424a703035e5af1aeed5baf0609f417f64e5c not found: ID does not exist" containerID="aaea79a6899ef0f795f4cbb0ee4424a703035e5af1aeed5baf0609f417f64e5c" Nov 03 13:05:48.388034 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.387990 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaea79a6899ef0f795f4cbb0ee4424a703035e5af1aeed5baf0609f417f64e5c"} err="failed to get container status \"aaea79a6899ef0f795f4cbb0ee4424a703035e5af1aeed5baf0609f417f64e5c\": rpc error: code = NotFound desc = could not find container \"aaea79a6899ef0f795f4cbb0ee4424a703035e5af1aeed5baf0609f417f64e5c\": container with ID starting with aaea79a6899ef0f795f4cbb0ee4424a703035e5af1aeed5baf0609f417f64e5c not found: ID does not exist" Nov 03 13:05:48.511636 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.511601 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 13:05:48.558143 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.558073 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 13:05:48.558610 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.558588 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 13:05:48.558905 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.558890 5979 cpu_manager.go:414] "containerMap: removing container" podUID="7361d2a9-d089-4073-bbfb-e7978943fd88" containerName="prom-label-proxy" Nov 03 13:05:48.558973 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.558908 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7361d2a9-d089-4073-bbfb-e7978943fd88" containerName="prom-label-proxy" Nov 03 13:05:48.558973 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.558923 5979 cpu_manager.go:414] "containerMap: removing container" podUID="7361d2a9-d089-4073-bbfb-e7978943fd88" containerName="config-reloader" Nov 03 13:05:48.558973 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.558931 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7361d2a9-d089-4073-bbfb-e7978943fd88" containerName="config-reloader" Nov 03 13:05:48.558973 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.558945 5979 cpu_manager.go:414] "containerMap: removing container" podUID="7361d2a9-d089-4073-bbfb-e7978943fd88" containerName="init-config-reloader" Nov 03 13:05:48.558973 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.558953 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7361d2a9-d089-4073-bbfb-e7978943fd88" containerName="init-config-reloader" Nov 03 13:05:48.558973 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.558971 5979 cpu_manager.go:414] "containerMap: removing container" podUID="7361d2a9-d089-4073-bbfb-e7978943fd88" containerName="alertmanager" Nov 03 13:05:48.559268 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.558979 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7361d2a9-d089-4073-bbfb-e7978943fd88" containerName="alertmanager" Nov 03 13:05:48.559268 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.558995 5979 cpu_manager.go:414] "containerMap: removing container" podUID="7361d2a9-d089-4073-bbfb-e7978943fd88" containerName="kube-rbac-proxy-metric" Nov 03 13:05:48.559268 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.559003 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7361d2a9-d089-4073-bbfb-e7978943fd88" containerName="kube-rbac-proxy-metric" Nov 03 13:05:48.559268 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.559020 5979 cpu_manager.go:414] "containerMap: removing container" podUID="7361d2a9-d089-4073-bbfb-e7978943fd88" containerName="kube-rbac-proxy-web" Nov 03 13:05:48.559268 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.559027 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7361d2a9-d089-4073-bbfb-e7978943fd88" containerName="kube-rbac-proxy-web" Nov 03 13:05:48.559268 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.559048 5979 cpu_manager.go:414] "containerMap: removing container" podUID="58dabb02-fd21-40ee-a4f5-e990b2b9a35a" containerName="console" Nov 03 13:05:48.559268 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.559055 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58dabb02-fd21-40ee-a4f5-e990b2b9a35a" containerName="console" Nov 03 13:05:48.559268 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.559074 5979 cpu_manager.go:414] "containerMap: removing container" podUID="7361d2a9-d089-4073-bbfb-e7978943fd88" containerName="kube-rbac-proxy" Nov 03 13:05:48.559268 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.559081 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7361d2a9-d089-4073-bbfb-e7978943fd88" containerName="kube-rbac-proxy" Nov 03 13:05:48.559268 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.559149 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="7361d2a9-d089-4073-bbfb-e7978943fd88" containerName="kube-rbac-proxy" Nov 03 13:05:48.559268 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.559164 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="7361d2a9-d089-4073-bbfb-e7978943fd88" containerName="prom-label-proxy" Nov 03 13:05:48.559268 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.559173 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="7361d2a9-d089-4073-bbfb-e7978943fd88" containerName="config-reloader" Nov 03 13:05:48.559268 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.559182 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="7361d2a9-d089-4073-bbfb-e7978943fd88" containerName="init-config-reloader" Nov 03 13:05:48.559268 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.559191 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="7361d2a9-d089-4073-bbfb-e7978943fd88" containerName="alertmanager" Nov 03 13:05:48.559268 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.559201 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="7361d2a9-d089-4073-bbfb-e7978943fd88" containerName="kube-rbac-proxy-metric" Nov 03 13:05:48.559268 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.559210 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="7361d2a9-d089-4073-bbfb-e7978943fd88" containerName="kube-rbac-proxy-web" Nov 03 13:05:48.559268 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.559221 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="58dabb02-fd21-40ee-a4f5-e990b2b9a35a" containerName="console" Nov 03 13:05:48.563339 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.563321 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:48.616434 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.616346 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 03 13:05:48.616434 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.616363 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 03 13:05:48.616434 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.616347 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 03 13:05:48.616434 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.616346 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 03 13:05:48.616756 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.616450 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 03 13:05:48.616756 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.616555 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 03 13:05:48.616756 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.616559 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 03 13:05:48.616756 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.616587 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 03 13:05:48.616756 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.616605 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-swq8s\"" Nov 03 13:05:48.662994 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.662964 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 03 13:05:48.728690 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.728662 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0eae77ff-4716-4b3e-94cf-cd767108537c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0eae77ff-4716-4b3e-94cf-cd767108537c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:48.729048 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.728696 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0eae77ff-4716-4b3e-94cf-cd767108537c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0eae77ff-4716-4b3e-94cf-cd767108537c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:48.729048 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.728722 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0eae77ff-4716-4b3e-94cf-cd767108537c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0eae77ff-4716-4b3e-94cf-cd767108537c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:48.729048 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.728771 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0eae77ff-4716-4b3e-94cf-cd767108537c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0eae77ff-4716-4b3e-94cf-cd767108537c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:48.729048 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.728858 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7phj\" (UniqueName: \"kubernetes.io/projected/0eae77ff-4716-4b3e-94cf-cd767108537c-kube-api-access-h7phj\") pod \"alertmanager-main-0\" (UID: \"0eae77ff-4716-4b3e-94cf-cd767108537c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:48.729048 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.728890 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0eae77ff-4716-4b3e-94cf-cd767108537c-config-out\") pod \"alertmanager-main-0\" (UID: \"0eae77ff-4716-4b3e-94cf-cd767108537c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:48.729048 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.728962 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0eae77ff-4716-4b3e-94cf-cd767108537c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0eae77ff-4716-4b3e-94cf-cd767108537c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:48.729048 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.728981 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0eae77ff-4716-4b3e-94cf-cd767108537c-web-config\") pod \"alertmanager-main-0\" (UID: \"0eae77ff-4716-4b3e-94cf-cd767108537c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:48.729048 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.729014 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0eae77ff-4716-4b3e-94cf-cd767108537c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0eae77ff-4716-4b3e-94cf-cd767108537c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:48.729304 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.729062 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0eae77ff-4716-4b3e-94cf-cd767108537c-config-volume\") pod \"alertmanager-main-0\" (UID: \"0eae77ff-4716-4b3e-94cf-cd767108537c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:48.729304 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.729082 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0eae77ff-4716-4b3e-94cf-cd767108537c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0eae77ff-4716-4b3e-94cf-cd767108537c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:48.729304 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.729121 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0eae77ff-4716-4b3e-94cf-cd767108537c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0eae77ff-4716-4b3e-94cf-cd767108537c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:48.729304 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.729149 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0eae77ff-4716-4b3e-94cf-cd767108537c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0eae77ff-4716-4b3e-94cf-cd767108537c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:48.777677 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.777640 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 13:05:48.830400 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.830366 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0eae77ff-4716-4b3e-94cf-cd767108537c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0eae77ff-4716-4b3e-94cf-cd767108537c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:48.830400 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.830397 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0eae77ff-4716-4b3e-94cf-cd767108537c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0eae77ff-4716-4b3e-94cf-cd767108537c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:48.830658 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.830423 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0eae77ff-4716-4b3e-94cf-cd767108537c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0eae77ff-4716-4b3e-94cf-cd767108537c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:48.830658 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.830457 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0eae77ff-4716-4b3e-94cf-cd767108537c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0eae77ff-4716-4b3e-94cf-cd767108537c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:48.830658 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.830500 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0eae77ff-4716-4b3e-94cf-cd767108537c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0eae77ff-4716-4b3e-94cf-cd767108537c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:48.830658 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.830524 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7phj\" (UniqueName: \"kubernetes.io/projected/0eae77ff-4716-4b3e-94cf-cd767108537c-kube-api-access-h7phj\") pod \"alertmanager-main-0\" (UID: \"0eae77ff-4716-4b3e-94cf-cd767108537c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:48.830658 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.830540 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0eae77ff-4716-4b3e-94cf-cd767108537c-config-out\") pod \"alertmanager-main-0\" (UID: \"0eae77ff-4716-4b3e-94cf-cd767108537c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:48.830658 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.830569 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0eae77ff-4716-4b3e-94cf-cd767108537c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0eae77ff-4716-4b3e-94cf-cd767108537c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:48.830658 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.830595 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0eae77ff-4716-4b3e-94cf-cd767108537c-web-config\") pod \"alertmanager-main-0\" (UID: \"0eae77ff-4716-4b3e-94cf-cd767108537c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:48.830658 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.830632 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0eae77ff-4716-4b3e-94cf-cd767108537c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0eae77ff-4716-4b3e-94cf-cd767108537c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:48.831080 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.830680 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0eae77ff-4716-4b3e-94cf-cd767108537c-config-volume\") pod \"alertmanager-main-0\" (UID: \"0eae77ff-4716-4b3e-94cf-cd767108537c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:48.831080 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.830704 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0eae77ff-4716-4b3e-94cf-cd767108537c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0eae77ff-4716-4b3e-94cf-cd767108537c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:48.831080 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.830743 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0eae77ff-4716-4b3e-94cf-cd767108537c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0eae77ff-4716-4b3e-94cf-cd767108537c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:48.831080 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.830816 5979 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 13:05:48.831280 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.831110 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0eae77ff-4716-4b3e-94cf-cd767108537c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0eae77ff-4716-4b3e-94cf-cd767108537c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:48.831780 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.831698 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0eae77ff-4716-4b3e-94cf-cd767108537c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0eae77ff-4716-4b3e-94cf-cd767108537c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:48.831780 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.831737 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0eae77ff-4716-4b3e-94cf-cd767108537c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0eae77ff-4716-4b3e-94cf-cd767108537c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:48.833661 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.833638 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0eae77ff-4716-4b3e-94cf-cd767108537c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0eae77ff-4716-4b3e-94cf-cd767108537c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:48.833861 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.833834 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0eae77ff-4716-4b3e-94cf-cd767108537c-web-config\") pod \"alertmanager-main-0\" (UID: \"0eae77ff-4716-4b3e-94cf-cd767108537c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:48.834217 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.834190 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0eae77ff-4716-4b3e-94cf-cd767108537c-config-out\") pod \"alertmanager-main-0\" (UID: \"0eae77ff-4716-4b3e-94cf-cd767108537c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:48.834298 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.834236 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0eae77ff-4716-4b3e-94cf-cd767108537c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0eae77ff-4716-4b3e-94cf-cd767108537c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:48.834390 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.834366 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0eae77ff-4716-4b3e-94cf-cd767108537c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0eae77ff-4716-4b3e-94cf-cd767108537c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:48.834512 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.834495 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0eae77ff-4716-4b3e-94cf-cd767108537c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0eae77ff-4716-4b3e-94cf-cd767108537c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:48.834604 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.834583 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0eae77ff-4716-4b3e-94cf-cd767108537c-config-volume\") pod \"alertmanager-main-0\" (UID: \"0eae77ff-4716-4b3e-94cf-cd767108537c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:48.834652 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.834627 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0eae77ff-4716-4b3e-94cf-cd767108537c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0eae77ff-4716-4b3e-94cf-cd767108537c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:48.835290 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.835274 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0eae77ff-4716-4b3e-94cf-cd767108537c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0eae77ff-4716-4b3e-94cf-cd767108537c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:48.884205 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:48.884183 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7phj\" (UniqueName: \"kubernetes.io/projected/0eae77ff-4716-4b3e-94cf-cd767108537c-kube-api-access-h7phj\") pod \"alertmanager-main-0\" (UID: \"0eae77ff-4716-4b3e-94cf-cd767108537c\") " pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:49.183107 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:49.183030 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 03 13:05:49.654997 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:49.654956 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 03 13:05:49.662669 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:49.662639 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eae77ff_4716_4b3e_94cf_cd767108537c.slice/crio-9f44744711492d6970235fd4ebb80f3f702c2fd12c9295c5e72b7468b2b87faf WatchSource:0}: Error finding container 9f44744711492d6970235fd4ebb80f3f702c2fd12c9295c5e72b7468b2b87faf: Status 404 returned error can't find the container with id 9f44744711492d6970235fd4ebb80f3f702c2fd12c9295c5e72b7468b2b87faf Nov 03 13:05:50.098100 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:50.098023 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-59897cc7df-n8cb9"] Nov 03 13:05:50.100548 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:50.100533 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-59897cc7df-n8cb9" Nov 03 13:05:50.112896 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:50.112868 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7361d2a9-d089-4073-bbfb-e7978943fd88" path="/var/lib/kubelet/pods/7361d2a9-d089-4073-bbfb-e7978943fd88/volumes" Nov 03 13:05:50.149854 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:50.149828 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 03 13:05:50.149854 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:50.149849 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 03 13:05:50.150029 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:50.149828 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 03 13:05:50.150029 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:50.149878 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-qm27s\"" Nov 03 13:05:50.150104 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:50.150064 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 03 13:05:50.196147 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:50.196123 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 03 13:05:50.243416 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:50.243341 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/11a6e3d9-2422-4fdb-8c44-949500637239-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-59897cc7df-n8cb9\" (UID: \"11a6e3d9-2422-4fdb-8c44-949500637239\") " pod="openshift-monitoring/telemeter-client-59897cc7df-n8cb9" Nov 03 13:05:50.243609 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:50.243461 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgs47\" (UniqueName: \"kubernetes.io/projected/11a6e3d9-2422-4fdb-8c44-949500637239-kube-api-access-zgs47\") pod \"telemeter-client-59897cc7df-n8cb9\" (UID: \"11a6e3d9-2422-4fdb-8c44-949500637239\") " pod="openshift-monitoring/telemeter-client-59897cc7df-n8cb9" Nov 03 13:05:50.243609 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:50.243514 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11a6e3d9-2422-4fdb-8c44-949500637239-metrics-client-ca\") pod \"telemeter-client-59897cc7df-n8cb9\" (UID: \"11a6e3d9-2422-4fdb-8c44-949500637239\") " pod="openshift-monitoring/telemeter-client-59897cc7df-n8cb9" Nov 03 13:05:50.243741 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:50.243607 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11a6e3d9-2422-4fdb-8c44-949500637239-telemeter-trusted-ca-bundle\") pod \"telemeter-client-59897cc7df-n8cb9\" (UID: \"11a6e3d9-2422-4fdb-8c44-949500637239\") " pod="openshift-monitoring/telemeter-client-59897cc7df-n8cb9" Nov 03 13:05:50.243741 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:50.243622 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 03 13:05:50.243741 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:50.243644 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/11a6e3d9-2422-4fdb-8c44-949500637239-secret-telemeter-client\") pod \"telemeter-client-59897cc7df-n8cb9\" (UID: \"11a6e3d9-2422-4fdb-8c44-949500637239\") " pod="openshift-monitoring/telemeter-client-59897cc7df-n8cb9" Nov 03 13:05:50.243741 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:50.243715 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/11a6e3d9-2422-4fdb-8c44-949500637239-telemeter-client-tls\") pod \"telemeter-client-59897cc7df-n8cb9\" (UID: \"11a6e3d9-2422-4fdb-8c44-949500637239\") " pod="openshift-monitoring/telemeter-client-59897cc7df-n8cb9" Nov 03 13:05:50.243930 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:50.243767 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11a6e3d9-2422-4fdb-8c44-949500637239-serving-certs-ca-bundle\") pod \"telemeter-client-59897cc7df-n8cb9\" (UID: \"11a6e3d9-2422-4fdb-8c44-949500637239\") " pod="openshift-monitoring/telemeter-client-59897cc7df-n8cb9" Nov 03 13:05:50.243930 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:50.243814 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/11a6e3d9-2422-4fdb-8c44-949500637239-federate-client-tls\") pod \"telemeter-client-59897cc7df-n8cb9\" (UID: \"11a6e3d9-2422-4fdb-8c44-949500637239\") " pod="openshift-monitoring/telemeter-client-59897cc7df-n8cb9" Nov 03 13:05:50.244448 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:50.244427 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-59897cc7df-n8cb9"] Nov 03 13:05:50.344196 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:50.344159 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11a6e3d9-2422-4fdb-8c44-949500637239-telemeter-trusted-ca-bundle\") pod \"telemeter-client-59897cc7df-n8cb9\" (UID: \"11a6e3d9-2422-4fdb-8c44-949500637239\") " pod="openshift-monitoring/telemeter-client-59897cc7df-n8cb9" Nov 03 13:05:50.344196 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:50.344200 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/11a6e3d9-2422-4fdb-8c44-949500637239-secret-telemeter-client\") pod \"telemeter-client-59897cc7df-n8cb9\" (UID: \"11a6e3d9-2422-4fdb-8c44-949500637239\") " pod="openshift-monitoring/telemeter-client-59897cc7df-n8cb9" Nov 03 13:05:50.344447 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:50.344224 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/11a6e3d9-2422-4fdb-8c44-949500637239-telemeter-client-tls\") pod \"telemeter-client-59897cc7df-n8cb9\" (UID: \"11a6e3d9-2422-4fdb-8c44-949500637239\") " pod="openshift-monitoring/telemeter-client-59897cc7df-n8cb9" Nov 03 13:05:50.344447 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:50.344263 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11a6e3d9-2422-4fdb-8c44-949500637239-serving-certs-ca-bundle\") pod \"telemeter-client-59897cc7df-n8cb9\" (UID: \"11a6e3d9-2422-4fdb-8c44-949500637239\") " pod="openshift-monitoring/telemeter-client-59897cc7df-n8cb9" Nov 03 13:05:50.344447 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:50.344318 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/11a6e3d9-2422-4fdb-8c44-949500637239-federate-client-tls\") pod \"telemeter-client-59897cc7df-n8cb9\" (UID: \"11a6e3d9-2422-4fdb-8c44-949500637239\") " pod="openshift-monitoring/telemeter-client-59897cc7df-n8cb9" Nov 03 13:05:50.344447 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:50.344358 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/11a6e3d9-2422-4fdb-8c44-949500637239-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-59897cc7df-n8cb9\" (UID: \"11a6e3d9-2422-4fdb-8c44-949500637239\") " pod="openshift-monitoring/telemeter-client-59897cc7df-n8cb9" Nov 03 13:05:50.344447 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:50.344387 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgs47\" (UniqueName: \"kubernetes.io/projected/11a6e3d9-2422-4fdb-8c44-949500637239-kube-api-access-zgs47\") pod \"telemeter-client-59897cc7df-n8cb9\" (UID: \"11a6e3d9-2422-4fdb-8c44-949500637239\") " pod="openshift-monitoring/telemeter-client-59897cc7df-n8cb9" Nov 03 13:05:50.344447 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:50.344416 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11a6e3d9-2422-4fdb-8c44-949500637239-metrics-client-ca\") pod \"telemeter-client-59897cc7df-n8cb9\" (UID: \"11a6e3d9-2422-4fdb-8c44-949500637239\") " pod="openshift-monitoring/telemeter-client-59897cc7df-n8cb9" Nov 03 13:05:50.345040 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:50.345009 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11a6e3d9-2422-4fdb-8c44-949500637239-serving-certs-ca-bundle\") pod \"telemeter-client-59897cc7df-n8cb9\" (UID: \"11a6e3d9-2422-4fdb-8c44-949500637239\") " pod="openshift-monitoring/telemeter-client-59897cc7df-n8cb9" Nov 03 13:05:50.345169 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:50.345094 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11a6e3d9-2422-4fdb-8c44-949500637239-metrics-client-ca\") pod \"telemeter-client-59897cc7df-n8cb9\" (UID: \"11a6e3d9-2422-4fdb-8c44-949500637239\") " pod="openshift-monitoring/telemeter-client-59897cc7df-n8cb9" Nov 03 13:05:50.345280 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:50.345259 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11a6e3d9-2422-4fdb-8c44-949500637239-telemeter-trusted-ca-bundle\") pod \"telemeter-client-59897cc7df-n8cb9\" (UID: \"11a6e3d9-2422-4fdb-8c44-949500637239\") " pod="openshift-monitoring/telemeter-client-59897cc7df-n8cb9" Nov 03 13:05:50.345815 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:50.345783 5979 generic.go:356] "Generic (PLEG): container finished" podID="0eae77ff-4716-4b3e-94cf-cd767108537c" containerID="6e4a9714dd44d1f2c92fbaa9044d411f830f045f9e41642b68c99d8b098e9295" exitCode=0 Nov 03 13:05:50.345922 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:50.345824 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0eae77ff-4716-4b3e-94cf-cd767108537c","Type":"ContainerDied","Data":"6e4a9714dd44d1f2c92fbaa9044d411f830f045f9e41642b68c99d8b098e9295"} Nov 03 13:05:50.345922 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:50.345886 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0eae77ff-4716-4b3e-94cf-cd767108537c","Type":"ContainerStarted","Data":"9f44744711492d6970235fd4ebb80f3f702c2fd12c9295c5e72b7468b2b87faf"} Nov 03 13:05:50.347043 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:50.347013 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/11a6e3d9-2422-4fdb-8c44-949500637239-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-59897cc7df-n8cb9\" (UID: \"11a6e3d9-2422-4fdb-8c44-949500637239\") " pod="openshift-monitoring/telemeter-client-59897cc7df-n8cb9" Nov 03 13:05:50.347252 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:50.347234 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/11a6e3d9-2422-4fdb-8c44-949500637239-telemeter-client-tls\") pod \"telemeter-client-59897cc7df-n8cb9\" (UID: \"11a6e3d9-2422-4fdb-8c44-949500637239\") " pod="openshift-monitoring/telemeter-client-59897cc7df-n8cb9" Nov 03 13:05:50.347311 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:50.347263 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/11a6e3d9-2422-4fdb-8c44-949500637239-federate-client-tls\") pod \"telemeter-client-59897cc7df-n8cb9\" (UID: \"11a6e3d9-2422-4fdb-8c44-949500637239\") " pod="openshift-monitoring/telemeter-client-59897cc7df-n8cb9" Nov 03 13:05:50.347311 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:50.347266 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/11a6e3d9-2422-4fdb-8c44-949500637239-secret-telemeter-client\") pod \"telemeter-client-59897cc7df-n8cb9\" (UID: \"11a6e3d9-2422-4fdb-8c44-949500637239\") " pod="openshift-monitoring/telemeter-client-59897cc7df-n8cb9" Nov 03 13:05:50.398959 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:50.398936 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgs47\" (UniqueName: \"kubernetes.io/projected/11a6e3d9-2422-4fdb-8c44-949500637239-kube-api-access-zgs47\") pod \"telemeter-client-59897cc7df-n8cb9\" (UID: \"11a6e3d9-2422-4fdb-8c44-949500637239\") " pod="openshift-monitoring/telemeter-client-59897cc7df-n8cb9" Nov 03 13:05:50.414841 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:50.414813 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-59897cc7df-n8cb9" Nov 03 13:05:50.841484 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:50.841445 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-59897cc7df-n8cb9"] Nov 03 13:05:50.849434 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:50.849405 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11a6e3d9_2422_4fdb_8c44_949500637239.slice/crio-f8cbe766cb41523fa3345b826531abe68f9b1fb1f61b39c40bea66cf80b6b154 WatchSource:0}: Error finding container f8cbe766cb41523fa3345b826531abe68f9b1fb1f61b39c40bea66cf80b6b154: Status 404 returned error can't find the container with id f8cbe766cb41523fa3345b826531abe68f9b1fb1f61b39c40bea66cf80b6b154 Nov 03 13:05:50.851938 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:50.851917 5979 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 13:05:51.353240 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:51.353198 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0eae77ff-4716-4b3e-94cf-cd767108537c","Type":"ContainerStarted","Data":"548d7b38596a0071c69a53ce01c2c8b9191968282a3da57a7818a57639cfa916"} Nov 03 13:05:51.353717 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:51.353277 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0eae77ff-4716-4b3e-94cf-cd767108537c","Type":"ContainerStarted","Data":"5dce23bd9faf988484c9da3232cf9c067a4ea30fa8b9dd437bdfc9c36ccff7b1"} Nov 03 13:05:51.353717 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:51.353306 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0eae77ff-4716-4b3e-94cf-cd767108537c","Type":"ContainerStarted","Data":"22f32fb7cf334c8719a8ec003a004f2f5e49cf341424abc7ea8b0a8fdf719afa"} Nov 03 13:05:51.353717 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:51.353341 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0eae77ff-4716-4b3e-94cf-cd767108537c","Type":"ContainerStarted","Data":"7586bd41dee1603bd53cd560c5b19ba34f11e4c070b846e84f421901202921dc"} Nov 03 13:05:51.353717 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:51.353376 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0eae77ff-4716-4b3e-94cf-cd767108537c","Type":"ContainerStarted","Data":"81fb2b066e52fae35066066c57d462c704bf45382348c3369cbf1ee9429600d7"} Nov 03 13:05:51.353717 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:51.353404 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0eae77ff-4716-4b3e-94cf-cd767108537c","Type":"ContainerStarted","Data":"de0e8b2c85cbd670a0e14f5ad26cd815d5e50d328ede7e6a823e78139a9e029b"} Nov 03 13:05:51.354398 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:51.354379 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-59897cc7df-n8cb9" event={"ID":"11a6e3d9-2422-4fdb-8c44-949500637239","Type":"ContainerStarted","Data":"f8cbe766cb41523fa3345b826531abe68f9b1fb1f61b39c40bea66cf80b6b154"} Nov 03 13:05:51.469914 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:51.469718 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.457492523 podStartE2EDuration="3.469701376s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.208853ms" isStatefulPod=false podCreationTimestamp="2026-11-03 13:05:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:05:51.46953868 +0000 UTC m=+33.480226925" watchObservedRunningTime="2026-11-03 13:05:51.469701376 +0000 UTC m=+33.480389621" Nov 03 13:05:52.024681 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.024648 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 13:05:52.025020 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.024985 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="30d608b6-d82e-4fcf-ab44-b8e064c463d0" containerName="prometheus" containerID="cri-o://e797acde2976802df9507cb8adea62190a05660ef0fced2b2afe7cb4075fdc72" gracePeriod=600 Nov 03 13:05:52.025158 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.025025 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="30d608b6-d82e-4fcf-ab44-b8e064c463d0" containerName="kube-rbac-proxy" containerID="cri-o://be6f49e9d541c310aac05d244ec6ee360d277537e656c034e6099e963b47149f" gracePeriod=600 Nov 03 13:05:52.025158 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.025042 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="30d608b6-d82e-4fcf-ab44-b8e064c463d0" containerName="kube-rbac-proxy-web" containerID="cri-o://040ae9aec0f27bf15e42a48919412f565f4fa698b28cf15787eabe7b4fabe2f9" gracePeriod=600 Nov 03 13:05:52.025158 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.025034 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="30d608b6-d82e-4fcf-ab44-b8e064c463d0" containerName="thanos-sidecar" containerID="cri-o://9861343ab263f81c1dbee8304feff9c17d8de8b8d176c47e5196c2b17e6287ce" gracePeriod=600 Nov 03 13:05:52.025293 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.025160 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="30d608b6-d82e-4fcf-ab44-b8e064c463d0" containerName="kube-rbac-proxy-thanos" containerID="cri-o://3bb5b563b33e764f2cacb6395268c2a267d365ce2eb2be0658dbef1abe48d34a" gracePeriod=600 Nov 03 13:05:52.025293 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.025160 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="30d608b6-d82e-4fcf-ab44-b8e064c463d0" containerName="config-reloader" containerID="cri-o://0acd1240c69ae69c18d842f9defe65913faf7126e2d9c432a6199e0da68b4570" gracePeriod=600 Nov 03 13:05:52.243494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.243449 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9v5mk" Nov 03 13:05:52.302830 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.302805 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:52.363150 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.363119 5979 generic.go:356] "Generic (PLEG): container finished" podID="30d608b6-d82e-4fcf-ab44-b8e064c463d0" containerID="3bb5b563b33e764f2cacb6395268c2a267d365ce2eb2be0658dbef1abe48d34a" exitCode=0 Nov 03 13:05:52.363150 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.363142 5979 generic.go:356] "Generic (PLEG): container finished" podID="30d608b6-d82e-4fcf-ab44-b8e064c463d0" containerID="be6f49e9d541c310aac05d244ec6ee360d277537e656c034e6099e963b47149f" exitCode=0 Nov 03 13:05:52.363150 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.363149 5979 generic.go:356] "Generic (PLEG): container finished" podID="30d608b6-d82e-4fcf-ab44-b8e064c463d0" containerID="040ae9aec0f27bf15e42a48919412f565f4fa698b28cf15787eabe7b4fabe2f9" exitCode=0 Nov 03 13:05:52.363150 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.363156 5979 generic.go:356] "Generic (PLEG): container finished" podID="30d608b6-d82e-4fcf-ab44-b8e064c463d0" containerID="9861343ab263f81c1dbee8304feff9c17d8de8b8d176c47e5196c2b17e6287ce" exitCode=0 Nov 03 13:05:52.363790 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.363164 5979 generic.go:356] "Generic (PLEG): container finished" podID="30d608b6-d82e-4fcf-ab44-b8e064c463d0" containerID="0acd1240c69ae69c18d842f9defe65913faf7126e2d9c432a6199e0da68b4570" exitCode=0 Nov 03 13:05:52.363790 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.363170 5979 generic.go:356] "Generic (PLEG): container finished" podID="30d608b6-d82e-4fcf-ab44-b8e064c463d0" containerID="e797acde2976802df9507cb8adea62190a05660ef0fced2b2afe7cb4075fdc72" exitCode=0 Nov 03 13:05:52.363790 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.363183 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"30d608b6-d82e-4fcf-ab44-b8e064c463d0","Type":"ContainerDied","Data":"3bb5b563b33e764f2cacb6395268c2a267d365ce2eb2be0658dbef1abe48d34a"} Nov 03 13:05:52.363790 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.363208 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:52.363790 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.363269 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"30d608b6-d82e-4fcf-ab44-b8e064c463d0","Type":"ContainerDied","Data":"be6f49e9d541c310aac05d244ec6ee360d277537e656c034e6099e963b47149f"} Nov 03 13:05:52.363790 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.363313 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"30d608b6-d82e-4fcf-ab44-b8e064c463d0","Type":"ContainerDied","Data":"040ae9aec0f27bf15e42a48919412f565f4fa698b28cf15787eabe7b4fabe2f9"} Nov 03 13:05:52.363790 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.363332 5979 scope.go:122] "RemoveContainer" containerID="3bb5b563b33e764f2cacb6395268c2a267d365ce2eb2be0658dbef1abe48d34a" Nov 03 13:05:52.363790 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.363354 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"30d608b6-d82e-4fcf-ab44-b8e064c463d0","Type":"ContainerDied","Data":"9861343ab263f81c1dbee8304feff9c17d8de8b8d176c47e5196c2b17e6287ce"} Nov 03 13:05:52.363790 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.363393 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"30d608b6-d82e-4fcf-ab44-b8e064c463d0","Type":"ContainerDied","Data":"0acd1240c69ae69c18d842f9defe65913faf7126e2d9c432a6199e0da68b4570"} Nov 03 13:05:52.363790 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.363437 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"30d608b6-d82e-4fcf-ab44-b8e064c463d0","Type":"ContainerDied","Data":"e797acde2976802df9507cb8adea62190a05660ef0fced2b2afe7cb4075fdc72"} Nov 03 13:05:52.363790 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.363493 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"30d608b6-d82e-4fcf-ab44-b8e064c463d0","Type":"ContainerDied","Data":"cfe8a5916ef9fe15a5e0a8f79b48c137087a707357efd660e29b358bd865bb40"} Nov 03 13:05:52.372641 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.372626 5979 scope.go:122] "RemoveContainer" containerID="be6f49e9d541c310aac05d244ec6ee360d277537e656c034e6099e963b47149f" Nov 03 13:05:52.380980 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.380955 5979 scope.go:122] "RemoveContainer" containerID="040ae9aec0f27bf15e42a48919412f565f4fa698b28cf15787eabe7b4fabe2f9" Nov 03 13:05:52.396275 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.396252 5979 scope.go:122] "RemoveContainer" containerID="9861343ab263f81c1dbee8304feff9c17d8de8b8d176c47e5196c2b17e6287ce" Nov 03 13:05:52.406614 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.406568 5979 scope.go:122] "RemoveContainer" containerID="0acd1240c69ae69c18d842f9defe65913faf7126e2d9c432a6199e0da68b4570" Nov 03 13:05:52.415354 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.415326 5979 scope.go:122] "RemoveContainer" containerID="e797acde2976802df9507cb8adea62190a05660ef0fced2b2afe7cb4075fdc72" Nov 03 13:05:52.424807 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.424787 5979 scope.go:122] "RemoveContainer" containerID="1caee93a0441c7c697a1ab2aa46e921bc1aca35c3270332e260a2665db7555a6" Nov 03 13:05:52.433112 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.433095 5979 scope.go:122] "RemoveContainer" containerID="3bb5b563b33e764f2cacb6395268c2a267d365ce2eb2be0658dbef1abe48d34a" Nov 03 13:05:52.433374 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:05:52.433356 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bb5b563b33e764f2cacb6395268c2a267d365ce2eb2be0658dbef1abe48d34a\": container with ID starting with 3bb5b563b33e764f2cacb6395268c2a267d365ce2eb2be0658dbef1abe48d34a not found: ID does not exist" containerID="3bb5b563b33e764f2cacb6395268c2a267d365ce2eb2be0658dbef1abe48d34a" Nov 03 13:05:52.433443 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.433386 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bb5b563b33e764f2cacb6395268c2a267d365ce2eb2be0658dbef1abe48d34a"} err="failed to get container status \"3bb5b563b33e764f2cacb6395268c2a267d365ce2eb2be0658dbef1abe48d34a\": rpc error: code = NotFound desc = could not find container \"3bb5b563b33e764f2cacb6395268c2a267d365ce2eb2be0658dbef1abe48d34a\": container with ID starting with 3bb5b563b33e764f2cacb6395268c2a267d365ce2eb2be0658dbef1abe48d34a not found: ID does not exist" Nov 03 13:05:52.433443 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.433408 5979 scope.go:122] "RemoveContainer" containerID="be6f49e9d541c310aac05d244ec6ee360d277537e656c034e6099e963b47149f" Nov 03 13:05:52.433710 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:05:52.433692 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be6f49e9d541c310aac05d244ec6ee360d277537e656c034e6099e963b47149f\": container with ID starting with be6f49e9d541c310aac05d244ec6ee360d277537e656c034e6099e963b47149f not found: ID does not exist" containerID="be6f49e9d541c310aac05d244ec6ee360d277537e656c034e6099e963b47149f" Nov 03 13:05:52.433749 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.433715 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be6f49e9d541c310aac05d244ec6ee360d277537e656c034e6099e963b47149f"} err="failed to get container status \"be6f49e9d541c310aac05d244ec6ee360d277537e656c034e6099e963b47149f\": rpc error: code = NotFound desc = could not find container \"be6f49e9d541c310aac05d244ec6ee360d277537e656c034e6099e963b47149f\": container with ID starting with be6f49e9d541c310aac05d244ec6ee360d277537e656c034e6099e963b47149f not found: ID does not exist" Nov 03 13:05:52.433749 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.433729 5979 scope.go:122] "RemoveContainer" containerID="040ae9aec0f27bf15e42a48919412f565f4fa698b28cf15787eabe7b4fabe2f9" Nov 03 13:05:52.434077 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:05:52.434052 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"040ae9aec0f27bf15e42a48919412f565f4fa698b28cf15787eabe7b4fabe2f9\": container with ID starting with 040ae9aec0f27bf15e42a48919412f565f4fa698b28cf15787eabe7b4fabe2f9 not found: ID does not exist" containerID="040ae9aec0f27bf15e42a48919412f565f4fa698b28cf15787eabe7b4fabe2f9" Nov 03 13:05:52.434163 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.434085 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"040ae9aec0f27bf15e42a48919412f565f4fa698b28cf15787eabe7b4fabe2f9"} err="failed to get container status \"040ae9aec0f27bf15e42a48919412f565f4fa698b28cf15787eabe7b4fabe2f9\": rpc error: code = NotFound desc = could not find container \"040ae9aec0f27bf15e42a48919412f565f4fa698b28cf15787eabe7b4fabe2f9\": container with ID starting with 040ae9aec0f27bf15e42a48919412f565f4fa698b28cf15787eabe7b4fabe2f9 not found: ID does not exist" Nov 03 13:05:52.434163 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.434107 5979 scope.go:122] "RemoveContainer" containerID="9861343ab263f81c1dbee8304feff9c17d8de8b8d176c47e5196c2b17e6287ce" Nov 03 13:05:52.434356 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:05:52.434335 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9861343ab263f81c1dbee8304feff9c17d8de8b8d176c47e5196c2b17e6287ce\": container with ID starting with 9861343ab263f81c1dbee8304feff9c17d8de8b8d176c47e5196c2b17e6287ce not found: ID does not exist" containerID="9861343ab263f81c1dbee8304feff9c17d8de8b8d176c47e5196c2b17e6287ce" Nov 03 13:05:52.434403 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.434363 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9861343ab263f81c1dbee8304feff9c17d8de8b8d176c47e5196c2b17e6287ce"} err="failed to get container status \"9861343ab263f81c1dbee8304feff9c17d8de8b8d176c47e5196c2b17e6287ce\": rpc error: code = NotFound desc = could not find container \"9861343ab263f81c1dbee8304feff9c17d8de8b8d176c47e5196c2b17e6287ce\": container with ID starting with 9861343ab263f81c1dbee8304feff9c17d8de8b8d176c47e5196c2b17e6287ce not found: ID does not exist" Nov 03 13:05:52.434403 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.434381 5979 scope.go:122] "RemoveContainer" containerID="0acd1240c69ae69c18d842f9defe65913faf7126e2d9c432a6199e0da68b4570" Nov 03 13:05:52.434693 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:05:52.434675 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0acd1240c69ae69c18d842f9defe65913faf7126e2d9c432a6199e0da68b4570\": container with ID starting with 0acd1240c69ae69c18d842f9defe65913faf7126e2d9c432a6199e0da68b4570 not found: ID does not exist" containerID="0acd1240c69ae69c18d842f9defe65913faf7126e2d9c432a6199e0da68b4570" Nov 03 13:05:52.434769 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.434705 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0acd1240c69ae69c18d842f9defe65913faf7126e2d9c432a6199e0da68b4570"} err="failed to get container status \"0acd1240c69ae69c18d842f9defe65913faf7126e2d9c432a6199e0da68b4570\": rpc error: code = NotFound desc = could not find container \"0acd1240c69ae69c18d842f9defe65913faf7126e2d9c432a6199e0da68b4570\": container with ID starting with 0acd1240c69ae69c18d842f9defe65913faf7126e2d9c432a6199e0da68b4570 not found: ID does not exist" Nov 03 13:05:52.434769 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.434720 5979 scope.go:122] "RemoveContainer" containerID="e797acde2976802df9507cb8adea62190a05660ef0fced2b2afe7cb4075fdc72" Nov 03 13:05:52.434959 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:05:52.434943 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e797acde2976802df9507cb8adea62190a05660ef0fced2b2afe7cb4075fdc72\": container with ID starting with e797acde2976802df9507cb8adea62190a05660ef0fced2b2afe7cb4075fdc72 not found: ID does not exist" containerID="e797acde2976802df9507cb8adea62190a05660ef0fced2b2afe7cb4075fdc72" Nov 03 13:05:52.435020 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.434960 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e797acde2976802df9507cb8adea62190a05660ef0fced2b2afe7cb4075fdc72"} err="failed to get container status \"e797acde2976802df9507cb8adea62190a05660ef0fced2b2afe7cb4075fdc72\": rpc error: code = NotFound desc = could not find container \"e797acde2976802df9507cb8adea62190a05660ef0fced2b2afe7cb4075fdc72\": container with ID starting with e797acde2976802df9507cb8adea62190a05660ef0fced2b2afe7cb4075fdc72 not found: ID does not exist" Nov 03 13:05:52.435020 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.434980 5979 scope.go:122] "RemoveContainer" containerID="1caee93a0441c7c697a1ab2aa46e921bc1aca35c3270332e260a2665db7555a6" Nov 03 13:05:52.435215 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:05:52.435197 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1caee93a0441c7c697a1ab2aa46e921bc1aca35c3270332e260a2665db7555a6\": container with ID starting with 1caee93a0441c7c697a1ab2aa46e921bc1aca35c3270332e260a2665db7555a6 not found: ID does not exist" containerID="1caee93a0441c7c697a1ab2aa46e921bc1aca35c3270332e260a2665db7555a6" Nov 03 13:05:52.435277 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.435219 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1caee93a0441c7c697a1ab2aa46e921bc1aca35c3270332e260a2665db7555a6"} err="failed to get container status \"1caee93a0441c7c697a1ab2aa46e921bc1aca35c3270332e260a2665db7555a6\": rpc error: code = NotFound desc = could not find container \"1caee93a0441c7c697a1ab2aa46e921bc1aca35c3270332e260a2665db7555a6\": container with ID starting with 1caee93a0441c7c697a1ab2aa46e921bc1aca35c3270332e260a2665db7555a6 not found: ID does not exist" Nov 03 13:05:52.435277 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.435232 5979 scope.go:122] "RemoveContainer" containerID="3bb5b563b33e764f2cacb6395268c2a267d365ce2eb2be0658dbef1abe48d34a" Nov 03 13:05:52.435513 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.435488 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bb5b563b33e764f2cacb6395268c2a267d365ce2eb2be0658dbef1abe48d34a"} err="failed to get container status \"3bb5b563b33e764f2cacb6395268c2a267d365ce2eb2be0658dbef1abe48d34a\": rpc error: code = NotFound desc = could not find container \"3bb5b563b33e764f2cacb6395268c2a267d365ce2eb2be0658dbef1abe48d34a\": container with ID starting with 3bb5b563b33e764f2cacb6395268c2a267d365ce2eb2be0658dbef1abe48d34a not found: ID does not exist" Nov 03 13:05:52.435513 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.435513 5979 scope.go:122] "RemoveContainer" containerID="be6f49e9d541c310aac05d244ec6ee360d277537e656c034e6099e963b47149f" Nov 03 13:05:52.435752 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.435724 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be6f49e9d541c310aac05d244ec6ee360d277537e656c034e6099e963b47149f"} err="failed to get container status \"be6f49e9d541c310aac05d244ec6ee360d277537e656c034e6099e963b47149f\": rpc error: code = NotFound desc = could not find container \"be6f49e9d541c310aac05d244ec6ee360d277537e656c034e6099e963b47149f\": container with ID starting with be6f49e9d541c310aac05d244ec6ee360d277537e656c034e6099e963b47149f not found: ID does not exist" Nov 03 13:05:52.435752 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.435745 5979 scope.go:122] "RemoveContainer" containerID="040ae9aec0f27bf15e42a48919412f565f4fa698b28cf15787eabe7b4fabe2f9" Nov 03 13:05:52.436057 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.436033 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"040ae9aec0f27bf15e42a48919412f565f4fa698b28cf15787eabe7b4fabe2f9"} err="failed to get container status \"040ae9aec0f27bf15e42a48919412f565f4fa698b28cf15787eabe7b4fabe2f9\": rpc error: code = NotFound desc = could not find container \"040ae9aec0f27bf15e42a48919412f565f4fa698b28cf15787eabe7b4fabe2f9\": container with ID starting with 040ae9aec0f27bf15e42a48919412f565f4fa698b28cf15787eabe7b4fabe2f9 not found: ID does not exist" Nov 03 13:05:52.436133 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.436057 5979 scope.go:122] "RemoveContainer" containerID="9861343ab263f81c1dbee8304feff9c17d8de8b8d176c47e5196c2b17e6287ce" Nov 03 13:05:52.436297 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.436273 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9861343ab263f81c1dbee8304feff9c17d8de8b8d176c47e5196c2b17e6287ce"} err="failed to get container status \"9861343ab263f81c1dbee8304feff9c17d8de8b8d176c47e5196c2b17e6287ce\": rpc error: code = NotFound desc = could not find container \"9861343ab263f81c1dbee8304feff9c17d8de8b8d176c47e5196c2b17e6287ce\": container with ID starting with 9861343ab263f81c1dbee8304feff9c17d8de8b8d176c47e5196c2b17e6287ce not found: ID does not exist" Nov 03 13:05:52.436350 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.436299 5979 scope.go:122] "RemoveContainer" containerID="0acd1240c69ae69c18d842f9defe65913faf7126e2d9c432a6199e0da68b4570" Nov 03 13:05:52.436577 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.436552 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0acd1240c69ae69c18d842f9defe65913faf7126e2d9c432a6199e0da68b4570"} err="failed to get container status \"0acd1240c69ae69c18d842f9defe65913faf7126e2d9c432a6199e0da68b4570\": rpc error: code = NotFound desc = could not find container \"0acd1240c69ae69c18d842f9defe65913faf7126e2d9c432a6199e0da68b4570\": container with ID starting with 0acd1240c69ae69c18d842f9defe65913faf7126e2d9c432a6199e0da68b4570 not found: ID does not exist" Nov 03 13:05:52.436668 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.436577 5979 scope.go:122] "RemoveContainer" containerID="e797acde2976802df9507cb8adea62190a05660ef0fced2b2afe7cb4075fdc72" Nov 03 13:05:52.436824 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.436798 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e797acde2976802df9507cb8adea62190a05660ef0fced2b2afe7cb4075fdc72"} err="failed to get container status \"e797acde2976802df9507cb8adea62190a05660ef0fced2b2afe7cb4075fdc72\": rpc error: code = NotFound desc = could not find container \"e797acde2976802df9507cb8adea62190a05660ef0fced2b2afe7cb4075fdc72\": container with ID starting with e797acde2976802df9507cb8adea62190a05660ef0fced2b2afe7cb4075fdc72 not found: ID does not exist" Nov 03 13:05:52.436824 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.436822 5979 scope.go:122] "RemoveContainer" containerID="1caee93a0441c7c697a1ab2aa46e921bc1aca35c3270332e260a2665db7555a6" Nov 03 13:05:52.437062 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.437043 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1caee93a0441c7c697a1ab2aa46e921bc1aca35c3270332e260a2665db7555a6"} err="failed to get container status \"1caee93a0441c7c697a1ab2aa46e921bc1aca35c3270332e260a2665db7555a6\": rpc error: code = NotFound desc = could not find container \"1caee93a0441c7c697a1ab2aa46e921bc1aca35c3270332e260a2665db7555a6\": container with ID starting with 1caee93a0441c7c697a1ab2aa46e921bc1aca35c3270332e260a2665db7555a6 not found: ID does not exist" Nov 03 13:05:52.437119 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.437063 5979 scope.go:122] "RemoveContainer" containerID="3bb5b563b33e764f2cacb6395268c2a267d365ce2eb2be0658dbef1abe48d34a" Nov 03 13:05:52.437287 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.437267 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bb5b563b33e764f2cacb6395268c2a267d365ce2eb2be0658dbef1abe48d34a"} err="failed to get container status \"3bb5b563b33e764f2cacb6395268c2a267d365ce2eb2be0658dbef1abe48d34a\": rpc error: code = NotFound desc = could not find container \"3bb5b563b33e764f2cacb6395268c2a267d365ce2eb2be0658dbef1abe48d34a\": container with ID starting with 3bb5b563b33e764f2cacb6395268c2a267d365ce2eb2be0658dbef1abe48d34a not found: ID does not exist" Nov 03 13:05:52.437361 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.437288 5979 scope.go:122] "RemoveContainer" containerID="be6f49e9d541c310aac05d244ec6ee360d277537e656c034e6099e963b47149f" Nov 03 13:05:52.437609 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.437591 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be6f49e9d541c310aac05d244ec6ee360d277537e656c034e6099e963b47149f"} err="failed to get container status \"be6f49e9d541c310aac05d244ec6ee360d277537e656c034e6099e963b47149f\": rpc error: code = NotFound desc = could not find container \"be6f49e9d541c310aac05d244ec6ee360d277537e656c034e6099e963b47149f\": container with ID starting with be6f49e9d541c310aac05d244ec6ee360d277537e656c034e6099e963b47149f not found: ID does not exist" Nov 03 13:05:52.437609 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.437609 5979 scope.go:122] "RemoveContainer" containerID="040ae9aec0f27bf15e42a48919412f565f4fa698b28cf15787eabe7b4fabe2f9" Nov 03 13:05:52.437847 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.437827 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"040ae9aec0f27bf15e42a48919412f565f4fa698b28cf15787eabe7b4fabe2f9"} err="failed to get container status \"040ae9aec0f27bf15e42a48919412f565f4fa698b28cf15787eabe7b4fabe2f9\": rpc error: code = NotFound desc = could not find container \"040ae9aec0f27bf15e42a48919412f565f4fa698b28cf15787eabe7b4fabe2f9\": container with ID starting with 040ae9aec0f27bf15e42a48919412f565f4fa698b28cf15787eabe7b4fabe2f9 not found: ID does not exist" Nov 03 13:05:52.437922 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.437849 5979 scope.go:122] "RemoveContainer" containerID="9861343ab263f81c1dbee8304feff9c17d8de8b8d176c47e5196c2b17e6287ce" Nov 03 13:05:52.438106 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.438079 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9861343ab263f81c1dbee8304feff9c17d8de8b8d176c47e5196c2b17e6287ce"} err="failed to get container status \"9861343ab263f81c1dbee8304feff9c17d8de8b8d176c47e5196c2b17e6287ce\": rpc error: code = NotFound desc = could not find container \"9861343ab263f81c1dbee8304feff9c17d8de8b8d176c47e5196c2b17e6287ce\": container with ID starting with 9861343ab263f81c1dbee8304feff9c17d8de8b8d176c47e5196c2b17e6287ce not found: ID does not exist" Nov 03 13:05:52.438106 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.438106 5979 scope.go:122] "RemoveContainer" containerID="0acd1240c69ae69c18d842f9defe65913faf7126e2d9c432a6199e0da68b4570" Nov 03 13:05:52.438415 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.438395 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0acd1240c69ae69c18d842f9defe65913faf7126e2d9c432a6199e0da68b4570"} err="failed to get container status \"0acd1240c69ae69c18d842f9defe65913faf7126e2d9c432a6199e0da68b4570\": rpc error: code = NotFound desc = could not find container \"0acd1240c69ae69c18d842f9defe65913faf7126e2d9c432a6199e0da68b4570\": container with ID starting with 0acd1240c69ae69c18d842f9defe65913faf7126e2d9c432a6199e0da68b4570 not found: ID does not exist" Nov 03 13:05:52.438497 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.438422 5979 scope.go:122] "RemoveContainer" containerID="e797acde2976802df9507cb8adea62190a05660ef0fced2b2afe7cb4075fdc72" Nov 03 13:05:52.438673 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.438650 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e797acde2976802df9507cb8adea62190a05660ef0fced2b2afe7cb4075fdc72"} err="failed to get container status \"e797acde2976802df9507cb8adea62190a05660ef0fced2b2afe7cb4075fdc72\": rpc error: code = NotFound desc = could not find container \"e797acde2976802df9507cb8adea62190a05660ef0fced2b2afe7cb4075fdc72\": container with ID starting with e797acde2976802df9507cb8adea62190a05660ef0fced2b2afe7cb4075fdc72 not found: ID does not exist" Nov 03 13:05:52.438673 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.438672 5979 scope.go:122] "RemoveContainer" containerID="1caee93a0441c7c697a1ab2aa46e921bc1aca35c3270332e260a2665db7555a6" Nov 03 13:05:52.438980 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.438956 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1caee93a0441c7c697a1ab2aa46e921bc1aca35c3270332e260a2665db7555a6"} err="failed to get container status \"1caee93a0441c7c697a1ab2aa46e921bc1aca35c3270332e260a2665db7555a6\": rpc error: code = NotFound desc = could not find container \"1caee93a0441c7c697a1ab2aa46e921bc1aca35c3270332e260a2665db7555a6\": container with ID starting with 1caee93a0441c7c697a1ab2aa46e921bc1aca35c3270332e260a2665db7555a6 not found: ID does not exist" Nov 03 13:05:52.439055 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.438982 5979 scope.go:122] "RemoveContainer" containerID="3bb5b563b33e764f2cacb6395268c2a267d365ce2eb2be0658dbef1abe48d34a" Nov 03 13:05:52.439266 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.439242 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bb5b563b33e764f2cacb6395268c2a267d365ce2eb2be0658dbef1abe48d34a"} err="failed to get container status \"3bb5b563b33e764f2cacb6395268c2a267d365ce2eb2be0658dbef1abe48d34a\": rpc error: code = NotFound desc = could not find container \"3bb5b563b33e764f2cacb6395268c2a267d365ce2eb2be0658dbef1abe48d34a\": container with ID starting with 3bb5b563b33e764f2cacb6395268c2a267d365ce2eb2be0658dbef1abe48d34a not found: ID does not exist" Nov 03 13:05:52.439266 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.439266 5979 scope.go:122] "RemoveContainer" containerID="be6f49e9d541c310aac05d244ec6ee360d277537e656c034e6099e963b47149f" Nov 03 13:05:52.439626 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.439498 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be6f49e9d541c310aac05d244ec6ee360d277537e656c034e6099e963b47149f"} err="failed to get container status \"be6f49e9d541c310aac05d244ec6ee360d277537e656c034e6099e963b47149f\": rpc error: code = NotFound desc = could not find container \"be6f49e9d541c310aac05d244ec6ee360d277537e656c034e6099e963b47149f\": container with ID starting with be6f49e9d541c310aac05d244ec6ee360d277537e656c034e6099e963b47149f not found: ID does not exist" Nov 03 13:05:52.439626 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.439517 5979 scope.go:122] "RemoveContainer" containerID="040ae9aec0f27bf15e42a48919412f565f4fa698b28cf15787eabe7b4fabe2f9" Nov 03 13:05:52.439802 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.439783 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"040ae9aec0f27bf15e42a48919412f565f4fa698b28cf15787eabe7b4fabe2f9"} err="failed to get container status \"040ae9aec0f27bf15e42a48919412f565f4fa698b28cf15787eabe7b4fabe2f9\": rpc error: code = NotFound desc = could not find container \"040ae9aec0f27bf15e42a48919412f565f4fa698b28cf15787eabe7b4fabe2f9\": container with ID starting with 040ae9aec0f27bf15e42a48919412f565f4fa698b28cf15787eabe7b4fabe2f9 not found: ID does not exist" Nov 03 13:05:52.439850 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.439805 5979 scope.go:122] "RemoveContainer" containerID="9861343ab263f81c1dbee8304feff9c17d8de8b8d176c47e5196c2b17e6287ce" Nov 03 13:05:52.440049 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.440032 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9861343ab263f81c1dbee8304feff9c17d8de8b8d176c47e5196c2b17e6287ce"} err="failed to get container status \"9861343ab263f81c1dbee8304feff9c17d8de8b8d176c47e5196c2b17e6287ce\": rpc error: code = NotFound desc = could not find container \"9861343ab263f81c1dbee8304feff9c17d8de8b8d176c47e5196c2b17e6287ce\": container with ID starting with 9861343ab263f81c1dbee8304feff9c17d8de8b8d176c47e5196c2b17e6287ce not found: ID does not exist" Nov 03 13:05:52.440049 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.440049 5979 scope.go:122] "RemoveContainer" containerID="0acd1240c69ae69c18d842f9defe65913faf7126e2d9c432a6199e0da68b4570" Nov 03 13:05:52.440290 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.440272 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0acd1240c69ae69c18d842f9defe65913faf7126e2d9c432a6199e0da68b4570"} err="failed to get container status \"0acd1240c69ae69c18d842f9defe65913faf7126e2d9c432a6199e0da68b4570\": rpc error: code = NotFound desc = could not find container \"0acd1240c69ae69c18d842f9defe65913faf7126e2d9c432a6199e0da68b4570\": container with ID starting with 0acd1240c69ae69c18d842f9defe65913faf7126e2d9c432a6199e0da68b4570 not found: ID does not exist" Nov 03 13:05:52.440290 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.440292 5979 scope.go:122] "RemoveContainer" containerID="e797acde2976802df9507cb8adea62190a05660ef0fced2b2afe7cb4075fdc72" Nov 03 13:05:52.440642 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.440616 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e797acde2976802df9507cb8adea62190a05660ef0fced2b2afe7cb4075fdc72"} err="failed to get container status \"e797acde2976802df9507cb8adea62190a05660ef0fced2b2afe7cb4075fdc72\": rpc error: code = NotFound desc = could not find container \"e797acde2976802df9507cb8adea62190a05660ef0fced2b2afe7cb4075fdc72\": container with ID starting with e797acde2976802df9507cb8adea62190a05660ef0fced2b2afe7cb4075fdc72 not found: ID does not exist" Nov 03 13:05:52.440642 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.440640 5979 scope.go:122] "RemoveContainer" containerID="1caee93a0441c7c697a1ab2aa46e921bc1aca35c3270332e260a2665db7555a6" Nov 03 13:05:52.440903 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.440876 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1caee93a0441c7c697a1ab2aa46e921bc1aca35c3270332e260a2665db7555a6"} err="failed to get container status \"1caee93a0441c7c697a1ab2aa46e921bc1aca35c3270332e260a2665db7555a6\": rpc error: code = NotFound desc = could not find container \"1caee93a0441c7c697a1ab2aa46e921bc1aca35c3270332e260a2665db7555a6\": container with ID starting with 1caee93a0441c7c697a1ab2aa46e921bc1aca35c3270332e260a2665db7555a6 not found: ID does not exist" Nov 03 13:05:52.440903 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.440893 5979 scope.go:122] "RemoveContainer" containerID="3bb5b563b33e764f2cacb6395268c2a267d365ce2eb2be0658dbef1abe48d34a" Nov 03 13:05:52.441152 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.441134 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bb5b563b33e764f2cacb6395268c2a267d365ce2eb2be0658dbef1abe48d34a"} err="failed to get container status \"3bb5b563b33e764f2cacb6395268c2a267d365ce2eb2be0658dbef1abe48d34a\": rpc error: code = NotFound desc = could not find container \"3bb5b563b33e764f2cacb6395268c2a267d365ce2eb2be0658dbef1abe48d34a\": container with ID starting with 3bb5b563b33e764f2cacb6395268c2a267d365ce2eb2be0658dbef1abe48d34a not found: ID does not exist" Nov 03 13:05:52.441202 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.441152 5979 scope.go:122] "RemoveContainer" containerID="be6f49e9d541c310aac05d244ec6ee360d277537e656c034e6099e963b47149f" Nov 03 13:05:52.441394 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.441376 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be6f49e9d541c310aac05d244ec6ee360d277537e656c034e6099e963b47149f"} err="failed to get container status \"be6f49e9d541c310aac05d244ec6ee360d277537e656c034e6099e963b47149f\": rpc error: code = NotFound desc = could not find container \"be6f49e9d541c310aac05d244ec6ee360d277537e656c034e6099e963b47149f\": container with ID starting with be6f49e9d541c310aac05d244ec6ee360d277537e656c034e6099e963b47149f not found: ID does not exist" Nov 03 13:05:52.441394 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.441393 5979 scope.go:122] "RemoveContainer" containerID="040ae9aec0f27bf15e42a48919412f565f4fa698b28cf15787eabe7b4fabe2f9" Nov 03 13:05:52.441659 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.441632 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"040ae9aec0f27bf15e42a48919412f565f4fa698b28cf15787eabe7b4fabe2f9"} err="failed to get container status \"040ae9aec0f27bf15e42a48919412f565f4fa698b28cf15787eabe7b4fabe2f9\": rpc error: code = NotFound desc = could not find container \"040ae9aec0f27bf15e42a48919412f565f4fa698b28cf15787eabe7b4fabe2f9\": container with ID starting with 040ae9aec0f27bf15e42a48919412f565f4fa698b28cf15787eabe7b4fabe2f9 not found: ID does not exist" Nov 03 13:05:52.441659 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.441659 5979 scope.go:122] "RemoveContainer" containerID="9861343ab263f81c1dbee8304feff9c17d8de8b8d176c47e5196c2b17e6287ce" Nov 03 13:05:52.442082 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.442052 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9861343ab263f81c1dbee8304feff9c17d8de8b8d176c47e5196c2b17e6287ce"} err="failed to get container status \"9861343ab263f81c1dbee8304feff9c17d8de8b8d176c47e5196c2b17e6287ce\": rpc error: code = NotFound desc = could not find container \"9861343ab263f81c1dbee8304feff9c17d8de8b8d176c47e5196c2b17e6287ce\": container with ID starting with 9861343ab263f81c1dbee8304feff9c17d8de8b8d176c47e5196c2b17e6287ce not found: ID does not exist" Nov 03 13:05:52.442082 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.442083 5979 scope.go:122] "RemoveContainer" containerID="0acd1240c69ae69c18d842f9defe65913faf7126e2d9c432a6199e0da68b4570" Nov 03 13:05:52.442386 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.442367 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0acd1240c69ae69c18d842f9defe65913faf7126e2d9c432a6199e0da68b4570"} err="failed to get container status \"0acd1240c69ae69c18d842f9defe65913faf7126e2d9c432a6199e0da68b4570\": rpc error: code = NotFound desc = could not find container \"0acd1240c69ae69c18d842f9defe65913faf7126e2d9c432a6199e0da68b4570\": container with ID starting with 0acd1240c69ae69c18d842f9defe65913faf7126e2d9c432a6199e0da68b4570 not found: ID does not exist" Nov 03 13:05:52.442386 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.442386 5979 scope.go:122] "RemoveContainer" containerID="e797acde2976802df9507cb8adea62190a05660ef0fced2b2afe7cb4075fdc72" Nov 03 13:05:52.442636 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.442618 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e797acde2976802df9507cb8adea62190a05660ef0fced2b2afe7cb4075fdc72"} err="failed to get container status \"e797acde2976802df9507cb8adea62190a05660ef0fced2b2afe7cb4075fdc72\": rpc error: code = NotFound desc = could not find container \"e797acde2976802df9507cb8adea62190a05660ef0fced2b2afe7cb4075fdc72\": container with ID starting with e797acde2976802df9507cb8adea62190a05660ef0fced2b2afe7cb4075fdc72 not found: ID does not exist" Nov 03 13:05:52.442636 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.442635 5979 scope.go:122] "RemoveContainer" containerID="1caee93a0441c7c697a1ab2aa46e921bc1aca35c3270332e260a2665db7555a6" Nov 03 13:05:52.442905 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.442886 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1caee93a0441c7c697a1ab2aa46e921bc1aca35c3270332e260a2665db7555a6"} err="failed to get container status \"1caee93a0441c7c697a1ab2aa46e921bc1aca35c3270332e260a2665db7555a6\": rpc error: code = NotFound desc = could not find container \"1caee93a0441c7c697a1ab2aa46e921bc1aca35c3270332e260a2665db7555a6\": container with ID starting with 1caee93a0441c7c697a1ab2aa46e921bc1aca35c3270332e260a2665db7555a6 not found: ID does not exist" Nov 03 13:05:52.442905 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.442905 5979 scope.go:122] "RemoveContainer" containerID="3bb5b563b33e764f2cacb6395268c2a267d365ce2eb2be0658dbef1abe48d34a" Nov 03 13:05:52.443158 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.443143 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bb5b563b33e764f2cacb6395268c2a267d365ce2eb2be0658dbef1abe48d34a"} err="failed to get container status \"3bb5b563b33e764f2cacb6395268c2a267d365ce2eb2be0658dbef1abe48d34a\": rpc error: code = NotFound desc = could not find container \"3bb5b563b33e764f2cacb6395268c2a267d365ce2eb2be0658dbef1abe48d34a\": container with ID starting with 3bb5b563b33e764f2cacb6395268c2a267d365ce2eb2be0658dbef1abe48d34a not found: ID does not exist" Nov 03 13:05:52.443158 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.443158 5979 scope.go:122] "RemoveContainer" containerID="be6f49e9d541c310aac05d244ec6ee360d277537e656c034e6099e963b47149f" Nov 03 13:05:52.443428 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.443405 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be6f49e9d541c310aac05d244ec6ee360d277537e656c034e6099e963b47149f"} err="failed to get container status \"be6f49e9d541c310aac05d244ec6ee360d277537e656c034e6099e963b47149f\": rpc error: code = NotFound desc = could not find container \"be6f49e9d541c310aac05d244ec6ee360d277537e656c034e6099e963b47149f\": container with ID starting with be6f49e9d541c310aac05d244ec6ee360d277537e656c034e6099e963b47149f not found: ID does not exist" Nov 03 13:05:52.443428 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.443428 5979 scope.go:122] "RemoveContainer" containerID="040ae9aec0f27bf15e42a48919412f565f4fa698b28cf15787eabe7b4fabe2f9" Nov 03 13:05:52.443710 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.443682 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"040ae9aec0f27bf15e42a48919412f565f4fa698b28cf15787eabe7b4fabe2f9"} err="failed to get container status \"040ae9aec0f27bf15e42a48919412f565f4fa698b28cf15787eabe7b4fabe2f9\": rpc error: code = NotFound desc = could not find container \"040ae9aec0f27bf15e42a48919412f565f4fa698b28cf15787eabe7b4fabe2f9\": container with ID starting with 040ae9aec0f27bf15e42a48919412f565f4fa698b28cf15787eabe7b4fabe2f9 not found: ID does not exist" Nov 03 13:05:52.443710 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.443709 5979 scope.go:122] "RemoveContainer" containerID="9861343ab263f81c1dbee8304feff9c17d8de8b8d176c47e5196c2b17e6287ce" Nov 03 13:05:52.443951 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.443935 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9861343ab263f81c1dbee8304feff9c17d8de8b8d176c47e5196c2b17e6287ce"} err="failed to get container status \"9861343ab263f81c1dbee8304feff9c17d8de8b8d176c47e5196c2b17e6287ce\": rpc error: code = NotFound desc = could not find container \"9861343ab263f81c1dbee8304feff9c17d8de8b8d176c47e5196c2b17e6287ce\": container with ID starting with 9861343ab263f81c1dbee8304feff9c17d8de8b8d176c47e5196c2b17e6287ce not found: ID does not exist" Nov 03 13:05:52.443951 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.443950 5979 scope.go:122] "RemoveContainer" containerID="0acd1240c69ae69c18d842f9defe65913faf7126e2d9c432a6199e0da68b4570" Nov 03 13:05:52.444224 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.444200 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0acd1240c69ae69c18d842f9defe65913faf7126e2d9c432a6199e0da68b4570"} err="failed to get container status \"0acd1240c69ae69c18d842f9defe65913faf7126e2d9c432a6199e0da68b4570\": rpc error: code = NotFound desc = could not find container \"0acd1240c69ae69c18d842f9defe65913faf7126e2d9c432a6199e0da68b4570\": container with ID starting with 0acd1240c69ae69c18d842f9defe65913faf7126e2d9c432a6199e0da68b4570 not found: ID does not exist" Nov 03 13:05:52.444279 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.444226 5979 scope.go:122] "RemoveContainer" containerID="e797acde2976802df9507cb8adea62190a05660ef0fced2b2afe7cb4075fdc72" Nov 03 13:05:52.444524 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.444501 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e797acde2976802df9507cb8adea62190a05660ef0fced2b2afe7cb4075fdc72"} err="failed to get container status \"e797acde2976802df9507cb8adea62190a05660ef0fced2b2afe7cb4075fdc72\": rpc error: code = NotFound desc = could not find container \"e797acde2976802df9507cb8adea62190a05660ef0fced2b2afe7cb4075fdc72\": container with ID starting with e797acde2976802df9507cb8adea62190a05660ef0fced2b2afe7cb4075fdc72 not found: ID does not exist" Nov 03 13:05:52.444612 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.444524 5979 scope.go:122] "RemoveContainer" containerID="1caee93a0441c7c697a1ab2aa46e921bc1aca35c3270332e260a2665db7555a6" Nov 03 13:05:52.444824 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.444807 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1caee93a0441c7c697a1ab2aa46e921bc1aca35c3270332e260a2665db7555a6"} err="failed to get container status \"1caee93a0441c7c697a1ab2aa46e921bc1aca35c3270332e260a2665db7555a6\": rpc error: code = NotFound desc = could not find container \"1caee93a0441c7c697a1ab2aa46e921bc1aca35c3270332e260a2665db7555a6\": container with ID starting with 1caee93a0441c7c697a1ab2aa46e921bc1aca35c3270332e260a2665db7555a6 not found: ID does not exist" Nov 03 13:05:52.462012 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.461992 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " Nov 03 13:05:52.462112 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.462022 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-configmap-serving-certs-ca-bundle\") pod \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " Nov 03 13:05:52.462112 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.462051 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-secret-prometheus-k8s-tls\") pod \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " Nov 03 13:05:52.462112 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.462083 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " Nov 03 13:05:52.462327 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.462123 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-thanos-prometheus-http-client-file\") pod \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " Nov 03 13:05:52.462327 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.462161 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-config\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-config\") pod \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " Nov 03 13:05:52.462327 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.462196 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-web-config\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-web-config\") pod \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " Nov 03 13:05:52.462327 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.462226 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-prometheus-k8s-rulefiles-2\") pod \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " Nov 03 13:05:52.462327 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.462287 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-prometheus-trusted-ca-bundle\") pod \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " Nov 03 13:05:52.462327 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.462316 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30d608b6-d82e-4fcf-ab44-b8e064c463d0-config-out\" (UniqueName: \"kubernetes.io/empty-dir/30d608b6-d82e-4fcf-ab44-b8e064c463d0-config-out\") pod \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " Nov 03 13:05:52.462640 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.462348 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-configmap-kubelet-serving-ca-bundle\") pod \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " Nov 03 13:05:52.462640 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.462392 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-secret-kube-rbac-proxy\") pod \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " Nov 03 13:05:52.462640 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.462419 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30d608b6-d82e-4fcf-ab44-b8e064c463d0-tls-assets\" (UniqueName: \"kubernetes.io/projected/30d608b6-d82e-4fcf-ab44-b8e064c463d0-tls-assets\") pod \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " Nov 03 13:05:52.462640 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.462428 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-configmap-serving-certs-ca-bundle" pod "30d608b6-d82e-4fcf-ab44-b8e064c463d0" (UID: "30d608b6-d82e-4fcf-ab44-b8e064c463d0"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:05:52.462640 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.462467 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-secret-metrics-client-certs\") pod \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " Nov 03 13:05:52.462640 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.462518 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-secret-grpc-tls\") pod \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " Nov 03 13:05:52.462640 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.462560 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30d608b6-d82e-4fcf-ab44-b8e064c463d0-kube-api-access-jdvh2\" (UniqueName: \"kubernetes.io/projected/30d608b6-d82e-4fcf-ab44-b8e064c463d0-kube-api-access-jdvh2\") pod \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " Nov 03 13:05:52.462640 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.462595 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-prometheus-k8s-rulefiles-0\") pod \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " Nov 03 13:05:52.462640 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.462630 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-configmap-metrics-client-ca\") pod \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " Nov 03 13:05:52.463073 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.462660 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-prometheus-k8s-rulefiles-1\") pod \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " Nov 03 13:05:52.463073 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.462701 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30d608b6-d82e-4fcf-ab44-b8e064c463d0-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/30d608b6-d82e-4fcf-ab44-b8e064c463d0-prometheus-k8s-db\") pod \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\" (UID: \"30d608b6-d82e-4fcf-ab44-b8e064c463d0\") " Nov 03 13:05:52.463073 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.462746 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-prometheus-k8s-rulefiles-2" pod "30d608b6-d82e-4fcf-ab44-b8e064c463d0" (UID: "30d608b6-d82e-4fcf-ab44-b8e064c463d0"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:05:52.463073 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.462790 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-prometheus-trusted-ca-bundle" pod "30d608b6-d82e-4fcf-ab44-b8e064c463d0" (UID: "30d608b6-d82e-4fcf-ab44-b8e064c463d0"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:05:52.463073 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.462880 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-configmap-kubelet-serving-ca-bundle" pod "30d608b6-d82e-4fcf-ab44-b8e064c463d0" (UID: "30d608b6-d82e-4fcf-ab44-b8e064c463d0"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:05:52.463315 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.463150 5979 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-prometheus-trusted-ca-bundle\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:05:52.463315 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.463173 5979 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:05:52.463315 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.463189 5979 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:05:52.463315 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.463205 5979 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:05:52.464826 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.463775 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-configmap-metrics-client-ca" pod "30d608b6-d82e-4fcf-ab44-b8e064c463d0" (UID: "30d608b6-d82e-4fcf-ab44-b8e064c463d0"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:05:52.464826 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.464143 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-prometheus-k8s-rulefiles-1" pod "30d608b6-d82e-4fcf-ab44-b8e064c463d0" (UID: "30d608b6-d82e-4fcf-ab44-b8e064c463d0"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:05:52.465188 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.464990 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-prometheus-k8s-rulefiles-0" pod "30d608b6-d82e-4fcf-ab44-b8e064c463d0" (UID: "30d608b6-d82e-4fcf-ab44-b8e064c463d0"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:05:52.465689 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.465654 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30d608b6-d82e-4fcf-ab44-b8e064c463d0-prometheus-k8s-db" pod "30d608b6-d82e-4fcf-ab44-b8e064c463d0" (UID: "30d608b6-d82e-4fcf-ab44-b8e064c463d0"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:05:52.468123 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.468076 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-secret-prometheus-k8s-thanos-sidecar-tls" pod "30d608b6-d82e-4fcf-ab44-b8e064c463d0" (UID: "30d608b6-d82e-4fcf-ab44-b8e064c463d0"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:05:52.469260 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.469128 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-secret-kube-rbac-proxy" pod "30d608b6-d82e-4fcf-ab44-b8e064c463d0" (UID: "30d608b6-d82e-4fcf-ab44-b8e064c463d0"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:05:52.470037 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.469925 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-secret-prometheus-k8s-tls" pod "30d608b6-d82e-4fcf-ab44-b8e064c463d0" (UID: "30d608b6-d82e-4fcf-ab44-b8e064c463d0"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:05:52.470178 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.470152 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30d608b6-d82e-4fcf-ab44-b8e064c463d0-tls-assets" pod "30d608b6-d82e-4fcf-ab44-b8e064c463d0" (UID: "30d608b6-d82e-4fcf-ab44-b8e064c463d0"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:05:52.470443 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.470321 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-secret-prometheus-k8s-kube-rbac-proxy-web" pod "30d608b6-d82e-4fcf-ab44-b8e064c463d0" (UID: "30d608b6-d82e-4fcf-ab44-b8e064c463d0"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:05:52.470601 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.470581 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30d608b6-d82e-4fcf-ab44-b8e064c463d0-kube-api-access-jdvh2" pod "30d608b6-d82e-4fcf-ab44-b8e064c463d0" (UID: "30d608b6-d82e-4fcf-ab44-b8e064c463d0"). InnerVolumeSpecName "kube-api-access-jdvh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:05:52.470664 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.470648 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-config" pod "30d608b6-d82e-4fcf-ab44-b8e064c463d0" (UID: "30d608b6-d82e-4fcf-ab44-b8e064c463d0"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:05:52.471272 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.471248 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-secret-grpc-tls" pod "30d608b6-d82e-4fcf-ab44-b8e064c463d0" (UID: "30d608b6-d82e-4fcf-ab44-b8e064c463d0"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:05:52.471404 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.471375 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-secret-metrics-client-certs" pod "30d608b6-d82e-4fcf-ab44-b8e064c463d0" (UID: "30d608b6-d82e-4fcf-ab44-b8e064c463d0"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:05:52.471600 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.471581 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30d608b6-d82e-4fcf-ab44-b8e064c463d0-config-out" pod "30d608b6-d82e-4fcf-ab44-b8e064c463d0" (UID: "30d608b6-d82e-4fcf-ab44-b8e064c463d0"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:05:52.471656 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.471604 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-thanos-prometheus-http-client-file" pod "30d608b6-d82e-4fcf-ab44-b8e064c463d0" (UID: "30d608b6-d82e-4fcf-ab44-b8e064c463d0"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:05:52.480360 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.480332 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-web-config" pod "30d608b6-d82e-4fcf-ab44-b8e064c463d0" (UID: "30d608b6-d82e-4fcf-ab44-b8e064c463d0"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:05:52.564566 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.564467 5979 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/30d608b6-d82e-4fcf-ab44-b8e064c463d0-prometheus-k8s-db\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:05:52.564566 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.564519 5979 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:05:52.564566 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.564535 5979 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-secret-prometheus-k8s-tls\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:05:52.564566 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.564546 5979 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:05:52.564566 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.564559 5979 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-thanos-prometheus-http-client-file\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:05:52.565090 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.564574 5979 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-config\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:05:52.565090 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.564594 5979 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-web-config\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:05:52.565090 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.564615 5979 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/30d608b6-d82e-4fcf-ab44-b8e064c463d0-config-out\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:05:52.565090 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.564628 5979 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-secret-kube-rbac-proxy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:05:52.565090 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.564636 5979 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/30d608b6-d82e-4fcf-ab44-b8e064c463d0-tls-assets\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:05:52.565090 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.564645 5979 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-secret-metrics-client-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:05:52.565090 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.564654 5979 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/30d608b6-d82e-4fcf-ab44-b8e064c463d0-secret-grpc-tls\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:05:52.565090 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.564663 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdvh2\" (UniqueName: \"kubernetes.io/projected/30d608b6-d82e-4fcf-ab44-b8e064c463d0-kube-api-access-jdvh2\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:05:52.565090 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.564674 5979 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:05:52.565090 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.564688 5979 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-configmap-metrics-client-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:05:52.565090 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.564700 5979 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/30d608b6-d82e-4fcf-ab44-b8e064c463d0-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:05:52.778880 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.778845 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 13:05:52.783306 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.783278 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 13:05:52.807212 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.807172 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 13:05:52.807587 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.807566 5979 cpu_manager.go:414] "containerMap: removing container" podUID="30d608b6-d82e-4fcf-ab44-b8e064c463d0" containerName="thanos-sidecar" Nov 03 13:05:52.807655 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.807588 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30d608b6-d82e-4fcf-ab44-b8e064c463d0" containerName="thanos-sidecar" Nov 03 13:05:52.807655 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.807603 5979 cpu_manager.go:414] "containerMap: removing container" podUID="30d608b6-d82e-4fcf-ab44-b8e064c463d0" containerName="prometheus" Nov 03 13:05:52.807655 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.807608 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30d608b6-d82e-4fcf-ab44-b8e064c463d0" containerName="prometheus" Nov 03 13:05:52.807655 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.807619 5979 cpu_manager.go:414] "containerMap: removing container" podUID="30d608b6-d82e-4fcf-ab44-b8e064c463d0" containerName="kube-rbac-proxy-web" Nov 03 13:05:52.807655 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.807626 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30d608b6-d82e-4fcf-ab44-b8e064c463d0" containerName="kube-rbac-proxy-web" Nov 03 13:05:52.807655 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.807640 5979 cpu_manager.go:414] "containerMap: removing container" podUID="30d608b6-d82e-4fcf-ab44-b8e064c463d0" containerName="kube-rbac-proxy" Nov 03 13:05:52.807655 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.807645 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30d608b6-d82e-4fcf-ab44-b8e064c463d0" containerName="kube-rbac-proxy" Nov 03 13:05:52.807897 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.807661 5979 cpu_manager.go:414] "containerMap: removing container" podUID="30d608b6-d82e-4fcf-ab44-b8e064c463d0" containerName="init-config-reloader" Nov 03 13:05:52.807897 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.807665 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30d608b6-d82e-4fcf-ab44-b8e064c463d0" containerName="init-config-reloader" Nov 03 13:05:52.807897 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.807675 5979 cpu_manager.go:414] "containerMap: removing container" podUID="30d608b6-d82e-4fcf-ab44-b8e064c463d0" containerName="kube-rbac-proxy-thanos" Nov 03 13:05:52.807897 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.807680 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30d608b6-d82e-4fcf-ab44-b8e064c463d0" containerName="kube-rbac-proxy-thanos" Nov 03 13:05:52.807897 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.807688 5979 cpu_manager.go:414] "containerMap: removing container" podUID="30d608b6-d82e-4fcf-ab44-b8e064c463d0" containerName="config-reloader" Nov 03 13:05:52.807897 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.807693 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30d608b6-d82e-4fcf-ab44-b8e064c463d0" containerName="config-reloader" Nov 03 13:05:52.807897 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.807797 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="30d608b6-d82e-4fcf-ab44-b8e064c463d0" containerName="init-config-reloader" Nov 03 13:05:52.807897 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.807812 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="30d608b6-d82e-4fcf-ab44-b8e064c463d0" containerName="kube-rbac-proxy-thanos" Nov 03 13:05:52.807897 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.807819 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="30d608b6-d82e-4fcf-ab44-b8e064c463d0" containerName="config-reloader" Nov 03 13:05:52.807897 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.807826 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="30d608b6-d82e-4fcf-ab44-b8e064c463d0" containerName="thanos-sidecar" Nov 03 13:05:52.807897 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.807831 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="30d608b6-d82e-4fcf-ab44-b8e064c463d0" containerName="prometheus" Nov 03 13:05:52.807897 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.807836 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="30d608b6-d82e-4fcf-ab44-b8e064c463d0" containerName="kube-rbac-proxy-web" Nov 03 13:05:52.807897 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.807842 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="30d608b6-d82e-4fcf-ab44-b8e064c463d0" containerName="kube-rbac-proxy" Nov 03 13:05:52.810441 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.810420 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:52.869126 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.869095 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 03 13:05:52.869126 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.869093 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 03 13:05:52.869356 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.869170 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 03 13:05:52.916107 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.916069 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 03 13:05:52.916296 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.916163 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 03 13:05:52.916296 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.916181 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 03 13:05:52.916296 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.916240 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 03 13:05:52.916296 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.916257 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 03 13:05:52.916296 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.916259 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 03 13:05:52.916296 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.916265 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 03 13:05:52.916629 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.916431 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-cgshq\"" Nov 03 13:05:52.916629 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.916437 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4ia0ld9dido9e\"" Nov 03 13:05:52.916629 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.916506 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 03 13:05:52.916629 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.916517 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 03 13:05:52.922352 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.917727 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 03 13:05:52.922352 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.918752 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 03 13:05:52.968877 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.968838 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16ea9a76-6906-4842-912f-e5cbe2784973-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"16ea9a76-6906-4842-912f-e5cbe2784973\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:52.968877 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.968886 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/16ea9a76-6906-4842-912f-e5cbe2784973-config\") pod \"prometheus-k8s-0\" (UID: \"16ea9a76-6906-4842-912f-e5cbe2784973\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:52.969082 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.968907 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/16ea9a76-6906-4842-912f-e5cbe2784973-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"16ea9a76-6906-4842-912f-e5cbe2784973\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:52.969082 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.968974 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/16ea9a76-6906-4842-912f-e5cbe2784973-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"16ea9a76-6906-4842-912f-e5cbe2784973\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:52.969082 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.969018 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16ea9a76-6906-4842-912f-e5cbe2784973-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"16ea9a76-6906-4842-912f-e5cbe2784973\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:52.969217 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.969083 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/16ea9a76-6906-4842-912f-e5cbe2784973-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"16ea9a76-6906-4842-912f-e5cbe2784973\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:52.969217 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.969108 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/16ea9a76-6906-4842-912f-e5cbe2784973-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"16ea9a76-6906-4842-912f-e5cbe2784973\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:52.969217 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.969130 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/16ea9a76-6906-4842-912f-e5cbe2784973-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"16ea9a76-6906-4842-912f-e5cbe2784973\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:52.969217 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.969175 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/16ea9a76-6906-4842-912f-e5cbe2784973-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"16ea9a76-6906-4842-912f-e5cbe2784973\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:52.969356 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.969222 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tzfw\" (UniqueName: \"kubernetes.io/projected/16ea9a76-6906-4842-912f-e5cbe2784973-kube-api-access-9tzfw\") pod \"prometheus-k8s-0\" (UID: \"16ea9a76-6906-4842-912f-e5cbe2784973\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:52.969356 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.969258 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/16ea9a76-6906-4842-912f-e5cbe2784973-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"16ea9a76-6906-4842-912f-e5cbe2784973\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:52.969356 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.969286 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16ea9a76-6906-4842-912f-e5cbe2784973-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"16ea9a76-6906-4842-912f-e5cbe2784973\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:52.969356 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.969326 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/16ea9a76-6906-4842-912f-e5cbe2784973-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"16ea9a76-6906-4842-912f-e5cbe2784973\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:52.969562 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.969367 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/16ea9a76-6906-4842-912f-e5cbe2784973-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"16ea9a76-6906-4842-912f-e5cbe2784973\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:52.969562 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.969393 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16ea9a76-6906-4842-912f-e5cbe2784973-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"16ea9a76-6906-4842-912f-e5cbe2784973\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:52.969562 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.969503 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/16ea9a76-6906-4842-912f-e5cbe2784973-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"16ea9a76-6906-4842-912f-e5cbe2784973\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:52.969562 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.969546 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/16ea9a76-6906-4842-912f-e5cbe2784973-web-config\") pod \"prometheus-k8s-0\" (UID: \"16ea9a76-6906-4842-912f-e5cbe2784973\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:52.969717 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.969609 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/16ea9a76-6906-4842-912f-e5cbe2784973-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"16ea9a76-6906-4842-912f-e5cbe2784973\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:52.969717 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.969636 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/16ea9a76-6906-4842-912f-e5cbe2784973-config-out\") pod \"prometheus-k8s-0\" (UID: \"16ea9a76-6906-4842-912f-e5cbe2784973\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:52.969717 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:52.969657 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/16ea9a76-6906-4842-912f-e5cbe2784973-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"16ea9a76-6906-4842-912f-e5cbe2784973\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:53.018368 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:53.018334 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 13:05:53.070182 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:53.070156 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/16ea9a76-6906-4842-912f-e5cbe2784973-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"16ea9a76-6906-4842-912f-e5cbe2784973\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:53.070278 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:53.070199 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/16ea9a76-6906-4842-912f-e5cbe2784973-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"16ea9a76-6906-4842-912f-e5cbe2784973\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:53.070388 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:53.070370 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/16ea9a76-6906-4842-912f-e5cbe2784973-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"16ea9a76-6906-4842-912f-e5cbe2784973\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:53.070442 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:53.070402 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/16ea9a76-6906-4842-912f-e5cbe2784973-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"16ea9a76-6906-4842-912f-e5cbe2784973\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:53.070442 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:53.070427 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tzfw\" (UniqueName: \"kubernetes.io/projected/16ea9a76-6906-4842-912f-e5cbe2784973-kube-api-access-9tzfw\") pod \"prometheus-k8s-0\" (UID: \"16ea9a76-6906-4842-912f-e5cbe2784973\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:53.070586 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:53.070463 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/16ea9a76-6906-4842-912f-e5cbe2784973-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"16ea9a76-6906-4842-912f-e5cbe2784973\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:53.070586 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:53.070498 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16ea9a76-6906-4842-912f-e5cbe2784973-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"16ea9a76-6906-4842-912f-e5cbe2784973\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:53.070586 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:53.070527 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/16ea9a76-6906-4842-912f-e5cbe2784973-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"16ea9a76-6906-4842-912f-e5cbe2784973\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:53.070586 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:53.070552 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/16ea9a76-6906-4842-912f-e5cbe2784973-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"16ea9a76-6906-4842-912f-e5cbe2784973\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:53.070868 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:53.070846 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16ea9a76-6906-4842-912f-e5cbe2784973-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"16ea9a76-6906-4842-912f-e5cbe2784973\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:53.070929 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:53.070914 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/16ea9a76-6906-4842-912f-e5cbe2784973-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"16ea9a76-6906-4842-912f-e5cbe2784973\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:53.070979 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:53.070942 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/16ea9a76-6906-4842-912f-e5cbe2784973-web-config\") pod \"prometheus-k8s-0\" (UID: \"16ea9a76-6906-4842-912f-e5cbe2784973\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:53.070979 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:53.070970 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/16ea9a76-6906-4842-912f-e5cbe2784973-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"16ea9a76-6906-4842-912f-e5cbe2784973\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:53.071071 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:53.070993 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/16ea9a76-6906-4842-912f-e5cbe2784973-config-out\") pod \"prometheus-k8s-0\" (UID: \"16ea9a76-6906-4842-912f-e5cbe2784973\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:53.071071 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:53.071016 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/16ea9a76-6906-4842-912f-e5cbe2784973-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"16ea9a76-6906-4842-912f-e5cbe2784973\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:53.071071 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:53.071052 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16ea9a76-6906-4842-912f-e5cbe2784973-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"16ea9a76-6906-4842-912f-e5cbe2784973\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:53.071223 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:53.071083 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/16ea9a76-6906-4842-912f-e5cbe2784973-config\") pod \"prometheus-k8s-0\" (UID: \"16ea9a76-6906-4842-912f-e5cbe2784973\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:53.071223 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:53.071108 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/16ea9a76-6906-4842-912f-e5cbe2784973-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"16ea9a76-6906-4842-912f-e5cbe2784973\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:53.071223 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:53.071139 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/16ea9a76-6906-4842-912f-e5cbe2784973-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"16ea9a76-6906-4842-912f-e5cbe2784973\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:53.071223 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:53.071166 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16ea9a76-6906-4842-912f-e5cbe2784973-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"16ea9a76-6906-4842-912f-e5cbe2784973\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:53.072175 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:53.071546 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/16ea9a76-6906-4842-912f-e5cbe2784973-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"16ea9a76-6906-4842-912f-e5cbe2784973\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:53.072175 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:53.071898 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/16ea9a76-6906-4842-912f-e5cbe2784973-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"16ea9a76-6906-4842-912f-e5cbe2784973\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:53.072175 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:53.071983 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16ea9a76-6906-4842-912f-e5cbe2784973-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"16ea9a76-6906-4842-912f-e5cbe2784973\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:53.072416 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:53.072193 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16ea9a76-6906-4842-912f-e5cbe2784973-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"16ea9a76-6906-4842-912f-e5cbe2784973\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:53.073519 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:53.073458 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/16ea9a76-6906-4842-912f-e5cbe2784973-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"16ea9a76-6906-4842-912f-e5cbe2784973\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:53.074062 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:53.074017 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/16ea9a76-6906-4842-912f-e5cbe2784973-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"16ea9a76-6906-4842-912f-e5cbe2784973\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:53.074162 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:53.074068 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/16ea9a76-6906-4842-912f-e5cbe2784973-config-out\") pod \"prometheus-k8s-0\" (UID: \"16ea9a76-6906-4842-912f-e5cbe2784973\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:53.074229 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:53.074207 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/16ea9a76-6906-4842-912f-e5cbe2784973-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"16ea9a76-6906-4842-912f-e5cbe2784973\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:53.074700 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:53.074676 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16ea9a76-6906-4842-912f-e5cbe2784973-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"16ea9a76-6906-4842-912f-e5cbe2784973\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:53.074783 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:53.074697 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16ea9a76-6906-4842-912f-e5cbe2784973-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"16ea9a76-6906-4842-912f-e5cbe2784973\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:53.076288 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:53.076212 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/16ea9a76-6906-4842-912f-e5cbe2784973-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"16ea9a76-6906-4842-912f-e5cbe2784973\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:53.076288 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:53.076247 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/16ea9a76-6906-4842-912f-e5cbe2784973-web-config\") pod \"prometheus-k8s-0\" (UID: \"16ea9a76-6906-4842-912f-e5cbe2784973\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:53.076288 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:53.076259 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/16ea9a76-6906-4842-912f-e5cbe2784973-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"16ea9a76-6906-4842-912f-e5cbe2784973\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:53.076524 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:53.076353 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/16ea9a76-6906-4842-912f-e5cbe2784973-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"16ea9a76-6906-4842-912f-e5cbe2784973\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:53.076524 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:53.076446 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/16ea9a76-6906-4842-912f-e5cbe2784973-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"16ea9a76-6906-4842-912f-e5cbe2784973\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:53.076961 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:53.076928 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/16ea9a76-6906-4842-912f-e5cbe2784973-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"16ea9a76-6906-4842-912f-e5cbe2784973\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:53.077064 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:53.077047 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/16ea9a76-6906-4842-912f-e5cbe2784973-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"16ea9a76-6906-4842-912f-e5cbe2784973\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:53.077141 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:53.077122 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/16ea9a76-6906-4842-912f-e5cbe2784973-config\") pod \"prometheus-k8s-0\" (UID: \"16ea9a76-6906-4842-912f-e5cbe2784973\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:53.078553 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:53.078533 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/16ea9a76-6906-4842-912f-e5cbe2784973-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"16ea9a76-6906-4842-912f-e5cbe2784973\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:53.124429 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:53.124403 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tzfw\" (UniqueName: \"kubernetes.io/projected/16ea9a76-6906-4842-912f-e5cbe2784973-kube-api-access-9tzfw\") pod \"prometheus-k8s-0\" (UID: \"16ea9a76-6906-4842-912f-e5cbe2784973\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:53.130738 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:53.130719 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:05:53.369371 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:53.369278 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-59897cc7df-n8cb9" event={"ID":"11a6e3d9-2422-4fdb-8c44-949500637239","Type":"ContainerStarted","Data":"84df70d15adb057e750fa890eb1bf29d184f51df0853e8e432581e4e6487984d"} Nov 03 13:05:53.369371 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:53.369328 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-59897cc7df-n8cb9" event={"ID":"11a6e3d9-2422-4fdb-8c44-949500637239","Type":"ContainerStarted","Data":"fe797721389d90f16ffc7b3e3fe8a0bc7365678cf81ccb220d91d58ffa163289"} Nov 03 13:05:53.369371 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:53.369345 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-59897cc7df-n8cb9" event={"ID":"11a6e3d9-2422-4fdb-8c44-949500637239","Type":"ContainerStarted","Data":"0f12c2456c49f02292b8730ca26f423458ecf95a9e2783c02fd71ef16a81588e"} Nov 03 13:05:53.479492 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:53.479405 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-59897cc7df-n8cb9" podStartSLOduration=1.259435883 podStartE2EDuration="3.479387251s" totalImagesPullingTime="2.219951368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:05:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:05:53.479381522 +0000 UTC m=+35.490069766" watchObservedRunningTime="2026-11-03 13:05:53.479387251 +0000 UTC m=+35.490075495" Nov 03 13:05:53.554344 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:53.554314 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 03 13:05:53.562490 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:05:53.562446 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16ea9a76_6906_4842_912f_e5cbe2784973.slice/crio-b2363e80f24a71e3741e284be471cb1ab3dacb4b2a72e875f33901adbda6895a WatchSource:0}: Error finding container b2363e80f24a71e3741e284be471cb1ab3dacb4b2a72e875f33901adbda6895a: Status 404 returned error can't find the container with id b2363e80f24a71e3741e284be471cb1ab3dacb4b2a72e875f33901adbda6895a Nov 03 13:05:53.877982 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:53.877943 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-b95bb587c-kqqpm"] Nov 03 13:05:53.880437 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:53.880416 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b95bb587c-kqqpm" Nov 03 13:05:53.934921 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:53.934885 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b95bb587c-kqqpm"] Nov 03 13:05:53.979399 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:53.979362 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/50b9bd35-6d94-4e09-9824-11ddb8182775-console-oauth-config\") pod \"console-b95bb587c-kqqpm\" (UID: \"50b9bd35-6d94-4e09-9824-11ddb8182775\") " pod="openshift-console/console-b95bb587c-kqqpm" Nov 03 13:05:53.979399 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:53.979398 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50b9bd35-6d94-4e09-9824-11ddb8182775-tmp\") pod \"console-b95bb587c-kqqpm\" (UID: \"50b9bd35-6d94-4e09-9824-11ddb8182775\") " pod="openshift-console/console-b95bb587c-kqqpm" Nov 03 13:05:53.979659 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:53.979425 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb8nc\" (UniqueName: \"kubernetes.io/projected/50b9bd35-6d94-4e09-9824-11ddb8182775-kube-api-access-zb8nc\") pod \"console-b95bb587c-kqqpm\" (UID: \"50b9bd35-6d94-4e09-9824-11ddb8182775\") " pod="openshift-console/console-b95bb587c-kqqpm" Nov 03 13:05:53.979659 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:53.979569 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/50b9bd35-6d94-4e09-9824-11ddb8182775-service-ca\") pod \"console-b95bb587c-kqqpm\" (UID: \"50b9bd35-6d94-4e09-9824-11ddb8182775\") " pod="openshift-console/console-b95bb587c-kqqpm" Nov 03 13:05:53.979659 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:53.979617 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/50b9bd35-6d94-4e09-9824-11ddb8182775-console-config\") pod \"console-b95bb587c-kqqpm\" (UID: \"50b9bd35-6d94-4e09-9824-11ddb8182775\") " pod="openshift-console/console-b95bb587c-kqqpm" Nov 03 13:05:53.979753 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:53.979699 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/50b9bd35-6d94-4e09-9824-11ddb8182775-console-serving-cert\") pod \"console-b95bb587c-kqqpm\" (UID: \"50b9bd35-6d94-4e09-9824-11ddb8182775\") " pod="openshift-console/console-b95bb587c-kqqpm" Nov 03 13:05:53.979753 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:53.979739 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50b9bd35-6d94-4e09-9824-11ddb8182775-trusted-ca-bundle\") pod \"console-b95bb587c-kqqpm\" (UID: \"50b9bd35-6d94-4e09-9824-11ddb8182775\") " pod="openshift-console/console-b95bb587c-kqqpm" Nov 03 13:05:53.979844 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:53.979757 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/50b9bd35-6d94-4e09-9824-11ddb8182775-oauth-serving-cert\") pod \"console-b95bb587c-kqqpm\" (UID: \"50b9bd35-6d94-4e09-9824-11ddb8182775\") " pod="openshift-console/console-b95bb587c-kqqpm" Nov 03 13:05:54.081134 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:54.081103 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/50b9bd35-6d94-4e09-9824-11ddb8182775-console-oauth-config\") pod \"console-b95bb587c-kqqpm\" (UID: \"50b9bd35-6d94-4e09-9824-11ddb8182775\") " pod="openshift-console/console-b95bb587c-kqqpm" Nov 03 13:05:54.081134 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:54.081135 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50b9bd35-6d94-4e09-9824-11ddb8182775-tmp\") pod \"console-b95bb587c-kqqpm\" (UID: \"50b9bd35-6d94-4e09-9824-11ddb8182775\") " pod="openshift-console/console-b95bb587c-kqqpm" Nov 03 13:05:54.081365 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:54.081150 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb8nc\" (UniqueName: \"kubernetes.io/projected/50b9bd35-6d94-4e09-9824-11ddb8182775-kube-api-access-zb8nc\") pod \"console-b95bb587c-kqqpm\" (UID: \"50b9bd35-6d94-4e09-9824-11ddb8182775\") " pod="openshift-console/console-b95bb587c-kqqpm" Nov 03 13:05:54.081365 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:54.081176 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/50b9bd35-6d94-4e09-9824-11ddb8182775-service-ca\") pod \"console-b95bb587c-kqqpm\" (UID: \"50b9bd35-6d94-4e09-9824-11ddb8182775\") " pod="openshift-console/console-b95bb587c-kqqpm" Nov 03 13:05:54.081365 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:54.081204 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/50b9bd35-6d94-4e09-9824-11ddb8182775-console-config\") pod \"console-b95bb587c-kqqpm\" (UID: \"50b9bd35-6d94-4e09-9824-11ddb8182775\") " pod="openshift-console/console-b95bb587c-kqqpm" Nov 03 13:05:54.081365 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:54.081224 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/50b9bd35-6d94-4e09-9824-11ddb8182775-console-serving-cert\") pod \"console-b95bb587c-kqqpm\" (UID: \"50b9bd35-6d94-4e09-9824-11ddb8182775\") " pod="openshift-console/console-b95bb587c-kqqpm" Nov 03 13:05:54.081365 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:54.081243 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50b9bd35-6d94-4e09-9824-11ddb8182775-trusted-ca-bundle\") pod \"console-b95bb587c-kqqpm\" (UID: \"50b9bd35-6d94-4e09-9824-11ddb8182775\") " pod="openshift-console/console-b95bb587c-kqqpm" Nov 03 13:05:54.081365 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:54.081259 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/50b9bd35-6d94-4e09-9824-11ddb8182775-oauth-serving-cert\") pod \"console-b95bb587c-kqqpm\" (UID: \"50b9bd35-6d94-4e09-9824-11ddb8182775\") " pod="openshift-console/console-b95bb587c-kqqpm" Nov 03 13:05:54.081688 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:54.081600 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50b9bd35-6d94-4e09-9824-11ddb8182775-tmp\") pod \"console-b95bb587c-kqqpm\" (UID: \"50b9bd35-6d94-4e09-9824-11ddb8182775\") " pod="openshift-console/console-b95bb587c-kqqpm" Nov 03 13:05:54.081980 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:54.081955 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/50b9bd35-6d94-4e09-9824-11ddb8182775-console-config\") pod \"console-b95bb587c-kqqpm\" (UID: \"50b9bd35-6d94-4e09-9824-11ddb8182775\") " pod="openshift-console/console-b95bb587c-kqqpm" Nov 03 13:05:54.082081 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:54.082030 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/50b9bd35-6d94-4e09-9824-11ddb8182775-oauth-serving-cert\") pod \"console-b95bb587c-kqqpm\" (UID: \"50b9bd35-6d94-4e09-9824-11ddb8182775\") " pod="openshift-console/console-b95bb587c-kqqpm" Nov 03 13:05:54.082153 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:54.082103 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/50b9bd35-6d94-4e09-9824-11ddb8182775-service-ca\") pod \"console-b95bb587c-kqqpm\" (UID: \"50b9bd35-6d94-4e09-9824-11ddb8182775\") " pod="openshift-console/console-b95bb587c-kqqpm" Nov 03 13:05:54.082215 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:54.082167 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50b9bd35-6d94-4e09-9824-11ddb8182775-trusted-ca-bundle\") pod \"console-b95bb587c-kqqpm\" (UID: \"50b9bd35-6d94-4e09-9824-11ddb8182775\") " pod="openshift-console/console-b95bb587c-kqqpm" Nov 03 13:05:54.083744 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:54.083707 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/50b9bd35-6d94-4e09-9824-11ddb8182775-console-oauth-config\") pod \"console-b95bb587c-kqqpm\" (UID: \"50b9bd35-6d94-4e09-9824-11ddb8182775\") " pod="openshift-console/console-b95bb587c-kqqpm" Nov 03 13:05:54.083822 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:54.083803 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/50b9bd35-6d94-4e09-9824-11ddb8182775-console-serving-cert\") pod \"console-b95bb587c-kqqpm\" (UID: \"50b9bd35-6d94-4e09-9824-11ddb8182775\") " pod="openshift-console/console-b95bb587c-kqqpm" Nov 03 13:05:54.112993 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:54.112958 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30d608b6-d82e-4fcf-ab44-b8e064c463d0" path="/var/lib/kubelet/pods/30d608b6-d82e-4fcf-ab44-b8e064c463d0/volumes" Nov 03 13:05:54.133971 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:54.133946 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb8nc\" (UniqueName: \"kubernetes.io/projected/50b9bd35-6d94-4e09-9824-11ddb8182775-kube-api-access-zb8nc\") pod \"console-b95bb587c-kqqpm\" (UID: \"50b9bd35-6d94-4e09-9824-11ddb8182775\") " pod="openshift-console/console-b95bb587c-kqqpm" Nov 03 13:05:54.191180 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:54.191085 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b95bb587c-kqqpm" Nov 03 13:05:54.374344 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:54.374310 5979 generic.go:356] "Generic (PLEG): container finished" podID="16ea9a76-6906-4842-912f-e5cbe2784973" containerID="1572408da019f0ddab6b394aeec8756bfcc4f2ef3d23c1fa63f2b0ee6a2c4c78" exitCode=0 Nov 03 13:05:54.374758 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:54.374393 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"16ea9a76-6906-4842-912f-e5cbe2784973","Type":"ContainerDied","Data":"1572408da019f0ddab6b394aeec8756bfcc4f2ef3d23c1fa63f2b0ee6a2c4c78"} Nov 03 13:05:54.374758 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:54.374458 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"16ea9a76-6906-4842-912f-e5cbe2784973","Type":"ContainerStarted","Data":"b2363e80f24a71e3741e284be471cb1ab3dacb4b2a72e875f33901adbda6895a"} Nov 03 13:05:54.556513 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:54.556487 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b95bb587c-kqqpm"] Nov 03 13:05:55.381526 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:55.381492 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"16ea9a76-6906-4842-912f-e5cbe2784973","Type":"ContainerStarted","Data":"280c3b45e69837cf26cf1ef3f1302cc0bad7c673937b0b3df0a23e223adf6a7a"} Nov 03 13:05:55.381995 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:55.381577 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"16ea9a76-6906-4842-912f-e5cbe2784973","Type":"ContainerStarted","Data":"74893d2aed5ddb9347988185fb75f0d6df910e8b59c597fafbac52d1f87c3c06"} Nov 03 13:05:55.381995 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:55.381618 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"16ea9a76-6906-4842-912f-e5cbe2784973","Type":"ContainerStarted","Data":"98d3fd56e1cab33533ab647217d51396c9b179c29c41cd0f03d100b306e0b0d2"} Nov 03 13:05:55.381995 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:55.381652 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"16ea9a76-6906-4842-912f-e5cbe2784973","Type":"ContainerStarted","Data":"ddaa6e0aee5bdcf9c6decd28a1322f0d3628b940b38c4591a8ac3ce3b1b55cdc"} Nov 03 13:05:55.381995 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:55.381687 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"16ea9a76-6906-4842-912f-e5cbe2784973","Type":"ContainerStarted","Data":"15a4329625c72bb51d8194d2e2cff528be0072eebdda4baabaf887b2b5156b83"} Nov 03 13:05:55.381995 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:55.381722 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"16ea9a76-6906-4842-912f-e5cbe2784973","Type":"ContainerStarted","Data":"9acc68cb741a8136bf2593c4ae3be52c115559b0d609ada12df2f5a5b44415fe"} Nov 03 13:05:55.383105 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:55.383082 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b95bb587c-kqqpm" event={"ID":"50b9bd35-6d94-4e09-9824-11ddb8182775","Type":"ContainerStarted","Data":"da69b75fecdc51cffa966b233b9bd111a3c60e5a2439cee5941dad9a79005755"} Nov 03 13:05:55.383183 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:55.383124 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b95bb587c-kqqpm" event={"ID":"50b9bd35-6d94-4e09-9824-11ddb8182775","Type":"ContainerStarted","Data":"24080822f67e938fd1b8934b5682ede3818bfdfd70c97f8e94118e3bd1ad9374"} Nov 03 13:05:55.500391 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:55.500334 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.484220283 podStartE2EDuration="3.500313524s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.093241ms" isStatefulPod=false podCreationTimestamp="2026-11-03 13:05:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:05:55.499052793 +0000 UTC m=+37.509741035" watchObservedRunningTime="2026-11-03 13:05:55.500313524 +0000 UTC m=+37.511001767" Nov 03 13:05:55.608035 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:55.607969 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-b95bb587c-kqqpm" podStartSLOduration=2.6079544439999998 podStartE2EDuration="2.607954444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:05:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:05:55.607577155 +0000 UTC m=+37.618265422" watchObservedRunningTime="2026-11-03 13:05:55.607954444 +0000 UTC m=+37.618642687" Nov 03 13:05:58.131114 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:05:58.131081 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:06:04.192223 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:04.192180 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-b95bb587c-kqqpm" Nov 03 13:06:04.192754 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:04.192249 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-b95bb587c-kqqpm" Nov 03 13:06:04.196815 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:04.196792 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-b95bb587c-kqqpm" Nov 03 13:06:04.422394 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:04.422363 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-b95bb587c-kqqpm" Nov 03 13:06:04.558519 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:04.558418 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f44499665-ff6gp"] Nov 03 13:06:07.232837 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:06:07.232796 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:06:09.867647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:09.867609 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:06:09.883070 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:09.883047 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:06:10.460221 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:10.460188 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 03 13:06:12.193920 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:12.193884 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6664fdb79d-kcrw8"] Nov 03 13:06:12.196532 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:12.196516 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6664fdb79d-kcrw8" Nov 03 13:06:12.244383 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:12.244348 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ca6896f-ddb2-41e7-8d94-6c5570cbd8e0-trusted-ca-bundle\") pod \"console-6664fdb79d-kcrw8\" (UID: \"3ca6896f-ddb2-41e7-8d94-6c5570cbd8e0\") " pod="openshift-console/console-6664fdb79d-kcrw8" Nov 03 13:06:12.244383 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:12.244386 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttclt\" (UniqueName: \"kubernetes.io/projected/3ca6896f-ddb2-41e7-8d94-6c5570cbd8e0-kube-api-access-ttclt\") pod \"console-6664fdb79d-kcrw8\" (UID: \"3ca6896f-ddb2-41e7-8d94-6c5570cbd8e0\") " pod="openshift-console/console-6664fdb79d-kcrw8" Nov 03 13:06:12.244643 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:12.244410 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3ca6896f-ddb2-41e7-8d94-6c5570cbd8e0-console-serving-cert\") pod \"console-6664fdb79d-kcrw8\" (UID: \"3ca6896f-ddb2-41e7-8d94-6c5570cbd8e0\") " pod="openshift-console/console-6664fdb79d-kcrw8" Nov 03 13:06:12.244643 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:12.244563 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3ca6896f-ddb2-41e7-8d94-6c5570cbd8e0-console-config\") pod \"console-6664fdb79d-kcrw8\" (UID: \"3ca6896f-ddb2-41e7-8d94-6c5570cbd8e0\") " pod="openshift-console/console-6664fdb79d-kcrw8" Nov 03 13:06:12.244643 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:12.244607 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3ca6896f-ddb2-41e7-8d94-6c5570cbd8e0-console-oauth-config\") pod \"console-6664fdb79d-kcrw8\" (UID: \"3ca6896f-ddb2-41e7-8d94-6c5570cbd8e0\") " pod="openshift-console/console-6664fdb79d-kcrw8" Nov 03 13:06:12.244643 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:12.244640 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3ca6896f-ddb2-41e7-8d94-6c5570cbd8e0-service-ca\") pod \"console-6664fdb79d-kcrw8\" (UID: \"3ca6896f-ddb2-41e7-8d94-6c5570cbd8e0\") " pod="openshift-console/console-6664fdb79d-kcrw8" Nov 03 13:06:12.244855 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:12.244709 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3ca6896f-ddb2-41e7-8d94-6c5570cbd8e0-oauth-serving-cert\") pod \"console-6664fdb79d-kcrw8\" (UID: \"3ca6896f-ddb2-41e7-8d94-6c5570cbd8e0\") " pod="openshift-console/console-6664fdb79d-kcrw8" Nov 03 13:06:12.244855 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:12.244779 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ca6896f-ddb2-41e7-8d94-6c5570cbd8e0-tmp\") pod \"console-6664fdb79d-kcrw8\" (UID: \"3ca6896f-ddb2-41e7-8d94-6c5570cbd8e0\") " pod="openshift-console/console-6664fdb79d-kcrw8" Nov 03 13:06:12.291692 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:12.291662 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6664fdb79d-kcrw8"] Nov 03 13:06:12.346032 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:12.345999 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ca6896f-ddb2-41e7-8d94-6c5570cbd8e0-trusted-ca-bundle\") pod \"console-6664fdb79d-kcrw8\" (UID: \"3ca6896f-ddb2-41e7-8d94-6c5570cbd8e0\") " pod="openshift-console/console-6664fdb79d-kcrw8" Nov 03 13:06:12.346032 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:12.346032 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttclt\" (UniqueName: \"kubernetes.io/projected/3ca6896f-ddb2-41e7-8d94-6c5570cbd8e0-kube-api-access-ttclt\") pod \"console-6664fdb79d-kcrw8\" (UID: \"3ca6896f-ddb2-41e7-8d94-6c5570cbd8e0\") " pod="openshift-console/console-6664fdb79d-kcrw8" Nov 03 13:06:12.346276 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:12.346058 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3ca6896f-ddb2-41e7-8d94-6c5570cbd8e0-console-serving-cert\") pod \"console-6664fdb79d-kcrw8\" (UID: \"3ca6896f-ddb2-41e7-8d94-6c5570cbd8e0\") " pod="openshift-console/console-6664fdb79d-kcrw8" Nov 03 13:06:12.346276 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:12.346076 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3ca6896f-ddb2-41e7-8d94-6c5570cbd8e0-console-config\") pod \"console-6664fdb79d-kcrw8\" (UID: \"3ca6896f-ddb2-41e7-8d94-6c5570cbd8e0\") " pod="openshift-console/console-6664fdb79d-kcrw8" Nov 03 13:06:12.346276 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:12.346098 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3ca6896f-ddb2-41e7-8d94-6c5570cbd8e0-console-oauth-config\") pod \"console-6664fdb79d-kcrw8\" (UID: \"3ca6896f-ddb2-41e7-8d94-6c5570cbd8e0\") " pod="openshift-console/console-6664fdb79d-kcrw8" Nov 03 13:06:12.346276 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:12.346117 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3ca6896f-ddb2-41e7-8d94-6c5570cbd8e0-service-ca\") pod \"console-6664fdb79d-kcrw8\" (UID: \"3ca6896f-ddb2-41e7-8d94-6c5570cbd8e0\") " pod="openshift-console/console-6664fdb79d-kcrw8" Nov 03 13:06:12.346276 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:12.346141 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3ca6896f-ddb2-41e7-8d94-6c5570cbd8e0-oauth-serving-cert\") pod \"console-6664fdb79d-kcrw8\" (UID: \"3ca6896f-ddb2-41e7-8d94-6c5570cbd8e0\") " pod="openshift-console/console-6664fdb79d-kcrw8" Nov 03 13:06:12.346276 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:12.346163 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ca6896f-ddb2-41e7-8d94-6c5570cbd8e0-tmp\") pod \"console-6664fdb79d-kcrw8\" (UID: \"3ca6896f-ddb2-41e7-8d94-6c5570cbd8e0\") " pod="openshift-console/console-6664fdb79d-kcrw8" Nov 03 13:06:12.346692 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:12.346668 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ca6896f-ddb2-41e7-8d94-6c5570cbd8e0-tmp\") pod \"console-6664fdb79d-kcrw8\" (UID: \"3ca6896f-ddb2-41e7-8d94-6c5570cbd8e0\") " pod="openshift-console/console-6664fdb79d-kcrw8" Nov 03 13:06:12.346935 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:12.346909 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3ca6896f-ddb2-41e7-8d94-6c5570cbd8e0-console-config\") pod \"console-6664fdb79d-kcrw8\" (UID: \"3ca6896f-ddb2-41e7-8d94-6c5570cbd8e0\") " pod="openshift-console/console-6664fdb79d-kcrw8" Nov 03 13:06:12.347018 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:12.346927 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3ca6896f-ddb2-41e7-8d94-6c5570cbd8e0-oauth-serving-cert\") pod \"console-6664fdb79d-kcrw8\" (UID: \"3ca6896f-ddb2-41e7-8d94-6c5570cbd8e0\") " pod="openshift-console/console-6664fdb79d-kcrw8" Nov 03 13:06:12.347218 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:12.347197 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ca6896f-ddb2-41e7-8d94-6c5570cbd8e0-trusted-ca-bundle\") pod \"console-6664fdb79d-kcrw8\" (UID: \"3ca6896f-ddb2-41e7-8d94-6c5570cbd8e0\") " pod="openshift-console/console-6664fdb79d-kcrw8" Nov 03 13:06:12.347278 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:12.347224 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3ca6896f-ddb2-41e7-8d94-6c5570cbd8e0-service-ca\") pod \"console-6664fdb79d-kcrw8\" (UID: \"3ca6896f-ddb2-41e7-8d94-6c5570cbd8e0\") " pod="openshift-console/console-6664fdb79d-kcrw8" Nov 03 13:06:12.348689 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:12.348662 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3ca6896f-ddb2-41e7-8d94-6c5570cbd8e0-console-oauth-config\") pod \"console-6664fdb79d-kcrw8\" (UID: \"3ca6896f-ddb2-41e7-8d94-6c5570cbd8e0\") " pod="openshift-console/console-6664fdb79d-kcrw8" Nov 03 13:06:12.348862 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:12.348847 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3ca6896f-ddb2-41e7-8d94-6c5570cbd8e0-console-serving-cert\") pod \"console-6664fdb79d-kcrw8\" (UID: \"3ca6896f-ddb2-41e7-8d94-6c5570cbd8e0\") " pod="openshift-console/console-6664fdb79d-kcrw8" Nov 03 13:06:12.399752 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:12.399717 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttclt\" (UniqueName: \"kubernetes.io/projected/3ca6896f-ddb2-41e7-8d94-6c5570cbd8e0-kube-api-access-ttclt\") pod \"console-6664fdb79d-kcrw8\" (UID: \"3ca6896f-ddb2-41e7-8d94-6c5570cbd8e0\") " pod="openshift-console/console-6664fdb79d-kcrw8" Nov 03 13:06:12.507294 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:12.507212 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6664fdb79d-kcrw8" Nov 03 13:06:12.871257 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:12.871178 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6664fdb79d-kcrw8"] Nov 03 13:06:12.878781 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:06:12.878755 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ca6896f_ddb2_41e7_8d94_6c5570cbd8e0.slice/crio-5004811a321bfd2357dc2adb786ae58b45a88a91dae8baef2ef49fa954d10fcd WatchSource:0}: Error finding container 5004811a321bfd2357dc2adb786ae58b45a88a91dae8baef2ef49fa954d10fcd: Status 404 returned error can't find the container with id 5004811a321bfd2357dc2adb786ae58b45a88a91dae8baef2ef49fa954d10fcd Nov 03 13:06:13.457394 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:13.457353 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6664fdb79d-kcrw8" event={"ID":"3ca6896f-ddb2-41e7-8d94-6c5570cbd8e0","Type":"ContainerStarted","Data":"e397783fb9d08dd12f6c8a3d890a931e8b1cbdfecbb55a1b3379e09cd2a64589"} Nov 03 13:06:13.457789 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:13.457437 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6664fdb79d-kcrw8" event={"ID":"3ca6896f-ddb2-41e7-8d94-6c5570cbd8e0","Type":"ContainerStarted","Data":"5004811a321bfd2357dc2adb786ae58b45a88a91dae8baef2ef49fa954d10fcd"} Nov 03 13:06:13.565255 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:13.565199 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6664fdb79d-kcrw8" podStartSLOduration=1.5651849100000002 podStartE2EDuration="1.56518491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:06:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:06:13.56457218 +0000 UTC m=+55.575260430" watchObservedRunningTime="2026-11-03 13:06:13.56518491 +0000 UTC m=+55.575873153" Nov 03 13:06:22.507735 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:22.507702 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6664fdb79d-kcrw8" Nov 03 13:06:22.508280 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:22.507766 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6664fdb79d-kcrw8" Nov 03 13:06:22.512621 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:22.512600 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6664fdb79d-kcrw8" Nov 03 13:06:23.516350 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:23.516323 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6664fdb79d-kcrw8" Nov 03 13:06:23.673407 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:23.673372 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b95bb587c-kqqpm"] Nov 03 13:06:29.581228 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:29.581179 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6f44499665-ff6gp" podUID="2dcde035-9b90-4961-8cfc-3f37bfb67581" containerName="console" containerID="cri-o://8e111aef53dca254384b00b073010d66493f2c2fd5e216ce5bacfa845f43c72a" gracePeriod=15 Nov 03 13:06:29.832500 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:29.832429 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f44499665-ff6gp" Nov 03 13:06:29.925684 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:29.925654 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dcde035-9b90-4961-8cfc-3f37bfb67581-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dcde035-9b90-4961-8cfc-3f37bfb67581-service-ca\") pod \"2dcde035-9b90-4961-8cfc-3f37bfb67581\" (UID: \"2dcde035-9b90-4961-8cfc-3f37bfb67581\") " Nov 03 13:06:29.925817 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:29.925697 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2dcde035-9b90-4961-8cfc-3f37bfb67581-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2dcde035-9b90-4961-8cfc-3f37bfb67581-console-serving-cert\") pod \"2dcde035-9b90-4961-8cfc-3f37bfb67581\" (UID: \"2dcde035-9b90-4961-8cfc-3f37bfb67581\") " Nov 03 13:06:29.925817 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:29.925731 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dcde035-9b90-4961-8cfc-3f37bfb67581-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2dcde035-9b90-4961-8cfc-3f37bfb67581-oauth-serving-cert\") pod \"2dcde035-9b90-4961-8cfc-3f37bfb67581\" (UID: \"2dcde035-9b90-4961-8cfc-3f37bfb67581\") " Nov 03 13:06:29.925817 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:29.925786 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2dcde035-9b90-4961-8cfc-3f37bfb67581-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2dcde035-9b90-4961-8cfc-3f37bfb67581-console-oauth-config\") pod \"2dcde035-9b90-4961-8cfc-3f37bfb67581\" (UID: \"2dcde035-9b90-4961-8cfc-3f37bfb67581\") " Nov 03 13:06:29.925931 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:29.925840 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dcde035-9b90-4961-8cfc-3f37bfb67581-kube-api-access-7gb4j\" (UniqueName: \"kubernetes.io/projected/2dcde035-9b90-4961-8cfc-3f37bfb67581-kube-api-access-7gb4j\") pod \"2dcde035-9b90-4961-8cfc-3f37bfb67581\" (UID: \"2dcde035-9b90-4961-8cfc-3f37bfb67581\") " Nov 03 13:06:29.925931 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:29.925867 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dcde035-9b90-4961-8cfc-3f37bfb67581-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dcde035-9b90-4961-8cfc-3f37bfb67581-trusted-ca-bundle\") pod \"2dcde035-9b90-4961-8cfc-3f37bfb67581\" (UID: \"2dcde035-9b90-4961-8cfc-3f37bfb67581\") " Nov 03 13:06:29.925931 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:29.925906 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dcde035-9b90-4961-8cfc-3f37bfb67581-console-config\" (UniqueName: \"kubernetes.io/configmap/2dcde035-9b90-4961-8cfc-3f37bfb67581-console-config\") pod \"2dcde035-9b90-4961-8cfc-3f37bfb67581\" (UID: \"2dcde035-9b90-4961-8cfc-3f37bfb67581\") " Nov 03 13:06:29.926074 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:29.925942 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dcde035-9b90-4961-8cfc-3f37bfb67581-tmp\" (UniqueName: \"kubernetes.io/empty-dir/2dcde035-9b90-4961-8cfc-3f37bfb67581-tmp\") pod \"2dcde035-9b90-4961-8cfc-3f37bfb67581\" (UID: \"2dcde035-9b90-4961-8cfc-3f37bfb67581\") " Nov 03 13:06:29.926136 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:29.926108 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dcde035-9b90-4961-8cfc-3f37bfb67581-service-ca" pod "2dcde035-9b90-4961-8cfc-3f37bfb67581" (UID: "2dcde035-9b90-4961-8cfc-3f37bfb67581"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:06:29.926241 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:29.926208 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dcde035-9b90-4961-8cfc-3f37bfb67581-oauth-serving-cert" pod "2dcde035-9b90-4961-8cfc-3f37bfb67581" (UID: "2dcde035-9b90-4961-8cfc-3f37bfb67581"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:06:29.926460 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:29.926332 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dcde035-9b90-4961-8cfc-3f37bfb67581-trusted-ca-bundle" pod "2dcde035-9b90-4961-8cfc-3f37bfb67581" (UID: "2dcde035-9b90-4961-8cfc-3f37bfb67581"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:06:29.926559 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:29.926497 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dcde035-9b90-4961-8cfc-3f37bfb67581-tmp" pod "2dcde035-9b90-4961-8cfc-3f37bfb67581" (UID: "2dcde035-9b90-4961-8cfc-3f37bfb67581"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:06:29.926678 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:29.926649 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dcde035-9b90-4961-8cfc-3f37bfb67581-console-config" pod "2dcde035-9b90-4961-8cfc-3f37bfb67581" (UID: "2dcde035-9b90-4961-8cfc-3f37bfb67581"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:06:29.928248 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:29.928228 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dcde035-9b90-4961-8cfc-3f37bfb67581-kube-api-access-7gb4j" pod "2dcde035-9b90-4961-8cfc-3f37bfb67581" (UID: "2dcde035-9b90-4961-8cfc-3f37bfb67581"). InnerVolumeSpecName "kube-api-access-7gb4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:06:29.928248 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:29.928237 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dcde035-9b90-4961-8cfc-3f37bfb67581-console-oauth-config" pod "2dcde035-9b90-4961-8cfc-3f37bfb67581" (UID: "2dcde035-9b90-4961-8cfc-3f37bfb67581"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:06:29.928368 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:29.928333 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dcde035-9b90-4961-8cfc-3f37bfb67581-console-serving-cert" pod "2dcde035-9b90-4961-8cfc-3f37bfb67581" (UID: "2dcde035-9b90-4961-8cfc-3f37bfb67581"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:06:30.026819 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:30.026783 5979 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2dcde035-9b90-4961-8cfc-3f37bfb67581-console-oauth-config\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:06:30.026819 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:30.026814 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gb4j\" (UniqueName: \"kubernetes.io/projected/2dcde035-9b90-4961-8cfc-3f37bfb67581-kube-api-access-7gb4j\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:06:30.026819 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:30.026826 5979 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2dcde035-9b90-4961-8cfc-3f37bfb67581-trusted-ca-bundle\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:06:30.027075 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:30.026835 5979 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2dcde035-9b90-4961-8cfc-3f37bfb67581-console-config\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:06:30.027075 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:30.026844 5979 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2dcde035-9b90-4961-8cfc-3f37bfb67581-tmp\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:06:30.027075 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:30.026853 5979 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2dcde035-9b90-4961-8cfc-3f37bfb67581-service-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:06:30.027075 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:30.026862 5979 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2dcde035-9b90-4961-8cfc-3f37bfb67581-console-serving-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:06:30.027075 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:30.026871 5979 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2dcde035-9b90-4961-8cfc-3f37bfb67581-oauth-serving-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:06:30.538327 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:30.538295 5979 generic.go:356] "Generic (PLEG): container finished" podID="2dcde035-9b90-4961-8cfc-3f37bfb67581" containerID="8e111aef53dca254384b00b073010d66493f2c2fd5e216ce5bacfa845f43c72a" exitCode=0 Nov 03 13:06:30.538547 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:30.538342 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f44499665-ff6gp" Nov 03 13:06:30.538547 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:30.538361 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f44499665-ff6gp" event={"ID":"2dcde035-9b90-4961-8cfc-3f37bfb67581","Type":"ContainerDied","Data":"8e111aef53dca254384b00b073010d66493f2c2fd5e216ce5bacfa845f43c72a"} Nov 03 13:06:30.538547 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:30.538401 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f44499665-ff6gp" event={"ID":"2dcde035-9b90-4961-8cfc-3f37bfb67581","Type":"ContainerDied","Data":"021415e37c70ce8f7864ca9d751fb5be0f42c656f462e392f6bbcfb72e3be2ad"} Nov 03 13:06:30.538547 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:30.538419 5979 scope.go:122] "RemoveContainer" containerID="8e111aef53dca254384b00b073010d66493f2c2fd5e216ce5bacfa845f43c72a" Nov 03 13:06:30.546560 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:30.546540 5979 scope.go:122] "RemoveContainer" containerID="8e111aef53dca254384b00b073010d66493f2c2fd5e216ce5bacfa845f43c72a" Nov 03 13:06:30.546822 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:06:30.546803 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e111aef53dca254384b00b073010d66493f2c2fd5e216ce5bacfa845f43c72a\": container with ID starting with 8e111aef53dca254384b00b073010d66493f2c2fd5e216ce5bacfa845f43c72a not found: ID does not exist" containerID="8e111aef53dca254384b00b073010d66493f2c2fd5e216ce5bacfa845f43c72a" Nov 03 13:06:30.546885 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:30.546836 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e111aef53dca254384b00b073010d66493f2c2fd5e216ce5bacfa845f43c72a"} err="failed to get container status \"8e111aef53dca254384b00b073010d66493f2c2fd5e216ce5bacfa845f43c72a\": rpc error: code = NotFound desc = could not find container \"8e111aef53dca254384b00b073010d66493f2c2fd5e216ce5bacfa845f43c72a\": container with ID starting with 8e111aef53dca254384b00b073010d66493f2c2fd5e216ce5bacfa845f43c72a not found: ID does not exist" Nov 03 13:06:30.691842 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:30.691807 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f44499665-ff6gp"] Nov 03 13:06:30.738300 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:30.738260 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6f44499665-ff6gp"] Nov 03 13:06:32.112806 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:32.112774 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dcde035-9b90-4961-8cfc-3f37bfb67581" path="/var/lib/kubelet/pods/2dcde035-9b90-4961-8cfc-3f37bfb67581/volumes" Nov 03 13:06:37.232191 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:06:37.232154 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:06:48.693839 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:48.693775 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-b95bb587c-kqqpm" podUID="50b9bd35-6d94-4e09-9824-11ddb8182775" containerName="console" containerID="cri-o://da69b75fecdc51cffa966b233b9bd111a3c60e5a2439cee5941dad9a79005755" gracePeriod=15 Nov 03 13:06:48.948381 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:48.948329 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b95bb587c-kqqpm" Nov 03 13:06:48.990273 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:48.990232 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50b9bd35-6d94-4e09-9824-11ddb8182775-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/50b9bd35-6d94-4e09-9824-11ddb8182775-console-serving-cert\") pod \"50b9bd35-6d94-4e09-9824-11ddb8182775\" (UID: \"50b9bd35-6d94-4e09-9824-11ddb8182775\") " Nov 03 13:06:48.990443 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:48.990288 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50b9bd35-6d94-4e09-9824-11ddb8182775-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/50b9bd35-6d94-4e09-9824-11ddb8182775-oauth-serving-cert\") pod \"50b9bd35-6d94-4e09-9824-11ddb8182775\" (UID: \"50b9bd35-6d94-4e09-9824-11ddb8182775\") " Nov 03 13:06:48.990443 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:48.990325 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50b9bd35-6d94-4e09-9824-11ddb8182775-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/50b9bd35-6d94-4e09-9824-11ddb8182775-console-oauth-config\") pod \"50b9bd35-6d94-4e09-9824-11ddb8182775\" (UID: \"50b9bd35-6d94-4e09-9824-11ddb8182775\") " Nov 03 13:06:48.990443 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:48.990340 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50b9bd35-6d94-4e09-9824-11ddb8182775-console-config\" (UniqueName: \"kubernetes.io/configmap/50b9bd35-6d94-4e09-9824-11ddb8182775-console-config\") pod \"50b9bd35-6d94-4e09-9824-11ddb8182775\" (UID: \"50b9bd35-6d94-4e09-9824-11ddb8182775\") " Nov 03 13:06:48.990443 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:48.990365 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50b9bd35-6d94-4e09-9824-11ddb8182775-tmp\" (UniqueName: \"kubernetes.io/empty-dir/50b9bd35-6d94-4e09-9824-11ddb8182775-tmp\") pod \"50b9bd35-6d94-4e09-9824-11ddb8182775\" (UID: \"50b9bd35-6d94-4e09-9824-11ddb8182775\") " Nov 03 13:06:48.990443 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:48.990394 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50b9bd35-6d94-4e09-9824-11ddb8182775-kube-api-access-zb8nc\" (UniqueName: \"kubernetes.io/projected/50b9bd35-6d94-4e09-9824-11ddb8182775-kube-api-access-zb8nc\") pod \"50b9bd35-6d94-4e09-9824-11ddb8182775\" (UID: \"50b9bd35-6d94-4e09-9824-11ddb8182775\") " Nov 03 13:06:48.990443 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:48.990425 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50b9bd35-6d94-4e09-9824-11ddb8182775-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50b9bd35-6d94-4e09-9824-11ddb8182775-trusted-ca-bundle\") pod \"50b9bd35-6d94-4e09-9824-11ddb8182775\" (UID: \"50b9bd35-6d94-4e09-9824-11ddb8182775\") " Nov 03 13:06:48.990772 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:48.990503 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50b9bd35-6d94-4e09-9824-11ddb8182775-service-ca\" (UniqueName: \"kubernetes.io/configmap/50b9bd35-6d94-4e09-9824-11ddb8182775-service-ca\") pod \"50b9bd35-6d94-4e09-9824-11ddb8182775\" (UID: \"50b9bd35-6d94-4e09-9824-11ddb8182775\") " Nov 03 13:06:48.990772 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:48.990743 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50b9bd35-6d94-4e09-9824-11ddb8182775-tmp" pod "50b9bd35-6d94-4e09-9824-11ddb8182775" (UID: "50b9bd35-6d94-4e09-9824-11ddb8182775"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:06:48.990946 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:48.990914 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50b9bd35-6d94-4e09-9824-11ddb8182775-console-config" pod "50b9bd35-6d94-4e09-9824-11ddb8182775" (UID: "50b9bd35-6d94-4e09-9824-11ddb8182775"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:06:48.991070 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:48.991050 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50b9bd35-6d94-4e09-9824-11ddb8182775-service-ca" pod "50b9bd35-6d94-4e09-9824-11ddb8182775" (UID: "50b9bd35-6d94-4e09-9824-11ddb8182775"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:06:48.991140 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:48.991125 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50b9bd35-6d94-4e09-9824-11ddb8182775-oauth-serving-cert" pod "50b9bd35-6d94-4e09-9824-11ddb8182775" (UID: "50b9bd35-6d94-4e09-9824-11ddb8182775"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:06:48.991196 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:48.991147 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50b9bd35-6d94-4e09-9824-11ddb8182775-trusted-ca-bundle" pod "50b9bd35-6d94-4e09-9824-11ddb8182775" (UID: "50b9bd35-6d94-4e09-9824-11ddb8182775"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:06:48.992415 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:48.992390 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50b9bd35-6d94-4e09-9824-11ddb8182775-console-serving-cert" pod "50b9bd35-6d94-4e09-9824-11ddb8182775" (UID: "50b9bd35-6d94-4e09-9824-11ddb8182775"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:06:48.992538 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:48.992521 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50b9bd35-6d94-4e09-9824-11ddb8182775-kube-api-access-zb8nc" pod "50b9bd35-6d94-4e09-9824-11ddb8182775" (UID: "50b9bd35-6d94-4e09-9824-11ddb8182775"). InnerVolumeSpecName "kube-api-access-zb8nc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:06:48.992714 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:48.992692 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50b9bd35-6d94-4e09-9824-11ddb8182775-console-oauth-config" pod "50b9bd35-6d94-4e09-9824-11ddb8182775" (UID: "50b9bd35-6d94-4e09-9824-11ddb8182775"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:06:49.091683 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:49.091647 5979 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/50b9bd35-6d94-4e09-9824-11ddb8182775-console-serving-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:06:49.091683 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:49.091676 5979 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/50b9bd35-6d94-4e09-9824-11ddb8182775-oauth-serving-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:06:49.091683 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:49.091685 5979 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/50b9bd35-6d94-4e09-9824-11ddb8182775-console-oauth-config\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:06:49.091916 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:49.091694 5979 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/50b9bd35-6d94-4e09-9824-11ddb8182775-console-config\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:06:49.091916 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:49.091703 5979 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50b9bd35-6d94-4e09-9824-11ddb8182775-tmp\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:06:49.091916 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:49.091713 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb8nc\" (UniqueName: \"kubernetes.io/projected/50b9bd35-6d94-4e09-9824-11ddb8182775-kube-api-access-zb8nc\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:06:49.091916 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:49.091722 5979 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50b9bd35-6d94-4e09-9824-11ddb8182775-trusted-ca-bundle\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:06:49.091916 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:49.091731 5979 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/50b9bd35-6d94-4e09-9824-11ddb8182775-service-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:06:49.610192 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:49.610158 5979 generic.go:356] "Generic (PLEG): container finished" podID="50b9bd35-6d94-4e09-9824-11ddb8182775" containerID="da69b75fecdc51cffa966b233b9bd111a3c60e5a2439cee5941dad9a79005755" exitCode=0 Nov 03 13:06:49.610375 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:49.610205 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b95bb587c-kqqpm" Nov 03 13:06:49.610375 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:49.610241 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b95bb587c-kqqpm" event={"ID":"50b9bd35-6d94-4e09-9824-11ddb8182775","Type":"ContainerDied","Data":"da69b75fecdc51cffa966b233b9bd111a3c60e5a2439cee5941dad9a79005755"} Nov 03 13:06:49.610375 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:49.610294 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b95bb587c-kqqpm" event={"ID":"50b9bd35-6d94-4e09-9824-11ddb8182775","Type":"ContainerDied","Data":"24080822f67e938fd1b8934b5682ede3818bfdfd70c97f8e94118e3bd1ad9374"} Nov 03 13:06:49.610375 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:49.610313 5979 scope.go:122] "RemoveContainer" containerID="da69b75fecdc51cffa966b233b9bd111a3c60e5a2439cee5941dad9a79005755" Nov 03 13:06:49.618994 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:49.618971 5979 scope.go:122] "RemoveContainer" containerID="da69b75fecdc51cffa966b233b9bd111a3c60e5a2439cee5941dad9a79005755" Nov 03 13:06:49.619255 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:06:49.619230 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da69b75fecdc51cffa966b233b9bd111a3c60e5a2439cee5941dad9a79005755\": container with ID starting with da69b75fecdc51cffa966b233b9bd111a3c60e5a2439cee5941dad9a79005755 not found: ID does not exist" containerID="da69b75fecdc51cffa966b233b9bd111a3c60e5a2439cee5941dad9a79005755" Nov 03 13:06:49.619299 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:49.619268 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da69b75fecdc51cffa966b233b9bd111a3c60e5a2439cee5941dad9a79005755"} err="failed to get container status \"da69b75fecdc51cffa966b233b9bd111a3c60e5a2439cee5941dad9a79005755\": rpc error: code = NotFound desc = could not find container \"da69b75fecdc51cffa966b233b9bd111a3c60e5a2439cee5941dad9a79005755\": container with ID starting with da69b75fecdc51cffa966b233b9bd111a3c60e5a2439cee5941dad9a79005755 not found: ID does not exist" Nov 03 13:06:49.764296 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:49.764262 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b95bb587c-kqqpm"] Nov 03 13:06:49.767552 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:49.767532 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-b95bb587c-kqqpm"] Nov 03 13:06:50.116657 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:06:50.116627 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50b9bd35-6d94-4e09-9824-11ddb8182775" path="/var/lib/kubelet/pods/50b9bd35-6d94-4e09-9824-11ddb8182775/volumes" Nov 03 13:07:07.232325 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:07:07.232248 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:07:37.232953 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:07:37.232914 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:08:07.232094 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:08:07.232052 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:08:37.232418 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:08:37.232370 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:09:07.232822 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:09:07.232778 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:09:30.597188 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:30.597152 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-7dkh8"] Nov 03 13:09:30.597703 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:30.597518 5979 cpu_manager.go:414] "containerMap: removing container" podUID="2dcde035-9b90-4961-8cfc-3f37bfb67581" containerName="console" Nov 03 13:09:30.597703 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:30.597531 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dcde035-9b90-4961-8cfc-3f37bfb67581" containerName="console" Nov 03 13:09:30.597703 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:30.597542 5979 cpu_manager.go:414] "containerMap: removing container" podUID="50b9bd35-6d94-4e09-9824-11ddb8182775" containerName="console" Nov 03 13:09:30.597703 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:30.597548 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50b9bd35-6d94-4e09-9824-11ddb8182775" containerName="console" Nov 03 13:09:30.597703 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:30.597617 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dcde035-9b90-4961-8cfc-3f37bfb67581" containerName="console" Nov 03 13:09:30.597703 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:30.597625 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="50b9bd35-6d94-4e09-9824-11ddb8182775" containerName="console" Nov 03 13:09:30.600441 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:30.600425 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-7dkh8" Nov 03 13:09:30.653926 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:30.653888 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/63beb893-d5e6-44d7-8de0-25f23ea1aae6-metrics-cert\") pod \"opendatahub-operator-controller-manager-5b7574c576-7dkh8\" (UID: \"63beb893-d5e6-44d7-8de0-25f23ea1aae6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-7dkh8" Nov 03 13:09:30.654094 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:30.653956 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/63beb893-d5e6-44d7-8de0-25f23ea1aae6-manifests\") pod \"opendatahub-operator-controller-manager-5b7574c576-7dkh8\" (UID: \"63beb893-d5e6-44d7-8de0-25f23ea1aae6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-7dkh8" Nov 03 13:09:30.654094 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:30.654048 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/63beb893-d5e6-44d7-8de0-25f23ea1aae6-webhook-cert\") pod \"opendatahub-operator-controller-manager-5b7574c576-7dkh8\" (UID: \"63beb893-d5e6-44d7-8de0-25f23ea1aae6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-7dkh8" Nov 03 13:09:30.654094 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:30.654080 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/63beb893-d5e6-44d7-8de0-25f23ea1aae6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5b7574c576-7dkh8\" (UID: \"63beb893-d5e6-44d7-8de0-25f23ea1aae6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-7dkh8" Nov 03 13:09:30.654200 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:30.654099 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63beb893-d5e6-44d7-8de0-25f23ea1aae6-tmp\") pod \"opendatahub-operator-controller-manager-5b7574c576-7dkh8\" (UID: \"63beb893-d5e6-44d7-8de0-25f23ea1aae6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-7dkh8" Nov 03 13:09:30.654237 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:30.654214 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2cph\" (UniqueName: \"kubernetes.io/projected/63beb893-d5e6-44d7-8de0-25f23ea1aae6-kube-api-access-n2cph\") pod \"opendatahub-operator-controller-manager-5b7574c576-7dkh8\" (UID: \"63beb893-d5e6-44d7-8de0-25f23ea1aae6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-7dkh8" Nov 03 13:09:30.702886 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:30.702833 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 13:09:30.702886 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:30.702877 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-clsch\"" Nov 03 13:09:30.703095 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:30.702899 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 13:09:30.703095 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:30.702933 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 13:09:30.705016 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:30.705001 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 13:09:30.705288 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:30.705271 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 13:09:30.755552 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:30.755514 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2cph\" (UniqueName: \"kubernetes.io/projected/63beb893-d5e6-44d7-8de0-25f23ea1aae6-kube-api-access-n2cph\") pod \"opendatahub-operator-controller-manager-5b7574c576-7dkh8\" (UID: \"63beb893-d5e6-44d7-8de0-25f23ea1aae6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-7dkh8" Nov 03 13:09:30.755754 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:30.755558 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/63beb893-d5e6-44d7-8de0-25f23ea1aae6-metrics-cert\") pod \"opendatahub-operator-controller-manager-5b7574c576-7dkh8\" (UID: \"63beb893-d5e6-44d7-8de0-25f23ea1aae6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-7dkh8" Nov 03 13:09:30.755754 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:30.755618 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/63beb893-d5e6-44d7-8de0-25f23ea1aae6-manifests\") pod \"opendatahub-operator-controller-manager-5b7574c576-7dkh8\" (UID: \"63beb893-d5e6-44d7-8de0-25f23ea1aae6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-7dkh8" Nov 03 13:09:30.755754 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:30.755648 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/63beb893-d5e6-44d7-8de0-25f23ea1aae6-webhook-cert\") pod \"opendatahub-operator-controller-manager-5b7574c576-7dkh8\" (UID: \"63beb893-d5e6-44d7-8de0-25f23ea1aae6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-7dkh8" Nov 03 13:09:30.755935 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:30.755765 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/63beb893-d5e6-44d7-8de0-25f23ea1aae6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5b7574c576-7dkh8\" (UID: \"63beb893-d5e6-44d7-8de0-25f23ea1aae6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-7dkh8" Nov 03 13:09:30.755935 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:30.755803 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63beb893-d5e6-44d7-8de0-25f23ea1aae6-tmp\") pod \"opendatahub-operator-controller-manager-5b7574c576-7dkh8\" (UID: \"63beb893-d5e6-44d7-8de0-25f23ea1aae6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-7dkh8" Nov 03 13:09:30.756046 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:30.756028 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/63beb893-d5e6-44d7-8de0-25f23ea1aae6-manifests\") pod \"opendatahub-operator-controller-manager-5b7574c576-7dkh8\" (UID: \"63beb893-d5e6-44d7-8de0-25f23ea1aae6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-7dkh8" Nov 03 13:09:30.758365 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:30.758338 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63beb893-d5e6-44d7-8de0-25f23ea1aae6-tmp\") pod \"opendatahub-operator-controller-manager-5b7574c576-7dkh8\" (UID: \"63beb893-d5e6-44d7-8de0-25f23ea1aae6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-7dkh8" Nov 03 13:09:30.758541 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:30.758528 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/63beb893-d5e6-44d7-8de0-25f23ea1aae6-metrics-cert\") pod \"opendatahub-operator-controller-manager-5b7574c576-7dkh8\" (UID: \"63beb893-d5e6-44d7-8de0-25f23ea1aae6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-7dkh8" Nov 03 13:09:30.758790 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:30.758768 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/63beb893-d5e6-44d7-8de0-25f23ea1aae6-webhook-cert\") pod \"opendatahub-operator-controller-manager-5b7574c576-7dkh8\" (UID: \"63beb893-d5e6-44d7-8de0-25f23ea1aae6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-7dkh8" Nov 03 13:09:30.759109 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:30.759088 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/63beb893-d5e6-44d7-8de0-25f23ea1aae6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5b7574c576-7dkh8\" (UID: \"63beb893-d5e6-44d7-8de0-25f23ea1aae6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-7dkh8" Nov 03 13:09:30.760064 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:30.760034 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-7dkh8"] Nov 03 13:09:30.813577 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:30.813546 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2cph\" (UniqueName: \"kubernetes.io/projected/63beb893-d5e6-44d7-8de0-25f23ea1aae6-kube-api-access-n2cph\") pod \"opendatahub-operator-controller-manager-5b7574c576-7dkh8\" (UID: \"63beb893-d5e6-44d7-8de0-25f23ea1aae6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-7dkh8" Nov 03 13:09:30.914044 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:30.914013 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-7dkh8" Nov 03 13:09:31.435266 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:31.435231 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-7dkh8"] Nov 03 13:09:31.442726 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:09:31.442701 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63beb893_d5e6_44d7_8de0_25f23ea1aae6.slice/crio-ef461e7b17910498fe9e23df50a0781fbf3d0b1171953a5643209bfb33c7cc33 WatchSource:0}: Error finding container ef461e7b17910498fe9e23df50a0781fbf3d0b1171953a5643209bfb33c7cc33: Status 404 returned error can't find the container with id ef461e7b17910498fe9e23df50a0781fbf3d0b1171953a5643209bfb33c7cc33 Nov 03 13:09:32.206986 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:32.206950 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-7dkh8" event={"ID":"63beb893-d5e6-44d7-8de0-25f23ea1aae6","Type":"ContainerStarted","Data":"ef461e7b17910498fe9e23df50a0781fbf3d0b1171953a5643209bfb33c7cc33"} Nov 03 13:09:35.219787 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:35.219751 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-7dkh8" event={"ID":"63beb893-d5e6-44d7-8de0-25f23ea1aae6","Type":"ContainerStarted","Data":"8542ab3e1be387c09a806c6b498a2907e672835af848e2a0f65d7454168ccbd5"} Nov 03 13:09:35.821955 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:35.821878 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-jlqlx"] Nov 03 13:09:35.825032 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:35.825012 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-jlqlx" Nov 03 13:09:35.875006 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:35.874982 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-l679n\"" Nov 03 13:09:35.875520 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:35.875505 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 03 13:09:35.875600 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:35.875545 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 03 13:09:35.881314 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:35.881294 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-jlqlx"] Nov 03 13:09:35.907951 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:35.907924 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjbbj\" (UniqueName: \"kubernetes.io/projected/ebf13c2a-c88f-4ff2-b554-7d2007ccf156-kube-api-access-bjbbj\") pod \"jobset-operator-674cd59dc5-jlqlx\" (UID: \"ebf13c2a-c88f-4ff2-b554-7d2007ccf156\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-jlqlx" Nov 03 13:09:35.908081 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:35.907956 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ebf13c2a-c88f-4ff2-b554-7d2007ccf156-tmp\") pod \"jobset-operator-674cd59dc5-jlqlx\" (UID: \"ebf13c2a-c88f-4ff2-b554-7d2007ccf156\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-jlqlx" Nov 03 13:09:36.009148 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:36.009119 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjbbj\" (UniqueName: \"kubernetes.io/projected/ebf13c2a-c88f-4ff2-b554-7d2007ccf156-kube-api-access-bjbbj\") pod \"jobset-operator-674cd59dc5-jlqlx\" (UID: \"ebf13c2a-c88f-4ff2-b554-7d2007ccf156\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-jlqlx" Nov 03 13:09:36.009148 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:36.009151 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ebf13c2a-c88f-4ff2-b554-7d2007ccf156-tmp\") pod \"jobset-operator-674cd59dc5-jlqlx\" (UID: \"ebf13c2a-c88f-4ff2-b554-7d2007ccf156\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-jlqlx" Nov 03 13:09:36.009552 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:36.009536 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ebf13c2a-c88f-4ff2-b554-7d2007ccf156-tmp\") pod \"jobset-operator-674cd59dc5-jlqlx\" (UID: \"ebf13c2a-c88f-4ff2-b554-7d2007ccf156\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-jlqlx" Nov 03 13:09:36.062796 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:36.062767 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjbbj\" (UniqueName: \"kubernetes.io/projected/ebf13c2a-c88f-4ff2-b554-7d2007ccf156-kube-api-access-bjbbj\") pod \"jobset-operator-674cd59dc5-jlqlx\" (UID: \"ebf13c2a-c88f-4ff2-b554-7d2007ccf156\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-jlqlx" Nov 03 13:09:36.150908 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:36.150878 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-jlqlx" Nov 03 13:09:36.227504 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:36.226301 5979 generic.go:356] "Generic (PLEG): container finished" podID="63beb893-d5e6-44d7-8de0-25f23ea1aae6" containerID="8542ab3e1be387c09a806c6b498a2907e672835af848e2a0f65d7454168ccbd5" exitCode=0 Nov 03 13:09:36.227504 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:36.226412 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-7dkh8" event={"ID":"63beb893-d5e6-44d7-8de0-25f23ea1aae6","Type":"ContainerDied","Data":"8542ab3e1be387c09a806c6b498a2907e672835af848e2a0f65d7454168ccbd5"} Nov 03 13:09:36.521970 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:36.521882 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-jlqlx"] Nov 03 13:09:36.530066 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:09:36.530028 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podebf13c2a_c88f_4ff2_b554_7d2007ccf156.slice/crio-42b5903f996cfe85a650debf3bff442fcae6f4498a97c022d9f62e3bd8cd8e0d WatchSource:0}: Error finding container 42b5903f996cfe85a650debf3bff442fcae6f4498a97c022d9f62e3bd8cd8e0d: Status 404 returned error can't find the container with id 42b5903f996cfe85a650debf3bff442fcae6f4498a97c022d9f62e3bd8cd8e0d Nov 03 13:09:37.233196 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:37.233165 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-jlqlx" event={"ID":"ebf13c2a-c88f-4ff2-b554-7d2007ccf156","Type":"ContainerStarted","Data":"42b5903f996cfe85a650debf3bff442fcae6f4498a97c022d9f62e3bd8cd8e0d"} Nov 03 13:09:37.235465 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:09:37.235441 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:09:38.239266 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:38.239232 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-7dkh8" event={"ID":"63beb893-d5e6-44d7-8de0-25f23ea1aae6","Type":"ContainerStarted","Data":"f7e29af51ce361961776236ca9237c9813e8b16ea3c2d0757cc98700cf78901c"} Nov 03 13:09:38.239675 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:38.239373 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-7dkh8" Nov 03 13:09:38.502264 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:38.502158 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-7dkh8" podStartSLOduration=3.479080492 podStartE2EDuration="8.502143639s" totalImagesPullingTime="4.53964206s" totalInitContainerRuntime="483.421087ms" isStatefulPod=false podCreationTimestamp="2026-11-03 13:09:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:09:38.455408965 +0000 UTC m=+260.466097208" watchObservedRunningTime="2026-11-03 13:09:38.502143639 +0000 UTC m=+260.512831881" Nov 03 13:09:41.253856 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:41.253822 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-jlqlx" event={"ID":"ebf13c2a-c88f-4ff2-b554-7d2007ccf156","Type":"ContainerStarted","Data":"a0a7b65e423ba865e7b9c3bb94c67de79788610991b9ed474777f302789c5c05"} Nov 03 13:09:49.245786 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:49.245756 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b7574c576-7dkh8" Nov 03 13:09:49.512335 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:49.512239 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-jlqlx" podStartSLOduration=10.248165527 podStartE2EDuration="14.512225888s" totalImagesPullingTime="4.264060361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:09:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:09:41.368158342 +0000 UTC m=+263.378846584" watchObservedRunningTime="2026-11-03 13:09:49.512225888 +0000 UTC m=+271.522914126" Nov 03 13:09:52.235584 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:52.235544 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-57gvd"] Nov 03 13:09:52.238986 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:52.238965 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-57gvd" Nov 03 13:09:52.336575 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:52.336539 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 03 13:09:52.336575 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:52.336536 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 03 13:09:52.336575 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:52.336539 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-j898p\"" Nov 03 13:09:52.337389 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:52.337367 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-57gvd"] Nov 03 13:09:52.366437 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:52.366412 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/751348e5-dd81-44a3-8874-45c723c32c80-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-57gvd\" (UID: \"751348e5-dd81-44a3-8874-45c723c32c80\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-57gvd" Nov 03 13:09:52.366618 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:52.366486 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjvtm\" (UniqueName: \"kubernetes.io/projected/751348e5-dd81-44a3-8874-45c723c32c80-kube-api-access-wjvtm\") pod \"cert-manager-webhook-5dbbbf75cc-57gvd\" (UID: \"751348e5-dd81-44a3-8874-45c723c32c80\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-57gvd" Nov 03 13:09:52.467224 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:52.467187 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjvtm\" (UniqueName: \"kubernetes.io/projected/751348e5-dd81-44a3-8874-45c723c32c80-kube-api-access-wjvtm\") pod \"cert-manager-webhook-5dbbbf75cc-57gvd\" (UID: \"751348e5-dd81-44a3-8874-45c723c32c80\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-57gvd" Nov 03 13:09:52.467419 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:52.467279 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/751348e5-dd81-44a3-8874-45c723c32c80-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-57gvd\" (UID: \"751348e5-dd81-44a3-8874-45c723c32c80\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-57gvd" Nov 03 13:09:52.521535 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:52.521457 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/751348e5-dd81-44a3-8874-45c723c32c80-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-57gvd\" (UID: \"751348e5-dd81-44a3-8874-45c723c32c80\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-57gvd" Nov 03 13:09:52.521682 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:52.521639 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjvtm\" (UniqueName: \"kubernetes.io/projected/751348e5-dd81-44a3-8874-45c723c32c80-kube-api-access-wjvtm\") pod \"cert-manager-webhook-5dbbbf75cc-57gvd\" (UID: \"751348e5-dd81-44a3-8874-45c723c32c80\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-57gvd" Nov 03 13:09:52.551628 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:52.551596 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-57gvd" Nov 03 13:09:52.930014 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:52.929980 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-57gvd"] Nov 03 13:09:52.937460 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:09:52.937432 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod751348e5_dd81_44a3_8874_45c723c32c80.slice/crio-fad144414575e13e54db4d42db58a3a8bba225ecafc5c14f5e62540993aa44fb WatchSource:0}: Error finding container fad144414575e13e54db4d42db58a3a8bba225ecafc5c14f5e62540993aa44fb: Status 404 returned error can't find the container with id fad144414575e13e54db4d42db58a3a8bba225ecafc5c14f5e62540993aa44fb Nov 03 13:09:53.306446 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:53.306363 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-57gvd" event={"ID":"751348e5-dd81-44a3-8874-45c723c32c80","Type":"ContainerStarted","Data":"fad144414575e13e54db4d42db58a3a8bba225ecafc5c14f5e62540993aa44fb"} Nov 03 13:09:58.329363 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:58.329324 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-57gvd" event={"ID":"751348e5-dd81-44a3-8874-45c723c32c80","Type":"ContainerStarted","Data":"a4ee280f3df366b21d9dfbb92b319e7c524e93943858512944667ce2a1e828fe"} Nov 03 13:09:58.329790 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:58.329448 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-57gvd" Nov 03 13:09:58.439746 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:58.439693 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-57gvd" podStartSLOduration=1.9996945529999999 podStartE2EDuration="6.439679101s" totalImagesPullingTime="4.439984548s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:09:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:09:58.439465637 +0000 UTC m=+280.450153879" watchObservedRunningTime="2026-11-03 13:09:58.439679101 +0000 UTC m=+280.450367337" Nov 03 13:09:59.439459 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:59.439371 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-98gbs"] Nov 03 13:09:59.442815 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:59.442793 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-98gbs" Nov 03 13:09:59.495090 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:59.495055 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-98gbs"] Nov 03 13:09:59.531203 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:59.531172 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89tcc\" (UniqueName: \"kubernetes.io/projected/2a9ac56d-a4e5-4975-a6a2-837947198479-kube-api-access-89tcc\") pod \"cert-manager-cainjector-5c57bbd4fc-98gbs\" (UID: \"2a9ac56d-a4e5-4975-a6a2-837947198479\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-98gbs" Nov 03 13:09:59.531350 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:59.531208 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2a9ac56d-a4e5-4975-a6a2-837947198479-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-98gbs\" (UID: \"2a9ac56d-a4e5-4975-a6a2-837947198479\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-98gbs" Nov 03 13:09:59.540321 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:59.540289 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-tqsn4\"" Nov 03 13:09:59.632173 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:59.632136 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89tcc\" (UniqueName: \"kubernetes.io/projected/2a9ac56d-a4e5-4975-a6a2-837947198479-kube-api-access-89tcc\") pod \"cert-manager-cainjector-5c57bbd4fc-98gbs\" (UID: \"2a9ac56d-a4e5-4975-a6a2-837947198479\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-98gbs" Nov 03 13:09:59.632360 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:59.632182 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2a9ac56d-a4e5-4975-a6a2-837947198479-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-98gbs\" (UID: \"2a9ac56d-a4e5-4975-a6a2-837947198479\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-98gbs" Nov 03 13:09:59.689096 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:59.689064 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2a9ac56d-a4e5-4975-a6a2-837947198479-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-98gbs\" (UID: \"2a9ac56d-a4e5-4975-a6a2-837947198479\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-98gbs" Nov 03 13:09:59.689251 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:59.689190 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89tcc\" (UniqueName: \"kubernetes.io/projected/2a9ac56d-a4e5-4975-a6a2-837947198479-kube-api-access-89tcc\") pod \"cert-manager-cainjector-5c57bbd4fc-98gbs\" (UID: \"2a9ac56d-a4e5-4975-a6a2-837947198479\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-98gbs" Nov 03 13:09:59.753879 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:09:59.753800 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-98gbs" Nov 03 13:10:00.126416 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:00.126326 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-98gbs"] Nov 03 13:10:00.133942 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:10:00.133907 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a9ac56d_a4e5_4975_a6a2_837947198479.slice/crio-3a7c55fd598d2f6181ca6f8ab7e54c90b35273d0ba5c595b5b599a95d0e9b7b2 WatchSource:0}: Error finding container 3a7c55fd598d2f6181ca6f8ab7e54c90b35273d0ba5c595b5b599a95d0e9b7b2: Status 404 returned error can't find the container with id 3a7c55fd598d2f6181ca6f8ab7e54c90b35273d0ba5c595b5b599a95d0e9b7b2 Nov 03 13:10:00.339626 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:00.339595 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-98gbs" event={"ID":"2a9ac56d-a4e5-4975-a6a2-837947198479","Type":"ContainerStarted","Data":"5b48c6ff475636d292d3d90bbed84f2a2763e63d84f6f88bef022665fb006fdc"} Nov 03 13:10:00.339811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:00.339635 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-98gbs" event={"ID":"2a9ac56d-a4e5-4975-a6a2-837947198479","Type":"ContainerStarted","Data":"3a7c55fd598d2f6181ca6f8ab7e54c90b35273d0ba5c595b5b599a95d0e9b7b2"} Nov 03 13:10:00.446287 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:00.446237 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-98gbs" podStartSLOduration=1.446223039 podStartE2EDuration="1.446223039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:09:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:10:00.445164969 +0000 UTC m=+282.455853211" watchObservedRunningTime="2026-11-03 13:10:00.446223039 +0000 UTC m=+282.456911281" Nov 03 13:10:03.163593 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:03.163559 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-x9zs4"] Nov 03 13:10:03.167106 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:03.167083 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-x9zs4" Nov 03 13:10:03.264075 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:03.264032 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bphf\" (UniqueName: \"kubernetes.io/projected/d1361d2d-b34a-4ff1-83cb-fd4973e4418a-kube-api-access-9bphf\") pod \"cert-manager-597b95cb84-x9zs4\" (UID: \"d1361d2d-b34a-4ff1-83cb-fd4973e4418a\") " pod="cert-manager/cert-manager-597b95cb84-x9zs4" Nov 03 13:10:03.264075 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:03.264035 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-njvdh\"" Nov 03 13:10:03.264280 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:03.264144 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d1361d2d-b34a-4ff1-83cb-fd4973e4418a-bound-sa-token\") pod \"cert-manager-597b95cb84-x9zs4\" (UID: \"d1361d2d-b34a-4ff1-83cb-fd4973e4418a\") " pod="cert-manager/cert-manager-597b95cb84-x9zs4" Nov 03 13:10:03.265596 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:03.265577 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-x9zs4"] Nov 03 13:10:03.364995 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:03.364957 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bphf\" (UniqueName: \"kubernetes.io/projected/d1361d2d-b34a-4ff1-83cb-fd4973e4418a-kube-api-access-9bphf\") pod \"cert-manager-597b95cb84-x9zs4\" (UID: \"d1361d2d-b34a-4ff1-83cb-fd4973e4418a\") " pod="cert-manager/cert-manager-597b95cb84-x9zs4" Nov 03 13:10:03.365197 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:03.365003 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d1361d2d-b34a-4ff1-83cb-fd4973e4418a-bound-sa-token\") pod \"cert-manager-597b95cb84-x9zs4\" (UID: \"d1361d2d-b34a-4ff1-83cb-fd4973e4418a\") " pod="cert-manager/cert-manager-597b95cb84-x9zs4" Nov 03 13:10:03.425866 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:03.425795 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d1361d2d-b34a-4ff1-83cb-fd4973e4418a-bound-sa-token\") pod \"cert-manager-597b95cb84-x9zs4\" (UID: \"d1361d2d-b34a-4ff1-83cb-fd4973e4418a\") " pod="cert-manager/cert-manager-597b95cb84-x9zs4" Nov 03 13:10:03.425998 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:03.425916 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bphf\" (UniqueName: \"kubernetes.io/projected/d1361d2d-b34a-4ff1-83cb-fd4973e4418a-kube-api-access-9bphf\") pod \"cert-manager-597b95cb84-x9zs4\" (UID: \"d1361d2d-b34a-4ff1-83cb-fd4973e4418a\") " pod="cert-manager/cert-manager-597b95cb84-x9zs4" Nov 03 13:10:03.478705 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:03.478655 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-x9zs4" Nov 03 13:10:03.845823 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:03.845742 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-x9zs4"] Nov 03 13:10:03.853175 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:10:03.853142 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd1361d2d_b34a_4ff1_83cb_fd4973e4418a.slice/crio-3b7366e710c0972ceca50d73909e7cfa76855bda3b74f58e317bfa5078f6151c WatchSource:0}: Error finding container 3b7366e710c0972ceca50d73909e7cfa76855bda3b74f58e317bfa5078f6151c: Status 404 returned error can't find the container with id 3b7366e710c0972ceca50d73909e7cfa76855bda3b74f58e317bfa5078f6151c Nov 03 13:10:04.336449 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:04.336416 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-57gvd" Nov 03 13:10:04.364971 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:04.364937 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-x9zs4" event={"ID":"d1361d2d-b34a-4ff1-83cb-fd4973e4418a","Type":"ContainerStarted","Data":"93f666c25ffda9a87890ce67dd1ad216feb28b3e55e2221054e2bbabcdd88239"} Nov 03 13:10:04.365131 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:04.364985 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-x9zs4" event={"ID":"d1361d2d-b34a-4ff1-83cb-fd4973e4418a","Type":"ContainerStarted","Data":"3b7366e710c0972ceca50d73909e7cfa76855bda3b74f58e317bfa5078f6151c"} Nov 03 13:10:04.552695 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:04.552646 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-x9zs4" podStartSLOduration=1.552621948 podStartE2EDuration="1.552621948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:10:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:10:04.552220188 +0000 UTC m=+286.562908431" watchObservedRunningTime="2026-11-03 13:10:04.552621948 +0000 UTC m=+286.563310189" Nov 03 13:10:07.234596 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:10:07.234555 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:10:10.890835 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:10.890798 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-77875f86b8-krvtz"] Nov 03 13:10:10.896989 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:10.896964 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-77875f86b8-krvtz" Nov 03 13:10:10.935290 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:10.935259 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/593e2573-815f-4352-8e5f-13ab39dbd37e-tmp\") pod \"kube-auth-proxy-77875f86b8-krvtz\" (UID: \"593e2573-815f-4352-8e5f-13ab39dbd37e\") " pod="openshift-ingress/kube-auth-proxy-77875f86b8-krvtz" Nov 03 13:10:10.935486 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:10.935297 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/593e2573-815f-4352-8e5f-13ab39dbd37e-tls-certs\") pod \"kube-auth-proxy-77875f86b8-krvtz\" (UID: \"593e2573-815f-4352-8e5f-13ab39dbd37e\") " pod="openshift-ingress/kube-auth-proxy-77875f86b8-krvtz" Nov 03 13:10:10.935486 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:10.935436 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qvcz\" (UniqueName: \"kubernetes.io/projected/593e2573-815f-4352-8e5f-13ab39dbd37e-kube-api-access-4qvcz\") pod \"kube-auth-proxy-77875f86b8-krvtz\" (UID: \"593e2573-815f-4352-8e5f-13ab39dbd37e\") " pod="openshift-ingress/kube-auth-proxy-77875f86b8-krvtz" Nov 03 13:10:10.986738 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:10.986701 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 13:10:10.986898 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:10.986817 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 13:10:10.993843 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:10.993815 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-p6lgj\"" Nov 03 13:10:10.993843 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:10.993813 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 03 13:10:10.994068 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:10.993825 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 03 13:10:10.995039 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:10.995018 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-77875f86b8-krvtz"] Nov 03 13:10:11.035949 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:11.035912 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/593e2573-815f-4352-8e5f-13ab39dbd37e-tmp\") pod \"kube-auth-proxy-77875f86b8-krvtz\" (UID: \"593e2573-815f-4352-8e5f-13ab39dbd37e\") " pod="openshift-ingress/kube-auth-proxy-77875f86b8-krvtz" Nov 03 13:10:11.036146 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:11.035957 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/593e2573-815f-4352-8e5f-13ab39dbd37e-tls-certs\") pod \"kube-auth-proxy-77875f86b8-krvtz\" (UID: \"593e2573-815f-4352-8e5f-13ab39dbd37e\") " pod="openshift-ingress/kube-auth-proxy-77875f86b8-krvtz" Nov 03 13:10:11.036146 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:11.036027 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qvcz\" (UniqueName: \"kubernetes.io/projected/593e2573-815f-4352-8e5f-13ab39dbd37e-kube-api-access-4qvcz\") pod \"kube-auth-proxy-77875f86b8-krvtz\" (UID: \"593e2573-815f-4352-8e5f-13ab39dbd37e\") " pod="openshift-ingress/kube-auth-proxy-77875f86b8-krvtz" Nov 03 13:10:11.038240 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:11.038216 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/593e2573-815f-4352-8e5f-13ab39dbd37e-tmp\") pod \"kube-auth-proxy-77875f86b8-krvtz\" (UID: \"593e2573-815f-4352-8e5f-13ab39dbd37e\") " pod="openshift-ingress/kube-auth-proxy-77875f86b8-krvtz" Nov 03 13:10:11.038380 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:11.038359 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/593e2573-815f-4352-8e5f-13ab39dbd37e-tls-certs\") pod \"kube-auth-proxy-77875f86b8-krvtz\" (UID: \"593e2573-815f-4352-8e5f-13ab39dbd37e\") " pod="openshift-ingress/kube-auth-proxy-77875f86b8-krvtz" Nov 03 13:10:11.092603 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:11.092572 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qvcz\" (UniqueName: \"kubernetes.io/projected/593e2573-815f-4352-8e5f-13ab39dbd37e-kube-api-access-4qvcz\") pod \"kube-auth-proxy-77875f86b8-krvtz\" (UID: \"593e2573-815f-4352-8e5f-13ab39dbd37e\") " pod="openshift-ingress/kube-auth-proxy-77875f86b8-krvtz" Nov 03 13:10:11.208170 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:11.208080 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-77875f86b8-krvtz" Nov 03 13:10:11.576338 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:11.576256 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-77875f86b8-krvtz"] Nov 03 13:10:11.583393 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:10:11.583365 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod593e2573_815f_4352_8e5f_13ab39dbd37e.slice/crio-1d9783d7c8afa933496c7f38667757f332bc50101d0bd96a0f1d8e98eaebbb9c WatchSource:0}: Error finding container 1d9783d7c8afa933496c7f38667757f332bc50101d0bd96a0f1d8e98eaebbb9c: Status 404 returned error can't find the container with id 1d9783d7c8afa933496c7f38667757f332bc50101d0bd96a0f1d8e98eaebbb9c Nov 03 13:10:12.400341 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:12.400304 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-77875f86b8-krvtz" event={"ID":"593e2573-815f-4352-8e5f-13ab39dbd37e","Type":"ContainerStarted","Data":"1d9783d7c8afa933496c7f38667757f332bc50101d0bd96a0f1d8e98eaebbb9c"} Nov 03 13:10:15.416655 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:15.416618 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-77875f86b8-krvtz" event={"ID":"593e2573-815f-4352-8e5f-13ab39dbd37e","Type":"ContainerStarted","Data":"102c9bf4da7e6e913256878ff99750b9ea12bba56bb3ac89f837cedfa4ecc105"} Nov 03 13:10:15.524520 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:15.524454 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-77875f86b8-krvtz" podStartSLOduration=2.361877475 podStartE2EDuration="5.524440012s" totalImagesPullingTime="3.162562537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:10:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:10:15.523202701 +0000 UTC m=+297.533890943" watchObservedRunningTime="2026-11-03 13:10:15.524440012 +0000 UTC m=+297.535128254" Nov 03 13:10:27.309563 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:27.309526 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-kz62x"] Nov 03 13:10:27.314655 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:27.314633 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-kz62x" Nov 03 13:10:27.365736 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:27.365693 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-kz62x"] Nov 03 13:10:27.387915 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:27.387875 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fe5fa66d-21ae-4f49-9c64-98d930ed7c6f-config-file\") pod \"limitador-limitador-74dfcfb894-kz62x\" (UID: \"fe5fa66d-21ae-4f49-9c64-98d930ed7c6f\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-kz62x" Nov 03 13:10:27.388107 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:27.388016 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ckwc\" (UniqueName: \"kubernetes.io/projected/fe5fa66d-21ae-4f49-9c64-98d930ed7c6f-kube-api-access-2ckwc\") pod \"limitador-limitador-74dfcfb894-kz62x\" (UID: \"fe5fa66d-21ae-4f49-9c64-98d930ed7c6f\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-kz62x" Nov 03 13:10:27.410852 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:27.410818 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 13:10:27.410975 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:27.410864 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 13:10:27.410975 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:27.410907 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 03 13:10:27.410975 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:27.410935 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-wtvwk\"" Nov 03 13:10:27.488671 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:27.488636 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fe5fa66d-21ae-4f49-9c64-98d930ed7c6f-config-file\") pod \"limitador-limitador-74dfcfb894-kz62x\" (UID: \"fe5fa66d-21ae-4f49-9c64-98d930ed7c6f\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-kz62x" Nov 03 13:10:27.488850 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:27.488679 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ckwc\" (UniqueName: \"kubernetes.io/projected/fe5fa66d-21ae-4f49-9c64-98d930ed7c6f-kube-api-access-2ckwc\") pod \"limitador-limitador-74dfcfb894-kz62x\" (UID: \"fe5fa66d-21ae-4f49-9c64-98d930ed7c6f\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-kz62x" Nov 03 13:10:27.489352 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:27.489330 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fe5fa66d-21ae-4f49-9c64-98d930ed7c6f-config-file\") pod \"limitador-limitador-74dfcfb894-kz62x\" (UID: \"fe5fa66d-21ae-4f49-9c64-98d930ed7c6f\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-kz62x" Nov 03 13:10:27.544383 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:27.544353 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ckwc\" (UniqueName: \"kubernetes.io/projected/fe5fa66d-21ae-4f49-9c64-98d930ed7c6f-kube-api-access-2ckwc\") pod \"limitador-limitador-74dfcfb894-kz62x\" (UID: \"fe5fa66d-21ae-4f49-9c64-98d930ed7c6f\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-kz62x" Nov 03 13:10:27.554133 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:27.554105 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-kz62x"] Nov 03 13:10:27.628089 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:27.628011 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-kz62x" Nov 03 13:10:27.998779 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:27.998741 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-kz62x"] Nov 03 13:10:28.006401 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:10:28.006370 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe5fa66d_21ae_4f49_9c64_98d930ed7c6f.slice/crio-f6deea3aace5119110332b0269488ed3a24f08f08c0696b3516c35ee9dfbaa16 WatchSource:0}: Error finding container f6deea3aace5119110332b0269488ed3a24f08f08c0696b3516c35ee9dfbaa16: Status 404 returned error can't find the container with id f6deea3aace5119110332b0269488ed3a24f08f08c0696b3516c35ee9dfbaa16 Nov 03 13:10:28.473459 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:28.473425 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-kz62x" event={"ID":"fe5fa66d-21ae-4f49-9c64-98d930ed7c6f","Type":"ContainerStarted","Data":"f6deea3aace5119110332b0269488ed3a24f08f08c0696b3516c35ee9dfbaa16"} Nov 03 13:10:31.343552 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:31.343519 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:10:31.348905 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:31.348878 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-466qf" Nov 03 13:10:31.400937 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:31.400899 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:10:31.427677 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:31.427644 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f31d565d-15b1-4b03-bbfc-ff43778c7a95-config-file\") pod \"limitador-limitador-7c9b87d574-466qf\" (UID: \"f31d565d-15b1-4b03-bbfc-ff43778c7a95\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-466qf" Nov 03 13:10:31.427837 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:31.427704 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8ww7\" (UniqueName: \"kubernetes.io/projected/f31d565d-15b1-4b03-bbfc-ff43778c7a95-kube-api-access-q8ww7\") pod \"limitador-limitador-7c9b87d574-466qf\" (UID: \"f31d565d-15b1-4b03-bbfc-ff43778c7a95\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-466qf" Nov 03 13:10:31.528974 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:31.528935 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8ww7\" (UniqueName: \"kubernetes.io/projected/f31d565d-15b1-4b03-bbfc-ff43778c7a95-kube-api-access-q8ww7\") pod \"limitador-limitador-7c9b87d574-466qf\" (UID: \"f31d565d-15b1-4b03-bbfc-ff43778c7a95\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-466qf" Nov 03 13:10:31.529187 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:31.529064 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f31d565d-15b1-4b03-bbfc-ff43778c7a95-config-file\") pod \"limitador-limitador-7c9b87d574-466qf\" (UID: \"f31d565d-15b1-4b03-bbfc-ff43778c7a95\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-466qf" Nov 03 13:10:31.529760 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:31.529738 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f31d565d-15b1-4b03-bbfc-ff43778c7a95-config-file\") pod \"limitador-limitador-7c9b87d574-466qf\" (UID: \"f31d565d-15b1-4b03-bbfc-ff43778c7a95\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-466qf" Nov 03 13:10:31.582816 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:31.582781 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8ww7\" (UniqueName: \"kubernetes.io/projected/f31d565d-15b1-4b03-bbfc-ff43778c7a95-kube-api-access-q8ww7\") pod \"limitador-limitador-7c9b87d574-466qf\" (UID: \"f31d565d-15b1-4b03-bbfc-ff43778c7a95\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-466qf" Nov 03 13:10:31.663121 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:31.663083 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-466qf" Nov 03 13:10:31.666878 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:31.666844 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:10:32.034768 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:32.034684 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:10:32.043621 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:10:32.043588 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf31d565d_15b1_4b03_bbfc_ff43778c7a95.slice/crio-088531da4b28cc94fde7679a9bf77b90843bd30b329f8898bfc9ad9303a6ba5f WatchSource:0}: Error finding container 088531da4b28cc94fde7679a9bf77b90843bd30b329f8898bfc9ad9303a6ba5f: Status 404 returned error can't find the container with id 088531da4b28cc94fde7679a9bf77b90843bd30b329f8898bfc9ad9303a6ba5f Nov 03 13:10:32.491363 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:32.491322 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-466qf" event={"ID":"f31d565d-15b1-4b03-bbfc-ff43778c7a95","Type":"ContainerStarted","Data":"088531da4b28cc94fde7679a9bf77b90843bd30b329f8898bfc9ad9303a6ba5f"} Nov 03 13:10:32.492784 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:32.492761 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-kz62x" event={"ID":"fe5fa66d-21ae-4f49-9c64-98d930ed7c6f","Type":"ContainerStarted","Data":"a8ac199819e0b6c8ebabdc56f7e2f20aeff46756bb1b4f65a22999dec06be0fb"} Nov 03 13:10:32.492907 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:32.492888 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-kz62x" Nov 03 13:10:32.602543 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:32.602467 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-kz62x" podStartSLOduration=1.471467394 podStartE2EDuration="5.60245514s" totalImagesPullingTime="4.130987746s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:10:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:10:32.601956247 +0000 UTC m=+314.612644490" watchObservedRunningTime="2026-11-03 13:10:32.60245514 +0000 UTC m=+314.613143381" Nov 03 13:10:34.502968 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:34.502931 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-466qf" event={"ID":"f31d565d-15b1-4b03-bbfc-ff43778c7a95","Type":"ContainerStarted","Data":"d8e3f2005633376a715e26cf93b16ab628670db26a1af7486f30ead77261db0b"} Nov 03 13:10:34.503387 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:34.503006 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-466qf" Nov 03 13:10:34.612315 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:34.612256 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-466qf" podStartSLOduration=2.196736912 podStartE2EDuration="3.612240405s" totalImagesPullingTime="1.415503493s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:10:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:10:34.610999982 +0000 UTC m=+316.621688224" watchObservedRunningTime="2026-11-03 13:10:34.612240405 +0000 UTC m=+316.622928648" Nov 03 13:10:37.233697 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:10:37.233656 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:10:43.498447 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:43.498415 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-kz62x" Nov 03 13:10:45.507932 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:45.507901 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-466qf" Nov 03 13:10:45.661596 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:45.661552 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-kz62x"] Nov 03 13:10:45.661810 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:45.661781 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-kz62x" podUID="fe5fa66d-21ae-4f49-9c64-98d930ed7c6f" containerName="limitador" containerID="cri-o://a8ac199819e0b6c8ebabdc56f7e2f20aeff46756bb1b4f65a22999dec06be0fb" gracePeriod=30 Nov 03 13:10:46.231000 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:46.230936 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-kz62x" Nov 03 13:10:46.265293 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:46.265259 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe5fa66d-21ae-4f49-9c64-98d930ed7c6f-config-file\" (UniqueName: \"kubernetes.io/configmap/fe5fa66d-21ae-4f49-9c64-98d930ed7c6f-config-file\") pod \"fe5fa66d-21ae-4f49-9c64-98d930ed7c6f\" (UID: \"fe5fa66d-21ae-4f49-9c64-98d930ed7c6f\") " Nov 03 13:10:46.265434 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:46.265304 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe5fa66d-21ae-4f49-9c64-98d930ed7c6f-kube-api-access-2ckwc\" (UniqueName: \"kubernetes.io/projected/fe5fa66d-21ae-4f49-9c64-98d930ed7c6f-kube-api-access-2ckwc\") pod \"fe5fa66d-21ae-4f49-9c64-98d930ed7c6f\" (UID: \"fe5fa66d-21ae-4f49-9c64-98d930ed7c6f\") " Nov 03 13:10:46.265652 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:46.265629 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe5fa66d-21ae-4f49-9c64-98d930ed7c6f-config-file" pod "fe5fa66d-21ae-4f49-9c64-98d930ed7c6f" (UID: "fe5fa66d-21ae-4f49-9c64-98d930ed7c6f"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:10:46.267444 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:46.267419 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe5fa66d-21ae-4f49-9c64-98d930ed7c6f-kube-api-access-2ckwc" pod "fe5fa66d-21ae-4f49-9c64-98d930ed7c6f" (UID: "fe5fa66d-21ae-4f49-9c64-98d930ed7c6f"). InnerVolumeSpecName "kube-api-access-2ckwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:10:46.366004 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:46.365969 5979 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fe5fa66d-21ae-4f49-9c64-98d930ed7c6f-config-file\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:10:46.366004 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:46.365996 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ckwc\" (UniqueName: \"kubernetes.io/projected/fe5fa66d-21ae-4f49-9c64-98d930ed7c6f-kube-api-access-2ckwc\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:10:46.555777 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:46.555699 5979 generic.go:356] "Generic (PLEG): container finished" podID="fe5fa66d-21ae-4f49-9c64-98d930ed7c6f" containerID="a8ac199819e0b6c8ebabdc56f7e2f20aeff46756bb1b4f65a22999dec06be0fb" exitCode=0 Nov 03 13:10:46.555777 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:46.555744 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-kz62x" event={"ID":"fe5fa66d-21ae-4f49-9c64-98d930ed7c6f","Type":"ContainerDied","Data":"a8ac199819e0b6c8ebabdc56f7e2f20aeff46756bb1b4f65a22999dec06be0fb"} Nov 03 13:10:46.555777 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:46.555747 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-kz62x" Nov 03 13:10:46.556275 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:46.555784 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-kz62x" event={"ID":"fe5fa66d-21ae-4f49-9c64-98d930ed7c6f","Type":"ContainerDied","Data":"f6deea3aace5119110332b0269488ed3a24f08f08c0696b3516c35ee9dfbaa16"} Nov 03 13:10:46.556275 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:46.555807 5979 scope.go:122] "RemoveContainer" containerID="a8ac199819e0b6c8ebabdc56f7e2f20aeff46756bb1b4f65a22999dec06be0fb" Nov 03 13:10:46.565046 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:46.565020 5979 scope.go:122] "RemoveContainer" containerID="a8ac199819e0b6c8ebabdc56f7e2f20aeff46756bb1b4f65a22999dec06be0fb" Nov 03 13:10:46.565315 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:10:46.565298 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8ac199819e0b6c8ebabdc56f7e2f20aeff46756bb1b4f65a22999dec06be0fb\": container with ID starting with a8ac199819e0b6c8ebabdc56f7e2f20aeff46756bb1b4f65a22999dec06be0fb not found: ID does not exist" containerID="a8ac199819e0b6c8ebabdc56f7e2f20aeff46756bb1b4f65a22999dec06be0fb" Nov 03 13:10:46.565380 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:46.565323 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8ac199819e0b6c8ebabdc56f7e2f20aeff46756bb1b4f65a22999dec06be0fb"} err="failed to get container status \"a8ac199819e0b6c8ebabdc56f7e2f20aeff46756bb1b4f65a22999dec06be0fb\": rpc error: code = NotFound desc = could not find container \"a8ac199819e0b6c8ebabdc56f7e2f20aeff46756bb1b4f65a22999dec06be0fb\": container with ID starting with a8ac199819e0b6c8ebabdc56f7e2f20aeff46756bb1b4f65a22999dec06be0fb not found: ID does not exist" Nov 03 13:10:46.708081 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:46.708027 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-kz62x"] Nov 03 13:10:46.710852 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:46.710822 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-kz62x"] Nov 03 13:10:48.113344 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:10:48.113310 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe5fa66d-21ae-4f49-9c64-98d930ed7c6f" path="/var/lib/kubelet/pods/fe5fa66d-21ae-4f49-9c64-98d930ed7c6f/volumes" Nov 03 13:11:07.233795 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:11:07.233758 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:11:12.958000 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:12.957963 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7fc47b6789-65mt4"] Nov 03 13:11:12.958392 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:12.958319 5979 cpu_manager.go:414] "containerMap: removing container" podUID="fe5fa66d-21ae-4f49-9c64-98d930ed7c6f" containerName="limitador" Nov 03 13:11:12.958392 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:12.958330 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe5fa66d-21ae-4f49-9c64-98d930ed7c6f" containerName="limitador" Nov 03 13:11:12.958497 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:12.958407 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe5fa66d-21ae-4f49-9c64-98d930ed7c6f" containerName="limitador" Nov 03 13:11:12.963149 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:12.963125 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7fc47b6789-65mt4" Nov 03 13:11:13.059265 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:13.059232 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 13:11:13.059454 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:13.059355 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-rbwxk\"" Nov 03 13:11:13.059577 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:13.059508 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 13:11:13.059577 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:13.059523 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 13:11:13.059648 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:13.059607 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 13:11:13.059877 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:13.059861 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 13:11:13.061239 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:13.061217 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7fc47b6789-65mt4"] Nov 03 13:11:13.111196 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:13.111163 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/640c6789-69a9-4897-9fc9-1a01b16dbead-metrics-certs\") pod \"maas-controller-7fc47b6789-65mt4\" (UID: \"640c6789-69a9-4897-9fc9-1a01b16dbead\") " pod="opendatahub/maas-controller-7fc47b6789-65mt4" Nov 03 13:11:13.111196 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:13.111192 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfm47\" (UniqueName: \"kubernetes.io/projected/640c6789-69a9-4897-9fc9-1a01b16dbead-kube-api-access-bfm47\") pod \"maas-controller-7fc47b6789-65mt4\" (UID: \"640c6789-69a9-4897-9fc9-1a01b16dbead\") " pod="opendatahub/maas-controller-7fc47b6789-65mt4" Nov 03 13:11:13.111425 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:13.111212 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/640c6789-69a9-4897-9fc9-1a01b16dbead-webhook-cert\") pod \"maas-controller-7fc47b6789-65mt4\" (UID: \"640c6789-69a9-4897-9fc9-1a01b16dbead\") " pod="opendatahub/maas-controller-7fc47b6789-65mt4" Nov 03 13:11:13.212171 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:13.212090 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/640c6789-69a9-4897-9fc9-1a01b16dbead-metrics-certs\") pod \"maas-controller-7fc47b6789-65mt4\" (UID: \"640c6789-69a9-4897-9fc9-1a01b16dbead\") " pod="opendatahub/maas-controller-7fc47b6789-65mt4" Nov 03 13:11:13.212171 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:13.212125 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfm47\" (UniqueName: \"kubernetes.io/projected/640c6789-69a9-4897-9fc9-1a01b16dbead-kube-api-access-bfm47\") pod \"maas-controller-7fc47b6789-65mt4\" (UID: \"640c6789-69a9-4897-9fc9-1a01b16dbead\") " pod="opendatahub/maas-controller-7fc47b6789-65mt4" Nov 03 13:11:13.212171 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:13.212147 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/640c6789-69a9-4897-9fc9-1a01b16dbead-webhook-cert\") pod \"maas-controller-7fc47b6789-65mt4\" (UID: \"640c6789-69a9-4897-9fc9-1a01b16dbead\") " pod="opendatahub/maas-controller-7fc47b6789-65mt4" Nov 03 13:11:13.214665 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:13.214637 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/640c6789-69a9-4897-9fc9-1a01b16dbead-metrics-certs\") pod \"maas-controller-7fc47b6789-65mt4\" (UID: \"640c6789-69a9-4897-9fc9-1a01b16dbead\") " pod="opendatahub/maas-controller-7fc47b6789-65mt4" Nov 03 13:11:13.214791 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:13.214768 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/640c6789-69a9-4897-9fc9-1a01b16dbead-webhook-cert\") pod \"maas-controller-7fc47b6789-65mt4\" (UID: \"640c6789-69a9-4897-9fc9-1a01b16dbead\") " pod="opendatahub/maas-controller-7fc47b6789-65mt4" Nov 03 13:11:13.265707 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:13.265675 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfm47\" (UniqueName: \"kubernetes.io/projected/640c6789-69a9-4897-9fc9-1a01b16dbead-kube-api-access-bfm47\") pod \"maas-controller-7fc47b6789-65mt4\" (UID: \"640c6789-69a9-4897-9fc9-1a01b16dbead\") " pod="opendatahub/maas-controller-7fc47b6789-65mt4" Nov 03 13:11:13.275536 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:13.275513 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7fc47b6789-65mt4" Nov 03 13:11:13.528616 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:13.528545 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-588d644cc-njxzq"] Nov 03 13:11:13.535634 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:13.533235 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-588d644cc-njxzq" Nov 03 13:11:13.587376 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:13.587342 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-588d644cc-njxzq"] Nov 03 13:11:13.615203 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:13.615170 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phq2t\" (UniqueName: \"kubernetes.io/projected/445b3b20-c0f6-41d9-ab82-d1598427036b-kube-api-access-phq2t\") pod \"maas-controller-588d644cc-njxzq\" (UID: \"445b3b20-c0f6-41d9-ab82-d1598427036b\") " pod="opendatahub/maas-controller-588d644cc-njxzq" Nov 03 13:11:13.615344 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:13.615215 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/445b3b20-c0f6-41d9-ab82-d1598427036b-metrics-certs\") pod \"maas-controller-588d644cc-njxzq\" (UID: \"445b3b20-c0f6-41d9-ab82-d1598427036b\") " pod="opendatahub/maas-controller-588d644cc-njxzq" Nov 03 13:11:13.615344 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:13.615323 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/445b3b20-c0f6-41d9-ab82-d1598427036b-webhook-cert\") pod \"maas-controller-588d644cc-njxzq\" (UID: \"445b3b20-c0f6-41d9-ab82-d1598427036b\") " pod="opendatahub/maas-controller-588d644cc-njxzq" Nov 03 13:11:13.646498 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:13.646442 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7fc47b6789-65mt4"] Nov 03 13:11:13.653882 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:11:13.653854 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod640c6789_69a9_4897_9fc9_1a01b16dbead.slice/crio-823e1e5edeed0a2553e33948461af79e932d056a1ff4c48e22389f2bbb130c91 WatchSource:0}: Error finding container 823e1e5edeed0a2553e33948461af79e932d056a1ff4c48e22389f2bbb130c91: Status 404 returned error can't find the container with id 823e1e5edeed0a2553e33948461af79e932d056a1ff4c48e22389f2bbb130c91 Nov 03 13:11:13.655160 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:13.655143 5979 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 13:11:13.675520 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:13.675458 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7fc47b6789-65mt4" event={"ID":"640c6789-69a9-4897-9fc9-1a01b16dbead","Type":"ContainerStarted","Data":"823e1e5edeed0a2553e33948461af79e932d056a1ff4c48e22389f2bbb130c91"} Nov 03 13:11:13.716524 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:13.716460 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phq2t\" (UniqueName: \"kubernetes.io/projected/445b3b20-c0f6-41d9-ab82-d1598427036b-kube-api-access-phq2t\") pod \"maas-controller-588d644cc-njxzq\" (UID: \"445b3b20-c0f6-41d9-ab82-d1598427036b\") " pod="opendatahub/maas-controller-588d644cc-njxzq" Nov 03 13:11:13.716690 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:13.716620 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/445b3b20-c0f6-41d9-ab82-d1598427036b-metrics-certs\") pod \"maas-controller-588d644cc-njxzq\" (UID: \"445b3b20-c0f6-41d9-ab82-d1598427036b\") " pod="opendatahub/maas-controller-588d644cc-njxzq" Nov 03 13:11:13.716690 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:13.716680 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/445b3b20-c0f6-41d9-ab82-d1598427036b-webhook-cert\") pod \"maas-controller-588d644cc-njxzq\" (UID: \"445b3b20-c0f6-41d9-ab82-d1598427036b\") " pod="opendatahub/maas-controller-588d644cc-njxzq" Nov 03 13:11:13.719032 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:13.719011 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/445b3b20-c0f6-41d9-ab82-d1598427036b-webhook-cert\") pod \"maas-controller-588d644cc-njxzq\" (UID: \"445b3b20-c0f6-41d9-ab82-d1598427036b\") " pod="opendatahub/maas-controller-588d644cc-njxzq" Nov 03 13:11:13.719143 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:13.719062 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/445b3b20-c0f6-41d9-ab82-d1598427036b-metrics-certs\") pod \"maas-controller-588d644cc-njxzq\" (UID: \"445b3b20-c0f6-41d9-ab82-d1598427036b\") " pod="opendatahub/maas-controller-588d644cc-njxzq" Nov 03 13:11:13.770300 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:13.770270 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phq2t\" (UniqueName: \"kubernetes.io/projected/445b3b20-c0f6-41d9-ab82-d1598427036b-kube-api-access-phq2t\") pod \"maas-controller-588d644cc-njxzq\" (UID: \"445b3b20-c0f6-41d9-ab82-d1598427036b\") " pod="opendatahub/maas-controller-588d644cc-njxzq" Nov 03 13:11:13.849100 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:13.849017 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-588d644cc-njxzq" Nov 03 13:11:14.213048 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:14.213014 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-588d644cc-njxzq"] Nov 03 13:11:14.220316 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:11:14.220281 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod445b3b20_c0f6_41d9_ab82_d1598427036b.slice/crio-44730c2bd8201ff2145cd2f6a2ea5b0eb36034353c6d74a015cddf44afeaa6fa WatchSource:0}: Error finding container 44730c2bd8201ff2145cd2f6a2ea5b0eb36034353c6d74a015cddf44afeaa6fa: Status 404 returned error can't find the container with id 44730c2bd8201ff2145cd2f6a2ea5b0eb36034353c6d74a015cddf44afeaa6fa Nov 03 13:11:14.680169 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:14.680133 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-588d644cc-njxzq" event={"ID":"445b3b20-c0f6-41d9-ab82-d1598427036b","Type":"ContainerStarted","Data":"44730c2bd8201ff2145cd2f6a2ea5b0eb36034353c6d74a015cddf44afeaa6fa"} Nov 03 13:11:20.387568 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:20.387536 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 13:11:20.712530 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:20.712466 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-588d644cc-njxzq" event={"ID":"445b3b20-c0f6-41d9-ab82-d1598427036b","Type":"ContainerStarted","Data":"d21e2d243f6298178d64e8d5ed1ddd9f2623beede4ff10ab99d739f1dd673579"} Nov 03 13:11:20.712708 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:20.712675 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-588d644cc-njxzq" Nov 03 13:11:20.714141 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:20.714116 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7fc47b6789-65mt4" event={"ID":"640c6789-69a9-4897-9fc9-1a01b16dbead","Type":"ContainerStarted","Data":"918ac842593d084ad46aac68f956296a84e35cccb5fc9d6afd333216eadde4d0"} Nov 03 13:11:20.714276 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:20.714259 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7fc47b6789-65mt4" Nov 03 13:11:20.823833 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:20.823769 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-588d644cc-njxzq" podStartSLOduration=1.70025957 podStartE2EDuration="7.823752384s" totalImagesPullingTime="6.123492814s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:11:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:11:20.823408569 +0000 UTC m=+362.834096811" watchObservedRunningTime="2026-11-03 13:11:20.823752384 +0000 UTC m=+362.834440627" Nov 03 13:11:20.933920 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:20.933864 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7fc47b6789-65mt4" podStartSLOduration=2.250502904 podStartE2EDuration="8.933849117s" totalImagesPullingTime="6.683346213s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:11:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:11:20.932998339 +0000 UTC m=+362.943686581" watchObservedRunningTime="2026-11-03 13:11:20.933849117 +0000 UTC m=+362.944537356" Nov 03 13:11:23.761602 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:23.761566 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fwvg7"] Nov 03 13:11:23.765657 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:23.765628 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fwvg7" Nov 03 13:11:23.818983 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:23.818949 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fwvg7"] Nov 03 13:11:23.863938 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:23.863894 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Nov 03 13:11:23.864136 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:23.863986 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Nov 03 13:11:23.864136 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:23.864030 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-6vnkd\"" Nov 03 13:11:23.864136 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:23.864038 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Nov 03 13:11:23.914429 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:23.914395 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f33ad3b9-499c-41f5-9cf5-029709b82ca6-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fwvg7\" (UID: \"f33ad3b9-499c-41f5-9cf5-029709b82ca6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fwvg7" Nov 03 13:11:23.914636 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:23.914456 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f33ad3b9-499c-41f5-9cf5-029709b82ca6-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fwvg7\" (UID: \"f33ad3b9-499c-41f5-9cf5-029709b82ca6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fwvg7" Nov 03 13:11:23.914636 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:23.914493 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/f33ad3b9-499c-41f5-9cf5-029709b82ca6-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fwvg7\" (UID: \"f33ad3b9-499c-41f5-9cf5-029709b82ca6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fwvg7" Nov 03 13:11:23.914724 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:23.914632 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgjsw\" (UniqueName: \"kubernetes.io/projected/f33ad3b9-499c-41f5-9cf5-029709b82ca6-kube-api-access-zgjsw\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fwvg7\" (UID: \"f33ad3b9-499c-41f5-9cf5-029709b82ca6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fwvg7" Nov 03 13:11:24.016033 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:24.015945 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f33ad3b9-499c-41f5-9cf5-029709b82ca6-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fwvg7\" (UID: \"f33ad3b9-499c-41f5-9cf5-029709b82ca6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fwvg7" Nov 03 13:11:24.016033 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:24.016009 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f33ad3b9-499c-41f5-9cf5-029709b82ca6-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fwvg7\" (UID: \"f33ad3b9-499c-41f5-9cf5-029709b82ca6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fwvg7" Nov 03 13:11:24.016033 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:24.016032 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/f33ad3b9-499c-41f5-9cf5-029709b82ca6-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fwvg7\" (UID: \"f33ad3b9-499c-41f5-9cf5-029709b82ca6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fwvg7" Nov 03 13:11:24.016318 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:24.016058 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgjsw\" (UniqueName: \"kubernetes.io/projected/f33ad3b9-499c-41f5-9cf5-029709b82ca6-kube-api-access-zgjsw\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fwvg7\" (UID: \"f33ad3b9-499c-41f5-9cf5-029709b82ca6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fwvg7" Nov 03 13:11:24.016318 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:11:24.016157 5979 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found Nov 03 13:11:24.016318 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:11:24.016247 5979 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f33ad3b9-499c-41f5-9cf5-029709b82ca6-proxy-source podName:f33ad3b9-499c-41f5-9cf5-029709b82ca6 nodeName:}" failed. No retries permitted until 2026-11-03 13:11:24.516222105 +0000 UTC m=+366.526910345 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/f33ad3b9-499c-41f5-9cf5-029709b82ca6-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-fwvg7" (UID: "f33ad3b9-499c-41f5-9cf5-029709b82ca6") : configmap "usage-tenancy-proxy-source" not found Nov 03 13:11:24.016736 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:24.016717 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f33ad3b9-499c-41f5-9cf5-029709b82ca6-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fwvg7\" (UID: \"f33ad3b9-499c-41f5-9cf5-029709b82ca6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fwvg7" Nov 03 13:11:24.018331 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:24.018299 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f33ad3b9-499c-41f5-9cf5-029709b82ca6-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fwvg7\" (UID: \"f33ad3b9-499c-41f5-9cf5-029709b82ca6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fwvg7" Nov 03 13:11:24.072112 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:24.072081 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgjsw\" (UniqueName: \"kubernetes.io/projected/f33ad3b9-499c-41f5-9cf5-029709b82ca6-kube-api-access-zgjsw\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fwvg7\" (UID: \"f33ad3b9-499c-41f5-9cf5-029709b82ca6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fwvg7" Nov 03 13:11:24.520156 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:24.520120 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/f33ad3b9-499c-41f5-9cf5-029709b82ca6-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fwvg7\" (UID: \"f33ad3b9-499c-41f5-9cf5-029709b82ca6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fwvg7" Nov 03 13:11:24.520935 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:24.520915 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/f33ad3b9-499c-41f5-9cf5-029709b82ca6-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-fwvg7\" (UID: \"f33ad3b9-499c-41f5-9cf5-029709b82ca6\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fwvg7" Nov 03 13:11:24.678987 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:24.678951 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fwvg7" Nov 03 13:11:25.043631 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:25.043601 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fwvg7"] Nov 03 13:11:25.050701 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:11:25.050667 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf33ad3b9_499c_41f5_9cf5_029709b82ca6.slice/crio-be1cdd7af0a8e1e7cdc3632d7ce6f54b76942edb0d6ac05f2b09c4b020286cd6 WatchSource:0}: Error finding container be1cdd7af0a8e1e7cdc3632d7ce6f54b76942edb0d6ac05f2b09c4b020286cd6: Status 404 returned error can't find the container with id be1cdd7af0a8e1e7cdc3632d7ce6f54b76942edb0d6ac05f2b09c4b020286cd6 Nov 03 13:11:25.738317 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:25.738284 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fwvg7" event={"ID":"f33ad3b9-499c-41f5-9cf5-029709b82ca6","Type":"ContainerStarted","Data":"be1cdd7af0a8e1e7cdc3632d7ce6f54b76942edb0d6ac05f2b09c4b020286cd6"} Nov 03 13:11:31.722566 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:31.722527 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7fc47b6789-65mt4" Nov 03 13:11:31.723111 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:31.722621 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-588d644cc-njxzq" Nov 03 13:11:31.970518 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:31.970461 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7fc47b6789-65mt4"] Nov 03 13:11:31.971331 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:31.970724 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-7fc47b6789-65mt4" podUID="640c6789-69a9-4897-9fc9-1a01b16dbead" containerName="manager" containerID="cri-o://918ac842593d084ad46aac68f956296a84e35cccb5fc9d6afd333216eadde4d0" gracePeriod=30 Nov 03 13:11:32.354938 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:32.354909 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7fc47b6789-65mt4" Nov 03 13:11:32.407509 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:32.407448 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/640c6789-69a9-4897-9fc9-1a01b16dbead-kube-api-access-bfm47\" (UniqueName: \"kubernetes.io/projected/640c6789-69a9-4897-9fc9-1a01b16dbead-kube-api-access-bfm47\") pod \"640c6789-69a9-4897-9fc9-1a01b16dbead\" (UID: \"640c6789-69a9-4897-9fc9-1a01b16dbead\") " Nov 03 13:11:32.407690 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:32.407670 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/640c6789-69a9-4897-9fc9-1a01b16dbead-webhook-cert\" (UniqueName: \"kubernetes.io/secret/640c6789-69a9-4897-9fc9-1a01b16dbead-webhook-cert\") pod \"640c6789-69a9-4897-9fc9-1a01b16dbead\" (UID: \"640c6789-69a9-4897-9fc9-1a01b16dbead\") " Nov 03 13:11:32.407762 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:32.407746 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/640c6789-69a9-4897-9fc9-1a01b16dbead-metrics-certs\" (UniqueName: \"kubernetes.io/secret/640c6789-69a9-4897-9fc9-1a01b16dbead-metrics-certs\") pod \"640c6789-69a9-4897-9fc9-1a01b16dbead\" (UID: \"640c6789-69a9-4897-9fc9-1a01b16dbead\") " Nov 03 13:11:32.411677 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:32.411407 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/640c6789-69a9-4897-9fc9-1a01b16dbead-kube-api-access-bfm47" pod "640c6789-69a9-4897-9fc9-1a01b16dbead" (UID: "640c6789-69a9-4897-9fc9-1a01b16dbead"). InnerVolumeSpecName "kube-api-access-bfm47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:11:32.411782 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:32.411692 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/640c6789-69a9-4897-9fc9-1a01b16dbead-metrics-certs" pod "640c6789-69a9-4897-9fc9-1a01b16dbead" (UID: "640c6789-69a9-4897-9fc9-1a01b16dbead"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:11:32.411925 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:32.411903 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/640c6789-69a9-4897-9fc9-1a01b16dbead-webhook-cert" pod "640c6789-69a9-4897-9fc9-1a01b16dbead" (UID: "640c6789-69a9-4897-9fc9-1a01b16dbead"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:11:32.508700 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:32.508624 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfm47\" (UniqueName: \"kubernetes.io/projected/640c6789-69a9-4897-9fc9-1a01b16dbead-kube-api-access-bfm47\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:11:32.508700 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:32.508662 5979 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/640c6789-69a9-4897-9fc9-1a01b16dbead-webhook-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:11:32.508700 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:32.508677 5979 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/640c6789-69a9-4897-9fc9-1a01b16dbead-metrics-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:11:32.777806 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:32.777717 5979 generic.go:356] "Generic (PLEG): container finished" podID="640c6789-69a9-4897-9fc9-1a01b16dbead" containerID="918ac842593d084ad46aac68f956296a84e35cccb5fc9d6afd333216eadde4d0" exitCode=0 Nov 03 13:11:32.777806 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:32.777776 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7fc47b6789-65mt4" Nov 03 13:11:32.778310 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:32.777810 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7fc47b6789-65mt4" event={"ID":"640c6789-69a9-4897-9fc9-1a01b16dbead","Type":"ContainerDied","Data":"918ac842593d084ad46aac68f956296a84e35cccb5fc9d6afd333216eadde4d0"} Nov 03 13:11:32.778310 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:32.777863 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7fc47b6789-65mt4" event={"ID":"640c6789-69a9-4897-9fc9-1a01b16dbead","Type":"ContainerDied","Data":"823e1e5edeed0a2553e33948461af79e932d056a1ff4c48e22389f2bbb130c91"} Nov 03 13:11:32.778310 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:32.777893 5979 scope.go:122] "RemoveContainer" containerID="918ac842593d084ad46aac68f956296a84e35cccb5fc9d6afd333216eadde4d0" Nov 03 13:11:32.787669 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:32.787555 5979 scope.go:122] "RemoveContainer" containerID="918ac842593d084ad46aac68f956296a84e35cccb5fc9d6afd333216eadde4d0" Nov 03 13:11:32.787971 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:11:32.787934 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"918ac842593d084ad46aac68f956296a84e35cccb5fc9d6afd333216eadde4d0\": container with ID starting with 918ac842593d084ad46aac68f956296a84e35cccb5fc9d6afd333216eadde4d0 not found: ID does not exist" containerID="918ac842593d084ad46aac68f956296a84e35cccb5fc9d6afd333216eadde4d0" Nov 03 13:11:32.788063 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:32.787974 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"918ac842593d084ad46aac68f956296a84e35cccb5fc9d6afd333216eadde4d0"} err="failed to get container status \"918ac842593d084ad46aac68f956296a84e35cccb5fc9d6afd333216eadde4d0\": rpc error: code = NotFound desc = could not find container \"918ac842593d084ad46aac68f956296a84e35cccb5fc9d6afd333216eadde4d0\": container with ID starting with 918ac842593d084ad46aac68f956296a84e35cccb5fc9d6afd333216eadde4d0 not found: ID does not exist" Nov 03 13:11:32.932743 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:32.932710 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7fc47b6789-65mt4"] Nov 03 13:11:32.978905 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:32.978864 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7fc47b6789-65mt4"] Nov 03 13:11:34.114947 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:34.114915 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="640c6789-69a9-4897-9fc9-1a01b16dbead" path="/var/lib/kubelet/pods/640c6789-69a9-4897-9fc9-1a01b16dbead/volumes" Nov 03 13:11:34.688847 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:34.688812 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nvcvb"] Nov 03 13:11:34.689238 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:34.689220 5979 cpu_manager.go:414] "containerMap: removing container" podUID="640c6789-69a9-4897-9fc9-1a01b16dbead" containerName="manager" Nov 03 13:11:34.689315 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:34.689241 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="640c6789-69a9-4897-9fc9-1a01b16dbead" containerName="manager" Nov 03 13:11:34.689362 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:34.689351 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="640c6789-69a9-4897-9fc9-1a01b16dbead" containerName="manager" Nov 03 13:11:34.707910 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:34.707873 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nvcvb" Nov 03 13:11:34.730650 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:34.730616 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3a101d3a-a7d5-4f43-a4e8-0a29d9bb5607-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nvcvb\" (UID: \"3a101d3a-a7d5-4f43-a4e8-0a29d9bb5607\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nvcvb" Nov 03 13:11:34.730821 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:34.730703 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3a101d3a-a7d5-4f43-a4e8-0a29d9bb5607-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nvcvb\" (UID: \"3a101d3a-a7d5-4f43-a4e8-0a29d9bb5607\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nvcvb" Nov 03 13:11:34.730821 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:34.730725 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3a101d3a-a7d5-4f43-a4e8-0a29d9bb5607-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nvcvb\" (UID: \"3a101d3a-a7d5-4f43-a4e8-0a29d9bb5607\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nvcvb" Nov 03 13:11:34.730939 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:34.730822 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a101d3a-a7d5-4f43-a4e8-0a29d9bb5607-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nvcvb\" (UID: \"3a101d3a-a7d5-4f43-a4e8-0a29d9bb5607\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nvcvb" Nov 03 13:11:34.730939 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:34.730843 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3a101d3a-a7d5-4f43-a4e8-0a29d9bb5607-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nvcvb\" (UID: \"3a101d3a-a7d5-4f43-a4e8-0a29d9bb5607\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nvcvb" Nov 03 13:11:34.730939 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:34.730873 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a101d3a-a7d5-4f43-a4e8-0a29d9bb5607-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nvcvb\" (UID: \"3a101d3a-a7d5-4f43-a4e8-0a29d9bb5607\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nvcvb" Nov 03 13:11:34.730939 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:34.730897 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8hds\" (UniqueName: \"kubernetes.io/projected/3a101d3a-a7d5-4f43-a4e8-0a29d9bb5607-kube-api-access-q8hds\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nvcvb\" (UID: \"3a101d3a-a7d5-4f43-a4e8-0a29d9bb5607\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nvcvb" Nov 03 13:11:34.756738 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:34.756699 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 13:11:34.756738 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:34.756724 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 13:11:34.756738 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:34.756699 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 13:11:34.757018 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:34.756814 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-ffbgv\"" Nov 03 13:11:34.803509 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:34.803458 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 13:11:34.805050 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:34.805023 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nvcvb"] Nov 03 13:11:34.831946 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:34.831912 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3a101d3a-a7d5-4f43-a4e8-0a29d9bb5607-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nvcvb\" (UID: \"3a101d3a-a7d5-4f43-a4e8-0a29d9bb5607\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nvcvb" Nov 03 13:11:34.831946 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:34.831956 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3a101d3a-a7d5-4f43-a4e8-0a29d9bb5607-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nvcvb\" (UID: \"3a101d3a-a7d5-4f43-a4e8-0a29d9bb5607\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nvcvb" Nov 03 13:11:34.832194 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:34.832008 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a101d3a-a7d5-4f43-a4e8-0a29d9bb5607-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nvcvb\" (UID: \"3a101d3a-a7d5-4f43-a4e8-0a29d9bb5607\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nvcvb" Nov 03 13:11:34.832194 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:34.832033 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3a101d3a-a7d5-4f43-a4e8-0a29d9bb5607-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nvcvb\" (UID: \"3a101d3a-a7d5-4f43-a4e8-0a29d9bb5607\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nvcvb" Nov 03 13:11:34.832194 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:34.832066 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a101d3a-a7d5-4f43-a4e8-0a29d9bb5607-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nvcvb\" (UID: \"3a101d3a-a7d5-4f43-a4e8-0a29d9bb5607\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nvcvb" Nov 03 13:11:34.832345 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:34.832248 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8hds\" (UniqueName: \"kubernetes.io/projected/3a101d3a-a7d5-4f43-a4e8-0a29d9bb5607-kube-api-access-q8hds\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nvcvb\" (UID: \"3a101d3a-a7d5-4f43-a4e8-0a29d9bb5607\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nvcvb" Nov 03 13:11:34.832345 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:34.832314 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3a101d3a-a7d5-4f43-a4e8-0a29d9bb5607-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nvcvb\" (UID: \"3a101d3a-a7d5-4f43-a4e8-0a29d9bb5607\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nvcvb" Nov 03 13:11:34.832908 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:34.832884 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3a101d3a-a7d5-4f43-a4e8-0a29d9bb5607-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nvcvb\" (UID: \"3a101d3a-a7d5-4f43-a4e8-0a29d9bb5607\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nvcvb" Nov 03 13:11:34.834721 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:34.834690 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3a101d3a-a7d5-4f43-a4e8-0a29d9bb5607-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nvcvb\" (UID: \"3a101d3a-a7d5-4f43-a4e8-0a29d9bb5607\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nvcvb" Nov 03 13:11:34.834840 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:34.834811 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3a101d3a-a7d5-4f43-a4e8-0a29d9bb5607-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nvcvb\" (UID: \"3a101d3a-a7d5-4f43-a4e8-0a29d9bb5607\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nvcvb" Nov 03 13:11:34.835115 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:34.835095 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a101d3a-a7d5-4f43-a4e8-0a29d9bb5607-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nvcvb\" (UID: \"3a101d3a-a7d5-4f43-a4e8-0a29d9bb5607\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nvcvb" Nov 03 13:11:34.835227 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:34.835210 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3a101d3a-a7d5-4f43-a4e8-0a29d9bb5607-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nvcvb\" (UID: \"3a101d3a-a7d5-4f43-a4e8-0a29d9bb5607\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nvcvb" Nov 03 13:11:34.886061 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:34.886032 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a101d3a-a7d5-4f43-a4e8-0a29d9bb5607-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nvcvb\" (UID: \"3a101d3a-a7d5-4f43-a4e8-0a29d9bb5607\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nvcvb" Nov 03 13:11:34.886284 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:34.886265 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8hds\" (UniqueName: \"kubernetes.io/projected/3a101d3a-a7d5-4f43-a4e8-0a29d9bb5607-kube-api-access-q8hds\") pod \"istiod-openshift-gateway-7ddd9cfdc8-nvcvb\" (UID: \"3a101d3a-a7d5-4f43-a4e8-0a29d9bb5607\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nvcvb" Nov 03 13:11:35.020298 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:35.020201 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nvcvb" Nov 03 13:11:35.446548 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:35.446512 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nvcvb"] Nov 03 13:11:35.453138 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:11:35.453109 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a101d3a_a7d5_4f43_a4e8_0a29d9bb5607.slice/crio-d6e7ac3e2a1354e19ea1f854a71fab094ef03033d16dd8746c4f7ec2c420a4b1 WatchSource:0}: Error finding container d6e7ac3e2a1354e19ea1f854a71fab094ef03033d16dd8746c4f7ec2c420a4b1: Status 404 returned error can't find the container with id d6e7ac3e2a1354e19ea1f854a71fab094ef03033d16dd8746c4f7ec2c420a4b1 Nov 03 13:11:35.796974 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:35.796894 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nvcvb" event={"ID":"3a101d3a-a7d5-4f43-a4e8-0a29d9bb5607","Type":"ContainerStarted","Data":"d6e7ac3e2a1354e19ea1f854a71fab094ef03033d16dd8746c4f7ec2c420a4b1"} Nov 03 13:11:37.237840 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:11:37.237792 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:11:38.818572 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:38.818449 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fwvg7" event={"ID":"f33ad3b9-499c-41f5-9cf5-029709b82ca6","Type":"ContainerStarted","Data":"ce4440186c3806d0b3f089de2b1ef829f5878af689b70f2767091984b0f15bd0"} Nov 03 13:11:38.818998 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:38.818678 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fwvg7" Nov 03 13:11:38.927301 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:38.927241 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fwvg7" podStartSLOduration=2.546345219 podStartE2EDuration="15.927222494s" totalImagesPullingTime="13.380877275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:11:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:11:38.92529982 +0000 UTC m=+380.935988061" watchObservedRunningTime="2026-11-03 13:11:38.927222494 +0000 UTC m=+380.937910732" Nov 03 13:11:39.346464 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:39.346428 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:11:39.346595 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:39.346521 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:11:39.824997 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:39.824918 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nvcvb" event={"ID":"3a101d3a-a7d5-4f43-a4e8-0a29d9bb5607","Type":"ContainerStarted","Data":"7ffc049ce6872f8620810076ce49b8a5bdb174f83ec664aaa9f4d935344bdf3e"} Nov 03 13:11:39.825640 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:39.825147 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nvcvb" Nov 03 13:11:39.937892 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:39.937837 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nvcvb" podStartSLOduration=2.047746304 podStartE2EDuration="5.937822301s" totalImagesPullingTime="3.890075997s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:11:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:11:39.937112231 +0000 UTC m=+381.947800474" watchObservedRunningTime="2026-11-03 13:11:39.937822301 +0000 UTC m=+381.948510521" Nov 03 13:11:40.830998 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:40.830961 5979 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-nvcvb container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 13:11:40.831365 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:40.831022 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nvcvb" podUID="3a101d3a-a7d5-4f43-a4e8-0a29d9bb5607" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 13:11:43.830810 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:43.830775 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-nvcvb" Nov 03 13:11:49.830811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:49.830780 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-fwvg7" Nov 03 13:11:57.536185 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:11:57.536149 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:12:07.234027 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:12:07.233986 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:12:37.233794 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:12:37.233751 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:13:07.233051 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:13:07.232972 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:13:37.233851 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:13:37.233812 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:13:47.369494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:13:47.369440 5979 generic.go:356] "Generic (PLEG): container finished" podID="445b3b20-c0f6-41d9-ab82-d1598427036b" containerID="d21e2d243f6298178d64e8d5ed1ddd9f2623beede4ff10ab99d739f1dd673579" exitCode=1 Nov 03 13:13:47.369994 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:13:47.369530 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-588d644cc-njxzq" event={"ID":"445b3b20-c0f6-41d9-ab82-d1598427036b","Type":"ContainerDied","Data":"d21e2d243f6298178d64e8d5ed1ddd9f2623beede4ff10ab99d739f1dd673579"} Nov 03 13:13:47.369994 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:13:47.369968 5979 scope.go:122] "RemoveContainer" containerID="d21e2d243f6298178d64e8d5ed1ddd9f2623beede4ff10ab99d739f1dd673579" Nov 03 13:13:49.380430 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:13:49.380391 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-588d644cc-njxzq" event={"ID":"445b3b20-c0f6-41d9-ab82-d1598427036b","Type":"ContainerStarted","Data":"26764003d8102fae4b01923267ff121e565bcc4811b8d3ab4a705779f260508e"} Nov 03 13:13:49.380937 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:13:49.380628 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-588d644cc-njxzq" Nov 03 13:14:00.387096 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:00.387063 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-588d644cc-njxzq" Nov 03 13:14:07.232951 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:14:07.232846 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:14:12.558322 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:12.558285 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:14:23.515534 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:23.515495 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:14:30.780783 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:30.780740 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtv6b"] Nov 03 13:14:30.783740 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:30.783718 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtv6b" Nov 03 13:14:30.835355 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:30.835327 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 03 13:14:30.860617 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:30.860586 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 03 13:14:30.881995 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:30.881960 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Nov 03 13:14:30.882149 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:30.882015 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-6xb8m\"" Nov 03 13:14:30.882149 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:30.882016 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9qpm\" (UniqueName: \"kubernetes.io/projected/f51c4495-5704-40ba-ad07-0d8b278595d6-kube-api-access-w9qpm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtv6b\" (UID: \"f51c4495-5704-40ba-ad07-0d8b278595d6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtv6b" Nov 03 13:14:30.882149 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:30.882106 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f51c4495-5704-40ba-ad07-0d8b278595d6-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtv6b\" (UID: \"f51c4495-5704-40ba-ad07-0d8b278595d6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtv6b" Nov 03 13:14:30.882502 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:30.882201 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f51c4495-5704-40ba-ad07-0d8b278595d6-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtv6b\" (UID: \"f51c4495-5704-40ba-ad07-0d8b278595d6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtv6b" Nov 03 13:14:30.885128 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:30.885097 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtv6b"] Nov 03 13:14:30.886620 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:30.886592 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f51c4495-5704-40ba-ad07-0d8b278595d6-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtv6b\" (UID: \"f51c4495-5704-40ba-ad07-0d8b278595d6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtv6b" Nov 03 13:14:30.886801 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:30.886782 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f51c4495-5704-40ba-ad07-0d8b278595d6-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtv6b\" (UID: \"f51c4495-5704-40ba-ad07-0d8b278595d6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtv6b" Nov 03 13:14:30.886946 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:30.886931 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f51c4495-5704-40ba-ad07-0d8b278595d6-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtv6b\" (UID: \"f51c4495-5704-40ba-ad07-0d8b278595d6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtv6b" Nov 03 13:14:30.988266 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:30.988236 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f51c4495-5704-40ba-ad07-0d8b278595d6-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtv6b\" (UID: \"f51c4495-5704-40ba-ad07-0d8b278595d6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtv6b" Nov 03 13:14:30.988429 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:30.988272 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f51c4495-5704-40ba-ad07-0d8b278595d6-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtv6b\" (UID: \"f51c4495-5704-40ba-ad07-0d8b278595d6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtv6b" Nov 03 13:14:30.988429 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:30.988294 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f51c4495-5704-40ba-ad07-0d8b278595d6-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtv6b\" (UID: \"f51c4495-5704-40ba-ad07-0d8b278595d6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtv6b" Nov 03 13:14:30.988429 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:30.988400 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f51c4495-5704-40ba-ad07-0d8b278595d6-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtv6b\" (UID: \"f51c4495-5704-40ba-ad07-0d8b278595d6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtv6b" Nov 03 13:14:30.988599 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:30.988442 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9qpm\" (UniqueName: \"kubernetes.io/projected/f51c4495-5704-40ba-ad07-0d8b278595d6-kube-api-access-w9qpm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtv6b\" (UID: \"f51c4495-5704-40ba-ad07-0d8b278595d6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtv6b" Nov 03 13:14:30.988599 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:30.988503 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f51c4495-5704-40ba-ad07-0d8b278595d6-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtv6b\" (UID: \"f51c4495-5704-40ba-ad07-0d8b278595d6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtv6b" Nov 03 13:14:30.988772 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:30.988746 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f51c4495-5704-40ba-ad07-0d8b278595d6-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtv6b\" (UID: \"f51c4495-5704-40ba-ad07-0d8b278595d6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtv6b" Nov 03 13:14:30.988843 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:30.988809 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f51c4495-5704-40ba-ad07-0d8b278595d6-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtv6b\" (UID: \"f51c4495-5704-40ba-ad07-0d8b278595d6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtv6b" Nov 03 13:14:30.988902 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:30.988883 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f51c4495-5704-40ba-ad07-0d8b278595d6-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtv6b\" (UID: \"f51c4495-5704-40ba-ad07-0d8b278595d6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtv6b" Nov 03 13:14:30.990698 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:30.990673 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f51c4495-5704-40ba-ad07-0d8b278595d6-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtv6b\" (UID: \"f51c4495-5704-40ba-ad07-0d8b278595d6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtv6b" Nov 03 13:14:30.990927 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:30.990906 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f51c4495-5704-40ba-ad07-0d8b278595d6-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtv6b\" (UID: \"f51c4495-5704-40ba-ad07-0d8b278595d6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtv6b" Nov 03 13:14:31.046218 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:31.046142 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9qpm\" (UniqueName: \"kubernetes.io/projected/f51c4495-5704-40ba-ad07-0d8b278595d6-kube-api-access-w9qpm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtv6b\" (UID: \"f51c4495-5704-40ba-ad07-0d8b278595d6\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtv6b" Nov 03 13:14:31.096047 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:31.096008 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtv6b" Nov 03 13:14:31.467140 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:31.467104 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtv6b"] Nov 03 13:14:31.475090 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:14:31.475055 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf51c4495_5704_40ba_ad07_0d8b278595d6.slice/crio-970816c5836981b11d610783cf6e313840e6a883289ada144ee897fd0ecf48b7 WatchSource:0}: Error finding container 970816c5836981b11d610783cf6e313840e6a883289ada144ee897fd0ecf48b7: Status 404 returned error can't find the container with id 970816c5836981b11d610783cf6e313840e6a883289ada144ee897fd0ecf48b7 Nov 03 13:14:31.565253 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:31.565217 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtv6b" event={"ID":"f51c4495-5704-40ba-ad07-0d8b278595d6","Type":"ContainerStarted","Data":"970816c5836981b11d610783cf6e313840e6a883289ada144ee897fd0ecf48b7"} Nov 03 13:14:33.579007 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:33.578973 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtv6b" event={"ID":"f51c4495-5704-40ba-ad07-0d8b278595d6","Type":"ContainerStarted","Data":"6f491e447b61da7b032cf94065578efe47836f49726a9d6e92ab861a2cf9d88e"} Nov 03 13:14:33.689641 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:33.689574 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtv6b" podStartSLOduration=2.175624284 podStartE2EDuration="3.689552667s" totalImagesPullingTime="1.513928383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:14:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:14:33.688889713 +0000 UTC m=+555.699577955" watchObservedRunningTime="2026-11-03 13:14:33.689552667 +0000 UTC m=+555.700240910" Nov 03 13:14:34.096616 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:34.096580 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtv6b" Nov 03 13:14:34.455298 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:34.455267 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:14:37.031319 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:37.031276 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5p5rm"] Nov 03 13:14:37.037593 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:37.037563 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5p5rm" Nov 03 13:14:37.087636 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:37.087603 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5p5rm"] Nov 03 13:14:37.132709 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:37.132682 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Nov 03 13:14:37.148125 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:37.148088 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8407952f-9ae8-40c2-9f22-d6fd2cd68b36-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5p5rm\" (UID: \"8407952f-9ae8-40c2-9f22-d6fd2cd68b36\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5p5rm" Nov 03 13:14:37.148242 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:37.148222 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8407952f-9ae8-40c2-9f22-d6fd2cd68b36-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5p5rm\" (UID: \"8407952f-9ae8-40c2-9f22-d6fd2cd68b36\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5p5rm" Nov 03 13:14:37.148298 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:37.148265 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8407952f-9ae8-40c2-9f22-d6fd2cd68b36-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5p5rm\" (UID: \"8407952f-9ae8-40c2-9f22-d6fd2cd68b36\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5p5rm" Nov 03 13:14:37.148298 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:37.148282 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8407952f-9ae8-40c2-9f22-d6fd2cd68b36-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5p5rm\" (UID: \"8407952f-9ae8-40c2-9f22-d6fd2cd68b36\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5p5rm" Nov 03 13:14:37.148439 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:37.148417 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8407952f-9ae8-40c2-9f22-d6fd2cd68b36-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5p5rm\" (UID: \"8407952f-9ae8-40c2-9f22-d6fd2cd68b36\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5p5rm" Nov 03 13:14:37.148526 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:37.148510 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5x2x\" (UniqueName: \"kubernetes.io/projected/8407952f-9ae8-40c2-9f22-d6fd2cd68b36-kube-api-access-h5x2x\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5p5rm\" (UID: \"8407952f-9ae8-40c2-9f22-d6fd2cd68b36\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5p5rm" Nov 03 13:14:37.234782 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:14:37.234744 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:14:37.249907 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:37.249877 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8407952f-9ae8-40c2-9f22-d6fd2cd68b36-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5p5rm\" (UID: \"8407952f-9ae8-40c2-9f22-d6fd2cd68b36\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5p5rm" Nov 03 13:14:37.250076 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:37.249920 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8407952f-9ae8-40c2-9f22-d6fd2cd68b36-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5p5rm\" (UID: \"8407952f-9ae8-40c2-9f22-d6fd2cd68b36\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5p5rm" Nov 03 13:14:37.250076 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:37.249937 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8407952f-9ae8-40c2-9f22-d6fd2cd68b36-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5p5rm\" (UID: \"8407952f-9ae8-40c2-9f22-d6fd2cd68b36\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5p5rm" Nov 03 13:14:37.250076 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:37.249974 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8407952f-9ae8-40c2-9f22-d6fd2cd68b36-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5p5rm\" (UID: \"8407952f-9ae8-40c2-9f22-d6fd2cd68b36\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5p5rm" Nov 03 13:14:37.250076 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:37.250011 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5x2x\" (UniqueName: \"kubernetes.io/projected/8407952f-9ae8-40c2-9f22-d6fd2cd68b36-kube-api-access-h5x2x\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5p5rm\" (UID: \"8407952f-9ae8-40c2-9f22-d6fd2cd68b36\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5p5rm" Nov 03 13:14:37.250076 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:37.250063 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8407952f-9ae8-40c2-9f22-d6fd2cd68b36-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5p5rm\" (UID: \"8407952f-9ae8-40c2-9f22-d6fd2cd68b36\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5p5rm" Nov 03 13:14:37.250395 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:37.250372 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8407952f-9ae8-40c2-9f22-d6fd2cd68b36-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5p5rm\" (UID: \"8407952f-9ae8-40c2-9f22-d6fd2cd68b36\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5p5rm" Nov 03 13:14:37.250462 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:37.250379 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8407952f-9ae8-40c2-9f22-d6fd2cd68b36-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5p5rm\" (UID: \"8407952f-9ae8-40c2-9f22-d6fd2cd68b36\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5p5rm" Nov 03 13:14:37.250526 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:37.250465 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8407952f-9ae8-40c2-9f22-d6fd2cd68b36-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5p5rm\" (UID: \"8407952f-9ae8-40c2-9f22-d6fd2cd68b36\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5p5rm" Nov 03 13:14:37.252200 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:37.252182 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8407952f-9ae8-40c2-9f22-d6fd2cd68b36-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5p5rm\" (UID: \"8407952f-9ae8-40c2-9f22-d6fd2cd68b36\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5p5rm" Nov 03 13:14:37.252566 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:37.252549 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8407952f-9ae8-40c2-9f22-d6fd2cd68b36-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5p5rm\" (UID: \"8407952f-9ae8-40c2-9f22-d6fd2cd68b36\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5p5rm" Nov 03 13:14:37.304179 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:37.304102 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5x2x\" (UniqueName: \"kubernetes.io/projected/8407952f-9ae8-40c2-9f22-d6fd2cd68b36-kube-api-access-h5x2x\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5p5rm\" (UID: \"8407952f-9ae8-40c2-9f22-d6fd2cd68b36\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5p5rm" Nov 03 13:14:37.350145 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:37.350106 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5p5rm" Nov 03 13:14:37.724845 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:37.724804 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5p5rm"] Nov 03 13:14:37.732824 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:14:37.732800 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8407952f_9ae8_40c2_9f22_d6fd2cd68b36.slice/crio-593afb201e7d61720ca6c2d2da71993fad54a637f3eca7779ffd6b90e84bf8bf WatchSource:0}: Error finding container 593afb201e7d61720ca6c2d2da71993fad54a637f3eca7779ffd6b90e84bf8bf: Status 404 returned error can't find the container with id 593afb201e7d61720ca6c2d2da71993fad54a637f3eca7779ffd6b90e84bf8bf Nov 03 13:14:38.602785 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:38.602748 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5p5rm" event={"ID":"8407952f-9ae8-40c2-9f22-d6fd2cd68b36","Type":"ContainerStarted","Data":"869752db3fbc7ea182d64a786ac2ca5e41c14a965ef08ed1c2185dcd67c99590"} Nov 03 13:14:38.603194 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:38.602810 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5p5rm" event={"ID":"8407952f-9ae8-40c2-9f22-d6fd2cd68b36","Type":"ContainerStarted","Data":"593afb201e7d61720ca6c2d2da71993fad54a637f3eca7779ffd6b90e84bf8bf"} Nov 03 13:14:38.712160 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:38.712105 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5p5rm" podStartSLOduration=2.073535875 podStartE2EDuration="2.712091247s" totalImagesPullingTime="638.555372ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:14:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:14:38.71178252 +0000 UTC m=+560.722470762" watchObservedRunningTime="2026-11-03 13:14:38.712091247 +0000 UTC m=+560.722779489" Nov 03 13:14:39.350459 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:39.350424 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5p5rm" Nov 03 13:14:41.097113 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:41.097077 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtv6b" Nov 03 13:14:41.110028 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:41.109997 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtv6b" Nov 03 13:14:41.116379 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:41.116352 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz"] Nov 03 13:14:41.118999 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:41.118982 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz" Nov 03 13:14:41.167743 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:41.167712 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-wd2hr\"" Nov 03 13:14:41.187313 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:41.187274 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/832a1db8-0f65-48a3-b561-30b77a5bc2bd-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jmzwz\" (UID: \"832a1db8-0f65-48a3-b561-30b77a5bc2bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz" Nov 03 13:14:41.187313 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:41.187318 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/832a1db8-0f65-48a3-b561-30b77a5bc2bd-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jmzwz\" (UID: \"832a1db8-0f65-48a3-b561-30b77a5bc2bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz" Nov 03 13:14:41.187551 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:41.187353 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/832a1db8-0f65-48a3-b561-30b77a5bc2bd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jmzwz\" (UID: \"832a1db8-0f65-48a3-b561-30b77a5bc2bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz" Nov 03 13:14:41.187551 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:41.187391 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/832a1db8-0f65-48a3-b561-30b77a5bc2bd-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jmzwz\" (UID: \"832a1db8-0f65-48a3-b561-30b77a5bc2bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz" Nov 03 13:14:41.187551 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:41.187449 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/832a1db8-0f65-48a3-b561-30b77a5bc2bd-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jmzwz\" (UID: \"832a1db8-0f65-48a3-b561-30b77a5bc2bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz" Nov 03 13:14:41.187677 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:41.187580 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/832a1db8-0f65-48a3-b561-30b77a5bc2bd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jmzwz\" (UID: \"832a1db8-0f65-48a3-b561-30b77a5bc2bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz" Nov 03 13:14:41.187677 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:41.187617 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrpt7\" (UniqueName: \"kubernetes.io/projected/832a1db8-0f65-48a3-b561-30b77a5bc2bd-kube-api-access-lrpt7\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jmzwz\" (UID: \"832a1db8-0f65-48a3-b561-30b77a5bc2bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz" Nov 03 13:14:41.187822 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:41.187799 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/832a1db8-0f65-48a3-b561-30b77a5bc2bd-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jmzwz\" (UID: \"832a1db8-0f65-48a3-b561-30b77a5bc2bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz" Nov 03 13:14:41.187884 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:41.187864 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/832a1db8-0f65-48a3-b561-30b77a5bc2bd-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jmzwz\" (UID: \"832a1db8-0f65-48a3-b561-30b77a5bc2bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz" Nov 03 13:14:41.213243 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:41.213214 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz"] Nov 03 13:14:41.288571 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:41.288533 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/832a1db8-0f65-48a3-b561-30b77a5bc2bd-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jmzwz\" (UID: \"832a1db8-0f65-48a3-b561-30b77a5bc2bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz" Nov 03 13:14:41.288571 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:41.288574 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/832a1db8-0f65-48a3-b561-30b77a5bc2bd-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jmzwz\" (UID: \"832a1db8-0f65-48a3-b561-30b77a5bc2bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz" Nov 03 13:14:41.288821 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:41.288599 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/832a1db8-0f65-48a3-b561-30b77a5bc2bd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jmzwz\" (UID: \"832a1db8-0f65-48a3-b561-30b77a5bc2bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz" Nov 03 13:14:41.288821 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:41.288615 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/832a1db8-0f65-48a3-b561-30b77a5bc2bd-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jmzwz\" (UID: \"832a1db8-0f65-48a3-b561-30b77a5bc2bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz" Nov 03 13:14:41.288821 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:41.288641 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/832a1db8-0f65-48a3-b561-30b77a5bc2bd-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jmzwz\" (UID: \"832a1db8-0f65-48a3-b561-30b77a5bc2bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz" Nov 03 13:14:41.288821 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:41.288689 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/832a1db8-0f65-48a3-b561-30b77a5bc2bd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jmzwz\" (UID: \"832a1db8-0f65-48a3-b561-30b77a5bc2bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz" Nov 03 13:14:41.288821 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:41.288722 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrpt7\" (UniqueName: \"kubernetes.io/projected/832a1db8-0f65-48a3-b561-30b77a5bc2bd-kube-api-access-lrpt7\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jmzwz\" (UID: \"832a1db8-0f65-48a3-b561-30b77a5bc2bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz" Nov 03 13:14:41.289082 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:41.288822 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/832a1db8-0f65-48a3-b561-30b77a5bc2bd-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jmzwz\" (UID: \"832a1db8-0f65-48a3-b561-30b77a5bc2bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz" Nov 03 13:14:41.289082 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:41.288856 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/832a1db8-0f65-48a3-b561-30b77a5bc2bd-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jmzwz\" (UID: \"832a1db8-0f65-48a3-b561-30b77a5bc2bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz" Nov 03 13:14:41.289082 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:41.289028 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/832a1db8-0f65-48a3-b561-30b77a5bc2bd-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jmzwz\" (UID: \"832a1db8-0f65-48a3-b561-30b77a5bc2bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz" Nov 03 13:14:41.289082 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:41.289073 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/832a1db8-0f65-48a3-b561-30b77a5bc2bd-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jmzwz\" (UID: \"832a1db8-0f65-48a3-b561-30b77a5bc2bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz" Nov 03 13:14:41.289230 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:41.289189 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/832a1db8-0f65-48a3-b561-30b77a5bc2bd-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jmzwz\" (UID: \"832a1db8-0f65-48a3-b561-30b77a5bc2bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz" Nov 03 13:14:41.289263 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:41.289244 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/832a1db8-0f65-48a3-b561-30b77a5bc2bd-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jmzwz\" (UID: \"832a1db8-0f65-48a3-b561-30b77a5bc2bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz" Nov 03 13:14:41.289570 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:41.289547 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/832a1db8-0f65-48a3-b561-30b77a5bc2bd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jmzwz\" (UID: \"832a1db8-0f65-48a3-b561-30b77a5bc2bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz" Nov 03 13:14:41.291069 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:41.291047 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/832a1db8-0f65-48a3-b561-30b77a5bc2bd-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jmzwz\" (UID: \"832a1db8-0f65-48a3-b561-30b77a5bc2bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz" Nov 03 13:14:41.291206 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:41.291185 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/832a1db8-0f65-48a3-b561-30b77a5bc2bd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jmzwz\" (UID: \"832a1db8-0f65-48a3-b561-30b77a5bc2bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz" Nov 03 13:14:41.341888 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:41.341853 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/832a1db8-0f65-48a3-b561-30b77a5bc2bd-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jmzwz\" (UID: \"832a1db8-0f65-48a3-b561-30b77a5bc2bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz" Nov 03 13:14:41.342030 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:41.341991 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrpt7\" (UniqueName: \"kubernetes.io/projected/832a1db8-0f65-48a3-b561-30b77a5bc2bd-kube-api-access-lrpt7\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-jmzwz\" (UID: \"832a1db8-0f65-48a3-b561-30b77a5bc2bd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz" Nov 03 13:14:41.435057 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:41.435022 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz" Nov 03 13:14:41.628600 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:41.628568 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtv6b" Nov 03 13:14:41.864574 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:41.864495 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz"] Nov 03 13:14:41.871460 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:14:41.871423 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod832a1db8_0f65_48a3_b561_30b77a5bc2bd.slice/crio-9e0d29253ff808529edcb29eeabf170a770dbb16ddcfd7b99938dbdd82602796 WatchSource:0}: Error finding container 9e0d29253ff808529edcb29eeabf170a770dbb16ddcfd7b99938dbdd82602796: Status 404 returned error can't find the container with id 9e0d29253ff808529edcb29eeabf170a770dbb16ddcfd7b99938dbdd82602796 Nov 03 13:14:42.623543 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:42.623505 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz" event={"ID":"832a1db8-0f65-48a3-b561-30b77a5bc2bd","Type":"ContainerStarted","Data":"9e0d29253ff808529edcb29eeabf170a770dbb16ddcfd7b99938dbdd82602796"} Nov 03 13:14:45.893801 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:45.893770 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:14:45.894089 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:45.893857 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:14:45.894089 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:45.893883 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:14:46.674754 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:46.674703 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz" event={"ID":"832a1db8-0f65-48a3-b561-30b77a5bc2bd","Type":"ContainerStarted","Data":"9d351475ba6fcc26c7d5073b1ea9c7f866dc80a852bf57b2b9c6cc402ae79c90"} Nov 03 13:14:46.832148 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:46.832095 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz" podStartSLOduration=1.8127040989999998 podStartE2EDuration="5.832081867s" totalImagesPullingTime="4.019377768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:14:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:14:46.830949644 +0000 UTC m=+568.841637886" watchObservedRunningTime="2026-11-03 13:14:46.832081867 +0000 UTC m=+568.842770109" Nov 03 13:14:47.359177 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:47.355852 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5p5rm" Nov 03 13:14:47.380612 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:47.380573 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5p5rm" Nov 03 13:14:47.435464 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:47.435425 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz" Nov 03 13:14:47.436813 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:47.436788 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 03 13:14:47.436934 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:47.436843 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz" podUID="832a1db8-0f65-48a3-b561-30b77a5bc2bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 03 13:14:47.566991 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:47.566948 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:14:47.691211 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:47.691179 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5p5rm" Nov 03 13:14:48.435855 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:48.435819 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 03 13:14:48.436253 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:48.435866 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz" podUID="832a1db8-0f65-48a3-b561-30b77a5bc2bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 03 13:14:49.436350 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:49.436316 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 03 13:14:49.436742 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:49.436378 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz" podUID="832a1db8-0f65-48a3-b561-30b77a5bc2bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 03 13:14:50.435649 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:50.435614 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 03 13:14:50.435852 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:50.435671 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz" podUID="832a1db8-0f65-48a3-b561-30b77a5bc2bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 03 13:14:51.436108 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:51.436072 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 03 13:14:51.436504 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:51.436151 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz" podUID="832a1db8-0f65-48a3-b561-30b77a5bc2bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 03 13:14:52.435372 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:52.435335 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Nov 03 13:14:52.435566 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:52.435389 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz" podUID="832a1db8-0f65-48a3-b561-30b77a5bc2bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Nov 03 13:14:54.436440 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:54.436391 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:14:54.436858 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:54.436524 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz" podUID="832a1db8-0f65-48a3-b561-30b77a5bc2bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:14:55.477637 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:55.477599 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:14:55.478018 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:55.477673 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz" podUID="832a1db8-0f65-48a3-b561-30b77a5bc2bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:14:56.090504 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:56.090437 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz" Nov 03 13:14:56.091163 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:56.091134 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz" Nov 03 13:14:56.091928 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:56.091910 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz" Nov 03 13:14:56.251641 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:56.251605 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dkw27"] Nov 03 13:14:56.254442 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:56.254422 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dkw27" Nov 03 13:14:56.317434 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:56.317403 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dkw27"] Nov 03 13:14:56.367125 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:56.367009 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f313510b-02d8-4c68-accd-920343219917-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-dkw27\" (UID: \"f313510b-02d8-4c68-accd-920343219917\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dkw27" Nov 03 13:14:56.367294 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:56.367211 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f313510b-02d8-4c68-accd-920343219917-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-dkw27\" (UID: \"f313510b-02d8-4c68-accd-920343219917\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dkw27" Nov 03 13:14:56.367350 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:56.367321 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f313510b-02d8-4c68-accd-920343219917-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-dkw27\" (UID: \"f313510b-02d8-4c68-accd-920343219917\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dkw27" Nov 03 13:14:56.367400 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:56.367384 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clvcq\" (UniqueName: \"kubernetes.io/projected/f313510b-02d8-4c68-accd-920343219917-kube-api-access-clvcq\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-dkw27\" (UID: \"f313510b-02d8-4c68-accd-920343219917\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dkw27" Nov 03 13:14:56.367461 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:56.367417 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f313510b-02d8-4c68-accd-920343219917-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-dkw27\" (UID: \"f313510b-02d8-4c68-accd-920343219917\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dkw27" Nov 03 13:14:56.367547 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:56.367512 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f313510b-02d8-4c68-accd-920343219917-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-dkw27\" (UID: \"f313510b-02d8-4c68-accd-920343219917\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dkw27" Nov 03 13:14:56.367547 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:56.367535 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f313510b-02d8-4c68-accd-920343219917-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-dkw27\" (UID: \"f313510b-02d8-4c68-accd-920343219917\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dkw27" Nov 03 13:14:56.367654 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:56.367577 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f313510b-02d8-4c68-accd-920343219917-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-dkw27\" (UID: \"f313510b-02d8-4c68-accd-920343219917\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dkw27" Nov 03 13:14:56.367711 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:56.367682 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f313510b-02d8-4c68-accd-920343219917-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-dkw27\" (UID: \"f313510b-02d8-4c68-accd-920343219917\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dkw27" Nov 03 13:14:56.468694 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:56.468661 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f313510b-02d8-4c68-accd-920343219917-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-dkw27\" (UID: \"f313510b-02d8-4c68-accd-920343219917\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dkw27" Nov 03 13:14:56.468901 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:56.468710 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f313510b-02d8-4c68-accd-920343219917-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-dkw27\" (UID: \"f313510b-02d8-4c68-accd-920343219917\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dkw27" Nov 03 13:14:56.468901 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:56.468733 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f313510b-02d8-4c68-accd-920343219917-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-dkw27\" (UID: \"f313510b-02d8-4c68-accd-920343219917\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dkw27" Nov 03 13:14:56.468901 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:56.468751 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f313510b-02d8-4c68-accd-920343219917-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-dkw27\" (UID: \"f313510b-02d8-4c68-accd-920343219917\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dkw27" Nov 03 13:14:56.468901 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:56.468821 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clvcq\" (UniqueName: \"kubernetes.io/projected/f313510b-02d8-4c68-accd-920343219917-kube-api-access-clvcq\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-dkw27\" (UID: \"f313510b-02d8-4c68-accd-920343219917\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dkw27" Nov 03 13:14:56.469205 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:56.468925 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f313510b-02d8-4c68-accd-920343219917-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-dkw27\" (UID: \"f313510b-02d8-4c68-accd-920343219917\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dkw27" Nov 03 13:14:56.469272 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:56.469223 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f313510b-02d8-4c68-accd-920343219917-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-dkw27\" (UID: \"f313510b-02d8-4c68-accd-920343219917\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dkw27" Nov 03 13:14:56.469520 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:56.469499 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f313510b-02d8-4c68-accd-920343219917-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-dkw27\" (UID: \"f313510b-02d8-4c68-accd-920343219917\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dkw27" Nov 03 13:14:56.469626 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:56.469552 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f313510b-02d8-4c68-accd-920343219917-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-dkw27\" (UID: \"f313510b-02d8-4c68-accd-920343219917\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dkw27" Nov 03 13:14:56.469626 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:56.469587 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f313510b-02d8-4c68-accd-920343219917-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-dkw27\" (UID: \"f313510b-02d8-4c68-accd-920343219917\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dkw27" Nov 03 13:14:56.469626 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:56.469589 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f313510b-02d8-4c68-accd-920343219917-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-dkw27\" (UID: \"f313510b-02d8-4c68-accd-920343219917\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dkw27" Nov 03 13:14:56.469725 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:56.469640 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f313510b-02d8-4c68-accd-920343219917-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-dkw27\" (UID: \"f313510b-02d8-4c68-accd-920343219917\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dkw27" Nov 03 13:14:56.470004 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:56.469983 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f313510b-02d8-4c68-accd-920343219917-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-dkw27\" (UID: \"f313510b-02d8-4c68-accd-920343219917\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dkw27" Nov 03 13:14:56.470603 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:56.470582 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f313510b-02d8-4c68-accd-920343219917-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-dkw27\" (UID: \"f313510b-02d8-4c68-accd-920343219917\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dkw27" Nov 03 13:14:56.471152 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:56.471120 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f313510b-02d8-4c68-accd-920343219917-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-dkw27\" (UID: \"f313510b-02d8-4c68-accd-920343219917\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dkw27" Nov 03 13:14:56.472111 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:56.472091 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f313510b-02d8-4c68-accd-920343219917-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-dkw27\" (UID: \"f313510b-02d8-4c68-accd-920343219917\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dkw27" Nov 03 13:14:56.522254 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:56.522207 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clvcq\" (UniqueName: \"kubernetes.io/projected/f313510b-02d8-4c68-accd-920343219917-kube-api-access-clvcq\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-dkw27\" (UID: \"f313510b-02d8-4c68-accd-920343219917\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dkw27" Nov 03 13:14:56.522874 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:56.522849 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f313510b-02d8-4c68-accd-920343219917-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-dkw27\" (UID: \"f313510b-02d8-4c68-accd-920343219917\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dkw27" Nov 03 13:14:56.566148 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:56.566116 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dkw27" Nov 03 13:14:56.991490 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:56.991436 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dkw27"] Nov 03 13:14:56.999403 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:14:56.999375 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf313510b_02d8_4c68_accd_920343219917.slice/crio-162748121b2129b6c7ca6ad5f181c24e3fa8d5acd0b664f32fea3813c2d00c3d WatchSource:0}: Error finding container 162748121b2129b6c7ca6ad5f181c24e3fa8d5acd0b664f32fea3813c2d00c3d: Status 404 returned error can't find the container with id 162748121b2129b6c7ca6ad5f181c24e3fa8d5acd0b664f32fea3813c2d00c3d Nov 03 13:14:57.001680 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:57.001650 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:14:57.001754 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:57.001716 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:14:57.001754 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:57.001746 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:14:57.729065 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:57.729025 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dkw27" event={"ID":"f313510b-02d8-4c68-accd-920343219917","Type":"ContainerStarted","Data":"d0beab700e65e467cebbfa614224e81d1d9f1b416654fd7841105b78b5819c4d"} Nov 03 13:14:57.729560 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:57.729086 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dkw27" event={"ID":"f313510b-02d8-4c68-accd-920343219917","Type":"ContainerStarted","Data":"162748121b2129b6c7ca6ad5f181c24e3fa8d5acd0b664f32fea3813c2d00c3d"} Nov 03 13:14:57.886480 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:57.886425 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dkw27" podStartSLOduration=1.886411473 podStartE2EDuration="1.886411473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:14:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:14:57.885130345 +0000 UTC m=+579.895818587" watchObservedRunningTime="2026-11-03 13:14:57.886411473 +0000 UTC m=+579.897099708" Nov 03 13:14:58.568613 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:58.568554 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dkw27" Nov 03 13:14:58.568821 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:58.568622 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-dkw27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 13:14:58.568821 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:58.568685 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dkw27" podUID="f313510b-02d8-4c68-accd-920343219917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 13:14:59.568931 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:59.568893 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-dkw27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 13:14:59.569349 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:14:59.568960 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dkw27" podUID="f313510b-02d8-4c68-accd-920343219917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 13:15:00.176183 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:00.176145 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895195-c5h4v"] Nov 03 13:15:00.178960 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:00.178940 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895195-c5h4v" Nov 03 13:15:00.210017 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:00.209981 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjvfs\" (UniqueName: \"kubernetes.io/projected/74c198a3-f63a-45cc-8bf8-a7b2c90d597c-kube-api-access-kjvfs\") pod \"maas-api-key-cleanup-29895195-c5h4v\" (UID: \"74c198a3-f63a-45cc-8bf8-a7b2c90d597c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895195-c5h4v" Nov 03 13:15:00.210172 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:00.210031 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74c198a3-f63a-45cc-8bf8-a7b2c90d597c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895195-c5h4v\" (UID: \"74c198a3-f63a-45cc-8bf8-a7b2c90d597c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895195-c5h4v" Nov 03 13:15:00.228366 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:00.228337 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 13:15:00.274697 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:00.274654 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 13:15:00.274937 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:00.274727 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:15:00.276304 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:00.276278 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895195-c5h4v"] Nov 03 13:15:00.311403 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:00.311367 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjvfs\" (UniqueName: \"kubernetes.io/projected/74c198a3-f63a-45cc-8bf8-a7b2c90d597c-kube-api-access-kjvfs\") pod \"maas-api-key-cleanup-29895195-c5h4v\" (UID: \"74c198a3-f63a-45cc-8bf8-a7b2c90d597c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895195-c5h4v" Nov 03 13:15:00.311403 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:00.311406 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74c198a3-f63a-45cc-8bf8-a7b2c90d597c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895195-c5h4v\" (UID: \"74c198a3-f63a-45cc-8bf8-a7b2c90d597c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895195-c5h4v" Nov 03 13:15:00.312074 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:00.312054 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74c198a3-f63a-45cc-8bf8-a7b2c90d597c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895195-c5h4v\" (UID: \"74c198a3-f63a-45cc-8bf8-a7b2c90d597c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895195-c5h4v" Nov 03 13:15:00.371323 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:00.371293 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjvfs\" (UniqueName: \"kubernetes.io/projected/74c198a3-f63a-45cc-8bf8-a7b2c90d597c-kube-api-access-kjvfs\") pod \"maas-api-key-cleanup-29895195-c5h4v\" (UID: \"74c198a3-f63a-45cc-8bf8-a7b2c90d597c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895195-c5h4v" Nov 03 13:15:00.491609 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:00.491532 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895195-c5h4v" Nov 03 13:15:00.570348 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:00.569879 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-dkw27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 13:15:00.570348 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:00.569938 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dkw27" podUID="f313510b-02d8-4c68-accd-920343219917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 13:15:00.870696 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:00.870611 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895195-c5h4v"] Nov 03 13:15:00.877992 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:15:00.877959 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74c198a3_f63a_45cc_8bf8_a7b2c90d597c.slice/crio-0b3096890aee80cf26473f7c51474d9d26e8223854f0e837662806ad49066956 WatchSource:0}: Error finding container 0b3096890aee80cf26473f7c51474d9d26e8223854f0e837662806ad49066956: Status 404 returned error can't find the container with id 0b3096890aee80cf26473f7c51474d9d26e8223854f0e837662806ad49066956 Nov 03 13:15:01.568887 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:01.568848 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-dkw27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 13:15:01.569092 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:01.568929 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dkw27" podUID="f313510b-02d8-4c68-accd-920343219917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 13:15:01.780159 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:01.780121 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895195-c5h4v" event={"ID":"74c198a3-f63a-45cc-8bf8-a7b2c90d597c","Type":"ContainerStarted","Data":"0b3096890aee80cf26473f7c51474d9d26e8223854f0e837662806ad49066956"} Nov 03 13:15:02.568852 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:02.568813 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-dkw27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 13:15:02.569043 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:02.568871 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dkw27" podUID="f313510b-02d8-4c68-accd-920343219917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 13:15:03.566964 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:03.566922 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-dkw27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Nov 03 13:15:03.567356 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:03.566990 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dkw27" podUID="f313510b-02d8-4c68-accd-920343219917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Nov 03 13:15:05.295864 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:05.295824 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dkw27" Nov 03 13:15:05.296346 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:05.296256 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dkw27" Nov 03 13:15:05.297129 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:05.297109 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dkw27" Nov 03 13:15:07.239564 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:15:07.239526 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:15:07.814010 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:07.813925 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895195-c5h4v" event={"ID":"74c198a3-f63a-45cc-8bf8-a7b2c90d597c","Type":"ContainerStarted","Data":"3a26621de339e5b296b697ea96fa5acb76fdc36069c4e1310d0a1f5dce69a36c"} Nov 03 13:15:07.937421 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:07.937363 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895195-c5h4v" podStartSLOduration=1.6270479629999999 podStartE2EDuration="7.937348923s" totalImagesPullingTime="6.31030096s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:15:07.935286136 +0000 UTC m=+589.945974378" watchObservedRunningTime="2026-11-03 13:15:07.937348923 +0000 UTC m=+589.948037165" Nov 03 13:15:08.820485 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:08.820443 5979 generic.go:356] "Generic (PLEG): container finished" podID="74c198a3-f63a-45cc-8bf8-a7b2c90d597c" containerID="3a26621de339e5b296b697ea96fa5acb76fdc36069c4e1310d0a1f5dce69a36c" exitCode=60 Nov 03 13:15:08.820914 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:08.820536 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895195-c5h4v" event={"ID":"74c198a3-f63a-45cc-8bf8-a7b2c90d597c","Type":"ContainerDied","Data":"3a26621de339e5b296b697ea96fa5acb76fdc36069c4e1310d0a1f5dce69a36c"} Nov 03 13:15:08.820914 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:08.820825 5979 scope.go:122] "RemoveContainer" containerID="3a26621de339e5b296b697ea96fa5acb76fdc36069c4e1310d0a1f5dce69a36c" Nov 03 13:15:09.827397 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:09.827307 5979 generic.go:356] "Generic (PLEG): container finished" podID="74c198a3-f63a-45cc-8bf8-a7b2c90d597c" containerID="418fedb8715214281db2389cc893937f1113683aa03dc3834d260b5ec53099aa" exitCode=60 Nov 03 13:15:09.827397 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:09.827352 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895195-c5h4v" event={"ID":"74c198a3-f63a-45cc-8bf8-a7b2c90d597c","Type":"ContainerDied","Data":"418fedb8715214281db2389cc893937f1113683aa03dc3834d260b5ec53099aa"} Nov 03 13:15:09.827910 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:09.827410 5979 scope.go:122] "RemoveContainer" containerID="3a26621de339e5b296b697ea96fa5acb76fdc36069c4e1310d0a1f5dce69a36c" Nov 03 13:15:09.827910 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:09.827691 5979 scope.go:122] "RemoveContainer" containerID="418fedb8715214281db2389cc893937f1113683aa03dc3834d260b5ec53099aa" Nov 03 13:15:09.828023 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:15:09.827951 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895195-c5h4v_odh-ai-gateway-infra(74c198a3-f63a-45cc-8bf8-a7b2c90d597c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895195-c5h4v" podUID="74c198a3-f63a-45cc-8bf8-a7b2c90d597c" Nov 03 13:15:10.834868 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:10.834836 5979 scope.go:122] "RemoveContainer" containerID="418fedb8715214281db2389cc893937f1113683aa03dc3834d260b5ec53099aa" Nov 03 13:15:10.835254 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:15:10.835071 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895195-c5h4v_odh-ai-gateway-infra(74c198a3-f63a-45cc-8bf8-a7b2c90d597c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895195-c5h4v" podUID="74c198a3-f63a-45cc-8bf8-a7b2c90d597c" Nov 03 13:15:11.355812 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:11.355772 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9f5wz"] Nov 03 13:15:11.359322 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:11.359294 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9f5wz" Nov 03 13:15:11.421840 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:11.421802 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a14eed4-c891-453a-af30-354574069bc0-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9f5wz\" (UID: \"1a14eed4-c891-453a-af30-354574069bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9f5wz" Nov 03 13:15:11.422014 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:11.421853 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a14eed4-c891-453a-af30-354574069bc0-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9f5wz\" (UID: \"1a14eed4-c891-453a-af30-354574069bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9f5wz" Nov 03 13:15:11.422014 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:11.421960 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a14eed4-c891-453a-af30-354574069bc0-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9f5wz\" (UID: \"1a14eed4-c891-453a-af30-354574069bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9f5wz" Nov 03 13:15:11.422014 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:11.422006 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv2rm\" (UniqueName: \"kubernetes.io/projected/1a14eed4-c891-453a-af30-354574069bc0-kube-api-access-nv2rm\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9f5wz\" (UID: \"1a14eed4-c891-453a-af30-354574069bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9f5wz" Nov 03 13:15:11.422174 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:11.422050 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a14eed4-c891-453a-af30-354574069bc0-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9f5wz\" (UID: \"1a14eed4-c891-453a-af30-354574069bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9f5wz" Nov 03 13:15:11.422232 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:11.422160 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a14eed4-c891-453a-af30-354574069bc0-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9f5wz\" (UID: \"1a14eed4-c891-453a-af30-354574069bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9f5wz" Nov 03 13:15:11.422232 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:11.422198 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a14eed4-c891-453a-af30-354574069bc0-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9f5wz\" (UID: \"1a14eed4-c891-453a-af30-354574069bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9f5wz" Nov 03 13:15:11.422331 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:11.422307 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a14eed4-c891-453a-af30-354574069bc0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9f5wz\" (UID: \"1a14eed4-c891-453a-af30-354574069bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9f5wz" Nov 03 13:15:11.422378 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:11.422343 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a14eed4-c891-453a-af30-354574069bc0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9f5wz\" (UID: \"1a14eed4-c891-453a-af30-354574069bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9f5wz" Nov 03 13:15:11.458858 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:11.458819 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9f5wz"] Nov 03 13:15:11.523518 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:11.523460 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a14eed4-c891-453a-af30-354574069bc0-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9f5wz\" (UID: \"1a14eed4-c891-453a-af30-354574069bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9f5wz" Nov 03 13:15:11.523518 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:11.523520 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a14eed4-c891-453a-af30-354574069bc0-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9f5wz\" (UID: \"1a14eed4-c891-453a-af30-354574069bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9f5wz" Nov 03 13:15:11.523763 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:11.523558 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a14eed4-c891-453a-af30-354574069bc0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9f5wz\" (UID: \"1a14eed4-c891-453a-af30-354574069bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9f5wz" Nov 03 13:15:11.523763 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:11.523583 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a14eed4-c891-453a-af30-354574069bc0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9f5wz\" (UID: \"1a14eed4-c891-453a-af30-354574069bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9f5wz" Nov 03 13:15:11.523763 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:11.523637 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a14eed4-c891-453a-af30-354574069bc0-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9f5wz\" (UID: \"1a14eed4-c891-453a-af30-354574069bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9f5wz" Nov 03 13:15:11.523763 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:11.523685 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a14eed4-c891-453a-af30-354574069bc0-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9f5wz\" (UID: \"1a14eed4-c891-453a-af30-354574069bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9f5wz" Nov 03 13:15:11.523763 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:11.523739 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a14eed4-c891-453a-af30-354574069bc0-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9f5wz\" (UID: \"1a14eed4-c891-453a-af30-354574069bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9f5wz" Nov 03 13:15:11.524023 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:11.523782 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv2rm\" (UniqueName: \"kubernetes.io/projected/1a14eed4-c891-453a-af30-354574069bc0-kube-api-access-nv2rm\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9f5wz\" (UID: \"1a14eed4-c891-453a-af30-354574069bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9f5wz" Nov 03 13:15:11.524023 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:11.523850 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a14eed4-c891-453a-af30-354574069bc0-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9f5wz\" (UID: \"1a14eed4-c891-453a-af30-354574069bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9f5wz" Nov 03 13:15:11.524023 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:11.523918 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a14eed4-c891-453a-af30-354574069bc0-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9f5wz\" (UID: \"1a14eed4-c891-453a-af30-354574069bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9f5wz" Nov 03 13:15:11.524154 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:11.524053 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a14eed4-c891-453a-af30-354574069bc0-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9f5wz\" (UID: \"1a14eed4-c891-453a-af30-354574069bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9f5wz" Nov 03 13:15:11.524224 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:11.524179 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a14eed4-c891-453a-af30-354574069bc0-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9f5wz\" (UID: \"1a14eed4-c891-453a-af30-354574069bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9f5wz" Nov 03 13:15:11.524370 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:11.524350 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a14eed4-c891-453a-af30-354574069bc0-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9f5wz\" (UID: \"1a14eed4-c891-453a-af30-354574069bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9f5wz" Nov 03 13:15:11.524572 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:11.524546 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a14eed4-c891-453a-af30-354574069bc0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9f5wz\" (UID: \"1a14eed4-c891-453a-af30-354574069bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9f5wz" Nov 03 13:15:11.525926 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:11.525907 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a14eed4-c891-453a-af30-354574069bc0-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9f5wz\" (UID: \"1a14eed4-c891-453a-af30-354574069bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9f5wz" Nov 03 13:15:11.526173 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:11.526156 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a14eed4-c891-453a-af30-354574069bc0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9f5wz\" (UID: \"1a14eed4-c891-453a-af30-354574069bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9f5wz" Nov 03 13:15:11.578714 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:11.578678 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a14eed4-c891-453a-af30-354574069bc0-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9f5wz\" (UID: \"1a14eed4-c891-453a-af30-354574069bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9f5wz" Nov 03 13:15:11.578913 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:11.578767 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv2rm\" (UniqueName: \"kubernetes.io/projected/1a14eed4-c891-453a-af30-354574069bc0-kube-api-access-nv2rm\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-9f5wz\" (UID: \"1a14eed4-c891-453a-af30-354574069bc0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9f5wz" Nov 03 13:15:11.670632 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:11.670600 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9f5wz" Nov 03 13:15:12.093243 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:12.093150 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9f5wz"] Nov 03 13:15:12.100403 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:15:12.100368 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a14eed4_c891_453a_af30_354574069bc0.slice/crio-ab00879295e2b7771f6c0d70dfc642874df828772a3097e285797b63e236ef52 WatchSource:0}: Error finding container ab00879295e2b7771f6c0d70dfc642874df828772a3097e285797b63e236ef52: Status 404 returned error can't find the container with id ab00879295e2b7771f6c0d70dfc642874df828772a3097e285797b63e236ef52 Nov 03 13:15:12.102900 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:12.102863 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:15:12.102995 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:12.102953 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:15:12.103095 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:12.102997 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:15:12.869779 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:12.869743 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9f5wz" event={"ID":"1a14eed4-c891-453a-af30-354574069bc0","Type":"ContainerStarted","Data":"34a92327e9ddcfaa0aa13e6a764eff86bf05f3f38ac84ebd02a764796377a2d1"} Nov 03 13:15:12.870007 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:12.869798 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9f5wz" event={"ID":"1a14eed4-c891-453a-af30-354574069bc0","Type":"ContainerStarted","Data":"ab00879295e2b7771f6c0d70dfc642874df828772a3097e285797b63e236ef52"} Nov 03 13:15:12.979996 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:12.979940 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9f5wz" podStartSLOduration=1.979924337 podStartE2EDuration="1.979924337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:15:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:15:12.978187599 +0000 UTC m=+594.988875840" watchObservedRunningTime="2026-11-03 13:15:12.979924337 +0000 UTC m=+594.990612578" Nov 03 13:15:13.671547 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:13.671504 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9f5wz" Nov 03 13:15:13.672915 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:13.672891 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-9f5wz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 03 13:15:13.673034 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:13.672936 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9f5wz" podUID="1a14eed4-c891-453a-af30-354574069bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 03 13:15:14.671372 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:14.671332 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-9f5wz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 03 13:15:14.671583 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:14.671414 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9f5wz" podUID="1a14eed4-c891-453a-af30-354574069bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 03 13:15:15.671918 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:15.671871 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-9f5wz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 03 13:15:15.672307 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:15.671936 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9f5wz" podUID="1a14eed4-c891-453a-af30-354574069bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 03 13:15:16.671370 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:16.671329 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-9f5wz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 03 13:15:16.671662 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:16.671406 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9f5wz" podUID="1a14eed4-c891-453a-af30-354574069bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 03 13:15:17.671181 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:17.671137 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-9f5wz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 03 13:15:17.671717 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:17.671211 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9f5wz" podUID="1a14eed4-c891-453a-af30-354574069bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 03 13:15:18.671814 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:18.671776 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-9f5wz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Nov 03 13:15:18.672209 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:18.671860 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9f5wz" podUID="1a14eed4-c891-453a-af30-354574069bc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Nov 03 13:15:20.437846 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:20.437811 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9f5wz" Nov 03 13:15:20.438289 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:20.438165 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9f5wz" Nov 03 13:15:20.438814 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:20.438788 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9f5wz" Nov 03 13:15:22.108026 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:22.107994 5979 scope.go:122] "RemoveContainer" containerID="418fedb8715214281db2389cc893937f1113683aa03dc3834d260b5ec53099aa" Nov 03 13:15:22.930383 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:22.930351 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895195-c5h4v" event={"ID":"74c198a3-f63a-45cc-8bf8-a7b2c90d597c","Type":"ContainerStarted","Data":"41fcd5e6b93c54e208f66efb4ca4311ca997cc02c76570541d2cefe58409a16c"} Nov 03 13:15:23.936528 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:23.936487 5979 generic.go:356] "Generic (PLEG): container finished" podID="74c198a3-f63a-45cc-8bf8-a7b2c90d597c" containerID="41fcd5e6b93c54e208f66efb4ca4311ca997cc02c76570541d2cefe58409a16c" exitCode=60 Nov 03 13:15:23.936989 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:23.936497 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895195-c5h4v" event={"ID":"74c198a3-f63a-45cc-8bf8-a7b2c90d597c","Type":"ContainerDied","Data":"41fcd5e6b93c54e208f66efb4ca4311ca997cc02c76570541d2cefe58409a16c"} Nov 03 13:15:23.936989 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:23.936651 5979 scope.go:122] "RemoveContainer" containerID="418fedb8715214281db2389cc893937f1113683aa03dc3834d260b5ec53099aa" Nov 03 13:15:23.937078 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:23.937021 5979 scope.go:122] "RemoveContainer" containerID="41fcd5e6b93c54e208f66efb4ca4311ca997cc02c76570541d2cefe58409a16c" Nov 03 13:15:23.937302 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:15:23.937280 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895195-c5h4v_odh-ai-gateway-infra(74c198a3-f63a-45cc-8bf8-a7b2c90d597c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895195-c5h4v" podUID="74c198a3-f63a-45cc-8bf8-a7b2c90d597c" Nov 03 13:15:24.063564 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:24.063526 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895195-c5h4v"] Nov 03 13:15:25.081706 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:25.081683 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895195-c5h4v" Nov 03 13:15:25.086032 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:25.086012 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74c198a3-f63a-45cc-8bf8-a7b2c90d597c-kube-api-access-kjvfs\" (UniqueName: \"kubernetes.io/projected/74c198a3-f63a-45cc-8bf8-a7b2c90d597c-kube-api-access-kjvfs\") pod \"74c198a3-f63a-45cc-8bf8-a7b2c90d597c\" (UID: \"74c198a3-f63a-45cc-8bf8-a7b2c90d597c\") " Nov 03 13:15:25.086168 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:25.086148 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74c198a3-f63a-45cc-8bf8-a7b2c90d597c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74c198a3-f63a-45cc-8bf8-a7b2c90d597c-maas-api-service-ca\") pod \"74c198a3-f63a-45cc-8bf8-a7b2c90d597c\" (UID: \"74c198a3-f63a-45cc-8bf8-a7b2c90d597c\") " Nov 03 13:15:25.086498 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:25.086455 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74c198a3-f63a-45cc-8bf8-a7b2c90d597c-maas-api-service-ca" pod "74c198a3-f63a-45cc-8bf8-a7b2c90d597c" (UID: "74c198a3-f63a-45cc-8bf8-a7b2c90d597c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:15:25.088181 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:25.088163 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74c198a3-f63a-45cc-8bf8-a7b2c90d597c-kube-api-access-kjvfs" pod "74c198a3-f63a-45cc-8bf8-a7b2c90d597c" (UID: "74c198a3-f63a-45cc-8bf8-a7b2c90d597c"). InnerVolumeSpecName "kube-api-access-kjvfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:15:25.187143 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:25.187107 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/74c198a3-f63a-45cc-8bf8-a7b2c90d597c-maas-api-service-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:15:25.187143 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:25.187138 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjvfs\" (UniqueName: \"kubernetes.io/projected/74c198a3-f63a-45cc-8bf8-a7b2c90d597c-kube-api-access-kjvfs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:15:25.949178 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:25.949145 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895195-c5h4v" Nov 03 13:15:25.949364 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:25.949181 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895195-c5h4v" event={"ID":"74c198a3-f63a-45cc-8bf8-a7b2c90d597c","Type":"ContainerDied","Data":"0b3096890aee80cf26473f7c51474d9d26e8223854f0e837662806ad49066956"} Nov 03 13:15:25.949364 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:25.949244 5979 scope.go:122] "RemoveContainer" containerID="41fcd5e6b93c54e208f66efb4ca4311ca997cc02c76570541d2cefe58409a16c" Nov 03 13:15:26.102800 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:26.102765 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895195-c5h4v"] Nov 03 13:15:26.149021 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:26.148985 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895195-c5h4v"] Nov 03 13:15:28.114146 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:28.114113 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74c198a3-f63a-45cc-8bf8-a7b2c90d597c" path="/var/lib/kubelet/pods/74c198a3-f63a-45cc-8bf8-a7b2c90d597c/volumes" Nov 03 13:15:37.233312 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:15:37.233269 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:15:40.644444 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:40.644404 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p"] Nov 03 13:15:40.644996 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:40.644955 5979 cpu_manager.go:414] "containerMap: removing container" podUID="74c198a3-f63a-45cc-8bf8-a7b2c90d597c" containerName="cleanup" Nov 03 13:15:40.644996 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:40.644974 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74c198a3-f63a-45cc-8bf8-a7b2c90d597c" containerName="cleanup" Nov 03 13:15:40.645155 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:40.645010 5979 cpu_manager.go:414] "containerMap: removing container" podUID="74c198a3-f63a-45cc-8bf8-a7b2c90d597c" containerName="cleanup" Nov 03 13:15:40.645155 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:40.645017 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74c198a3-f63a-45cc-8bf8-a7b2c90d597c" containerName="cleanup" Nov 03 13:15:40.645155 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:40.645130 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="74c198a3-f63a-45cc-8bf8-a7b2c90d597c" containerName="cleanup" Nov 03 13:15:40.645155 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:40.645143 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="74c198a3-f63a-45cc-8bf8-a7b2c90d597c" containerName="cleanup" Nov 03 13:15:40.652481 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:40.652447 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p" Nov 03 13:15:40.740258 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:40.740219 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-istio-token\") pod \"e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p\" (UID: \"435a8d1c-24ab-4dce-a4e0-10aa8cff506f\") " pod="openshift-ingress/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p" Nov 03 13:15:40.740258 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:40.740259 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-workload-socket\") pod \"e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p\" (UID: \"435a8d1c-24ab-4dce-a4e0-10aa8cff506f\") " pod="openshift-ingress/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p" Nov 03 13:15:40.740531 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:40.740295 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-istio-podinfo\") pod \"e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p\" (UID: \"435a8d1c-24ab-4dce-a4e0-10aa8cff506f\") " pod="openshift-ingress/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p" Nov 03 13:15:40.740531 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:40.740498 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-credential-socket\") pod \"e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p\" (UID: \"435a8d1c-24ab-4dce-a4e0-10aa8cff506f\") " pod="openshift-ingress/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p" Nov 03 13:15:40.740645 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:40.740548 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-workload-certs\") pod \"e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p\" (UID: \"435a8d1c-24ab-4dce-a4e0-10aa8cff506f\") " pod="openshift-ingress/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p" Nov 03 13:15:40.740645 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:40.740605 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-istio-data\") pod \"e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p\" (UID: \"435a8d1c-24ab-4dce-a4e0-10aa8cff506f\") " pod="openshift-ingress/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p" Nov 03 13:15:40.740722 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:40.740646 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-istiod-ca-cert\") pod \"e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p\" (UID: \"435a8d1c-24ab-4dce-a4e0-10aa8cff506f\") " pod="openshift-ingress/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p" Nov 03 13:15:40.740722 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:40.740680 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-istio-envoy\") pod \"e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p\" (UID: \"435a8d1c-24ab-4dce-a4e0-10aa8cff506f\") " pod="openshift-ingress/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p" Nov 03 13:15:40.740821 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:40.740756 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-948v2\" (UniqueName: \"kubernetes.io/projected/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-kube-api-access-948v2\") pod \"e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p\" (UID: \"435a8d1c-24ab-4dce-a4e0-10aa8cff506f\") " pod="openshift-ingress/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p" Nov 03 13:15:40.750856 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:40.750835 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-821a13f6-openshift-default-dockercfg-fbkp2\"" Nov 03 13:15:40.752496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:40.752459 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p"] Nov 03 13:15:40.841845 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:40.841811 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-credential-socket\") pod \"e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p\" (UID: \"435a8d1c-24ab-4dce-a4e0-10aa8cff506f\") " pod="openshift-ingress/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p" Nov 03 13:15:40.841845 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:40.841846 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-workload-certs\") pod \"e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p\" (UID: \"435a8d1c-24ab-4dce-a4e0-10aa8cff506f\") " pod="openshift-ingress/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p" Nov 03 13:15:40.842109 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:40.841877 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-istio-data\") pod \"e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p\" (UID: \"435a8d1c-24ab-4dce-a4e0-10aa8cff506f\") " pod="openshift-ingress/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p" Nov 03 13:15:40.842109 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:40.841893 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-istiod-ca-cert\") pod \"e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p\" (UID: \"435a8d1c-24ab-4dce-a4e0-10aa8cff506f\") " pod="openshift-ingress/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p" Nov 03 13:15:40.842109 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:40.841921 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-istio-envoy\") pod \"e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p\" (UID: \"435a8d1c-24ab-4dce-a4e0-10aa8cff506f\") " pod="openshift-ingress/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p" Nov 03 13:15:40.842109 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:40.841952 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-948v2\" (UniqueName: \"kubernetes.io/projected/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-kube-api-access-948v2\") pod \"e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p\" (UID: \"435a8d1c-24ab-4dce-a4e0-10aa8cff506f\") " pod="openshift-ingress/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p" Nov 03 13:15:40.842109 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:40.841995 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-istio-token\") pod \"e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p\" (UID: \"435a8d1c-24ab-4dce-a4e0-10aa8cff506f\") " pod="openshift-ingress/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p" Nov 03 13:15:40.842109 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:40.842059 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-workload-socket\") pod \"e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p\" (UID: \"435a8d1c-24ab-4dce-a4e0-10aa8cff506f\") " pod="openshift-ingress/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p" Nov 03 13:15:40.842403 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:40.842120 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-istio-podinfo\") pod \"e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p\" (UID: \"435a8d1c-24ab-4dce-a4e0-10aa8cff506f\") " pod="openshift-ingress/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p" Nov 03 13:15:40.842403 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:40.842281 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-credential-socket\") pod \"e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p\" (UID: \"435a8d1c-24ab-4dce-a4e0-10aa8cff506f\") " pod="openshift-ingress/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p" Nov 03 13:15:40.842403 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:40.842325 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-workload-certs\") pod \"e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p\" (UID: \"435a8d1c-24ab-4dce-a4e0-10aa8cff506f\") " pod="openshift-ingress/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p" Nov 03 13:15:40.842403 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:40.842327 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-istio-data\") pod \"e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p\" (UID: \"435a8d1c-24ab-4dce-a4e0-10aa8cff506f\") " pod="openshift-ingress/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p" Nov 03 13:15:40.842667 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:40.842489 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-workload-socket\") pod \"e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p\" (UID: \"435a8d1c-24ab-4dce-a4e0-10aa8cff506f\") " pod="openshift-ingress/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p" Nov 03 13:15:40.842934 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:40.842916 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-istiod-ca-cert\") pod \"e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p\" (UID: \"435a8d1c-24ab-4dce-a4e0-10aa8cff506f\") " pod="openshift-ingress/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p" Nov 03 13:15:40.844547 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:40.844526 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-istio-envoy\") pod \"e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p\" (UID: \"435a8d1c-24ab-4dce-a4e0-10aa8cff506f\") " pod="openshift-ingress/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p" Nov 03 13:15:40.844699 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:40.844682 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-istio-podinfo\") pod \"e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p\" (UID: \"435a8d1c-24ab-4dce-a4e0-10aa8cff506f\") " pod="openshift-ingress/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p" Nov 03 13:15:40.895931 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:40.895853 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-istio-token\") pod \"e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p\" (UID: \"435a8d1c-24ab-4dce-a4e0-10aa8cff506f\") " pod="openshift-ingress/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p" Nov 03 13:15:40.896204 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:40.896185 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-948v2\" (UniqueName: \"kubernetes.io/projected/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-kube-api-access-948v2\") pod \"e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p\" (UID: \"435a8d1c-24ab-4dce-a4e0-10aa8cff506f\") " pod="openshift-ingress/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p" Nov 03 13:15:40.965241 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:40.965198 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p" Nov 03 13:15:41.392203 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:41.392167 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p"] Nov 03 13:15:41.405781 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:41.405732 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:15:41.405907 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:41.405832 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:15:41.405907 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:41.405875 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:15:42.033028 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:42.032994 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p" event={"ID":"435a8d1c-24ab-4dce-a4e0-10aa8cff506f","Type":"ContainerStarted","Data":"1d1af1b397212f6fcfce65e98bf87f16dfaccb80dd1b021a7cb2c315eb3e091b"} Nov 03 13:15:42.033416 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:42.033045 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p" event={"ID":"435a8d1c-24ab-4dce-a4e0-10aa8cff506f","Type":"ContainerStarted","Data":"e54d976fb7051b8b231bf2f84f4d5dcebb70c0372234b7a3e91b8cc837462d4a"} Nov 03 13:15:42.146106 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:42.145972 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p" podStartSLOduration=2.145954974 podStartE2EDuration="2.145954974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:15:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:15:42.143631934 +0000 UTC m=+624.154320178" watchObservedRunningTime="2026-11-03 13:15:42.145954974 +0000 UTC m=+624.156643218" Nov 03 13:15:42.966236 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:42.966199 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p" Nov 03 13:15:42.971128 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:42.971100 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p" Nov 03 13:15:43.037909 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:43.037881 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p" Nov 03 13:15:43.038840 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:43.038820 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p" Nov 03 13:15:43.129313 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:43.129281 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb"] Nov 03 13:15:43.129724 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:43.129712 5979 cpu_manager.go:414] "containerMap: removing container" podUID="74c198a3-f63a-45cc-8bf8-a7b2c90d597c" containerName="cleanup" Nov 03 13:15:43.129780 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:43.129726 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74c198a3-f63a-45cc-8bf8-a7b2c90d597c" containerName="cleanup" Nov 03 13:15:43.129821 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:43.129811 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="74c198a3-f63a-45cc-8bf8-a7b2c90d597c" containerName="cleanup" Nov 03 13:15:43.133304 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:43.133281 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb" Nov 03 13:15:43.181896 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:43.181871 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-11ca03ad-openshift-default-dockercfg-shvkw\"" Nov 03 13:15:43.188557 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:43.188535 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb"] Nov 03 13:15:43.272140 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:43.272066 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad693484-f565-418b-acbc-dd3739a1adec-istio-envoy\") pod \"e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb\" (UID: \"ad693484-f565-418b-acbc-dd3739a1adec\") " pod="openshift-ingress/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb" Nov 03 13:15:43.272140 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:43.272123 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad693484-f565-418b-acbc-dd3739a1adec-credential-socket\") pod \"e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb\" (UID: \"ad693484-f565-418b-acbc-dd3739a1adec\") " pod="openshift-ingress/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb" Nov 03 13:15:43.272322 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:43.272179 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad693484-f565-418b-acbc-dd3739a1adec-istiod-ca-cert\") pod \"e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb\" (UID: \"ad693484-f565-418b-acbc-dd3739a1adec\") " pod="openshift-ingress/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb" Nov 03 13:15:43.272322 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:43.272208 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad693484-f565-418b-acbc-dd3739a1adec-workload-socket\") pod \"e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb\" (UID: \"ad693484-f565-418b-acbc-dd3739a1adec\") " pod="openshift-ingress/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb" Nov 03 13:15:43.272322 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:43.272230 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad693484-f565-418b-acbc-dd3739a1adec-istio-data\") pod \"e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb\" (UID: \"ad693484-f565-418b-acbc-dd3739a1adec\") " pod="openshift-ingress/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb" Nov 03 13:15:43.272322 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:43.272316 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad693484-f565-418b-acbc-dd3739a1adec-workload-certs\") pod \"e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb\" (UID: \"ad693484-f565-418b-acbc-dd3739a1adec\") " pod="openshift-ingress/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb" Nov 03 13:15:43.272456 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:43.272331 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad693484-f565-418b-acbc-dd3739a1adec-istio-token\") pod \"e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb\" (UID: \"ad693484-f565-418b-acbc-dd3739a1adec\") " pod="openshift-ingress/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb" Nov 03 13:15:43.272456 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:43.272394 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pwgz\" (UniqueName: \"kubernetes.io/projected/ad693484-f565-418b-acbc-dd3739a1adec-kube-api-access-8pwgz\") pod \"e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb\" (UID: \"ad693484-f565-418b-acbc-dd3739a1adec\") " pod="openshift-ingress/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb" Nov 03 13:15:43.272456 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:43.272439 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad693484-f565-418b-acbc-dd3739a1adec-istio-podinfo\") pod \"e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb\" (UID: \"ad693484-f565-418b-acbc-dd3739a1adec\") " pod="openshift-ingress/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb" Nov 03 13:15:43.374064 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:43.374023 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad693484-f565-418b-acbc-dd3739a1adec-istio-envoy\") pod \"e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb\" (UID: \"ad693484-f565-418b-acbc-dd3739a1adec\") " pod="openshift-ingress/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb" Nov 03 13:15:43.374251 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:43.374088 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad693484-f565-418b-acbc-dd3739a1adec-credential-socket\") pod \"e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb\" (UID: \"ad693484-f565-418b-acbc-dd3739a1adec\") " pod="openshift-ingress/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb" Nov 03 13:15:43.374251 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:43.374106 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad693484-f565-418b-acbc-dd3739a1adec-istiod-ca-cert\") pod \"e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb\" (UID: \"ad693484-f565-418b-acbc-dd3739a1adec\") " pod="openshift-ingress/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb" Nov 03 13:15:43.374251 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:43.374134 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad693484-f565-418b-acbc-dd3739a1adec-workload-socket\") pod \"e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb\" (UID: \"ad693484-f565-418b-acbc-dd3739a1adec\") " pod="openshift-ingress/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb" Nov 03 13:15:43.374251 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:43.374160 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad693484-f565-418b-acbc-dd3739a1adec-istio-data\") pod \"e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb\" (UID: \"ad693484-f565-418b-acbc-dd3739a1adec\") " pod="openshift-ingress/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb" Nov 03 13:15:43.374251 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:43.374200 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad693484-f565-418b-acbc-dd3739a1adec-workload-certs\") pod \"e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb\" (UID: \"ad693484-f565-418b-acbc-dd3739a1adec\") " pod="openshift-ingress/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb" Nov 03 13:15:43.374251 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:43.374219 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad693484-f565-418b-acbc-dd3739a1adec-istio-token\") pod \"e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb\" (UID: \"ad693484-f565-418b-acbc-dd3739a1adec\") " pod="openshift-ingress/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb" Nov 03 13:15:43.374590 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:43.374257 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pwgz\" (UniqueName: \"kubernetes.io/projected/ad693484-f565-418b-acbc-dd3739a1adec-kube-api-access-8pwgz\") pod \"e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb\" (UID: \"ad693484-f565-418b-acbc-dd3739a1adec\") " pod="openshift-ingress/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb" Nov 03 13:15:43.374590 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:43.374280 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad693484-f565-418b-acbc-dd3739a1adec-istio-podinfo\") pod \"e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb\" (UID: \"ad693484-f565-418b-acbc-dd3739a1adec\") " pod="openshift-ingress/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb" Nov 03 13:15:43.374590 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:43.374522 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad693484-f565-418b-acbc-dd3739a1adec-credential-socket\") pod \"e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb\" (UID: \"ad693484-f565-418b-acbc-dd3739a1adec\") " pod="openshift-ingress/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb" Nov 03 13:15:43.374749 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:43.374598 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad693484-f565-418b-acbc-dd3739a1adec-workload-socket\") pod \"e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb\" (UID: \"ad693484-f565-418b-acbc-dd3739a1adec\") " pod="openshift-ingress/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb" Nov 03 13:15:43.374918 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:43.374893 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad693484-f565-418b-acbc-dd3739a1adec-workload-certs\") pod \"e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb\" (UID: \"ad693484-f565-418b-acbc-dd3739a1adec\") " pod="openshift-ingress/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb" Nov 03 13:15:43.374984 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:43.374922 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad693484-f565-418b-acbc-dd3739a1adec-istiod-ca-cert\") pod \"e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb\" (UID: \"ad693484-f565-418b-acbc-dd3739a1adec\") " pod="openshift-ingress/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb" Nov 03 13:15:43.374984 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:43.374897 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad693484-f565-418b-acbc-dd3739a1adec-istio-data\") pod \"e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb\" (UID: \"ad693484-f565-418b-acbc-dd3739a1adec\") " pod="openshift-ingress/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb" Nov 03 13:15:43.376427 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:43.376406 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad693484-f565-418b-acbc-dd3739a1adec-istio-envoy\") pod \"e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb\" (UID: \"ad693484-f565-418b-acbc-dd3739a1adec\") " pod="openshift-ingress/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb" Nov 03 13:15:43.377296 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:43.377271 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad693484-f565-418b-acbc-dd3739a1adec-istio-podinfo\") pod \"e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb\" (UID: \"ad693484-f565-418b-acbc-dd3739a1adec\") " pod="openshift-ingress/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb" Nov 03 13:15:43.427855 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:43.427820 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad693484-f565-418b-acbc-dd3739a1adec-istio-token\") pod \"e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb\" (UID: \"ad693484-f565-418b-acbc-dd3739a1adec\") " pod="openshift-ingress/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb" Nov 03 13:15:43.427995 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:43.427908 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pwgz\" (UniqueName: \"kubernetes.io/projected/ad693484-f565-418b-acbc-dd3739a1adec-kube-api-access-8pwgz\") pod \"e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb\" (UID: \"ad693484-f565-418b-acbc-dd3739a1adec\") " pod="openshift-ingress/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb" Nov 03 13:15:43.449856 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:43.449830 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb" Nov 03 13:15:43.879516 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:43.879481 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb"] Nov 03 13:15:43.886453 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:15:43.886425 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad693484_f565_418b_acbc_dd3739a1adec.slice/crio-cb2123912b6862f22fba9ef3e688551b25b83d65ec75cfc0e055299589105f8b WatchSource:0}: Error finding container cb2123912b6862f22fba9ef3e688551b25b83d65ec75cfc0e055299589105f8b: Status 404 returned error can't find the container with id cb2123912b6862f22fba9ef3e688551b25b83d65ec75cfc0e055299589105f8b Nov 03 13:15:43.888628 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:43.888595 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:15:43.888714 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:43.888660 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:15:43.888714 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:43.888689 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:15:44.044427 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:44.044392 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb" event={"ID":"ad693484-f565-418b-acbc-dd3739a1adec","Type":"ContainerStarted","Data":"691aca82b8868a5d414a9d5f626f259f2e3ac04c6529c65cb66fba10f959ecbb"} Nov 03 13:15:44.044837 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:44.044447 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb" event={"ID":"ad693484-f565-418b-acbc-dd3739a1adec","Type":"ContainerStarted","Data":"cb2123912b6862f22fba9ef3e688551b25b83d65ec75cfc0e055299589105f8b"} Nov 03 13:15:44.155108 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:44.155056 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb" podStartSLOduration=1.155041279 podStartE2EDuration="1.155041279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:15:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:15:44.153300457 +0000 UTC m=+626.163988699" watchObservedRunningTime="2026-11-03 13:15:44.155041279 +0000 UTC m=+626.165729520" Nov 03 13:15:44.451090 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:44.450996 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb" Nov 03 13:15:44.456002 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:44.455978 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb" Nov 03 13:15:45.049095 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:45.049068 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb" Nov 03 13:15:45.050014 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:45.049992 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb" Nov 03 13:15:47.511051 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:47.511012 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-67349d1c-6cb59557f7-2j7fx"] Nov 03 13:15:47.516552 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:47.516522 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-67349d1c-6cb59557f7-2j7fx" Nov 03 13:15:47.564296 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:47.564259 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-67349d1c-6cb59557f7-2j7fx"] Nov 03 13:15:47.608679 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:47.608646 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-67349d1c-dockercfg-sh2cf\"" Nov 03 13:15:47.608679 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:47.608674 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-67349d1c\"" Nov 03 13:15:47.621544 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:47.621516 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr9mv\" (UniqueName: \"kubernetes.io/projected/2c39eb4f-d901-4453-9e96-8236716c1e91-kube-api-access-xr9mv\") pod \"payload-processing-e2e-ait-67349d1c-6cb59557f7-2j7fx\" (UID: \"2c39eb4f-d901-4453-9e96-8236716c1e91\") " pod="openshift-ingress/payload-processing-e2e-ait-67349d1c-6cb59557f7-2j7fx" Nov 03 13:15:47.621684 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:47.621618 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c39eb4f-d901-4453-9e96-8236716c1e91-plugins-config-volume\") pod \"payload-processing-e2e-ait-67349d1c-6cb59557f7-2j7fx\" (UID: \"2c39eb4f-d901-4453-9e96-8236716c1e91\") " pod="openshift-ingress/payload-processing-e2e-ait-67349d1c-6cb59557f7-2j7fx" Nov 03 13:15:47.722894 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:47.722853 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr9mv\" (UniqueName: \"kubernetes.io/projected/2c39eb4f-d901-4453-9e96-8236716c1e91-kube-api-access-xr9mv\") pod \"payload-processing-e2e-ait-67349d1c-6cb59557f7-2j7fx\" (UID: \"2c39eb4f-d901-4453-9e96-8236716c1e91\") " pod="openshift-ingress/payload-processing-e2e-ait-67349d1c-6cb59557f7-2j7fx" Nov 03 13:15:47.723065 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:47.722966 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c39eb4f-d901-4453-9e96-8236716c1e91-plugins-config-volume\") pod \"payload-processing-e2e-ait-67349d1c-6cb59557f7-2j7fx\" (UID: \"2c39eb4f-d901-4453-9e96-8236716c1e91\") " pod="openshift-ingress/payload-processing-e2e-ait-67349d1c-6cb59557f7-2j7fx" Nov 03 13:15:47.723848 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:47.723827 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c39eb4f-d901-4453-9e96-8236716c1e91-plugins-config-volume\") pod \"payload-processing-e2e-ait-67349d1c-6cb59557f7-2j7fx\" (UID: \"2c39eb4f-d901-4453-9e96-8236716c1e91\") " pod="openshift-ingress/payload-processing-e2e-ait-67349d1c-6cb59557f7-2j7fx" Nov 03 13:15:47.779006 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:47.778925 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr9mv\" (UniqueName: \"kubernetes.io/projected/2c39eb4f-d901-4453-9e96-8236716c1e91-kube-api-access-xr9mv\") pod \"payload-processing-e2e-ait-67349d1c-6cb59557f7-2j7fx\" (UID: \"2c39eb4f-d901-4453-9e96-8236716c1e91\") " pod="openshift-ingress/payload-processing-e2e-ait-67349d1c-6cb59557f7-2j7fx" Nov 03 13:15:47.831277 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:47.831244 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-67349d1c-6cb59557f7-2j7fx" Nov 03 13:15:48.200249 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:48.200210 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-67349d1c-6cb59557f7-2j7fx"] Nov 03 13:15:48.207910 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:15:48.207874 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c39eb4f_d901_4453_9e96_8236716c1e91.slice/crio-4d15ecae4547fa5ec6a8b4252d26c07f949ef9834a64c573b744df7647f25b69 WatchSource:0}: Error finding container 4d15ecae4547fa5ec6a8b4252d26c07f949ef9834a64c573b744df7647f25b69: Status 404 returned error can't find the container with id 4d15ecae4547fa5ec6a8b4252d26c07f949ef9834a64c573b744df7647f25b69 Nov 03 13:15:49.072188 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:49.072151 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-67349d1c-6cb59557f7-2j7fx" event={"ID":"2c39eb4f-d901-4453-9e96-8236716c1e91","Type":"ContainerStarted","Data":"4d15ecae4547fa5ec6a8b4252d26c07f949ef9834a64c573b744df7647f25b69"} Nov 03 13:15:52.088247 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:52.088211 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-67349d1c-6cb59557f7-2j7fx" event={"ID":"2c39eb4f-d901-4453-9e96-8236716c1e91","Type":"ContainerStarted","Data":"6e069a51c01322f0e8e1e2e98d2e3eff0b47f17ad57d05787cafe6b73b5ef425"} Nov 03 13:15:52.088678 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:52.088356 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-67349d1c-6cb59557f7-2j7fx" Nov 03 13:15:52.198463 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:52.198404 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-67349d1c-6cb59557f7-2j7fx" podStartSLOduration=2.330575761 podStartE2EDuration="5.198390466s" totalImagesPullingTime="2.867814705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:15:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:15:52.196842673 +0000 UTC m=+634.207530916" watchObservedRunningTime="2026-11-03 13:15:52.198390466 +0000 UTC m=+634.209078705" Nov 03 13:15:52.245149 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:52.245107 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-821a13f6-dfd5b9795-2ffsx"] Nov 03 13:15:52.249118 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:52.249095 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-821a13f6-dfd5b9795-2ffsx" Nov 03 13:15:52.299931 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:52.299882 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-821a13f6\"" Nov 03 13:15:52.302032 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:52.302008 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-821a13f6-dockercfg-qpbpr\"" Nov 03 13:15:52.302376 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:52.302349 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-821a13f6-dfd5b9795-2ffsx"] Nov 03 13:15:52.375909 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:52.375817 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-821a13f6-649c78b7b7-8ht9h"] Nov 03 13:15:52.381323 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:52.381281 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5dd10cb-759b-48c0-a689-f1aa6abe8b8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-821a13f6-dfd5b9795-2ffsx\" (UID: \"c5dd10cb-759b-48c0-a689-f1aa6abe8b8d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-821a13f6-dfd5b9795-2ffsx" Nov 03 13:15:52.381541 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:52.381434 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fzz9\" (UniqueName: \"kubernetes.io/projected/c5dd10cb-759b-48c0-a689-f1aa6abe8b8d-kube-api-access-9fzz9\") pod \"payload-pre-processing-e2e-ait-821a13f6-dfd5b9795-2ffsx\" (UID: \"c5dd10cb-759b-48c0-a689-f1aa6abe8b8d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-821a13f6-dfd5b9795-2ffsx" Nov 03 13:15:52.381900 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:52.381883 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-821a13f6-649c78b7b7-8ht9h" Nov 03 13:15:52.433716 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:52.433681 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-821a13f6-649c78b7b7-8ht9h"] Nov 03 13:15:52.482493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:52.482441 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7ss6\" (UniqueName: \"kubernetes.io/projected/dd7cfbb5-315f-4b0c-bb3c-e2a8803ecf73-kube-api-access-s7ss6\") pod \"payload-processing-e2e-ait-821a13f6-649c78b7b7-8ht9h\" (UID: \"dd7cfbb5-315f-4b0c-bb3c-e2a8803ecf73\") " pod="openshift-ingress/payload-processing-e2e-ait-821a13f6-649c78b7b7-8ht9h" Nov 03 13:15:52.482673 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:52.482530 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5dd10cb-759b-48c0-a689-f1aa6abe8b8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-821a13f6-dfd5b9795-2ffsx\" (UID: \"c5dd10cb-759b-48c0-a689-f1aa6abe8b8d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-821a13f6-dfd5b9795-2ffsx" Nov 03 13:15:52.482673 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:52.482632 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fzz9\" (UniqueName: \"kubernetes.io/projected/c5dd10cb-759b-48c0-a689-f1aa6abe8b8d-kube-api-access-9fzz9\") pod \"payload-pre-processing-e2e-ait-821a13f6-dfd5b9795-2ffsx\" (UID: \"c5dd10cb-759b-48c0-a689-f1aa6abe8b8d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-821a13f6-dfd5b9795-2ffsx" Nov 03 13:15:52.482673 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:52.482668 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd7cfbb5-315f-4b0c-bb3c-e2a8803ecf73-plugins-config-volume\") pod \"payload-processing-e2e-ait-821a13f6-649c78b7b7-8ht9h\" (UID: \"dd7cfbb5-315f-4b0c-bb3c-e2a8803ecf73\") " pod="openshift-ingress/payload-processing-e2e-ait-821a13f6-649c78b7b7-8ht9h" Nov 03 13:15:52.483192 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:52.483170 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5dd10cb-759b-48c0-a689-f1aa6abe8b8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-821a13f6-dfd5b9795-2ffsx\" (UID: \"c5dd10cb-759b-48c0-a689-f1aa6abe8b8d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-821a13f6-dfd5b9795-2ffsx" Nov 03 13:15:52.536048 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:52.536017 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fzz9\" (UniqueName: \"kubernetes.io/projected/c5dd10cb-759b-48c0-a689-f1aa6abe8b8d-kube-api-access-9fzz9\") pod \"payload-pre-processing-e2e-ait-821a13f6-dfd5b9795-2ffsx\" (UID: \"c5dd10cb-759b-48c0-a689-f1aa6abe8b8d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-821a13f6-dfd5b9795-2ffsx" Nov 03 13:15:52.562710 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:52.562684 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-821a13f6-dfd5b9795-2ffsx" Nov 03 13:15:52.583230 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:52.583196 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd7cfbb5-315f-4b0c-bb3c-e2a8803ecf73-plugins-config-volume\") pod \"payload-processing-e2e-ait-821a13f6-649c78b7b7-8ht9h\" (UID: \"dd7cfbb5-315f-4b0c-bb3c-e2a8803ecf73\") " pod="openshift-ingress/payload-processing-e2e-ait-821a13f6-649c78b7b7-8ht9h" Nov 03 13:15:52.583364 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:52.583287 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7ss6\" (UniqueName: \"kubernetes.io/projected/dd7cfbb5-315f-4b0c-bb3c-e2a8803ecf73-kube-api-access-s7ss6\") pod \"payload-processing-e2e-ait-821a13f6-649c78b7b7-8ht9h\" (UID: \"dd7cfbb5-315f-4b0c-bb3c-e2a8803ecf73\") " pod="openshift-ingress/payload-processing-e2e-ait-821a13f6-649c78b7b7-8ht9h" Nov 03 13:15:52.583991 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:52.583973 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd7cfbb5-315f-4b0c-bb3c-e2a8803ecf73-plugins-config-volume\") pod \"payload-processing-e2e-ait-821a13f6-649c78b7b7-8ht9h\" (UID: \"dd7cfbb5-315f-4b0c-bb3c-e2a8803ecf73\") " pod="openshift-ingress/payload-processing-e2e-ait-821a13f6-649c78b7b7-8ht9h" Nov 03 13:15:52.640089 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:52.640060 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7ss6\" (UniqueName: \"kubernetes.io/projected/dd7cfbb5-315f-4b0c-bb3c-e2a8803ecf73-kube-api-access-s7ss6\") pod \"payload-processing-e2e-ait-821a13f6-649c78b7b7-8ht9h\" (UID: \"dd7cfbb5-315f-4b0c-bb3c-e2a8803ecf73\") " pod="openshift-ingress/payload-processing-e2e-ait-821a13f6-649c78b7b7-8ht9h" Nov 03 13:15:52.695881 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:52.695846 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-821a13f6-649c78b7b7-8ht9h" Nov 03 13:15:52.937100 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:52.937009 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-821a13f6-dfd5b9795-2ffsx"] Nov 03 13:15:52.944264 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:15:52.944222 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5dd10cb_759b_48c0_a689_f1aa6abe8b8d.slice/crio-e3897720e091bb7bd65267a6bff8850c879295b4e8d893b376a67f6a9ebce68d WatchSource:0}: Error finding container e3897720e091bb7bd65267a6bff8850c879295b4e8d893b376a67f6a9ebce68d: Status 404 returned error can't find the container with id e3897720e091bb7bd65267a6bff8850c879295b4e8d893b376a67f6a9ebce68d Nov 03 13:15:53.089184 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:53.089151 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-821a13f6-649c78b7b7-8ht9h"] Nov 03 13:15:53.097085 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:53.097048 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-821a13f6-dfd5b9795-2ffsx" event={"ID":"c5dd10cb-759b-48c0-a689-f1aa6abe8b8d","Type":"ContainerStarted","Data":"6d6b228720a783edfd2caf20844e04ea35c75b8c6a49d9a4022f20827a6ca235"} Nov 03 13:15:53.097261 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:53.097113 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-821a13f6-dfd5b9795-2ffsx" event={"ID":"c5dd10cb-759b-48c0-a689-f1aa6abe8b8d","Type":"ContainerStarted","Data":"e3897720e091bb7bd65267a6bff8850c879295b4e8d893b376a67f6a9ebce68d"} Nov 03 13:15:53.097261 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:53.097145 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-821a13f6-dfd5b9795-2ffsx" Nov 03 13:15:53.098050 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:15:53.098028 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd7cfbb5_315f_4b0c_bb3c_e2a8803ecf73.slice/crio-bcb31f0bdf0cf43eaaa94dc37edf0b061f3292428bb6d4530637c1b0773a1cf4 WatchSource:0}: Error finding container bcb31f0bdf0cf43eaaa94dc37edf0b061f3292428bb6d4530637c1b0773a1cf4: Status 404 returned error can't find the container with id bcb31f0bdf0cf43eaaa94dc37edf0b061f3292428bb6d4530637c1b0773a1cf4 Nov 03 13:15:53.201459 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:53.201361 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-821a13f6-dfd5b9795-2ffsx" podStartSLOduration=1.201347394 podStartE2EDuration="1.201347394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:15:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:15:53.200317816 +0000 UTC m=+635.211006059" watchObservedRunningTime="2026-11-03 13:15:53.201347394 +0000 UTC m=+635.212035635" Nov 03 13:15:54.103440 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:54.103402 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-821a13f6-649c78b7b7-8ht9h" event={"ID":"dd7cfbb5-315f-4b0c-bb3c-e2a8803ecf73","Type":"ContainerStarted","Data":"76474c18fc0254d7c87f8341bce6c9397dd46e29e61917fda7d3f2f78271ac36"} Nov 03 13:15:54.103926 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:54.103465 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-821a13f6-649c78b7b7-8ht9h" event={"ID":"dd7cfbb5-315f-4b0c-bb3c-e2a8803ecf73","Type":"ContainerStarted","Data":"bcb31f0bdf0cf43eaaa94dc37edf0b061f3292428bb6d4530637c1b0773a1cf4"} Nov 03 13:15:54.103926 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:54.103705 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-821a13f6-649c78b7b7-8ht9h" Nov 03 13:15:54.219842 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:54.219791 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-821a13f6-649c78b7b7-8ht9h" podStartSLOduration=2.219777325 podStartE2EDuration="2.219777325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:15:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:15:54.218486729 +0000 UTC m=+636.229174967" watchObservedRunningTime="2026-11-03 13:15:54.219777325 +0000 UTC m=+636.230465616" Nov 03 13:15:54.767423 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:54.767386 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-11ca03ad-6cb94c6c4c-4mf5l"] Nov 03 13:15:54.776358 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:54.776324 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-11ca03ad-6cb94c6c4c-4mf5l" Nov 03 13:15:54.825208 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:54.825172 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-11ca03ad-6cb94c6c4c-4mf5l"] Nov 03 13:15:54.872464 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:54.872425 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-11ca03ad-dockercfg-c9c6x\"" Nov 03 13:15:54.891705 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:54.891673 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-11ca03ad\"" Nov 03 13:15:54.911028 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:54.910993 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-676g5\" (UniqueName: \"kubernetes.io/projected/01e6f156-634a-4d24-9db0-c5f698c8deca-kube-api-access-676g5\") pod \"payload-pre-processing-e2e-ait-11ca03ad-6cb94c6c4c-4mf5l\" (UID: \"01e6f156-634a-4d24-9db0-c5f698c8deca\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-11ca03ad-6cb94c6c4c-4mf5l" Nov 03 13:15:54.911211 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:54.911101 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01e6f156-634a-4d24-9db0-c5f698c8deca-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-11ca03ad-6cb94c6c4c-4mf5l\" (UID: \"01e6f156-634a-4d24-9db0-c5f698c8deca\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-11ca03ad-6cb94c6c4c-4mf5l" Nov 03 13:15:55.011970 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:55.011936 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01e6f156-634a-4d24-9db0-c5f698c8deca-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-11ca03ad-6cb94c6c4c-4mf5l\" (UID: \"01e6f156-634a-4d24-9db0-c5f698c8deca\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-11ca03ad-6cb94c6c4c-4mf5l" Nov 03 13:15:55.012120 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:55.012066 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-676g5\" (UniqueName: \"kubernetes.io/projected/01e6f156-634a-4d24-9db0-c5f698c8deca-kube-api-access-676g5\") pod \"payload-pre-processing-e2e-ait-11ca03ad-6cb94c6c4c-4mf5l\" (UID: \"01e6f156-634a-4d24-9db0-c5f698c8deca\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-11ca03ad-6cb94c6c4c-4mf5l" Nov 03 13:15:55.012680 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:55.012660 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01e6f156-634a-4d24-9db0-c5f698c8deca-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-11ca03ad-6cb94c6c4c-4mf5l\" (UID: \"01e6f156-634a-4d24-9db0-c5f698c8deca\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-11ca03ad-6cb94c6c4c-4mf5l" Nov 03 13:15:55.065629 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:55.065548 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-676g5\" (UniqueName: \"kubernetes.io/projected/01e6f156-634a-4d24-9db0-c5f698c8deca-kube-api-access-676g5\") pod \"payload-pre-processing-e2e-ait-11ca03ad-6cb94c6c4c-4mf5l\" (UID: \"01e6f156-634a-4d24-9db0-c5f698c8deca\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-11ca03ad-6cb94c6c4c-4mf5l" Nov 03 13:15:55.091176 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:55.091140 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-11ca03ad-6cb94c6c4c-4mf5l" Nov 03 13:15:55.463661 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:55.463629 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-11ca03ad-6cb94c6c4c-4mf5l"] Nov 03 13:15:55.471218 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:15:55.471192 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01e6f156_634a_4d24_9db0_c5f698c8deca.slice/crio-587c6d7b4a8bb67194dfc56cb84fb6f229ff56e97f20bcd9c8d621a0e6b0280b WatchSource:0}: Error finding container 587c6d7b4a8bb67194dfc56cb84fb6f229ff56e97f20bcd9c8d621a0e6b0280b: Status 404 returned error can't find the container with id 587c6d7b4a8bb67194dfc56cb84fb6f229ff56e97f20bcd9c8d621a0e6b0280b Nov 03 13:15:56.115847 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:56.115807 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-11ca03ad-6cb94c6c4c-4mf5l" event={"ID":"01e6f156-634a-4d24-9db0-c5f698c8deca","Type":"ContainerStarted","Data":"295b3b803b0f5fa9b75257eff4ed7415f1f5d4903570220a47694717d6490fdb"} Nov 03 13:15:56.116038 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:56.115866 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-11ca03ad-6cb94c6c4c-4mf5l" event={"ID":"01e6f156-634a-4d24-9db0-c5f698c8deca","Type":"ContainerStarted","Data":"587c6d7b4a8bb67194dfc56cb84fb6f229ff56e97f20bcd9c8d621a0e6b0280b"} Nov 03 13:15:56.116038 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:56.115902 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-11ca03ad-6cb94c6c4c-4mf5l" Nov 03 13:15:56.230265 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:56.230205 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-11ca03ad-6cb94c6c4c-4mf5l" podStartSLOduration=2.23019029 podStartE2EDuration="2.23019029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:15:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:15:56.228448675 +0000 UTC m=+638.239136917" watchObservedRunningTime="2026-11-03 13:15:56.23019029 +0000 UTC m=+638.240878532" Nov 03 13:15:57.462748 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:57.462717 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cb1bd0b2-5c7c65f4d5-gv2mm"] Nov 03 13:15:57.466595 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:57.466574 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-cb1bd0b2-5c7c65f4d5-gv2mm" Nov 03 13:15:57.521827 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:57.521792 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cb1bd0b2-5c7c65f4d5-gv2mm"] Nov 03 13:15:57.541691 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:57.541653 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lfvc\" (UniqueName: \"kubernetes.io/projected/6d10d8b2-1143-48e1-9314-b471fa85370f-kube-api-access-7lfvc\") pod \"payload-processing-e2e-migrate-cb1bd0b2-5c7c65f4d5-gv2mm\" (UID: \"6d10d8b2-1143-48e1-9314-b471fa85370f\") " pod="openshift-ingress/payload-processing-e2e-migrate-cb1bd0b2-5c7c65f4d5-gv2mm" Nov 03 13:15:57.541887 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:57.541867 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d10d8b2-1143-48e1-9314-b471fa85370f-plugins-config-volume\") pod \"payload-processing-e2e-migrate-cb1bd0b2-5c7c65f4d5-gv2mm\" (UID: \"6d10d8b2-1143-48e1-9314-b471fa85370f\") " pod="openshift-ingress/payload-processing-e2e-migrate-cb1bd0b2-5c7c65f4d5-gv2mm" Nov 03 13:15:57.565261 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:57.565230 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-cb1bd0b2-dockercfg-srwg7\"" Nov 03 13:15:57.565261 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:57.565231 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-cb1bd0b2\"" Nov 03 13:15:57.643137 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:57.643104 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d10d8b2-1143-48e1-9314-b471fa85370f-plugins-config-volume\") pod \"payload-processing-e2e-migrate-cb1bd0b2-5c7c65f4d5-gv2mm\" (UID: \"6d10d8b2-1143-48e1-9314-b471fa85370f\") " pod="openshift-ingress/payload-processing-e2e-migrate-cb1bd0b2-5c7c65f4d5-gv2mm" Nov 03 13:15:57.643341 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:57.643169 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lfvc\" (UniqueName: \"kubernetes.io/projected/6d10d8b2-1143-48e1-9314-b471fa85370f-kube-api-access-7lfvc\") pod \"payload-processing-e2e-migrate-cb1bd0b2-5c7c65f4d5-gv2mm\" (UID: \"6d10d8b2-1143-48e1-9314-b471fa85370f\") " pod="openshift-ingress/payload-processing-e2e-migrate-cb1bd0b2-5c7c65f4d5-gv2mm" Nov 03 13:15:57.643897 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:57.643865 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d10d8b2-1143-48e1-9314-b471fa85370f-plugins-config-volume\") pod \"payload-processing-e2e-migrate-cb1bd0b2-5c7c65f4d5-gv2mm\" (UID: \"6d10d8b2-1143-48e1-9314-b471fa85370f\") " pod="openshift-ingress/payload-processing-e2e-migrate-cb1bd0b2-5c7c65f4d5-gv2mm" Nov 03 13:15:57.696675 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:57.696649 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lfvc\" (UniqueName: \"kubernetes.io/projected/6d10d8b2-1143-48e1-9314-b471fa85370f-kube-api-access-7lfvc\") pod \"payload-processing-e2e-migrate-cb1bd0b2-5c7c65f4d5-gv2mm\" (UID: \"6d10d8b2-1143-48e1-9314-b471fa85370f\") " pod="openshift-ingress/payload-processing-e2e-migrate-cb1bd0b2-5c7c65f4d5-gv2mm" Nov 03 13:15:57.780128 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:57.780047 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-cb1bd0b2-5c7c65f4d5-gv2mm" Nov 03 13:15:58.148012 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:58.147981 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cb1bd0b2-5c7c65f4d5-gv2mm"] Nov 03 13:15:58.155421 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:15:58.155389 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d10d8b2_1143_48e1_9314_b471fa85370f.slice/crio-193f0e0f26c60cbb6128116e070cc50b420e2993f47e6da89586bf35cbfc4d28 WatchSource:0}: Error finding container 193f0e0f26c60cbb6128116e070cc50b420e2993f47e6da89586bf35cbfc4d28: Status 404 returned error can't find the container with id 193f0e0f26c60cbb6128116e070cc50b420e2993f47e6da89586bf35cbfc4d28 Nov 03 13:15:59.040555 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:59.040452 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:15:59.132140 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:59.132102 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-cb1bd0b2-5c7c65f4d5-gv2mm" event={"ID":"6d10d8b2-1143-48e1-9314-b471fa85370f","Type":"ContainerStarted","Data":"cc4ff5b068567e110f5a23c1d6f4a4c567505d7e408c4aa066fd1e3a4ea1972a"} Nov 03 13:15:59.132314 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:59.132162 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-cb1bd0b2-5c7c65f4d5-gv2mm" Nov 03 13:15:59.132314 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:59.132178 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-cb1bd0b2-5c7c65f4d5-gv2mm" event={"ID":"6d10d8b2-1143-48e1-9314-b471fa85370f","Type":"ContainerStarted","Data":"193f0e0f26c60cbb6128116e070cc50b420e2993f47e6da89586bf35cbfc4d28"} Nov 03 13:15:59.239218 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:15:59.239156 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-cb1bd0b2-5c7c65f4d5-gv2mm" podStartSLOduration=2.239143251 podStartE2EDuration="2.239143251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:15:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:15:59.238396995 +0000 UTC m=+641.249085242" watchObservedRunningTime="2026-11-03 13:15:59.239143251 +0000 UTC m=+641.249831493" Nov 03 13:16:03.100714 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:03.100683 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-67349d1c-6cb59557f7-2j7fx" Nov 03 13:16:04.107126 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:04.107097 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-821a13f6-dfd5b9795-2ffsx" Nov 03 13:16:05.113115 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:05.113083 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-821a13f6-649c78b7b7-8ht9h" Nov 03 13:16:06.816463 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:06.816425 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a7a9081d-5d999b8bd7-drz5x"] Nov 03 13:16:06.820412 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:06.820390 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a7a9081d-5d999b8bd7-drz5x" Nov 03 13:16:06.876224 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:06.876192 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a7a9081d-5d999b8bd7-drz5x"] Nov 03 13:16:06.890322 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:06.890167 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ebdf83e-43fb-4a83-a9c8-ddadd6dfd982-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-a7a9081d-5d999b8bd7-drz5x\" (UID: \"4ebdf83e-43fb-4a83-a9c8-ddadd6dfd982\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a7a9081d-5d999b8bd7-drz5x" Nov 03 13:16:06.890322 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:06.890219 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5878\" (UniqueName: \"kubernetes.io/projected/4ebdf83e-43fb-4a83-a9c8-ddadd6dfd982-kube-api-access-q5878\") pod \"payload-pre-processing-e2e-derive-a7a9081d-5d999b8bd7-drz5x\" (UID: \"4ebdf83e-43fb-4a83-a9c8-ddadd6dfd982\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a7a9081d-5d999b8bd7-drz5x" Nov 03 13:16:06.919828 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:06.918864 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-a7a9081d\"" Nov 03 13:16:06.919828 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:06.919070 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-a7a9081d-dockercfg-lqf76\"" Nov 03 13:16:06.991404 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:06.991369 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ebdf83e-43fb-4a83-a9c8-ddadd6dfd982-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-a7a9081d-5d999b8bd7-drz5x\" (UID: \"4ebdf83e-43fb-4a83-a9c8-ddadd6dfd982\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a7a9081d-5d999b8bd7-drz5x" Nov 03 13:16:06.991615 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:06.991417 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5878\" (UniqueName: \"kubernetes.io/projected/4ebdf83e-43fb-4a83-a9c8-ddadd6dfd982-kube-api-access-q5878\") pod \"payload-pre-processing-e2e-derive-a7a9081d-5d999b8bd7-drz5x\" (UID: \"4ebdf83e-43fb-4a83-a9c8-ddadd6dfd982\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a7a9081d-5d999b8bd7-drz5x" Nov 03 13:16:06.992203 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:06.992176 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ebdf83e-43fb-4a83-a9c8-ddadd6dfd982-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-a7a9081d-5d999b8bd7-drz5x\" (UID: \"4ebdf83e-43fb-4a83-a9c8-ddadd6dfd982\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a7a9081d-5d999b8bd7-drz5x" Nov 03 13:16:07.046281 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:07.046247 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5878\" (UniqueName: \"kubernetes.io/projected/4ebdf83e-43fb-4a83-a9c8-ddadd6dfd982-kube-api-access-q5878\") pod \"payload-pre-processing-e2e-derive-a7a9081d-5d999b8bd7-drz5x\" (UID: \"4ebdf83e-43fb-4a83-a9c8-ddadd6dfd982\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a7a9081d-5d999b8bd7-drz5x" Nov 03 13:16:07.124321 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:07.124240 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-11ca03ad-6cb94c6c4c-4mf5l" Nov 03 13:16:07.133887 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:07.133855 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a7a9081d-5d999b8bd7-drz5x" Nov 03 13:16:07.288906 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:16:07.288816 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:16:07.370858 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:07.368821 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:16:07.537006 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:07.532059 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a7a9081d-5d999b8bd7-drz5x"] Nov 03 13:16:08.222952 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:08.222913 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a7a9081d-5d999b8bd7-drz5x" event={"ID":"4ebdf83e-43fb-4a83-a9c8-ddadd6dfd982","Type":"ContainerStarted","Data":"77e0fe6595f0af39fae24df8f57cfba25ca44fd25de99a598ea80da590cee819"} Nov 03 13:16:08.223369 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:08.222972 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a7a9081d-5d999b8bd7-drz5x" event={"ID":"4ebdf83e-43fb-4a83-a9c8-ddadd6dfd982","Type":"ContainerStarted","Data":"4deace760efcbeef3b4df02786c58d6f7e138f71e250ebf8e1f11b61b075e2cd"} Nov 03 13:16:08.223369 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:08.222995 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-a7a9081d-5d999b8bd7-drz5x" Nov 03 13:16:08.330882 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:08.330817 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-a7a9081d-5d999b8bd7-drz5x" podStartSLOduration=2.33079886 podStartE2EDuration="2.33079886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:16:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:16:08.329875472 +0000 UTC m=+650.340563717" watchObservedRunningTime="2026-11-03 13:16:08.33079886 +0000 UTC m=+650.341487103" Nov 03 13:16:09.229033 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:09.229002 5979 generic.go:356] "Generic (PLEG): container finished" podID="435a8d1c-24ab-4dce-a4e0-10aa8cff506f" containerID="1d1af1b397212f6fcfce65e98bf87f16dfaccb80dd1b021a7cb2c315eb3e091b" exitCode=0 Nov 03 13:16:09.229462 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:09.229079 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p" event={"ID":"435a8d1c-24ab-4dce-a4e0-10aa8cff506f","Type":"ContainerDied","Data":"1d1af1b397212f6fcfce65e98bf87f16dfaccb80dd1b021a7cb2c315eb3e091b"} Nov 03 13:16:09.229542 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:09.229525 5979 scope.go:122] "RemoveContainer" containerID="1d1af1b397212f6fcfce65e98bf87f16dfaccb80dd1b021a7cb2c315eb3e091b" Nov 03 13:16:09.230724 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:09.230689 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:16:09.230820 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:09.230756 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:16:09.230820 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:09.230786 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:16:10.140184 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:10.140157 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-cb1bd0b2-5c7c65f4d5-gv2mm" Nov 03 13:16:10.236834 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:10.236797 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p" event={"ID":"435a8d1c-24ab-4dce-a4e0-10aa8cff506f","Type":"ContainerStarted","Data":"bea00b9e7ba163e7ca18cf2c7e67e2f70f7fbe15247477af7ae980bf7124c2ea"} Nov 03 13:16:10.968497 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:10.967530 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p" Nov 03 13:16:10.968687 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:10.968603 5979 patch_prober.go:29] interesting pod/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 03 13:16:10.968687 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:10.968659 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p" podUID="435a8d1c-24ab-4dce-a4e0-10aa8cff506f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 03 13:16:11.967502 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:11.966566 5979 patch_prober.go:29] interesting pod/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 03 13:16:11.967502 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:11.966637 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p" podUID="435a8d1c-24ab-4dce-a4e0-10aa8cff506f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 03 13:16:12.966494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:12.965876 5979 patch_prober.go:29] interesting pod/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 03 13:16:12.966494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:12.965951 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p" podUID="435a8d1c-24ab-4dce-a4e0-10aa8cff506f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 03 13:16:13.038834 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:13.038800 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p" Nov 03 13:16:13.971495 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:13.970869 5979 patch_prober.go:29] interesting pod/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 03 13:16:13.971495 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:13.970934 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p" podUID="435a8d1c-24ab-4dce-a4e0-10aa8cff506f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 03 13:16:14.968887 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:14.968847 5979 patch_prober.go:29] interesting pod/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Nov 03 13:16:14.969278 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:14.968906 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p" podUID="435a8d1c-24ab-4dce-a4e0-10aa8cff506f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Nov 03 13:16:15.973505 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:15.973454 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p" Nov 03 13:16:15.974448 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:15.974430 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p" Nov 03 13:16:19.232205 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:19.232178 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-a7a9081d-5d999b8bd7-drz5x" Nov 03 13:16:23.530118 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:23.530084 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:16:26.176566 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:26.176536 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-821a13f6-z7rv9"] Nov 03 13:16:26.182027 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:26.182006 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-821a13f6-z7rv9" Nov 03 13:16:26.222058 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:26.222029 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a1cb675-1e8b-4943-8367-563d60d8b054-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-821a13f6-z7rv9\" (UID: \"0a1cb675-1e8b-4943-8367-563d60d8b054\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-821a13f6-z7rv9" Nov 03 13:16:26.222213 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:26.222064 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5ntg\" (UniqueName: \"kubernetes.io/projected/0a1cb675-1e8b-4943-8367-563d60d8b054-kube-api-access-m5ntg\") pod \"maas-api-revoke-keys-e2e-ait-821a13f6-z7rv9\" (UID: \"0a1cb675-1e8b-4943-8367-563d60d8b054\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-821a13f6-z7rv9" Nov 03 13:16:26.222213 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:26.222095 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a1cb675-1e8b-4943-8367-563d60d8b054-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-821a13f6-z7rv9\" (UID: \"0a1cb675-1e8b-4943-8367-563d60d8b054\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-821a13f6-z7rv9" Nov 03 13:16:26.231056 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:26.231030 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:16:26.231199 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:26.231094 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:16:26.279374 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:26.279344 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-821a13f6-z7rv9"] Nov 03 13:16:26.316431 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:26.316402 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 13:16:26.316632 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:26.316449 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 13:16:26.322895 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:26.322870 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a1cb675-1e8b-4943-8367-563d60d8b054-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-821a13f6-z7rv9\" (UID: \"0a1cb675-1e8b-4943-8367-563d60d8b054\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-821a13f6-z7rv9" Nov 03 13:16:26.322962 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:26.322904 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5ntg\" (UniqueName: \"kubernetes.io/projected/0a1cb675-1e8b-4943-8367-563d60d8b054-kube-api-access-m5ntg\") pod \"maas-api-revoke-keys-e2e-ait-821a13f6-z7rv9\" (UID: \"0a1cb675-1e8b-4943-8367-563d60d8b054\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-821a13f6-z7rv9" Nov 03 13:16:26.322962 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:26.322938 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a1cb675-1e8b-4943-8367-563d60d8b054-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-821a13f6-z7rv9\" (UID: \"0a1cb675-1e8b-4943-8367-563d60d8b054\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-821a13f6-z7rv9" Nov 03 13:16:26.323632 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:26.323609 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a1cb675-1e8b-4943-8367-563d60d8b054-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-821a13f6-z7rv9\" (UID: \"0a1cb675-1e8b-4943-8367-563d60d8b054\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-821a13f6-z7rv9" Nov 03 13:16:26.325238 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:26.325220 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a1cb675-1e8b-4943-8367-563d60d8b054-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-821a13f6-z7rv9\" (UID: \"0a1cb675-1e8b-4943-8367-563d60d8b054\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-821a13f6-z7rv9" Nov 03 13:16:26.376797 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:26.376767 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5ntg\" (UniqueName: \"kubernetes.io/projected/0a1cb675-1e8b-4943-8367-563d60d8b054-kube-api-access-m5ntg\") pod \"maas-api-revoke-keys-e2e-ait-821a13f6-z7rv9\" (UID: \"0a1cb675-1e8b-4943-8367-563d60d8b054\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-821a13f6-z7rv9" Nov 03 13:16:26.495003 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:26.494912 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-821a13f6-z7rv9" Nov 03 13:16:26.865055 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:26.864975 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-821a13f6-z7rv9"] Nov 03 13:16:26.872521 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:16:26.872494 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a1cb675_1e8b_4943_8367_563d60d8b054.slice/crio-c18137a0d98e53b02c70d02b8fa654798a7b79f6f6b86194245ede7104983145 WatchSource:0}: Error finding container c18137a0d98e53b02c70d02b8fa654798a7b79f6f6b86194245ede7104983145: Status 404 returned error can't find the container with id c18137a0d98e53b02c70d02b8fa654798a7b79f6f6b86194245ede7104983145 Nov 03 13:16:27.353936 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:27.353898 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-821a13f6-z7rv9" event={"ID":"0a1cb675-1e8b-4943-8367-563d60d8b054","Type":"ContainerStarted","Data":"37729346ca6aa0a0934ea94b239a713906fad7b5986bd47f7e1954b06c86d0d9"} Nov 03 13:16:27.354408 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:27.353966 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-821a13f6-z7rv9" event={"ID":"0a1cb675-1e8b-4943-8367-563d60d8b054","Type":"ContainerStarted","Data":"c18137a0d98e53b02c70d02b8fa654798a7b79f6f6b86194245ede7104983145"} Nov 03 13:16:27.460528 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:27.460401 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-821a13f6-z7rv9" podStartSLOduration=1.460384283 podStartE2EDuration="1.460384283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:16:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:16:27.458967931 +0000 UTC m=+669.469656173" watchObservedRunningTime="2026-11-03 13:16:27.460384283 +0000 UTC m=+669.471072524" Nov 03 13:16:28.360048 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:28.360011 5979 generic.go:356] "Generic (PLEG): container finished" podID="0a1cb675-1e8b-4943-8367-563d60d8b054" containerID="37729346ca6aa0a0934ea94b239a713906fad7b5986bd47f7e1954b06c86d0d9" exitCode=0 Nov 03 13:16:28.360442 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:28.360079 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-821a13f6-z7rv9" event={"ID":"0a1cb675-1e8b-4943-8367-563d60d8b054","Type":"ContainerDied","Data":"37729346ca6aa0a0934ea94b239a713906fad7b5986bd47f7e1954b06c86d0d9"} Nov 03 13:16:29.500214 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:29.500187 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-821a13f6-z7rv9" Nov 03 13:16:29.560754 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:29.560722 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a1cb675-1e8b-4943-8367-563d60d8b054-kube-api-access-m5ntg\" (UniqueName: \"kubernetes.io/projected/0a1cb675-1e8b-4943-8367-563d60d8b054-kube-api-access-m5ntg\") pod \"0a1cb675-1e8b-4943-8367-563d60d8b054\" (UID: \"0a1cb675-1e8b-4943-8367-563d60d8b054\") " Nov 03 13:16:29.560932 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:29.560797 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a1cb675-1e8b-4943-8367-563d60d8b054-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a1cb675-1e8b-4943-8367-563d60d8b054-maas-api-xks-ca\") pod \"0a1cb675-1e8b-4943-8367-563d60d8b054\" (UID: \"0a1cb675-1e8b-4943-8367-563d60d8b054\") " Nov 03 13:16:29.560932 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:29.560854 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a1cb675-1e8b-4943-8367-563d60d8b054-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a1cb675-1e8b-4943-8367-563d60d8b054-maas-api-openshift-ca\") pod \"0a1cb675-1e8b-4943-8367-563d60d8b054\" (UID: \"0a1cb675-1e8b-4943-8367-563d60d8b054\") " Nov 03 13:16:29.561276 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:29.561231 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a1cb675-1e8b-4943-8367-563d60d8b054-maas-api-openshift-ca" pod "0a1cb675-1e8b-4943-8367-563d60d8b054" (UID: "0a1cb675-1e8b-4943-8367-563d60d8b054"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:16:29.562848 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:29.562827 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a1cb675-1e8b-4943-8367-563d60d8b054-maas-api-xks-ca" pod "0a1cb675-1e8b-4943-8367-563d60d8b054" (UID: "0a1cb675-1e8b-4943-8367-563d60d8b054"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:16:29.562914 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:29.562897 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a1cb675-1e8b-4943-8367-563d60d8b054-kube-api-access-m5ntg" pod "0a1cb675-1e8b-4943-8367-563d60d8b054" (UID: "0a1cb675-1e8b-4943-8367-563d60d8b054"). InnerVolumeSpecName "kube-api-access-m5ntg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:16:29.662896 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:29.662857 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5ntg\" (UniqueName: \"kubernetes.io/projected/0a1cb675-1e8b-4943-8367-563d60d8b054-kube-api-access-m5ntg\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:16:29.662896 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:29.662885 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a1cb675-1e8b-4943-8367-563d60d8b054-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:16:29.662896 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:29.662901 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a1cb675-1e8b-4943-8367-563d60d8b054-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:16:30.372346 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:30.372260 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-821a13f6-z7rv9" event={"ID":"0a1cb675-1e8b-4943-8367-563d60d8b054","Type":"ContainerDied","Data":"c18137a0d98e53b02c70d02b8fa654798a7b79f6f6b86194245ede7104983145"} Nov 03 13:16:30.372346 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:30.372293 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-821a13f6-z7rv9" Nov 03 13:16:30.372346 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:30.372333 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c18137a0d98e53b02c70d02b8fa654798a7b79f6f6b86194245ede7104983145" Nov 03 13:16:36.884189 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:36.884150 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67349d1c-x46gc"] Nov 03 13:16:36.884656 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:36.884641 5979 cpu_manager.go:414] "containerMap: removing container" podUID="0a1cb675-1e8b-4943-8367-563d60d8b054" containerName="revoke-keys" Nov 03 13:16:36.884717 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:36.884656 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a1cb675-1e8b-4943-8367-563d60d8b054" containerName="revoke-keys" Nov 03 13:16:36.884766 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:36.884759 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a1cb675-1e8b-4943-8367-563d60d8b054" containerName="revoke-keys" Nov 03 13:16:36.887705 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:36.887684 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67349d1c-x46gc" Nov 03 13:16:36.939857 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:36.939815 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdcf19ec-e5e2-416a-8d66-1121e78847be-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-67349d1c-x46gc\" (UID: \"bdcf19ec-e5e2-416a-8d66-1121e78847be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67349d1c-x46gc" Nov 03 13:16:36.940037 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:36.939970 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdcf19ec-e5e2-416a-8d66-1121e78847be-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-67349d1c-x46gc\" (UID: \"bdcf19ec-e5e2-416a-8d66-1121e78847be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67349d1c-x46gc" Nov 03 13:16:36.940037 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:36.940007 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qwbp\" (UniqueName: \"kubernetes.io/projected/bdcf19ec-e5e2-416a-8d66-1121e78847be-kube-api-access-9qwbp\") pod \"maas-api-revoke-keys-e2e-ait-67349d1c-x46gc\" (UID: \"bdcf19ec-e5e2-416a-8d66-1121e78847be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67349d1c-x46gc" Nov 03 13:16:36.940772 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:36.940749 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67349d1c-x46gc"] Nov 03 13:16:36.984591 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:36.984552 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 13:16:36.984591 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:36.984552 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 13:16:36.984591 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:36.984552 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:16:36.984842 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:36.984611 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:16:37.041655 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:37.041625 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdcf19ec-e5e2-416a-8d66-1121e78847be-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-67349d1c-x46gc\" (UID: \"bdcf19ec-e5e2-416a-8d66-1121e78847be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67349d1c-x46gc" Nov 03 13:16:37.041655 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:37.041660 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qwbp\" (UniqueName: \"kubernetes.io/projected/bdcf19ec-e5e2-416a-8d66-1121e78847be-kube-api-access-9qwbp\") pod \"maas-api-revoke-keys-e2e-ait-67349d1c-x46gc\" (UID: \"bdcf19ec-e5e2-416a-8d66-1121e78847be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67349d1c-x46gc" Nov 03 13:16:37.041851 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:37.041726 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdcf19ec-e5e2-416a-8d66-1121e78847be-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-67349d1c-x46gc\" (UID: \"bdcf19ec-e5e2-416a-8d66-1121e78847be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67349d1c-x46gc" Nov 03 13:16:37.042532 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:37.042508 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdcf19ec-e5e2-416a-8d66-1121e78847be-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-67349d1c-x46gc\" (UID: \"bdcf19ec-e5e2-416a-8d66-1121e78847be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67349d1c-x46gc" Nov 03 13:16:37.043916 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:37.043890 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdcf19ec-e5e2-416a-8d66-1121e78847be-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-67349d1c-x46gc\" (UID: \"bdcf19ec-e5e2-416a-8d66-1121e78847be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67349d1c-x46gc" Nov 03 13:16:37.095067 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:37.095040 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qwbp\" (UniqueName: \"kubernetes.io/projected/bdcf19ec-e5e2-416a-8d66-1121e78847be-kube-api-access-9qwbp\") pod \"maas-api-revoke-keys-e2e-ait-67349d1c-x46gc\" (UID: \"bdcf19ec-e5e2-416a-8d66-1121e78847be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67349d1c-x46gc" Nov 03 13:16:37.203195 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:37.203103 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67349d1c-x46gc" Nov 03 13:16:37.208554 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:37.208531 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-821a13f6-649c78b7b7-8ht9h"] Nov 03 13:16:37.236652 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:16:37.236615 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:16:37.323881 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:37.323844 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-821a13f6-dfd5b9795-2ffsx"] Nov 03 13:16:37.578761 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:37.578669 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67349d1c-x46gc"] Nov 03 13:16:37.586107 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:16:37.586083 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdcf19ec_e5e2_416a_8d66_1121e78847be.slice/crio-49e3b0af4ff68d564023d1644631bdef6b7c4a2085b3479a19f4c284ff486895 WatchSource:0}: Error finding container 49e3b0af4ff68d564023d1644631bdef6b7c4a2085b3479a19f4c284ff486895: Status 404 returned error can't find the container with id 49e3b0af4ff68d564023d1644631bdef6b7c4a2085b3479a19f4c284ff486895 Nov 03 13:16:38.414437 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:38.414390 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67349d1c-x46gc" event={"ID":"bdcf19ec-e5e2-416a-8d66-1121e78847be","Type":"ContainerStarted","Data":"48b415cb9202ec7e64f3da0c0404d0d311035fac5f6b1fd7fa60a38d77baadcc"} Nov 03 13:16:38.414846 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:38.414461 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67349d1c-x46gc" event={"ID":"bdcf19ec-e5e2-416a-8d66-1121e78847be","Type":"ContainerStarted","Data":"49e3b0af4ff68d564023d1644631bdef6b7c4a2085b3479a19f4c284ff486895"} Nov 03 13:16:38.521968 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:38.521900 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67349d1c-x46gc" podStartSLOduration=2.521880452 podStartE2EDuration="2.521880452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:16:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:16:38.520739286 +0000 UTC m=+680.531427530" watchObservedRunningTime="2026-11-03 13:16:38.521880452 +0000 UTC m=+680.532568693" Nov 03 13:16:38.573034 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:38.570211 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:16:39.421204 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:39.421170 5979 generic.go:356] "Generic (PLEG): container finished" podID="bdcf19ec-e5e2-416a-8d66-1121e78847be" containerID="48b415cb9202ec7e64f3da0c0404d0d311035fac5f6b1fd7fa60a38d77baadcc" exitCode=0 Nov 03 13:16:39.421641 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:39.421264 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67349d1c-x46gc" event={"ID":"bdcf19ec-e5e2-416a-8d66-1121e78847be","Type":"ContainerDied","Data":"48b415cb9202ec7e64f3da0c0404d0d311035fac5f6b1fd7fa60a38d77baadcc"} Nov 03 13:16:40.428484 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:40.428434 5979 generic.go:356] "Generic (PLEG): container finished" podID="ad693484-f565-418b-acbc-dd3739a1adec" containerID="691aca82b8868a5d414a9d5f626f259f2e3ac04c6529c65cb66fba10f959ecbb" exitCode=0 Nov 03 13:16:40.429005 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:40.428505 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb" event={"ID":"ad693484-f565-418b-acbc-dd3739a1adec","Type":"ContainerDied","Data":"691aca82b8868a5d414a9d5f626f259f2e3ac04c6529c65cb66fba10f959ecbb"} Nov 03 13:16:40.429005 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:40.428969 5979 scope.go:122] "RemoveContainer" containerID="691aca82b8868a5d414a9d5f626f259f2e3ac04c6529c65cb66fba10f959ecbb" Nov 03 13:16:40.430291 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:40.430262 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:16:40.430374 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:40.430327 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:16:40.430374 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:40.430356 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:16:40.581207 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:40.581177 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67349d1c-x46gc" Nov 03 13:16:40.708723 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:40.708677 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bdcf19ec-e5e2-416a-8d66-1121e78847be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdcf19ec-e5e2-416a-8d66-1121e78847be-maas-api-xks-ca\") pod \"bdcf19ec-e5e2-416a-8d66-1121e78847be\" (UID: \"bdcf19ec-e5e2-416a-8d66-1121e78847be\") " Nov 03 13:16:40.708907 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:40.708865 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdcf19ec-e5e2-416a-8d66-1121e78847be-kube-api-access-9qwbp\" (UniqueName: \"kubernetes.io/projected/bdcf19ec-e5e2-416a-8d66-1121e78847be-kube-api-access-9qwbp\") pod \"bdcf19ec-e5e2-416a-8d66-1121e78847be\" (UID: \"bdcf19ec-e5e2-416a-8d66-1121e78847be\") " Nov 03 13:16:40.708979 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:40.708951 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdcf19ec-e5e2-416a-8d66-1121e78847be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdcf19ec-e5e2-416a-8d66-1121e78847be-maas-api-openshift-ca\") pod \"bdcf19ec-e5e2-416a-8d66-1121e78847be\" (UID: \"bdcf19ec-e5e2-416a-8d66-1121e78847be\") " Nov 03 13:16:40.709695 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:40.709659 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdcf19ec-e5e2-416a-8d66-1121e78847be-maas-api-openshift-ca" pod "bdcf19ec-e5e2-416a-8d66-1121e78847be" (UID: "bdcf19ec-e5e2-416a-8d66-1121e78847be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:16:40.711119 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:40.711091 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdcf19ec-e5e2-416a-8d66-1121e78847be-maas-api-xks-ca" pod "bdcf19ec-e5e2-416a-8d66-1121e78847be" (UID: "bdcf19ec-e5e2-416a-8d66-1121e78847be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:16:40.711444 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:40.711420 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdcf19ec-e5e2-416a-8d66-1121e78847be-kube-api-access-9qwbp" pod "bdcf19ec-e5e2-416a-8d66-1121e78847be" (UID: "bdcf19ec-e5e2-416a-8d66-1121e78847be"). InnerVolumeSpecName "kube-api-access-9qwbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:16:40.810554 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:40.810455 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qwbp\" (UniqueName: \"kubernetes.io/projected/bdcf19ec-e5e2-416a-8d66-1121e78847be-kube-api-access-9qwbp\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:16:40.810554 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:40.810503 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdcf19ec-e5e2-416a-8d66-1121e78847be-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:16:40.810554 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:40.810515 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdcf19ec-e5e2-416a-8d66-1121e78847be-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:16:41.394141 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:41.394108 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cb1bd0b2-vsct2"] Nov 03 13:16:41.394719 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:41.394697 5979 cpu_manager.go:414] "containerMap: removing container" podUID="bdcf19ec-e5e2-416a-8d66-1121e78847be" containerName="revoke-keys" Nov 03 13:16:41.394840 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:41.394719 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdcf19ec-e5e2-416a-8d66-1121e78847be" containerName="revoke-keys" Nov 03 13:16:41.394894 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:41.394887 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdcf19ec-e5e2-416a-8d66-1121e78847be" containerName="revoke-keys" Nov 03 13:16:41.397590 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:41.397569 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cb1bd0b2-vsct2" Nov 03 13:16:41.435913 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:41.435879 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67349d1c-x46gc" Nov 03 13:16:41.436316 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:41.435928 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67349d1c-x46gc" event={"ID":"bdcf19ec-e5e2-416a-8d66-1121e78847be","Type":"ContainerDied","Data":"49e3b0af4ff68d564023d1644631bdef6b7c4a2085b3479a19f4c284ff486895"} Nov 03 13:16:41.436316 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:41.435984 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49e3b0af4ff68d564023d1644631bdef6b7c4a2085b3479a19f4c284ff486895" Nov 03 13:16:41.438361 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:41.438341 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb" event={"ID":"ad693484-f565-418b-acbc-dd3739a1adec","Type":"ContainerStarted","Data":"87c926e28778597ea4da599eb56fe1c1b283acac8f63325aed312e2811580a09"} Nov 03 13:16:41.460720 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:41.450800 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb" Nov 03 13:16:41.464492 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:41.463650 5979 patch_prober.go:29] interesting pod/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 03 13:16:41.464492 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:41.463727 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb" podUID="ad693484-f565-418b-acbc-dd3739a1adec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 03 13:16:41.475382 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:41.475351 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cb1bd0b2-vsct2"] Nov 03 13:16:41.518406 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:41.518369 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x57vf\" (UniqueName: \"kubernetes.io/projected/a8a8d0bc-05fb-4a18-80af-874dd10e7f21-kube-api-access-x57vf\") pod \"maas-api-revoke-keys-e2e-migrate-cb1bd0b2-vsct2\" (UID: \"a8a8d0bc-05fb-4a18-80af-874dd10e7f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cb1bd0b2-vsct2" Nov 03 13:16:41.518586 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:41.518420 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8a8d0bc-05fb-4a18-80af-874dd10e7f21-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cb1bd0b2-vsct2\" (UID: \"a8a8d0bc-05fb-4a18-80af-874dd10e7f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cb1bd0b2-vsct2" Nov 03 13:16:41.518586 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:41.518568 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8a8d0bc-05fb-4a18-80af-874dd10e7f21-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cb1bd0b2-vsct2\" (UID: \"a8a8d0bc-05fb-4a18-80af-874dd10e7f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cb1bd0b2-vsct2" Nov 03 13:16:41.619159 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:41.619119 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x57vf\" (UniqueName: \"kubernetes.io/projected/a8a8d0bc-05fb-4a18-80af-874dd10e7f21-kube-api-access-x57vf\") pod \"maas-api-revoke-keys-e2e-migrate-cb1bd0b2-vsct2\" (UID: \"a8a8d0bc-05fb-4a18-80af-874dd10e7f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cb1bd0b2-vsct2" Nov 03 13:16:41.619361 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:41.619180 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8a8d0bc-05fb-4a18-80af-874dd10e7f21-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cb1bd0b2-vsct2\" (UID: \"a8a8d0bc-05fb-4a18-80af-874dd10e7f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cb1bd0b2-vsct2" Nov 03 13:16:41.619361 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:41.619238 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8a8d0bc-05fb-4a18-80af-874dd10e7f21-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cb1bd0b2-vsct2\" (UID: \"a8a8d0bc-05fb-4a18-80af-874dd10e7f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cb1bd0b2-vsct2" Nov 03 13:16:41.620005 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:41.619984 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8a8d0bc-05fb-4a18-80af-874dd10e7f21-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cb1bd0b2-vsct2\" (UID: \"a8a8d0bc-05fb-4a18-80af-874dd10e7f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cb1bd0b2-vsct2" Nov 03 13:16:41.621853 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:41.621834 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8a8d0bc-05fb-4a18-80af-874dd10e7f21-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-cb1bd0b2-vsct2\" (UID: \"a8a8d0bc-05fb-4a18-80af-874dd10e7f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cb1bd0b2-vsct2" Nov 03 13:16:41.672739 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:41.672659 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x57vf\" (UniqueName: \"kubernetes.io/projected/a8a8d0bc-05fb-4a18-80af-874dd10e7f21-kube-api-access-x57vf\") pod \"maas-api-revoke-keys-e2e-migrate-cb1bd0b2-vsct2\" (UID: \"a8a8d0bc-05fb-4a18-80af-874dd10e7f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cb1bd0b2-vsct2" Nov 03 13:16:41.710170 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:41.710138 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cb1bd0b2-vsct2" Nov 03 13:16:42.099409 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:42.099317 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cb1bd0b2-vsct2"] Nov 03 13:16:42.106633 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:16:42.106606 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8a8d0bc_05fb_4a18_80af_874dd10e7f21.slice/crio-56471892a6f6e055791d3d8a98c3378b2fc1db9ce38424a1169bea3992b72991 WatchSource:0}: Error finding container 56471892a6f6e055791d3d8a98c3378b2fc1db9ce38424a1169bea3992b72991: Status 404 returned error can't find the container with id 56471892a6f6e055791d3d8a98c3378b2fc1db9ce38424a1169bea3992b72991 Nov 03 13:16:42.209814 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:42.209743 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-821a13f6-649c78b7b7-8ht9h" podUID="dd7cfbb5-315f-4b0c-bb3c-e2a8803ecf73" containerName="payload-processing" containerID="cri-o://76474c18fc0254d7c87f8341bce6c9397dd46e29e61917fda7d3f2f78271ac36" gracePeriod=25 Nov 03 13:16:42.325334 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:42.325262 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-821a13f6-dfd5b9795-2ffsx" podUID="c5dd10cb-759b-48c0-a689-f1aa6abe8b8d" containerName="payload-pre-processing" containerID="cri-o://6d6b228720a783edfd2caf20844e04ea35c75b8c6a49d9a4022f20827a6ca235" gracePeriod=25 Nov 03 13:16:42.466517 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:42.451861 5979 patch_prober.go:29] interesting pod/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 03 13:16:42.476533 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:42.474834 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb" podUID="ad693484-f565-418b-acbc-dd3739a1adec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 03 13:16:42.480080 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:42.479868 5979 generic.go:356] "Generic (PLEG): container finished" podID="dd7cfbb5-315f-4b0c-bb3c-e2a8803ecf73" containerID="76474c18fc0254d7c87f8341bce6c9397dd46e29e61917fda7d3f2f78271ac36" exitCode=0 Nov 03 13:16:42.480080 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:42.479950 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-821a13f6-649c78b7b7-8ht9h" event={"ID":"dd7cfbb5-315f-4b0c-bb3c-e2a8803ecf73","Type":"ContainerDied","Data":"76474c18fc0254d7c87f8341bce6c9397dd46e29e61917fda7d3f2f78271ac36"} Nov 03 13:16:42.481916 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:42.481878 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cb1bd0b2-vsct2" event={"ID":"a8a8d0bc-05fb-4a18-80af-874dd10e7f21","Type":"ContainerStarted","Data":"56471892a6f6e055791d3d8a98c3378b2fc1db9ce38424a1169bea3992b72991"} Nov 03 13:16:42.483805 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:42.483768 5979 generic.go:356] "Generic (PLEG): container finished" podID="c5dd10cb-759b-48c0-a689-f1aa6abe8b8d" containerID="6d6b228720a783edfd2caf20844e04ea35c75b8c6a49d9a4022f20827a6ca235" exitCode=0 Nov 03 13:16:42.484434 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:42.484412 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-821a13f6-dfd5b9795-2ffsx" event={"ID":"c5dd10cb-759b-48c0-a689-f1aa6abe8b8d","Type":"ContainerDied","Data":"6d6b228720a783edfd2caf20844e04ea35c75b8c6a49d9a4022f20827a6ca235"} Nov 03 13:16:42.514751 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:42.514722 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-821a13f6-649c78b7b7-8ht9h" Nov 03 13:16:42.631600 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:42.631574 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd7cfbb5-315f-4b0c-bb3c-e2a8803ecf73-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd7cfbb5-315f-4b0c-bb3c-e2a8803ecf73-plugins-config-volume\") pod \"dd7cfbb5-315f-4b0c-bb3c-e2a8803ecf73\" (UID: \"dd7cfbb5-315f-4b0c-bb3c-e2a8803ecf73\") " Nov 03 13:16:42.631759 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:42.631626 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd7cfbb5-315f-4b0c-bb3c-e2a8803ecf73-kube-api-access-s7ss6\" (UniqueName: \"kubernetes.io/projected/dd7cfbb5-315f-4b0c-bb3c-e2a8803ecf73-kube-api-access-s7ss6\") pod \"dd7cfbb5-315f-4b0c-bb3c-e2a8803ecf73\" (UID: \"dd7cfbb5-315f-4b0c-bb3c-e2a8803ecf73\") " Nov 03 13:16:42.632558 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:42.632520 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd7cfbb5-315f-4b0c-bb3c-e2a8803ecf73-plugins-config-volume" pod "dd7cfbb5-315f-4b0c-bb3c-e2a8803ecf73" (UID: "dd7cfbb5-315f-4b0c-bb3c-e2a8803ecf73"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:16:42.635038 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:42.635007 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd7cfbb5-315f-4b0c-bb3c-e2a8803ecf73-kube-api-access-s7ss6" pod "dd7cfbb5-315f-4b0c-bb3c-e2a8803ecf73" (UID: "dd7cfbb5-315f-4b0c-bb3c-e2a8803ecf73"). InnerVolumeSpecName "kube-api-access-s7ss6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:16:42.638612 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:42.638593 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-821a13f6-dfd5b9795-2ffsx" Nov 03 13:16:42.732960 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:42.732880 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5dd10cb-759b-48c0-a689-f1aa6abe8b8d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5dd10cb-759b-48c0-a689-f1aa6abe8b8d-plugins-config-volume\") pod \"c5dd10cb-759b-48c0-a689-f1aa6abe8b8d\" (UID: \"c5dd10cb-759b-48c0-a689-f1aa6abe8b8d\") " Nov 03 13:16:42.733109 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:42.733016 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5dd10cb-759b-48c0-a689-f1aa6abe8b8d-kube-api-access-9fzz9\" (UniqueName: \"kubernetes.io/projected/c5dd10cb-759b-48c0-a689-f1aa6abe8b8d-kube-api-access-9fzz9\") pod \"c5dd10cb-759b-48c0-a689-f1aa6abe8b8d\" (UID: \"c5dd10cb-759b-48c0-a689-f1aa6abe8b8d\") " Nov 03 13:16:42.733246 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:42.733233 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd7cfbb5-315f-4b0c-bb3c-e2a8803ecf73-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:16:42.733306 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:42.733247 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7ss6\" (UniqueName: \"kubernetes.io/projected/dd7cfbb5-315f-4b0c-bb3c-e2a8803ecf73-kube-api-access-s7ss6\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:16:42.733306 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:42.733256 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5dd10cb-759b-48c0-a689-f1aa6abe8b8d-plugins-config-volume" pod "c5dd10cb-759b-48c0-a689-f1aa6abe8b8d" (UID: "c5dd10cb-759b-48c0-a689-f1aa6abe8b8d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:16:42.735030 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:42.735011 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5dd10cb-759b-48c0-a689-f1aa6abe8b8d-kube-api-access-9fzz9" pod "c5dd10cb-759b-48c0-a689-f1aa6abe8b8d" (UID: "c5dd10cb-759b-48c0-a689-f1aa6abe8b8d"). InnerVolumeSpecName "kube-api-access-9fzz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:16:42.834559 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:42.834525 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fzz9\" (UniqueName: \"kubernetes.io/projected/c5dd10cb-759b-48c0-a689-f1aa6abe8b8d-kube-api-access-9fzz9\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:16:42.834559 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:42.834551 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5dd10cb-759b-48c0-a689-f1aa6abe8b8d-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:16:43.453414 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:43.452970 5979 patch_prober.go:29] interesting pod/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 03 13:16:43.453414 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:43.453033 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb" podUID="ad693484-f565-418b-acbc-dd3739a1adec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 03 13:16:43.491704 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:43.491670 5979 generic.go:356] "Generic (PLEG): container finished" podID="a8a8d0bc-05fb-4a18-80af-874dd10e7f21" containerID="0ae3ad017e58ac65a197849a32b9636b04504c99126da636f4e7f3df76072014" exitCode=0 Nov 03 13:16:43.492149 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:43.491746 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cb1bd0b2-vsct2" event={"ID":"a8a8d0bc-05fb-4a18-80af-874dd10e7f21","Type":"ContainerDied","Data":"0ae3ad017e58ac65a197849a32b9636b04504c99126da636f4e7f3df76072014"} Nov 03 13:16:43.493019 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:43.492997 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-821a13f6-dfd5b9795-2ffsx" Nov 03 13:16:43.493131 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:43.493002 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-821a13f6-dfd5b9795-2ffsx" event={"ID":"c5dd10cb-759b-48c0-a689-f1aa6abe8b8d","Type":"ContainerDied","Data":"e3897720e091bb7bd65267a6bff8850c879295b4e8d893b376a67f6a9ebce68d"} Nov 03 13:16:43.493184 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:43.493146 5979 scope.go:122] "RemoveContainer" containerID="6d6b228720a783edfd2caf20844e04ea35c75b8c6a49d9a4022f20827a6ca235" Nov 03 13:16:43.494485 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:43.494353 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-821a13f6-649c78b7b7-8ht9h" event={"ID":"dd7cfbb5-315f-4b0c-bb3c-e2a8803ecf73","Type":"ContainerDied","Data":"bcb31f0bdf0cf43eaaa94dc37edf0b061f3292428bb6d4530637c1b0773a1cf4"} Nov 03 13:16:43.494485 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:43.494372 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-821a13f6-649c78b7b7-8ht9h" Nov 03 13:16:43.502369 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:43.502331 5979 scope.go:122] "RemoveContainer" containerID="76474c18fc0254d7c87f8341bce6c9397dd46e29e61917fda7d3f2f78271ac36" Nov 03 13:16:43.721600 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:43.721511 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-821a13f6-649c78b7b7-8ht9h"] Nov 03 13:16:43.726797 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:43.726768 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-821a13f6-649c78b7b7-8ht9h"] Nov 03 13:16:43.829375 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:43.829335 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-821a13f6-dfd5b9795-2ffsx"] Nov 03 13:16:43.832562 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:43.832523 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-821a13f6-dfd5b9795-2ffsx"] Nov 03 13:16:44.115510 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:44.115410 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5dd10cb-759b-48c0-a689-f1aa6abe8b8d" path="/var/lib/kubelet/pods/c5dd10cb-759b-48c0-a689-f1aa6abe8b8d/volumes" Nov 03 13:16:44.115911 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:44.115883 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd7cfbb5-315f-4b0c-bb3c-e2a8803ecf73" path="/var/lib/kubelet/pods/dd7cfbb5-315f-4b0c-bb3c-e2a8803ecf73/volumes" Nov 03 13:16:44.459497 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:44.453902 5979 patch_prober.go:29] interesting pod/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 03 13:16:44.459497 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:44.453975 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb" podUID="ad693484-f565-418b-acbc-dd3739a1adec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 03 13:16:44.679039 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:44.679010 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cb1bd0b2-vsct2" Nov 03 13:16:44.777158 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:44.777072 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8a8d0bc-05fb-4a18-80af-874dd10e7f21-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8a8d0bc-05fb-4a18-80af-874dd10e7f21-maas-api-xks-ca\") pod \"a8a8d0bc-05fb-4a18-80af-874dd10e7f21\" (UID: \"a8a8d0bc-05fb-4a18-80af-874dd10e7f21\") " Nov 03 13:16:44.777316 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:44.777199 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8a8d0bc-05fb-4a18-80af-874dd10e7f21-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8a8d0bc-05fb-4a18-80af-874dd10e7f21-maas-api-openshift-ca\") pod \"a8a8d0bc-05fb-4a18-80af-874dd10e7f21\" (UID: \"a8a8d0bc-05fb-4a18-80af-874dd10e7f21\") " Nov 03 13:16:44.777316 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:44.777236 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8a8d0bc-05fb-4a18-80af-874dd10e7f21-kube-api-access-x57vf\" (UniqueName: \"kubernetes.io/projected/a8a8d0bc-05fb-4a18-80af-874dd10e7f21-kube-api-access-x57vf\") pod \"a8a8d0bc-05fb-4a18-80af-874dd10e7f21\" (UID: \"a8a8d0bc-05fb-4a18-80af-874dd10e7f21\") " Nov 03 13:16:44.777571 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:44.777542 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8a8d0bc-05fb-4a18-80af-874dd10e7f21-maas-api-openshift-ca" pod "a8a8d0bc-05fb-4a18-80af-874dd10e7f21" (UID: "a8a8d0bc-05fb-4a18-80af-874dd10e7f21"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:16:44.777698 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:44.777653 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8a8d0bc-05fb-4a18-80af-874dd10e7f21-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:16:44.779214 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:44.779193 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8a8d0bc-05fb-4a18-80af-874dd10e7f21-kube-api-access-x57vf" pod "a8a8d0bc-05fb-4a18-80af-874dd10e7f21" (UID: "a8a8d0bc-05fb-4a18-80af-874dd10e7f21"). InnerVolumeSpecName "kube-api-access-x57vf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:16:44.779277 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:44.779232 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8a8d0bc-05fb-4a18-80af-874dd10e7f21-maas-api-xks-ca" pod "a8a8d0bc-05fb-4a18-80af-874dd10e7f21" (UID: "a8a8d0bc-05fb-4a18-80af-874dd10e7f21"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:16:44.878783 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:44.878745 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x57vf\" (UniqueName: \"kubernetes.io/projected/a8a8d0bc-05fb-4a18-80af-874dd10e7f21-kube-api-access-x57vf\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:16:44.878783 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:44.878780 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8a8d0bc-05fb-4a18-80af-874dd10e7f21-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:16:45.066553 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:45.065585 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb" Nov 03 13:16:45.455500 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:45.450824 5979 patch_prober.go:29] interesting pod/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 03 13:16:45.455500 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:45.450878 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb" podUID="ad693484-f565-418b-acbc-dd3739a1adec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 03 13:16:45.506931 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:45.506900 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cb1bd0b2-vsct2" event={"ID":"a8a8d0bc-05fb-4a18-80af-874dd10e7f21","Type":"ContainerDied","Data":"56471892a6f6e055791d3d8a98c3378b2fc1db9ce38424a1169bea3992b72991"} Nov 03 13:16:45.506931 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:45.506941 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cb1bd0b2-vsct2" Nov 03 13:16:45.506931 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:45.506952 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56471892a6f6e055791d3d8a98c3378b2fc1db9ce38424a1169bea3992b72991" Nov 03 13:16:46.451203 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:46.451163 5979 patch_prober.go:29] interesting pod/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Nov 03 13:16:46.451603 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:46.451251 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb" podUID="ad693484-f565-418b-acbc-dd3739a1adec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Nov 03 13:16:47.455552 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:47.455517 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb" Nov 03 13:16:47.456839 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:47.456823 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb" Nov 03 13:16:48.458089 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:48.458049 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-67349d1c-6cb59557f7-2j7fx"] Nov 03 13:16:51.257266 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:51.257230 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-821a13f6-z7rv9"] Nov 03 13:16:51.303630 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:51.303597 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-821a13f6-z7rv9"] Nov 03 13:16:51.356553 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:51.356515 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p"] Nov 03 13:16:51.356814 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:51.356779 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p" podUID="435a8d1c-24ab-4dce-a4e0-10aa8cff506f" containerName="istio-proxy" containerID="cri-o://bea00b9e7ba163e7ca18cf2c7e67e2f70f7fbe15247477af7ae980bf7124c2ea" gracePeriod=30 Nov 03 13:16:52.114742 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:52.114714 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a1cb675-1e8b-4943-8367-563d60d8b054" path="/var/lib/kubelet/pods/0a1cb675-1e8b-4943-8367-563d60d8b054/volumes" Nov 03 13:16:52.236714 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:52.236677 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cb1bd0b2-5c7c65f4d5-gv2mm"] Nov 03 13:16:52.297056 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:52.297027 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a7a9081d-h6kvw"] Nov 03 13:16:52.297694 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:52.297674 5979 cpu_manager.go:414] "containerMap: removing container" podUID="dd7cfbb5-315f-4b0c-bb3c-e2a8803ecf73" containerName="payload-processing" Nov 03 13:16:52.297790 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:52.297694 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd7cfbb5-315f-4b0c-bb3c-e2a8803ecf73" containerName="payload-processing" Nov 03 13:16:52.297790 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:52.297728 5979 cpu_manager.go:414] "containerMap: removing container" podUID="c5dd10cb-759b-48c0-a689-f1aa6abe8b8d" containerName="payload-pre-processing" Nov 03 13:16:52.297790 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:52.297736 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5dd10cb-759b-48c0-a689-f1aa6abe8b8d" containerName="payload-pre-processing" Nov 03 13:16:52.297790 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:52.297758 5979 cpu_manager.go:414] "containerMap: removing container" podUID="a8a8d0bc-05fb-4a18-80af-874dd10e7f21" containerName="revoke-keys" Nov 03 13:16:52.297790 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:52.297765 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8a8d0bc-05fb-4a18-80af-874dd10e7f21" containerName="revoke-keys" Nov 03 13:16:52.298041 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:52.297872 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd7cfbb5-315f-4b0c-bb3c-e2a8803ecf73" containerName="payload-processing" Nov 03 13:16:52.298041 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:52.297888 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5dd10cb-759b-48c0-a689-f1aa6abe8b8d" containerName="payload-pre-processing" Nov 03 13:16:52.298041 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:52.297899 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8a8d0bc-05fb-4a18-80af-874dd10e7f21" containerName="revoke-keys" Nov 03 13:16:52.302583 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:52.302562 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a7a9081d-h6kvw" Nov 03 13:16:52.352044 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:52.352008 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 13:16:52.354191 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:52.354163 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a7a9081d-h6kvw"] Nov 03 13:16:52.355029 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:52.355009 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 13:16:52.369696 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:52.369618 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e03e3b1a-acce-4e8b-91e8-4b863b6d59c1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-a7a9081d-h6kvw\" (UID: \"e03e3b1a-acce-4e8b-91e8-4b863b6d59c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a7a9081d-h6kvw" Nov 03 13:16:52.369696 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:52.369669 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7gj8\" (UniqueName: \"kubernetes.io/projected/e03e3b1a-acce-4e8b-91e8-4b863b6d59c1-kube-api-access-m7gj8\") pod \"maas-api-revoke-keys-e2e-derive-a7a9081d-h6kvw\" (UID: \"e03e3b1a-acce-4e8b-91e8-4b863b6d59c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a7a9081d-h6kvw" Nov 03 13:16:52.369888 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:52.369795 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e03e3b1a-acce-4e8b-91e8-4b863b6d59c1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-a7a9081d-h6kvw\" (UID: \"e03e3b1a-acce-4e8b-91e8-4b863b6d59c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a7a9081d-h6kvw" Nov 03 13:16:52.377340 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:52.377318 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:16:52.377446 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:52.377363 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:16:52.470342 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:52.470314 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7gj8\" (UniqueName: \"kubernetes.io/projected/e03e3b1a-acce-4e8b-91e8-4b863b6d59c1-kube-api-access-m7gj8\") pod \"maas-api-revoke-keys-e2e-derive-a7a9081d-h6kvw\" (UID: \"e03e3b1a-acce-4e8b-91e8-4b863b6d59c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a7a9081d-h6kvw" Nov 03 13:16:52.470536 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:52.470349 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e03e3b1a-acce-4e8b-91e8-4b863b6d59c1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-a7a9081d-h6kvw\" (UID: \"e03e3b1a-acce-4e8b-91e8-4b863b6d59c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a7a9081d-h6kvw" Nov 03 13:16:52.470536 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:52.470451 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e03e3b1a-acce-4e8b-91e8-4b863b6d59c1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-a7a9081d-h6kvw\" (UID: \"e03e3b1a-acce-4e8b-91e8-4b863b6d59c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a7a9081d-h6kvw" Nov 03 13:16:52.470999 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:52.470979 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e03e3b1a-acce-4e8b-91e8-4b863b6d59c1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-a7a9081d-h6kvw\" (UID: \"e03e3b1a-acce-4e8b-91e8-4b863b6d59c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a7a9081d-h6kvw" Nov 03 13:16:52.472736 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:52.472712 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e03e3b1a-acce-4e8b-91e8-4b863b6d59c1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-a7a9081d-h6kvw\" (UID: \"e03e3b1a-acce-4e8b-91e8-4b863b6d59c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a7a9081d-h6kvw" Nov 03 13:16:52.523463 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:52.523427 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7gj8\" (UniqueName: \"kubernetes.io/projected/e03e3b1a-acce-4e8b-91e8-4b863b6d59c1-kube-api-access-m7gj8\") pod \"maas-api-revoke-keys-e2e-derive-a7a9081d-h6kvw\" (UID: \"e03e3b1a-acce-4e8b-91e8-4b863b6d59c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a7a9081d-h6kvw" Nov 03 13:16:52.615622 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:52.615591 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a7a9081d-h6kvw" Nov 03 13:16:52.726594 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:52.726560 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4"] Nov 03 13:16:52.731826 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:52.731804 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4" Nov 03 13:16:52.773249 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:52.773213 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01e6f154-38b8-47bf-8f9b-436fa4cf1436-workload-certs\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4\" (UID: \"01e6f154-38b8-47bf-8f9b-436fa4cf1436\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4" Nov 03 13:16:52.773399 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:52.773274 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01e6f154-38b8-47bf-8f9b-436fa4cf1436-istio-envoy\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4\" (UID: \"01e6f154-38b8-47bf-8f9b-436fa4cf1436\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4" Nov 03 13:16:52.773399 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:52.773342 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c98bt\" (UniqueName: \"kubernetes.io/projected/01e6f154-38b8-47bf-8f9b-436fa4cf1436-kube-api-access-c98bt\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4\" (UID: \"01e6f154-38b8-47bf-8f9b-436fa4cf1436\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4" Nov 03 13:16:52.773512 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:52.773405 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01e6f154-38b8-47bf-8f9b-436fa4cf1436-istio-token\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4\" (UID: \"01e6f154-38b8-47bf-8f9b-436fa4cf1436\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4" Nov 03 13:16:52.773512 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:52.773429 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01e6f154-38b8-47bf-8f9b-436fa4cf1436-istio-data\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4\" (UID: \"01e6f154-38b8-47bf-8f9b-436fa4cf1436\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4" Nov 03 13:16:52.773625 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:52.773572 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01e6f154-38b8-47bf-8f9b-436fa4cf1436-istiod-ca-cert\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4\" (UID: \"01e6f154-38b8-47bf-8f9b-436fa4cf1436\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4" Nov 03 13:16:52.773625 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:52.773611 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01e6f154-38b8-47bf-8f9b-436fa4cf1436-credential-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4\" (UID: \"01e6f154-38b8-47bf-8f9b-436fa4cf1436\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4" Nov 03 13:16:52.773805 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:52.773762 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01e6f154-38b8-47bf-8f9b-436fa4cf1436-workload-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4\" (UID: \"01e6f154-38b8-47bf-8f9b-436fa4cf1436\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4" Nov 03 13:16:52.773900 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:52.773833 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01e6f154-38b8-47bf-8f9b-436fa4cf1436-istio-podinfo\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4\" (UID: \"01e6f154-38b8-47bf-8f9b-436fa4cf1436\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4" Nov 03 13:16:52.784120 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:52.784091 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-cdb532-openshift-default-dockercfg-q2fdz\"" Nov 03 13:16:52.785904 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:52.785881 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4"] Nov 03 13:16:52.877844 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:52.875909 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01e6f154-38b8-47bf-8f9b-436fa4cf1436-istio-envoy\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4\" (UID: \"01e6f154-38b8-47bf-8f9b-436fa4cf1436\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4" Nov 03 13:16:52.878119 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:52.878094 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c98bt\" (UniqueName: \"kubernetes.io/projected/01e6f154-38b8-47bf-8f9b-436fa4cf1436-kube-api-access-c98bt\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4\" (UID: \"01e6f154-38b8-47bf-8f9b-436fa4cf1436\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4" Nov 03 13:16:52.879344 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:52.879316 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01e6f154-38b8-47bf-8f9b-436fa4cf1436-istio-token\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4\" (UID: \"01e6f154-38b8-47bf-8f9b-436fa4cf1436\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4" Nov 03 13:16:52.879850 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:52.879830 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01e6f154-38b8-47bf-8f9b-436fa4cf1436-istio-data\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4\" (UID: \"01e6f154-38b8-47bf-8f9b-436fa4cf1436\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4" Nov 03 13:16:52.880433 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:52.880301 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01e6f154-38b8-47bf-8f9b-436fa4cf1436-istio-data\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4\" (UID: \"01e6f154-38b8-47bf-8f9b-436fa4cf1436\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4" Nov 03 13:16:52.881882 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:52.881834 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01e6f154-38b8-47bf-8f9b-436fa4cf1436-istiod-ca-cert\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4\" (UID: \"01e6f154-38b8-47bf-8f9b-436fa4cf1436\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4" Nov 03 13:16:52.882013 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:52.881994 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01e6f154-38b8-47bf-8f9b-436fa4cf1436-istiod-ca-cert\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4\" (UID: \"01e6f154-38b8-47bf-8f9b-436fa4cf1436\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4" Nov 03 13:16:52.882148 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:52.882133 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01e6f154-38b8-47bf-8f9b-436fa4cf1436-credential-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4\" (UID: \"01e6f154-38b8-47bf-8f9b-436fa4cf1436\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4" Nov 03 13:16:52.882631 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:52.882514 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01e6f154-38b8-47bf-8f9b-436fa4cf1436-credential-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4\" (UID: \"01e6f154-38b8-47bf-8f9b-436fa4cf1436\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4" Nov 03 13:16:52.882912 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:52.882893 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01e6f154-38b8-47bf-8f9b-436fa4cf1436-workload-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4\" (UID: \"01e6f154-38b8-47bf-8f9b-436fa4cf1436\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4" Nov 03 13:16:52.883398 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:52.883285 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01e6f154-38b8-47bf-8f9b-436fa4cf1436-workload-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4\" (UID: \"01e6f154-38b8-47bf-8f9b-436fa4cf1436\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4" Nov 03 13:16:52.883550 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:52.883533 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01e6f154-38b8-47bf-8f9b-436fa4cf1436-istio-podinfo\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4\" (UID: \"01e6f154-38b8-47bf-8f9b-436fa4cf1436\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4" Nov 03 13:16:52.884115 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:52.884097 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01e6f154-38b8-47bf-8f9b-436fa4cf1436-workload-certs\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4\" (UID: \"01e6f154-38b8-47bf-8f9b-436fa4cf1436\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4" Nov 03 13:16:52.884588 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:52.884453 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01e6f154-38b8-47bf-8f9b-436fa4cf1436-workload-certs\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4\" (UID: \"01e6f154-38b8-47bf-8f9b-436fa4cf1436\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4" Nov 03 13:16:52.899223 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:52.899172 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01e6f154-38b8-47bf-8f9b-436fa4cf1436-istio-envoy\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4\" (UID: \"01e6f154-38b8-47bf-8f9b-436fa4cf1436\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4" Nov 03 13:16:52.902418 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:52.902352 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01e6f154-38b8-47bf-8f9b-436fa4cf1436-istio-podinfo\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4\" (UID: \"01e6f154-38b8-47bf-8f9b-436fa4cf1436\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4" Nov 03 13:16:52.939005 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:52.938972 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01e6f154-38b8-47bf-8f9b-436fa4cf1436-istio-token\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4\" (UID: \"01e6f154-38b8-47bf-8f9b-436fa4cf1436\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4" Nov 03 13:16:52.939198 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:52.939056 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c98bt\" (UniqueName: \"kubernetes.io/projected/01e6f154-38b8-47bf-8f9b-436fa4cf1436-kube-api-access-c98bt\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4\" (UID: \"01e6f154-38b8-47bf-8f9b-436fa4cf1436\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4" Nov 03 13:16:53.013933 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:53.013890 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a7a9081d-h6kvw"] Nov 03 13:16:53.020678 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:16:53.020648 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode03e3b1a_acce_4e8b_91e8_4b863b6d59c1.slice/crio-f124015e59fd7110939806373c06272b56d69db43a62fa9e3fff98dd393860f1 WatchSource:0}: Error finding container f124015e59fd7110939806373c06272b56d69db43a62fa9e3fff98dd393860f1: Status 404 returned error can't find the container with id f124015e59fd7110939806373c06272b56d69db43a62fa9e3fff98dd393860f1 Nov 03 13:16:53.045327 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:53.045300 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4" Nov 03 13:16:53.458887 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:53.458844 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-67349d1c-6cb59557f7-2j7fx" podUID="2c39eb4f-d901-4453-9e96-8236716c1e91" containerName="payload-processing" containerID="cri-o://6e069a51c01322f0e8e1e2e98d2e3eff0b47f17ad57d05787cafe6b73b5ef425" gracePeriod=25 Nov 03 13:16:53.471152 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:53.471128 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4"] Nov 03 13:16:53.541576 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:16:53.541542 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01e6f154_38b8_47bf_8f9b_436fa4cf1436.slice/crio-00a5afe5ccb26049911a2ba4eabcb755268092def35fa421f9a867b7c159129b WatchSource:0}: Error finding container 00a5afe5ccb26049911a2ba4eabcb755268092def35fa421f9a867b7c159129b: Status 404 returned error can't find the container with id 00a5afe5ccb26049911a2ba4eabcb755268092def35fa421f9a867b7c159129b Nov 03 13:16:53.549946 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:53.549909 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a7a9081d-h6kvw" event={"ID":"e03e3b1a-acce-4e8b-91e8-4b863b6d59c1","Type":"ContainerStarted","Data":"191520a43514af6d7723138623a1fbd03798e4d38f114119a9c6d2833a74b319"} Nov 03 13:16:53.550117 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:53.549963 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a7a9081d-h6kvw" event={"ID":"e03e3b1a-acce-4e8b-91e8-4b863b6d59c1","Type":"ContainerStarted","Data":"f124015e59fd7110939806373c06272b56d69db43a62fa9e3fff98dd393860f1"} Nov 03 13:16:53.564416 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:53.564376 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:16:53.564516 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:53.564482 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:16:53.564573 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:53.564528 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:16:53.658699 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:53.658626 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a7a9081d-h6kvw" podStartSLOduration=1.6586102870000001 podStartE2EDuration="1.658610287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:16:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:16:53.656756578 +0000 UTC m=+695.667444820" watchObservedRunningTime="2026-11-03 13:16:53.658610287 +0000 UTC m=+695.669298528" Nov 03 13:16:53.761525 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:53.758929 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-67349d1c-6cb59557f7-2j7fx" Nov 03 13:16:53.802621 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:53.802535 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c39eb4f-d901-4453-9e96-8236716c1e91-kube-api-access-xr9mv\" (UniqueName: \"kubernetes.io/projected/2c39eb4f-d901-4453-9e96-8236716c1e91-kube-api-access-xr9mv\") pod \"2c39eb4f-d901-4453-9e96-8236716c1e91\" (UID: \"2c39eb4f-d901-4453-9e96-8236716c1e91\") " Nov 03 13:16:53.802778 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:53.802689 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c39eb4f-d901-4453-9e96-8236716c1e91-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c39eb4f-d901-4453-9e96-8236716c1e91-plugins-config-volume\") pod \"2c39eb4f-d901-4453-9e96-8236716c1e91\" (UID: \"2c39eb4f-d901-4453-9e96-8236716c1e91\") " Nov 03 13:16:53.803156 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:53.803126 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c39eb4f-d901-4453-9e96-8236716c1e91-plugins-config-volume" pod "2c39eb4f-d901-4453-9e96-8236716c1e91" (UID: "2c39eb4f-d901-4453-9e96-8236716c1e91"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:16:53.804713 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:53.804691 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c39eb4f-d901-4453-9e96-8236716c1e91-kube-api-access-xr9mv" pod "2c39eb4f-d901-4453-9e96-8236716c1e91" (UID: "2c39eb4f-d901-4453-9e96-8236716c1e91"). InnerVolumeSpecName "kube-api-access-xr9mv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:16:53.904281 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:53.904250 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr9mv\" (UniqueName: \"kubernetes.io/projected/2c39eb4f-d901-4453-9e96-8236716c1e91-kube-api-access-xr9mv\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:16:53.904281 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:53.904279 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c39eb4f-d901-4453-9e96-8236716c1e91-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:16:54.556064 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:54.556031 5979 generic.go:356] "Generic (PLEG): container finished" podID="2c39eb4f-d901-4453-9e96-8236716c1e91" containerID="6e069a51c01322f0e8e1e2e98d2e3eff0b47f17ad57d05787cafe6b73b5ef425" exitCode=0 Nov 03 13:16:54.556528 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:54.556082 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-67349d1c-6cb59557f7-2j7fx" Nov 03 13:16:54.556528 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:54.556116 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-67349d1c-6cb59557f7-2j7fx" event={"ID":"2c39eb4f-d901-4453-9e96-8236716c1e91","Type":"ContainerDied","Data":"6e069a51c01322f0e8e1e2e98d2e3eff0b47f17ad57d05787cafe6b73b5ef425"} Nov 03 13:16:54.556528 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:54.556174 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-67349d1c-6cb59557f7-2j7fx" event={"ID":"2c39eb4f-d901-4453-9e96-8236716c1e91","Type":"ContainerDied","Data":"4d15ecae4547fa5ec6a8b4252d26c07f949ef9834a64c573b744df7647f25b69"} Nov 03 13:16:54.556528 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:54.556202 5979 scope.go:122] "RemoveContainer" containerID="6e069a51c01322f0e8e1e2e98d2e3eff0b47f17ad57d05787cafe6b73b5ef425" Nov 03 13:16:54.557983 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:54.557910 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4" event={"ID":"01e6f154-38b8-47bf-8f9b-436fa4cf1436","Type":"ContainerStarted","Data":"9a1ed4e0b255d0d777095b119f9eebff6f036c31428b8681be748dbb940a5d1a"} Nov 03 13:16:54.557983 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:54.557959 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4" event={"ID":"01e6f154-38b8-47bf-8f9b-436fa4cf1436","Type":"ContainerStarted","Data":"00a5afe5ccb26049911a2ba4eabcb755268092def35fa421f9a867b7c159129b"} Nov 03 13:16:54.560015 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:54.559988 5979 generic.go:356] "Generic (PLEG): container finished" podID="e03e3b1a-acce-4e8b-91e8-4b863b6d59c1" containerID="191520a43514af6d7723138623a1fbd03798e4d38f114119a9c6d2833a74b319" exitCode=0 Nov 03 13:16:54.560105 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:54.560027 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a7a9081d-h6kvw" event={"ID":"e03e3b1a-acce-4e8b-91e8-4b863b6d59c1","Type":"ContainerDied","Data":"191520a43514af6d7723138623a1fbd03798e4d38f114119a9c6d2833a74b319"} Nov 03 13:16:54.566245 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:54.566225 5979 scope.go:122] "RemoveContainer" containerID="6e069a51c01322f0e8e1e2e98d2e3eff0b47f17ad57d05787cafe6b73b5ef425" Nov 03 13:16:54.566556 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:16:54.566529 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e069a51c01322f0e8e1e2e98d2e3eff0b47f17ad57d05787cafe6b73b5ef425\": container with ID starting with 6e069a51c01322f0e8e1e2e98d2e3eff0b47f17ad57d05787cafe6b73b5ef425 not found: ID does not exist" containerID="6e069a51c01322f0e8e1e2e98d2e3eff0b47f17ad57d05787cafe6b73b5ef425" Nov 03 13:16:54.566626 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:54.566564 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e069a51c01322f0e8e1e2e98d2e3eff0b47f17ad57d05787cafe6b73b5ef425"} err="failed to get container status \"6e069a51c01322f0e8e1e2e98d2e3eff0b47f17ad57d05787cafe6b73b5ef425\": rpc error: code = NotFound desc = could not find container \"6e069a51c01322f0e8e1e2e98d2e3eff0b47f17ad57d05787cafe6b73b5ef425\": container with ID starting with 6e069a51c01322f0e8e1e2e98d2e3eff0b47f17ad57d05787cafe6b73b5ef425 not found: ID does not exist" Nov 03 13:16:54.822873 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:54.822760 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4" podStartSLOduration=2.822745249 podStartE2EDuration="2.822745249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:16:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:16:54.821326895 +0000 UTC m=+696.832015138" watchObservedRunningTime="2026-11-03 13:16:54.822745249 +0000 UTC m=+696.833433490" Nov 03 13:16:54.925928 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:54.925891 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-67349d1c-6cb59557f7-2j7fx"] Nov 03 13:16:54.929910 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:54.929884 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-67349d1c-6cb59557f7-2j7fx"] Nov 03 13:16:55.045717 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:55.045676 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4" Nov 03 13:16:55.050539 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:55.050516 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4" Nov 03 13:16:55.550562 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:55.550531 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:16:55.567417 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:55.567379 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4" Nov 03 13:16:55.568295 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:55.568283 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4" Nov 03 13:16:55.758018 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:55.757990 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a7a9081d-h6kvw" Nov 03 13:16:55.843676 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:55.843586 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e03e3b1a-acce-4e8b-91e8-4b863b6d59c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e03e3b1a-acce-4e8b-91e8-4b863b6d59c1-maas-api-openshift-ca\") pod \"e03e3b1a-acce-4e8b-91e8-4b863b6d59c1\" (UID: \"e03e3b1a-acce-4e8b-91e8-4b863b6d59c1\") " Nov 03 13:16:55.843676 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:55.843674 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e03e3b1a-acce-4e8b-91e8-4b863b6d59c1-kube-api-access-m7gj8\" (UniqueName: \"kubernetes.io/projected/e03e3b1a-acce-4e8b-91e8-4b863b6d59c1-kube-api-access-m7gj8\") pod \"e03e3b1a-acce-4e8b-91e8-4b863b6d59c1\" (UID: \"e03e3b1a-acce-4e8b-91e8-4b863b6d59c1\") " Nov 03 13:16:55.843883 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:55.843777 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e03e3b1a-acce-4e8b-91e8-4b863b6d59c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e03e3b1a-acce-4e8b-91e8-4b863b6d59c1-maas-api-xks-ca\") pod \"e03e3b1a-acce-4e8b-91e8-4b863b6d59c1\" (UID: \"e03e3b1a-acce-4e8b-91e8-4b863b6d59c1\") " Nov 03 13:16:55.844043 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:55.844018 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e03e3b1a-acce-4e8b-91e8-4b863b6d59c1-maas-api-openshift-ca" pod "e03e3b1a-acce-4e8b-91e8-4b863b6d59c1" (UID: "e03e3b1a-acce-4e8b-91e8-4b863b6d59c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:16:55.844171 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:55.844150 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e03e3b1a-acce-4e8b-91e8-4b863b6d59c1-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:16:55.845868 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:55.845832 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e03e3b1a-acce-4e8b-91e8-4b863b6d59c1-maas-api-xks-ca" pod "e03e3b1a-acce-4e8b-91e8-4b863b6d59c1" (UID: "e03e3b1a-acce-4e8b-91e8-4b863b6d59c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:16:55.845982 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:55.845870 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e03e3b1a-acce-4e8b-91e8-4b863b6d59c1-kube-api-access-m7gj8" pod "e03e3b1a-acce-4e8b-91e8-4b863b6d59c1" (UID: "e03e3b1a-acce-4e8b-91e8-4b863b6d59c1"). InnerVolumeSpecName "kube-api-access-m7gj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:16:55.949388 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:55.949347 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e03e3b1a-acce-4e8b-91e8-4b863b6d59c1-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:16:55.949388 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:55.949381 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7gj8\" (UniqueName: \"kubernetes.io/projected/e03e3b1a-acce-4e8b-91e8-4b863b6d59c1-kube-api-access-m7gj8\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:16:56.118436 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:56.118341 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c39eb4f-d901-4453-9e96-8236716c1e91" path="/var/lib/kubelet/pods/2c39eb4f-d901-4453-9e96-8236716c1e91/volumes" Nov 03 13:16:56.577509 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:56.575282 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a7a9081d-h6kvw" Nov 03 13:16:56.577509 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:56.575281 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a7a9081d-h6kvw" event={"ID":"e03e3b1a-acce-4e8b-91e8-4b863b6d59c1","Type":"ContainerDied","Data":"f124015e59fd7110939806373c06272b56d69db43a62fa9e3fff98dd393860f1"} Nov 03 13:16:56.577509 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:56.575407 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f124015e59fd7110939806373c06272b56d69db43a62fa9e3fff98dd393860f1" Nov 03 13:16:56.578033 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:56.577899 5979 generic.go:356] "Generic (PLEG): container finished" podID="435a8d1c-24ab-4dce-a4e0-10aa8cff506f" containerID="bea00b9e7ba163e7ca18cf2c7e67e2f70f7fbe15247477af7ae980bf7124c2ea" exitCode=0 Nov 03 13:16:56.578582 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:56.578560 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p" event={"ID":"435a8d1c-24ab-4dce-a4e0-10aa8cff506f","Type":"ContainerDied","Data":"bea00b9e7ba163e7ca18cf2c7e67e2f70f7fbe15247477af7ae980bf7124c2ea"} Nov 03 13:16:56.578705 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:56.578610 5979 scope.go:122] "RemoveContainer" containerID="1d1af1b397212f6fcfce65e98bf87f16dfaccb80dd1b021a7cb2c315eb3e091b" Nov 03 13:16:56.606872 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:56.606849 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p" Nov 03 13:16:56.654568 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:56.654529 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-workload-certs\") pod \"435a8d1c-24ab-4dce-a4e0-10aa8cff506f\" (UID: \"435a8d1c-24ab-4dce-a4e0-10aa8cff506f\") " Nov 03 13:16:56.654746 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:56.654575 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-istio-token\" (UniqueName: \"kubernetes.io/projected/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-istio-token\") pod \"435a8d1c-24ab-4dce-a4e0-10aa8cff506f\" (UID: \"435a8d1c-24ab-4dce-a4e0-10aa8cff506f\") " Nov 03 13:16:56.654746 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:56.654638 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-istio-data\") pod \"435a8d1c-24ab-4dce-a4e0-10aa8cff506f\" (UID: \"435a8d1c-24ab-4dce-a4e0-10aa8cff506f\") " Nov 03 13:16:56.654746 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:56.654697 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-istio-envoy\") pod \"435a8d1c-24ab-4dce-a4e0-10aa8cff506f\" (UID: \"435a8d1c-24ab-4dce-a4e0-10aa8cff506f\") " Nov 03 13:16:56.654883 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:56.654778 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-credential-socket\") pod \"435a8d1c-24ab-4dce-a4e0-10aa8cff506f\" (UID: \"435a8d1c-24ab-4dce-a4e0-10aa8cff506f\") " Nov 03 13:16:56.654883 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:56.654805 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-kube-api-access-948v2\" (UniqueName: \"kubernetes.io/projected/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-kube-api-access-948v2\") pod \"435a8d1c-24ab-4dce-a4e0-10aa8cff506f\" (UID: \"435a8d1c-24ab-4dce-a4e0-10aa8cff506f\") " Nov 03 13:16:56.654883 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:56.654825 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-workload-certs" pod "435a8d1c-24ab-4dce-a4e0-10aa8cff506f" (UID: "435a8d1c-24ab-4dce-a4e0-10aa8cff506f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:16:56.654883 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:56.654839 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-istio-podinfo\") pod \"435a8d1c-24ab-4dce-a4e0-10aa8cff506f\" (UID: \"435a8d1c-24ab-4dce-a4e0-10aa8cff506f\") " Nov 03 13:16:56.654883 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:56.654874 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-workload-socket\") pod \"435a8d1c-24ab-4dce-a4e0-10aa8cff506f\" (UID: \"435a8d1c-24ab-4dce-a4e0-10aa8cff506f\") " Nov 03 13:16:56.655105 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:56.654924 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-istiod-ca-cert\") pod \"435a8d1c-24ab-4dce-a4e0-10aa8cff506f\" (UID: \"435a8d1c-24ab-4dce-a4e0-10aa8cff506f\") " Nov 03 13:16:56.655105 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:56.655048 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-istio-data" pod "435a8d1c-24ab-4dce-a4e0-10aa8cff506f" (UID: "435a8d1c-24ab-4dce-a4e0-10aa8cff506f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:16:56.655251 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:56.655197 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-workload-socket" pod "435a8d1c-24ab-4dce-a4e0-10aa8cff506f" (UID: "435a8d1c-24ab-4dce-a4e0-10aa8cff506f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:16:56.655641 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:56.655336 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-credential-socket" pod "435a8d1c-24ab-4dce-a4e0-10aa8cff506f" (UID: "435a8d1c-24ab-4dce-a4e0-10aa8cff506f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:16:56.655641 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:56.655535 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:16:56.655641 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:56.655555 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:16:56.655641 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:56.655570 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:16:56.655641 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:56.655588 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:16:56.655917 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:56.655699 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-istiod-ca-cert" pod "435a8d1c-24ab-4dce-a4e0-10aa8cff506f" (UID: "435a8d1c-24ab-4dce-a4e0-10aa8cff506f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:16:56.656953 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:56.656926 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-istio-token" pod "435a8d1c-24ab-4dce-a4e0-10aa8cff506f" (UID: "435a8d1c-24ab-4dce-a4e0-10aa8cff506f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:16:56.657055 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:56.657040 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-istio-podinfo" pod "435a8d1c-24ab-4dce-a4e0-10aa8cff506f" (UID: "435a8d1c-24ab-4dce-a4e0-10aa8cff506f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:16:56.657305 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:56.657283 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-kube-api-access-948v2" pod "435a8d1c-24ab-4dce-a4e0-10aa8cff506f" (UID: "435a8d1c-24ab-4dce-a4e0-10aa8cff506f"). InnerVolumeSpecName "kube-api-access-948v2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:16:56.657379 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:56.657324 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-istio-envoy" pod "435a8d1c-24ab-4dce-a4e0-10aa8cff506f" (UID: "435a8d1c-24ab-4dce-a4e0-10aa8cff506f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:16:56.756326 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:56.756289 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:16:56.756326 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:56.756322 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:16:56.756540 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:56.756338 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-948v2\" (UniqueName: \"kubernetes.io/projected/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-kube-api-access-948v2\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:16:56.756540 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:56.756352 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:16:56.756540 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:56.756364 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/435a8d1c-24ab-4dce-a4e0-10aa8cff506f-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:16:57.237165 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:57.237116 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-cb1bd0b2-5c7c65f4d5-gv2mm" podUID="6d10d8b2-1143-48e1-9314-b471fa85370f" containerName="payload-processing" containerID="cri-o://cc4ff5b068567e110f5a23c1d6f4a4c567505d7e408c4aa066fd1e3a4ea1972a" gracePeriod=25 Nov 03 13:16:57.511180 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:57.511114 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-cb1bd0b2-5c7c65f4d5-gv2mm" Nov 03 13:16:57.584165 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:57.584130 5979 generic.go:356] "Generic (PLEG): container finished" podID="6d10d8b2-1143-48e1-9314-b471fa85370f" containerID="cc4ff5b068567e110f5a23c1d6f4a4c567505d7e408c4aa066fd1e3a4ea1972a" exitCode=0 Nov 03 13:16:57.584595 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:57.584176 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-cb1bd0b2-5c7c65f4d5-gv2mm" Nov 03 13:16:57.584595 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:57.584219 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-cb1bd0b2-5c7c65f4d5-gv2mm" event={"ID":"6d10d8b2-1143-48e1-9314-b471fa85370f","Type":"ContainerDied","Data":"cc4ff5b068567e110f5a23c1d6f4a4c567505d7e408c4aa066fd1e3a4ea1972a"} Nov 03 13:16:57.584595 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:57.584272 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-cb1bd0b2-5c7c65f4d5-gv2mm" event={"ID":"6d10d8b2-1143-48e1-9314-b471fa85370f","Type":"ContainerDied","Data":"193f0e0f26c60cbb6128116e070cc50b420e2993f47e6da89586bf35cbfc4d28"} Nov 03 13:16:57.584595 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:57.584299 5979 scope.go:122] "RemoveContainer" containerID="cc4ff5b068567e110f5a23c1d6f4a4c567505d7e408c4aa066fd1e3a4ea1972a" Nov 03 13:16:57.585630 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:57.585498 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p" Nov 03 13:16:57.585630 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:57.585496 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p" event={"ID":"435a8d1c-24ab-4dce-a4e0-10aa8cff506f","Type":"ContainerDied","Data":"e54d976fb7051b8b231bf2f84f4d5dcebb70c0372234b7a3e91b8cc837462d4a"} Nov 03 13:16:57.587976 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:57.587953 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d10d8b2-1143-48e1-9314-b471fa85370f-kube-api-access-7lfvc\" (UniqueName: \"kubernetes.io/projected/6d10d8b2-1143-48e1-9314-b471fa85370f-kube-api-access-7lfvc\") pod \"6d10d8b2-1143-48e1-9314-b471fa85370f\" (UID: \"6d10d8b2-1143-48e1-9314-b471fa85370f\") " Nov 03 13:16:57.588080 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:57.588043 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d10d8b2-1143-48e1-9314-b471fa85370f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d10d8b2-1143-48e1-9314-b471fa85370f-plugins-config-volume\") pod \"6d10d8b2-1143-48e1-9314-b471fa85370f\" (UID: \"6d10d8b2-1143-48e1-9314-b471fa85370f\") " Nov 03 13:16:57.588379 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:57.588350 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d10d8b2-1143-48e1-9314-b471fa85370f-plugins-config-volume" pod "6d10d8b2-1143-48e1-9314-b471fa85370f" (UID: "6d10d8b2-1143-48e1-9314-b471fa85370f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:16:57.590010 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:57.589982 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d10d8b2-1143-48e1-9314-b471fa85370f-kube-api-access-7lfvc" pod "6d10d8b2-1143-48e1-9314-b471fa85370f" (UID: "6d10d8b2-1143-48e1-9314-b471fa85370f"). InnerVolumeSpecName "kube-api-access-7lfvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:16:57.594176 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:57.594157 5979 scope.go:122] "RemoveContainer" containerID="cc4ff5b068567e110f5a23c1d6f4a4c567505d7e408c4aa066fd1e3a4ea1972a" Nov 03 13:16:57.594518 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:16:57.594496 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc4ff5b068567e110f5a23c1d6f4a4c567505d7e408c4aa066fd1e3a4ea1972a\": container with ID starting with cc4ff5b068567e110f5a23c1d6f4a4c567505d7e408c4aa066fd1e3a4ea1972a not found: ID does not exist" containerID="cc4ff5b068567e110f5a23c1d6f4a4c567505d7e408c4aa066fd1e3a4ea1972a" Nov 03 13:16:57.594583 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:57.594525 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc4ff5b068567e110f5a23c1d6f4a4c567505d7e408c4aa066fd1e3a4ea1972a"} err="failed to get container status \"cc4ff5b068567e110f5a23c1d6f4a4c567505d7e408c4aa066fd1e3a4ea1972a\": rpc error: code = NotFound desc = could not find container \"cc4ff5b068567e110f5a23c1d6f4a4c567505d7e408c4aa066fd1e3a4ea1972a\": container with ID starting with cc4ff5b068567e110f5a23c1d6f4a4c567505d7e408c4aa066fd1e3a4ea1972a not found: ID does not exist" Nov 03 13:16:57.594583 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:57.594541 5979 scope.go:122] "RemoveContainer" containerID="bea00b9e7ba163e7ca18cf2c7e67e2f70f7fbe15247477af7ae980bf7124c2ea" Nov 03 13:16:57.689739 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:57.689703 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lfvc\" (UniqueName: \"kubernetes.io/projected/6d10d8b2-1143-48e1-9314-b471fa85370f-kube-api-access-7lfvc\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:16:57.689739 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:57.689736 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d10d8b2-1143-48e1-9314-b471fa85370f-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:16:57.740848 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:57.740811 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p"] Nov 03 13:16:57.749432 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:57.746587 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-821a13f6-openshift-default-5545fdcff7-4qm7p"] Nov 03 13:16:57.996600 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:57.996564 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cb1bd0b2-5c7c65f4d5-gv2mm"] Nov 03 13:16:58.000593 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:58.000562 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-cb1bd0b2-5c7c65f4d5-gv2mm"] Nov 03 13:16:58.026950 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:58.026918 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67349d1c-x46gc"] Nov 03 13:16:58.029126 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:58.029102 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67349d1c-x46gc"] Nov 03 13:16:58.114286 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:58.114250 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="435a8d1c-24ab-4dce-a4e0-10aa8cff506f" path="/var/lib/kubelet/pods/435a8d1c-24ab-4dce-a4e0-10aa8cff506f/volumes" Nov 03 13:16:58.114664 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:58.114651 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d10d8b2-1143-48e1-9314-b471fa85370f" path="/var/lib/kubelet/pods/6d10d8b2-1143-48e1-9314-b471fa85370f/volumes" Nov 03 13:16:58.115019 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:58.115007 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdcf19ec-e5e2-416a-8d66-1121e78847be" path="/var/lib/kubelet/pods/bdcf19ec-e5e2-416a-8d66-1121e78847be/volumes" Nov 03 13:16:59.322383 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:59.322350 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cb1bd0b2-vsct2"] Nov 03 13:16:59.369540 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:59.369462 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-cb1bd0b2-vsct2"] Nov 03 13:16:59.751083 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:59.751046 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-11ca03ad-sv2m9"] Nov 03 13:16:59.751761 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:59.751741 5979 cpu_manager.go:414] "containerMap: removing container" podUID="435a8d1c-24ab-4dce-a4e0-10aa8cff506f" containerName="istio-proxy" Nov 03 13:16:59.751884 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:59.751764 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="435a8d1c-24ab-4dce-a4e0-10aa8cff506f" containerName="istio-proxy" Nov 03 13:16:59.751884 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:59.751778 5979 cpu_manager.go:414] "containerMap: removing container" podUID="6d10d8b2-1143-48e1-9314-b471fa85370f" containerName="payload-processing" Nov 03 13:16:59.751884 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:59.751786 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d10d8b2-1143-48e1-9314-b471fa85370f" containerName="payload-processing" Nov 03 13:16:59.751884 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:59.751804 5979 cpu_manager.go:414] "containerMap: removing container" podUID="e03e3b1a-acce-4e8b-91e8-4b863b6d59c1" containerName="revoke-keys" Nov 03 13:16:59.751884 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:59.751811 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e03e3b1a-acce-4e8b-91e8-4b863b6d59c1" containerName="revoke-keys" Nov 03 13:16:59.751884 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:59.751842 5979 cpu_manager.go:414] "containerMap: removing container" podUID="2c39eb4f-d901-4453-9e96-8236716c1e91" containerName="payload-processing" Nov 03 13:16:59.751884 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:59.751849 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c39eb4f-d901-4453-9e96-8236716c1e91" containerName="payload-processing" Nov 03 13:16:59.752090 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:59.751991 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c39eb4f-d901-4453-9e96-8236716c1e91" containerName="payload-processing" Nov 03 13:16:59.752090 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:59.752003 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="435a8d1c-24ab-4dce-a4e0-10aa8cff506f" containerName="istio-proxy" Nov 03 13:16:59.752090 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:59.752015 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="435a8d1c-24ab-4dce-a4e0-10aa8cff506f" containerName="istio-proxy" Nov 03 13:16:59.752090 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:59.752022 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d10d8b2-1143-48e1-9314-b471fa85370f" containerName="payload-processing" Nov 03 13:16:59.752090 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:59.752033 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="e03e3b1a-acce-4e8b-91e8-4b863b6d59c1" containerName="revoke-keys" Nov 03 13:16:59.756843 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:59.756823 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-11ca03ad-sv2m9" Nov 03 13:16:59.804933 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:59.804885 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-11ca03ad-sv2m9"] Nov 03 13:16:59.805745 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:59.805720 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 13:16:59.805745 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:59.805729 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:16:59.805908 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:59.805753 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 13:16:59.805908 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:59.805815 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:16:59.816013 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:59.815983 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91c23dcf-a038-403a-9d3c-14c0e52f8220-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-11ca03ad-sv2m9\" (UID: \"91c23dcf-a038-403a-9d3c-14c0e52f8220\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-11ca03ad-sv2m9" Nov 03 13:16:59.816180 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:59.816036 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91c23dcf-a038-403a-9d3c-14c0e52f8220-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-11ca03ad-sv2m9\" (UID: \"91c23dcf-a038-403a-9d3c-14c0e52f8220\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-11ca03ad-sv2m9" Nov 03 13:16:59.816180 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:59.816160 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swvtn\" (UniqueName: \"kubernetes.io/projected/91c23dcf-a038-403a-9d3c-14c0e52f8220-kube-api-access-swvtn\") pod \"maas-api-revoke-keys-e2e-ait-11ca03ad-sv2m9\" (UID: \"91c23dcf-a038-403a-9d3c-14c0e52f8220\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-11ca03ad-sv2m9" Nov 03 13:16:59.917010 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:59.916972 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91c23dcf-a038-403a-9d3c-14c0e52f8220-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-11ca03ad-sv2m9\" (UID: \"91c23dcf-a038-403a-9d3c-14c0e52f8220\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-11ca03ad-sv2m9" Nov 03 13:16:59.917218 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:59.917017 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91c23dcf-a038-403a-9d3c-14c0e52f8220-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-11ca03ad-sv2m9\" (UID: \"91c23dcf-a038-403a-9d3c-14c0e52f8220\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-11ca03ad-sv2m9" Nov 03 13:16:59.917218 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:59.917049 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swvtn\" (UniqueName: \"kubernetes.io/projected/91c23dcf-a038-403a-9d3c-14c0e52f8220-kube-api-access-swvtn\") pod \"maas-api-revoke-keys-e2e-ait-11ca03ad-sv2m9\" (UID: \"91c23dcf-a038-403a-9d3c-14c0e52f8220\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-11ca03ad-sv2m9" Nov 03 13:16:59.917722 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:59.917698 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91c23dcf-a038-403a-9d3c-14c0e52f8220-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-11ca03ad-sv2m9\" (UID: \"91c23dcf-a038-403a-9d3c-14c0e52f8220\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-11ca03ad-sv2m9" Nov 03 13:16:59.919348 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:59.919328 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91c23dcf-a038-403a-9d3c-14c0e52f8220-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-11ca03ad-sv2m9\" (UID: \"91c23dcf-a038-403a-9d3c-14c0e52f8220\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-11ca03ad-sv2m9" Nov 03 13:16:59.970439 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:16:59.970409 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swvtn\" (UniqueName: \"kubernetes.io/projected/91c23dcf-a038-403a-9d3c-14c0e52f8220-kube-api-access-swvtn\") pod \"maas-api-revoke-keys-e2e-ait-11ca03ad-sv2m9\" (UID: \"91c23dcf-a038-403a-9d3c-14c0e52f8220\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-11ca03ad-sv2m9" Nov 03 13:17:00.070310 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:00.070231 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-11ca03ad-sv2m9" Nov 03 13:17:00.115115 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:00.115079 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8a8d0bc-05fb-4a18-80af-874dd10e7f21" path="/var/lib/kubelet/pods/a8a8d0bc-05fb-4a18-80af-874dd10e7f21/volumes" Nov 03 13:17:00.440452 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:00.440414 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-11ca03ad-sv2m9"] Nov 03 13:17:00.448249 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:17:00.448220 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91c23dcf_a038_403a_9d3c_14c0e52f8220.slice/crio-c9234005129a81665a2fe3807638c9470eca763504b85bf8f3826eb0e6469776 WatchSource:0}: Error finding container c9234005129a81665a2fe3807638c9470eca763504b85bf8f3826eb0e6469776: Status 404 returned error can't find the container with id c9234005129a81665a2fe3807638c9470eca763504b85bf8f3826eb0e6469776 Nov 03 13:17:00.607146 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:00.607112 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-11ca03ad-sv2m9" event={"ID":"91c23dcf-a038-403a-9d3c-14c0e52f8220","Type":"ContainerStarted","Data":"c9234005129a81665a2fe3807638c9470eca763504b85bf8f3826eb0e6469776"} Nov 03 13:17:00.847255 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:00.847196 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj"] Nov 03 13:17:00.847870 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:00.847852 5979 cpu_manager.go:414] "containerMap: removing container" podUID="435a8d1c-24ab-4dce-a4e0-10aa8cff506f" containerName="istio-proxy" Nov 03 13:17:00.847953 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:00.847872 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="435a8d1c-24ab-4dce-a4e0-10aa8cff506f" containerName="istio-proxy" Nov 03 13:17:00.851215 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:00.851190 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj" Nov 03 13:17:00.910933 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:00.909845 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-5bb463-openshift-default-dockercfg-jldlg\"" Nov 03 13:17:00.927510 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:00.926451 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj"] Nov 03 13:17:00.943173 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:00.935865 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6729077-e216-4dd9-a68d-69ce28c4c103-istio-podinfo\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj\" (UID: \"a6729077-e216-4dd9-a68d-69ce28c4c103\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj" Nov 03 13:17:00.943173 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:00.935933 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jth4k\" (UniqueName: \"kubernetes.io/projected/a6729077-e216-4dd9-a68d-69ce28c4c103-kube-api-access-jth4k\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj\" (UID: \"a6729077-e216-4dd9-a68d-69ce28c4c103\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj" Nov 03 13:17:00.943173 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:00.935998 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6729077-e216-4dd9-a68d-69ce28c4c103-workload-socket\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj\" (UID: \"a6729077-e216-4dd9-a68d-69ce28c4c103\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj" Nov 03 13:17:00.943173 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:00.936021 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6729077-e216-4dd9-a68d-69ce28c4c103-istio-token\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj\" (UID: \"a6729077-e216-4dd9-a68d-69ce28c4c103\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj" Nov 03 13:17:00.943173 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:00.936076 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6729077-e216-4dd9-a68d-69ce28c4c103-istio-data\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj\" (UID: \"a6729077-e216-4dd9-a68d-69ce28c4c103\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj" Nov 03 13:17:00.943173 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:00.936118 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6729077-e216-4dd9-a68d-69ce28c4c103-istiod-ca-cert\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj\" (UID: \"a6729077-e216-4dd9-a68d-69ce28c4c103\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj" Nov 03 13:17:00.943173 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:00.936147 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6729077-e216-4dd9-a68d-69ce28c4c103-credential-socket\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj\" (UID: \"a6729077-e216-4dd9-a68d-69ce28c4c103\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj" Nov 03 13:17:00.943173 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:00.936289 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6729077-e216-4dd9-a68d-69ce28c4c103-istio-envoy\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj\" (UID: \"a6729077-e216-4dd9-a68d-69ce28c4c103\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj" Nov 03 13:17:00.943173 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:00.936320 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6729077-e216-4dd9-a68d-69ce28c4c103-workload-certs\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj\" (UID: \"a6729077-e216-4dd9-a68d-69ce28c4c103\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj" Nov 03 13:17:01.037803 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:01.037717 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6729077-e216-4dd9-a68d-69ce28c4c103-istio-envoy\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj\" (UID: \"a6729077-e216-4dd9-a68d-69ce28c4c103\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj" Nov 03 13:17:01.037803 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:01.037755 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6729077-e216-4dd9-a68d-69ce28c4c103-workload-certs\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj\" (UID: \"a6729077-e216-4dd9-a68d-69ce28c4c103\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj" Nov 03 13:17:01.037803 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:01.037784 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6729077-e216-4dd9-a68d-69ce28c4c103-istio-podinfo\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj\" (UID: \"a6729077-e216-4dd9-a68d-69ce28c4c103\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj" Nov 03 13:17:01.038073 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:01.037807 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jth4k\" (UniqueName: \"kubernetes.io/projected/a6729077-e216-4dd9-a68d-69ce28c4c103-kube-api-access-jth4k\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj\" (UID: \"a6729077-e216-4dd9-a68d-69ce28c4c103\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj" Nov 03 13:17:01.038073 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:01.037851 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6729077-e216-4dd9-a68d-69ce28c4c103-workload-socket\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj\" (UID: \"a6729077-e216-4dd9-a68d-69ce28c4c103\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj" Nov 03 13:17:01.038073 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:01.037866 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6729077-e216-4dd9-a68d-69ce28c4c103-istio-token\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj\" (UID: \"a6729077-e216-4dd9-a68d-69ce28c4c103\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj" Nov 03 13:17:01.038073 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:01.037894 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6729077-e216-4dd9-a68d-69ce28c4c103-istio-data\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj\" (UID: \"a6729077-e216-4dd9-a68d-69ce28c4c103\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj" Nov 03 13:17:01.038073 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:01.037932 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6729077-e216-4dd9-a68d-69ce28c4c103-istiod-ca-cert\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj\" (UID: \"a6729077-e216-4dd9-a68d-69ce28c4c103\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj" Nov 03 13:17:01.038073 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:01.037958 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6729077-e216-4dd9-a68d-69ce28c4c103-credential-socket\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj\" (UID: \"a6729077-e216-4dd9-a68d-69ce28c4c103\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj" Nov 03 13:17:01.038370 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:01.038244 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6729077-e216-4dd9-a68d-69ce28c4c103-workload-certs\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj\" (UID: \"a6729077-e216-4dd9-a68d-69ce28c4c103\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj" Nov 03 13:17:01.038370 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:01.038305 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6729077-e216-4dd9-a68d-69ce28c4c103-istio-data\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj\" (UID: \"a6729077-e216-4dd9-a68d-69ce28c4c103\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj" Nov 03 13:17:01.038370 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:01.038356 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6729077-e216-4dd9-a68d-69ce28c4c103-workload-socket\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj\" (UID: \"a6729077-e216-4dd9-a68d-69ce28c4c103\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj" Nov 03 13:17:01.038523 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:01.038385 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6729077-e216-4dd9-a68d-69ce28c4c103-credential-socket\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj\" (UID: \"a6729077-e216-4dd9-a68d-69ce28c4c103\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj" Nov 03 13:17:01.038791 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:01.038772 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6729077-e216-4dd9-a68d-69ce28c4c103-istiod-ca-cert\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj\" (UID: \"a6729077-e216-4dd9-a68d-69ce28c4c103\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj" Nov 03 13:17:01.040011 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:01.039992 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6729077-e216-4dd9-a68d-69ce28c4c103-istio-envoy\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj\" (UID: \"a6729077-e216-4dd9-a68d-69ce28c4c103\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj" Nov 03 13:17:01.040515 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:01.040487 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6729077-e216-4dd9-a68d-69ce28c4c103-istio-podinfo\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj\" (UID: \"a6729077-e216-4dd9-a68d-69ce28c4c103\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj" Nov 03 13:17:01.094599 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:01.094565 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6729077-e216-4dd9-a68d-69ce28c4c103-istio-token\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj\" (UID: \"a6729077-e216-4dd9-a68d-69ce28c4c103\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj" Nov 03 13:17:01.095431 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:01.095402 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jth4k\" (UniqueName: \"kubernetes.io/projected/a6729077-e216-4dd9-a68d-69ce28c4c103-kube-api-access-jth4k\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj\" (UID: \"a6729077-e216-4dd9-a68d-69ce28c4c103\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj" Nov 03 13:17:01.164021 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:01.163984 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj" Nov 03 13:17:01.590651 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:01.590620 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj"] Nov 03 13:17:01.598236 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:17:01.598200 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6729077_e216_4dd9_a68d_69ce28c4c103.slice/crio-2721fbe30d7f870d13430df63a0faa2058aaa383947e38c08e2cfa944c2d946f WatchSource:0}: Error finding container 2721fbe30d7f870d13430df63a0faa2058aaa383947e38c08e2cfa944c2d946f: Status 404 returned error can't find the container with id 2721fbe30d7f870d13430df63a0faa2058aaa383947e38c08e2cfa944c2d946f Nov 03 13:17:01.600851 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:01.600808 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:17:01.600971 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:01.600894 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:17:01.600971 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:01.600934 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:17:01.613963 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:01.613930 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj" event={"ID":"a6729077-e216-4dd9-a68d-69ce28c4c103","Type":"ContainerStarted","Data":"2721fbe30d7f870d13430df63a0faa2058aaa383947e38c08e2cfa944c2d946f"} Nov 03 13:17:01.615601 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:01.615580 5979 generic.go:356] "Generic (PLEG): container finished" podID="91c23dcf-a038-403a-9d3c-14c0e52f8220" containerID="e36e7c73e854b5db34ac463164d4436736bdb11672c45c67f7f6a7f90b9359b0" exitCode=0 Nov 03 13:17:01.615714 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:01.615679 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-11ca03ad-sv2m9" event={"ID":"91c23dcf-a038-403a-9d3c-14c0e52f8220","Type":"ContainerDied","Data":"e36e7c73e854b5db34ac463164d4436736bdb11672c45c67f7f6a7f90b9359b0"} Nov 03 13:17:01.761309 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:01.761230 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b"] Nov 03 13:17:01.765092 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:01.765071 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b" Nov 03 13:17:01.816776 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:01.816737 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-3c5dbc-openshift-default-dockercfg-hfbq4\"" Nov 03 13:17:01.828046 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:01.828007 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b"] Nov 03 13:17:01.861728 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:01.861674 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-workload-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b\" (UID: \"7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b" Nov 03 13:17:01.861728 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:01.861726 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-istiod-ca-cert\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b\" (UID: \"7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b" Nov 03 13:17:01.861967 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:01.861858 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8nld\" (UniqueName: \"kubernetes.io/projected/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-kube-api-access-q8nld\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b\" (UID: \"7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b" Nov 03 13:17:01.861967 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:01.861898 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-credential-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b\" (UID: \"7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b" Nov 03 13:17:01.862061 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:01.862032 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-istio-podinfo\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b\" (UID: \"7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b" Nov 03 13:17:01.862100 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:01.862078 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-workload-certs\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b\" (UID: \"7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b" Nov 03 13:17:01.862142 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:01.862109 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-istio-envoy\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b\" (UID: \"7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b" Nov 03 13:17:01.862180 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:01.862158 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-istio-data\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b\" (UID: \"7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b" Nov 03 13:17:01.862282 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:01.862262 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-istio-token\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b\" (UID: \"7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b" Nov 03 13:17:01.963754 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:01.963719 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-workload-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b\" (UID: \"7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b" Nov 03 13:17:01.963754 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:01.963756 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-istiod-ca-cert\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b\" (UID: \"7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b" Nov 03 13:17:01.963986 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:01.963789 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8nld\" (UniqueName: \"kubernetes.io/projected/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-kube-api-access-q8nld\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b\" (UID: \"7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b" Nov 03 13:17:01.963986 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:01.963814 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-credential-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b\" (UID: \"7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b" Nov 03 13:17:01.964100 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:01.963977 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-istio-podinfo\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b\" (UID: \"7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b" Nov 03 13:17:01.964100 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:01.964028 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-workload-certs\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b\" (UID: \"7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b" Nov 03 13:17:01.964100 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:01.964060 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-istio-envoy\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b\" (UID: \"7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b" Nov 03 13:17:01.964100 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:01.964086 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-istio-data\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b\" (UID: \"7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b" Nov 03 13:17:01.964297 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:01.964149 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-istio-token\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b\" (UID: \"7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b" Nov 03 13:17:01.964297 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:01.964158 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-workload-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b\" (UID: \"7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b" Nov 03 13:17:01.964297 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:01.964233 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-credential-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b\" (UID: \"7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b" Nov 03 13:17:01.964436 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:01.964412 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-workload-certs\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b\" (UID: \"7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b" Nov 03 13:17:01.964569 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:01.964553 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-istiod-ca-cert\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b\" (UID: \"7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b" Nov 03 13:17:01.964648 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:01.964631 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-istio-data\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b\" (UID: \"7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b" Nov 03 13:17:01.966753 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:01.966727 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-istio-envoy\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b\" (UID: \"7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b" Nov 03 13:17:01.966866 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:01.966756 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-istio-podinfo\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b\" (UID: \"7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b" Nov 03 13:17:02.017296 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:02.017214 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-istio-token\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b\" (UID: \"7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b" Nov 03 13:17:02.017452 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:02.017369 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8nld\" (UniqueName: \"kubernetes.io/projected/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-kube-api-access-q8nld\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b\" (UID: \"7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b" Nov 03 13:17:02.077868 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:02.077829 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b" Nov 03 13:17:02.516286 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:02.516254 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b"] Nov 03 13:17:02.523988 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:17:02.523963 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7073d2c6_f0fd_4ed0_b98b_ca65ebddbf48.slice/crio-892c4273cc12998ec492ffef0b8802e882f74cd75ff404d20d83bf5183a96b98 WatchSource:0}: Error finding container 892c4273cc12998ec492ffef0b8802e882f74cd75ff404d20d83bf5183a96b98: Status 404 returned error can't find the container with id 892c4273cc12998ec492ffef0b8802e882f74cd75ff404d20d83bf5183a96b98 Nov 03 13:17:02.526297 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:02.526259 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:17:02.526392 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:02.526343 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:17:02.526465 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:02.526388 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:17:02.624229 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:02.624178 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b" event={"ID":"7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48","Type":"ContainerStarted","Data":"3b0bb741dc94ba17beaa92dee8f2d1ae54bb513ae3a6dbe9734b2ee5d01627a9"} Nov 03 13:17:02.624619 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:02.624251 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b" event={"ID":"7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48","Type":"ContainerStarted","Data":"892c4273cc12998ec492ffef0b8802e882f74cd75ff404d20d83bf5183a96b98"} Nov 03 13:17:02.626116 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:02.626089 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj" event={"ID":"a6729077-e216-4dd9-a68d-69ce28c4c103","Type":"ContainerStarted","Data":"4523c2499303c3864c2b430ca1044c2ffec208f7c8d94ebf83c16da3cb6e5c5e"} Nov 03 13:17:02.734670 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:02.734620 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b" podStartSLOduration=1.734605965 podStartE2EDuration="1.734605965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:17:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:17:02.733094054 +0000 UTC m=+704.743782296" watchObservedRunningTime="2026-11-03 13:17:02.734605965 +0000 UTC m=+704.745294206" Nov 03 13:17:02.768523 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:02.768423 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-11ca03ad-sv2m9" Nov 03 13:17:02.843855 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:02.843785 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj" podStartSLOduration=2.843767938 podStartE2EDuration="2.843767938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:17:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:17:02.842934563 +0000 UTC m=+704.853622805" watchObservedRunningTime="2026-11-03 13:17:02.843767938 +0000 UTC m=+704.854456182" Nov 03 13:17:02.875439 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:02.875402 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91c23dcf-a038-403a-9d3c-14c0e52f8220-kube-api-access-swvtn\" (UniqueName: \"kubernetes.io/projected/91c23dcf-a038-403a-9d3c-14c0e52f8220-kube-api-access-swvtn\") pod \"91c23dcf-a038-403a-9d3c-14c0e52f8220\" (UID: \"91c23dcf-a038-403a-9d3c-14c0e52f8220\") " Nov 03 13:17:02.875669 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:02.875491 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91c23dcf-a038-403a-9d3c-14c0e52f8220-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91c23dcf-a038-403a-9d3c-14c0e52f8220-maas-api-xks-ca\") pod \"91c23dcf-a038-403a-9d3c-14c0e52f8220\" (UID: \"91c23dcf-a038-403a-9d3c-14c0e52f8220\") " Nov 03 13:17:02.875669 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:02.875567 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91c23dcf-a038-403a-9d3c-14c0e52f8220-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91c23dcf-a038-403a-9d3c-14c0e52f8220-maas-api-openshift-ca\") pod \"91c23dcf-a038-403a-9d3c-14c0e52f8220\" (UID: \"91c23dcf-a038-403a-9d3c-14c0e52f8220\") " Nov 03 13:17:02.876311 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:02.876279 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91c23dcf-a038-403a-9d3c-14c0e52f8220-maas-api-openshift-ca" pod "91c23dcf-a038-403a-9d3c-14c0e52f8220" (UID: "91c23dcf-a038-403a-9d3c-14c0e52f8220"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:17:02.878778 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:02.878744 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91c23dcf-a038-403a-9d3c-14c0e52f8220-maas-api-xks-ca" pod "91c23dcf-a038-403a-9d3c-14c0e52f8220" (UID: "91c23dcf-a038-403a-9d3c-14c0e52f8220"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:17:02.879607 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:02.879579 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91c23dcf-a038-403a-9d3c-14c0e52f8220-kube-api-access-swvtn" pod "91c23dcf-a038-403a-9d3c-14c0e52f8220" (UID: "91c23dcf-a038-403a-9d3c-14c0e52f8220"). InnerVolumeSpecName "kube-api-access-swvtn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:17:02.984501 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:02.976924 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91c23dcf-a038-403a-9d3c-14c0e52f8220-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:17:02.984501 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:02.976964 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91c23dcf-a038-403a-9d3c-14c0e52f8220-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:17:02.984501 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:02.976981 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swvtn\" (UniqueName: \"kubernetes.io/projected/91c23dcf-a038-403a-9d3c-14c0e52f8220-kube-api-access-swvtn\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:17:03.078117 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:03.078031 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b" Nov 03 13:17:03.083085 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:03.083060 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b" Nov 03 13:17:03.164669 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:03.164634 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj" Nov 03 13:17:03.169778 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:03.169751 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj" Nov 03 13:17:03.632428 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:03.632402 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-11ca03ad-sv2m9" Nov 03 13:17:03.632916 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:03.632445 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-11ca03ad-sv2m9" event={"ID":"91c23dcf-a038-403a-9d3c-14c0e52f8220","Type":"ContainerDied","Data":"c9234005129a81665a2fe3807638c9470eca763504b85bf8f3826eb0e6469776"} Nov 03 13:17:03.632916 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:03.632521 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9234005129a81665a2fe3807638c9470eca763504b85bf8f3826eb0e6469776" Nov 03 13:17:03.632916 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:03.632741 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b" Nov 03 13:17:03.633104 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:03.633087 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj" Nov 03 13:17:03.633907 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:03.633884 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b" Nov 03 13:17:03.634043 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:03.634023 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj" Nov 03 13:17:04.431242 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:04.431206 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a7a9081d-5d999b8bd7-drz5x"] Nov 03 13:17:07.233172 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:17:07.233134 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:17:09.432257 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:09.432192 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-a7a9081d-5d999b8bd7-drz5x" podUID="4ebdf83e-43fb-4a83-a9c8-ddadd6dfd982" containerName="payload-pre-processing" containerID="cri-o://77e0fe6595f0af39fae24df8f57cfba25ca44fd25de99a598ea80da590cee819" gracePeriod=25 Nov 03 13:17:09.669751 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:09.669717 5979 generic.go:356] "Generic (PLEG): container finished" podID="4ebdf83e-43fb-4a83-a9c8-ddadd6dfd982" containerID="77e0fe6595f0af39fae24df8f57cfba25ca44fd25de99a598ea80da590cee819" exitCode=0 Nov 03 13:17:09.669870 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:09.669783 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a7a9081d-5d999b8bd7-drz5x" event={"ID":"4ebdf83e-43fb-4a83-a9c8-ddadd6dfd982","Type":"ContainerDied","Data":"77e0fe6595f0af39fae24df8f57cfba25ca44fd25de99a598ea80da590cee819"} Nov 03 13:17:09.691102 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:09.691043 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a7a9081d-5d999b8bd7-drz5x" Nov 03 13:17:09.772264 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:09.772229 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ebdf83e-43fb-4a83-a9c8-ddadd6dfd982-kube-api-access-q5878\" (UniqueName: \"kubernetes.io/projected/4ebdf83e-43fb-4a83-a9c8-ddadd6dfd982-kube-api-access-q5878\") pod \"4ebdf83e-43fb-4a83-a9c8-ddadd6dfd982\" (UID: \"4ebdf83e-43fb-4a83-a9c8-ddadd6dfd982\") " Nov 03 13:17:09.772264 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:09.772264 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ebdf83e-43fb-4a83-a9c8-ddadd6dfd982-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ebdf83e-43fb-4a83-a9c8-ddadd6dfd982-plugins-config-volume\") pod \"4ebdf83e-43fb-4a83-a9c8-ddadd6dfd982\" (UID: \"4ebdf83e-43fb-4a83-a9c8-ddadd6dfd982\") " Nov 03 13:17:09.772719 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:09.772695 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ebdf83e-43fb-4a83-a9c8-ddadd6dfd982-plugins-config-volume" pod "4ebdf83e-43fb-4a83-a9c8-ddadd6dfd982" (UID: "4ebdf83e-43fb-4a83-a9c8-ddadd6dfd982"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:17:09.774347 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:09.774327 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ebdf83e-43fb-4a83-a9c8-ddadd6dfd982-kube-api-access-q5878" pod "4ebdf83e-43fb-4a83-a9c8-ddadd6dfd982" (UID: "4ebdf83e-43fb-4a83-a9c8-ddadd6dfd982"). InnerVolumeSpecName "kube-api-access-q5878". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:17:09.874068 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:09.874034 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5878\" (UniqueName: \"kubernetes.io/projected/4ebdf83e-43fb-4a83-a9c8-ddadd6dfd982-kube-api-access-q5878\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:17:09.874068 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:09.874062 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ebdf83e-43fb-4a83-a9c8-ddadd6dfd982-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:17:10.676116 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:10.676084 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a7a9081d-5d999b8bd7-drz5x" Nov 03 13:17:10.676565 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:10.676083 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a7a9081d-5d999b8bd7-drz5x" event={"ID":"4ebdf83e-43fb-4a83-a9c8-ddadd6dfd982","Type":"ContainerDied","Data":"4deace760efcbeef3b4df02786c58d6f7e138f71e250ebf8e1f11b61b075e2cd"} Nov 03 13:17:10.676565 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:10.676218 5979 scope.go:122] "RemoveContainer" containerID="77e0fe6595f0af39fae24df8f57cfba25ca44fd25de99a598ea80da590cee819" Nov 03 13:17:10.830054 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:10.830019 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a7a9081d-5d999b8bd7-drz5x"] Nov 03 13:17:10.876656 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:10.876620 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a7a9081d-5d999b8bd7-drz5x"] Nov 03 13:17:11.974951 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:11.974919 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:17:12.114879 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:12.114843 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ebdf83e-43fb-4a83-a9c8-ddadd6dfd982" path="/var/lib/kubelet/pods/4ebdf83e-43fb-4a83-a9c8-ddadd6dfd982/volumes" Nov 03 13:17:14.055063 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:14.055027 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a7a9081d-h6kvw"] Nov 03 13:17:14.058889 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:14.058864 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a7a9081d-h6kvw"] Nov 03 13:17:14.117148 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:14.117109 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e03e3b1a-acce-4e8b-91e8-4b863b6d59c1" path="/var/lib/kubelet/pods/e03e3b1a-acce-4e8b-91e8-4b863b6d59c1/volumes" Nov 03 13:17:15.563920 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:15.563875 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m"] Nov 03 13:17:15.564348 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:15.564334 5979 cpu_manager.go:414] "containerMap: removing container" podUID="91c23dcf-a038-403a-9d3c-14c0e52f8220" containerName="revoke-keys" Nov 03 13:17:15.564400 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:15.564349 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91c23dcf-a038-403a-9d3c-14c0e52f8220" containerName="revoke-keys" Nov 03 13:17:15.564400 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:15.564373 5979 cpu_manager.go:414] "containerMap: removing container" podUID="4ebdf83e-43fb-4a83-a9c8-ddadd6dfd982" containerName="payload-pre-processing" Nov 03 13:17:15.564400 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:15.564378 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ebdf83e-43fb-4a83-a9c8-ddadd6dfd982" containerName="payload-pre-processing" Nov 03 13:17:15.564520 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:15.564462 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="91c23dcf-a038-403a-9d3c-14c0e52f8220" containerName="revoke-keys" Nov 03 13:17:15.564520 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:15.564486 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ebdf83e-43fb-4a83-a9c8-ddadd6dfd982" containerName="payload-pre-processing" Nov 03 13:17:15.569300 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:15.569280 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m" Nov 03 13:17:15.651516 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:15.641359 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m"] Nov 03 13:17:15.651764 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:15.643877 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8a57a81-8b6f-4049-9c5a-9884e968fee7-istio-podinfo\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m\" (UID: \"d8a57a81-8b6f-4049-9c5a-9884e968fee7\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m" Nov 03 13:17:15.651884 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:15.651810 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8a57a81-8b6f-4049-9c5a-9884e968fee7-workload-certs\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m\" (UID: \"d8a57a81-8b6f-4049-9c5a-9884e968fee7\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m" Nov 03 13:17:15.651884 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:15.651843 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86mhx\" (UniqueName: \"kubernetes.io/projected/d8a57a81-8b6f-4049-9c5a-9884e968fee7-kube-api-access-86mhx\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m\" (UID: \"d8a57a81-8b6f-4049-9c5a-9884e968fee7\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m" Nov 03 13:17:15.652012 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:15.651914 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a57a81-8b6f-4049-9c5a-9884e968fee7-credential-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m\" (UID: \"d8a57a81-8b6f-4049-9c5a-9884e968fee7\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m" Nov 03 13:17:15.652012 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:15.651943 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8a57a81-8b6f-4049-9c5a-9884e968fee7-istio-token\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m\" (UID: \"d8a57a81-8b6f-4049-9c5a-9884e968fee7\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m" Nov 03 13:17:15.652119 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:15.652101 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8a57a81-8b6f-4049-9c5a-9884e968fee7-istiod-ca-cert\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m\" (UID: \"d8a57a81-8b6f-4049-9c5a-9884e968fee7\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m" Nov 03 13:17:15.652228 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:15.652197 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8a57a81-8b6f-4049-9c5a-9884e968fee7-istio-envoy\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m\" (UID: \"d8a57a81-8b6f-4049-9c5a-9884e968fee7\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m" Nov 03 13:17:15.652463 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:15.652441 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8a57a81-8b6f-4049-9c5a-9884e968fee7-istio-data\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m\" (UID: \"d8a57a81-8b6f-4049-9c5a-9884e968fee7\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m" Nov 03 13:17:15.652634 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:15.652609 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a57a81-8b6f-4049-9c5a-9884e968fee7-workload-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m\" (UID: \"d8a57a81-8b6f-4049-9c5a-9884e968fee7\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m" Nov 03 13:17:15.668963 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:15.668930 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-34b13c-openshift-default-dockercfg-kf4m8\"" Nov 03 13:17:15.753697 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:15.753663 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8a57a81-8b6f-4049-9c5a-9884e968fee7-istio-envoy\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m\" (UID: \"d8a57a81-8b6f-4049-9c5a-9884e968fee7\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m" Nov 03 13:17:15.753879 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:15.753714 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8a57a81-8b6f-4049-9c5a-9884e968fee7-istio-data\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m\" (UID: \"d8a57a81-8b6f-4049-9c5a-9884e968fee7\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m" Nov 03 13:17:15.753879 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:15.753746 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a57a81-8b6f-4049-9c5a-9884e968fee7-workload-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m\" (UID: \"d8a57a81-8b6f-4049-9c5a-9884e968fee7\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m" Nov 03 13:17:15.753879 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:15.753767 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8a57a81-8b6f-4049-9c5a-9884e968fee7-istio-podinfo\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m\" (UID: \"d8a57a81-8b6f-4049-9c5a-9884e968fee7\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m" Nov 03 13:17:15.753879 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:15.753785 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8a57a81-8b6f-4049-9c5a-9884e968fee7-workload-certs\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m\" (UID: \"d8a57a81-8b6f-4049-9c5a-9884e968fee7\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m" Nov 03 13:17:15.753879 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:15.753809 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86mhx\" (UniqueName: \"kubernetes.io/projected/d8a57a81-8b6f-4049-9c5a-9884e968fee7-kube-api-access-86mhx\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m\" (UID: \"d8a57a81-8b6f-4049-9c5a-9884e968fee7\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m" Nov 03 13:17:15.753879 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:15.753859 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a57a81-8b6f-4049-9c5a-9884e968fee7-credential-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m\" (UID: \"d8a57a81-8b6f-4049-9c5a-9884e968fee7\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m" Nov 03 13:17:15.754171 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:15.753887 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8a57a81-8b6f-4049-9c5a-9884e968fee7-istio-token\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m\" (UID: \"d8a57a81-8b6f-4049-9c5a-9884e968fee7\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m" Nov 03 13:17:15.754171 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:15.753998 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8a57a81-8b6f-4049-9c5a-9884e968fee7-istiod-ca-cert\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m\" (UID: \"d8a57a81-8b6f-4049-9c5a-9884e968fee7\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m" Nov 03 13:17:15.754278 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:15.754171 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8a57a81-8b6f-4049-9c5a-9884e968fee7-istio-data\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m\" (UID: \"d8a57a81-8b6f-4049-9c5a-9884e968fee7\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m" Nov 03 13:17:15.754278 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:15.754216 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a57a81-8b6f-4049-9c5a-9884e968fee7-workload-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m\" (UID: \"d8a57a81-8b6f-4049-9c5a-9884e968fee7\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m" Nov 03 13:17:15.754492 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:15.754438 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a57a81-8b6f-4049-9c5a-9884e968fee7-credential-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m\" (UID: \"d8a57a81-8b6f-4049-9c5a-9884e968fee7\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m" Nov 03 13:17:15.754708 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:15.754686 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8a57a81-8b6f-4049-9c5a-9884e968fee7-istiod-ca-cert\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m\" (UID: \"d8a57a81-8b6f-4049-9c5a-9884e968fee7\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m" Nov 03 13:17:15.754778 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:15.754688 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8a57a81-8b6f-4049-9c5a-9884e968fee7-workload-certs\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m\" (UID: \"d8a57a81-8b6f-4049-9c5a-9884e968fee7\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m" Nov 03 13:17:15.756092 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:15.756075 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8a57a81-8b6f-4049-9c5a-9884e968fee7-istio-envoy\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m\" (UID: \"d8a57a81-8b6f-4049-9c5a-9884e968fee7\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m" Nov 03 13:17:15.756324 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:15.756308 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8a57a81-8b6f-4049-9c5a-9884e968fee7-istio-podinfo\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m\" (UID: \"d8a57a81-8b6f-4049-9c5a-9884e968fee7\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m" Nov 03 13:17:15.848573 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:15.838014 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86mhx\" (UniqueName: \"kubernetes.io/projected/d8a57a81-8b6f-4049-9c5a-9884e968fee7-kube-api-access-86mhx\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m\" (UID: \"d8a57a81-8b6f-4049-9c5a-9884e968fee7\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m" Nov 03 13:17:15.848573 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:15.840656 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8a57a81-8b6f-4049-9c5a-9884e968fee7-istio-token\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m\" (UID: \"d8a57a81-8b6f-4049-9c5a-9884e968fee7\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m" Nov 03 13:17:15.882314 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:15.882271 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m" Nov 03 13:17:16.309201 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:16.309164 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m"] Nov 03 13:17:16.316210 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:17:16.316186 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8a57a81_8b6f_4049_9c5a_9884e968fee7.slice/crio-039d35a97d82e80edab224d8171c22432978580f41e7c2d095a43cf6cbafa820 WatchSource:0}: Error finding container 039d35a97d82e80edab224d8171c22432978580f41e7c2d095a43cf6cbafa820: Status 404 returned error can't find the container with id 039d35a97d82e80edab224d8171c22432978580f41e7c2d095a43cf6cbafa820 Nov 03 13:17:16.318463 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:16.318434 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:17:16.318543 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:16.318510 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:17:16.318584 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:16.318540 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:17:16.724170 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:16.724121 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m" event={"ID":"d8a57a81-8b6f-4049-9c5a-9884e968fee7","Type":"ContainerStarted","Data":"5b71848d06a9680f1862c89aedcf2da53f592a49762b808ecb20a5c7195c4795"} Nov 03 13:17:16.724645 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:16.724187 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m" event={"ID":"d8a57a81-8b6f-4049-9c5a-9884e968fee7","Type":"ContainerStarted","Data":"039d35a97d82e80edab224d8171c22432978580f41e7c2d095a43cf6cbafa820"} Nov 03 13:17:16.838806 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:16.838752 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m" podStartSLOduration=1.838738417 podStartE2EDuration="1.838738417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:17:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:17:16.837456798 +0000 UTC m=+718.848145041" watchObservedRunningTime="2026-11-03 13:17:16.838738417 +0000 UTC m=+718.849426658" Nov 03 13:17:16.882861 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:16.882832 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m" Nov 03 13:17:17.887995 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:17.887966 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m" Nov 03 13:17:18.735135 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:18.735104 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m" Nov 03 13:17:18.736014 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:18.735994 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m" Nov 03 13:17:20.516733 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:20.516702 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-11ca03ad-6cb94c6c4c-4mf5l"] Nov 03 13:17:25.517186 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:25.517075 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-11ca03ad-6cb94c6c4c-4mf5l" podUID="01e6f156-634a-4d24-9db0-c5f698c8deca" containerName="payload-pre-processing" containerID="cri-o://295b3b803b0f5fa9b75257eff4ed7415f1f5d4903570220a47694717d6490fdb" gracePeriod=25 Nov 03 13:17:25.538223 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:25.538060 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:17:25.804705 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:25.804675 5979 generic.go:356] "Generic (PLEG): container finished" podID="01e6f156-634a-4d24-9db0-c5f698c8deca" containerID="295b3b803b0f5fa9b75257eff4ed7415f1f5d4903570220a47694717d6490fdb" exitCode=0 Nov 03 13:17:25.804862 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:25.804757 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-11ca03ad-6cb94c6c4c-4mf5l" event={"ID":"01e6f156-634a-4d24-9db0-c5f698c8deca","Type":"ContainerDied","Data":"295b3b803b0f5fa9b75257eff4ed7415f1f5d4903570220a47694717d6490fdb"} Nov 03 13:17:25.804862 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:25.804826 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-11ca03ad-6cb94c6c4c-4mf5l" event={"ID":"01e6f156-634a-4d24-9db0-c5f698c8deca","Type":"ContainerDied","Data":"587c6d7b4a8bb67194dfc56cb84fb6f229ff56e97f20bcd9c8d621a0e6b0280b"} Nov 03 13:17:25.804862 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:25.804852 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="587c6d7b4a8bb67194dfc56cb84fb6f229ff56e97f20bcd9c8d621a0e6b0280b" Nov 03 13:17:25.812503 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:25.812460 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-11ca03ad-6cb94c6c4c-4mf5l" Nov 03 13:17:25.903306 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:25.903271 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01e6f156-634a-4d24-9db0-c5f698c8deca-kube-api-access-676g5\" (UniqueName: \"kubernetes.io/projected/01e6f156-634a-4d24-9db0-c5f698c8deca-kube-api-access-676g5\") pod \"01e6f156-634a-4d24-9db0-c5f698c8deca\" (UID: \"01e6f156-634a-4d24-9db0-c5f698c8deca\") " Nov 03 13:17:25.903533 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:25.903512 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01e6f156-634a-4d24-9db0-c5f698c8deca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01e6f156-634a-4d24-9db0-c5f698c8deca-plugins-config-volume\") pod \"01e6f156-634a-4d24-9db0-c5f698c8deca\" (UID: \"01e6f156-634a-4d24-9db0-c5f698c8deca\") " Nov 03 13:17:25.903980 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:25.903948 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01e6f156-634a-4d24-9db0-c5f698c8deca-plugins-config-volume" pod "01e6f156-634a-4d24-9db0-c5f698c8deca" (UID: "01e6f156-634a-4d24-9db0-c5f698c8deca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:17:25.905484 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:25.905445 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01e6f156-634a-4d24-9db0-c5f698c8deca-kube-api-access-676g5" pod "01e6f156-634a-4d24-9db0-c5f698c8deca" (UID: "01e6f156-634a-4d24-9db0-c5f698c8deca"). InnerVolumeSpecName "kube-api-access-676g5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:17:26.005280 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:26.005247 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01e6f156-634a-4d24-9db0-c5f698c8deca-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:17:26.005280 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:26.005278 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-676g5\" (UniqueName: \"kubernetes.io/projected/01e6f156-634a-4d24-9db0-c5f698c8deca-kube-api-access-676g5\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:17:26.609655 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:26.609616 5979 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:17:26.610063 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:26.609683 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4" podUID="01e6f154-38b8-47bf-8f9b-436fa4cf1436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:17:26.810902 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:26.810867 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-11ca03ad-6cb94c6c4c-4mf5l" Nov 03 13:17:26.976038 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:26.975998 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-11ca03ad-6cb94c6c4c-4mf5l"] Nov 03 13:17:27.021256 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:27.021220 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-11ca03ad-6cb94c6c4c-4mf5l"] Nov 03 13:17:28.115628 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:28.115595 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01e6f156-634a-4d24-9db0-c5f698c8deca" path="/var/lib/kubelet/pods/01e6f156-634a-4d24-9db0-c5f698c8deca/volumes" Nov 03 13:17:28.823644 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:28.823560 5979 generic.go:356] "Generic (PLEG): container finished" podID="01e6f154-38b8-47bf-8f9b-436fa4cf1436" containerID="9a1ed4e0b255d0d777095b119f9eebff6f036c31428b8681be748dbb940a5d1a" exitCode=0 Nov 03 13:17:28.823847 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:28.823632 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4" event={"ID":"01e6f154-38b8-47bf-8f9b-436fa4cf1436","Type":"ContainerDied","Data":"9a1ed4e0b255d0d777095b119f9eebff6f036c31428b8681be748dbb940a5d1a"} Nov 03 13:17:28.824059 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:28.824032 5979 scope.go:122] "RemoveContainer" containerID="9a1ed4e0b255d0d777095b119f9eebff6f036c31428b8681be748dbb940a5d1a" Nov 03 13:17:28.825415 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:28.825391 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:17:28.825498 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:28.825452 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:17:28.825539 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:28.825514 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:17:29.623551 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:29.619603 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-34b13c-6bb577bd6f-r6tz2"] Nov 03 13:17:29.623551 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:29.620449 5979 cpu_manager.go:414] "containerMap: removing container" podUID="01e6f156-634a-4d24-9db0-c5f698c8deca" containerName="payload-pre-processing" Nov 03 13:17:29.623551 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:29.620485 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01e6f156-634a-4d24-9db0-c5f698c8deca" containerName="payload-pre-processing" Nov 03 13:17:29.623551 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:29.620592 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="01e6f156-634a-4d24-9db0-c5f698c8deca" containerName="payload-pre-processing" Nov 03 13:17:29.628426 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:29.628407 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-34b13c-6bb577bd6f-r6tz2" Nov 03 13:17:29.675981 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:29.675939 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-34b13c-6bb577bd6f-r6tz2"] Nov 03 13:17:29.678153 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:29.678129 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-34b13c\"" Nov 03 13:17:29.678288 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:29.678198 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-34b13c-dockercfg-7tnmj\"" Nov 03 13:17:29.774820 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:29.774786 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/420ec7c1-8dae-4500-9aec-5ed3d05433ff-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-34b13c-6bb577bd6f-r6tz2\" (UID: \"420ec7c1-8dae-4500-9aec-5ed3d05433ff\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-34b13c-6bb577bd6f-r6tz2" Nov 03 13:17:29.774990 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:29.774958 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k76np\" (UniqueName: \"kubernetes.io/projected/420ec7c1-8dae-4500-9aec-5ed3d05433ff-kube-api-access-k76np\") pod \"payload-processing-e2e-worker-w1-34b13c-6bb577bd6f-r6tz2\" (UID: \"420ec7c1-8dae-4500-9aec-5ed3d05433ff\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-34b13c-6bb577bd6f-r6tz2" Nov 03 13:17:29.831550 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:29.831517 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4" event={"ID":"01e6f154-38b8-47bf-8f9b-436fa4cf1436","Type":"ContainerStarted","Data":"c5e4a2dd70e2ae178f24143eff509a5ba3b66e4d4a02725b02841013f1b4e6e2"} Nov 03 13:17:29.875989 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:29.875886 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/420ec7c1-8dae-4500-9aec-5ed3d05433ff-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-34b13c-6bb577bd6f-r6tz2\" (UID: \"420ec7c1-8dae-4500-9aec-5ed3d05433ff\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-34b13c-6bb577bd6f-r6tz2" Nov 03 13:17:29.876149 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:29.876028 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k76np\" (UniqueName: \"kubernetes.io/projected/420ec7c1-8dae-4500-9aec-5ed3d05433ff-kube-api-access-k76np\") pod \"payload-processing-e2e-worker-w1-34b13c-6bb577bd6f-r6tz2\" (UID: \"420ec7c1-8dae-4500-9aec-5ed3d05433ff\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-34b13c-6bb577bd6f-r6tz2" Nov 03 13:17:29.876615 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:29.876594 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/420ec7c1-8dae-4500-9aec-5ed3d05433ff-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-34b13c-6bb577bd6f-r6tz2\" (UID: \"420ec7c1-8dae-4500-9aec-5ed3d05433ff\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-34b13c-6bb577bd6f-r6tz2" Nov 03 13:17:29.930282 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:29.930248 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k76np\" (UniqueName: \"kubernetes.io/projected/420ec7c1-8dae-4500-9aec-5ed3d05433ff-kube-api-access-k76np\") pod \"payload-processing-e2e-worker-w1-34b13c-6bb577bd6f-r6tz2\" (UID: \"420ec7c1-8dae-4500-9aec-5ed3d05433ff\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-34b13c-6bb577bd6f-r6tz2" Nov 03 13:17:29.963495 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:29.960680 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-34b13c-6bb577bd6f-r6tz2" Nov 03 13:17:30.072268 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:30.069910 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4" Nov 03 13:17:30.072268 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:30.070035 5979 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 03 13:17:30.072268 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:30.070080 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4" podUID="01e6f154-38b8-47bf-8f9b-436fa4cf1436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 03 13:17:30.336593 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:30.336556 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-34b13c-6bb577bd6f-r6tz2"] Nov 03 13:17:30.841213 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:30.841170 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-34b13c-6bb577bd6f-r6tz2" event={"ID":"420ec7c1-8dae-4500-9aec-5ed3d05433ff","Type":"ContainerStarted","Data":"ddd30a3395444916b242909258c11f2bf4f37b87ac28222f509f8f0cbef63bd1"} Nov 03 13:17:30.841963 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:30.841251 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-34b13c-6bb577bd6f-r6tz2" event={"ID":"420ec7c1-8dae-4500-9aec-5ed3d05433ff","Type":"ContainerStarted","Data":"3db19283fb94cbc52c22aa38f9f555b8d109e9d8f882dedb894b87950fb4d7b5"} Nov 03 13:17:30.969997 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:30.969935 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-34b13c-6bb577bd6f-r6tz2" podStartSLOduration=1.9699128080000001 podStartE2EDuration="1.969912808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:17:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:17:30.948895797 +0000 UTC m=+732.959584041" watchObservedRunningTime="2026-11-03 13:17:30.969912808 +0000 UTC m=+732.980601051" Nov 03 13:17:31.052496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:31.050899 5979 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 03 13:17:31.052496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:31.050962 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4" podUID="01e6f154-38b8-47bf-8f9b-436fa4cf1436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 03 13:17:31.869373 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:31.869324 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-34b13c-6bb577bd6f-r6tz2" Nov 03 13:17:32.062493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:32.053988 5979 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 03 13:17:32.062493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:32.054054 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4" podUID="01e6f154-38b8-47bf-8f9b-436fa4cf1436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 03 13:17:33.063505 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:33.062583 5979 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 03 13:17:33.063505 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:33.062649 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4" podUID="01e6f154-38b8-47bf-8f9b-436fa4cf1436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 03 13:17:34.051495 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:34.047833 5979 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Nov 03 13:17:34.051495 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:34.047899 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4" podUID="01e6f154-38b8-47bf-8f9b-436fa4cf1436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Nov 03 13:17:35.162590 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:35.162558 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4" Nov 03 13:17:35.164460 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:35.164438 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4" Nov 03 13:17:35.171162 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:35.171139 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4" Nov 03 13:17:35.478789 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:35.478709 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-11ca03ad-sv2m9"] Nov 03 13:17:35.481325 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:35.481288 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-11ca03ad-sv2m9"] Nov 03 13:17:36.028407 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:36.028367 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb"] Nov 03 13:17:36.028698 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:36.028643 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb" podUID="ad693484-f565-418b-acbc-dd3739a1adec" containerName="istio-proxy" containerID="cri-o://87c926e28778597ea4da599eb56fe1c1b283acac8f63325aed312e2811580a09" gracePeriod=30 Nov 03 13:17:36.114139 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:36.114105 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91c23dcf-a038-403a-9d3c-14c0e52f8220" path="/var/lib/kubelet/pods/91c23dcf-a038-403a-9d3c-14c0e52f8220/volumes" Nov 03 13:17:37.241754 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:17:37.241711 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:17:40.968663 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:40.968623 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:17:41.379652 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:41.379620 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb" Nov 03 13:17:41.434567 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:41.434536 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad693484-f565-418b-acbc-dd3739a1adec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad693484-f565-418b-acbc-dd3739a1adec-istiod-ca-cert\") pod \"ad693484-f565-418b-acbc-dd3739a1adec\" (UID: \"ad693484-f565-418b-acbc-dd3739a1adec\") " Nov 03 13:17:41.434755 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:41.434643 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad693484-f565-418b-acbc-dd3739a1adec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad693484-f565-418b-acbc-dd3739a1adec-workload-socket\") pod \"ad693484-f565-418b-acbc-dd3739a1adec\" (UID: \"ad693484-f565-418b-acbc-dd3739a1adec\") " Nov 03 13:17:41.434755 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:41.434686 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad693484-f565-418b-acbc-dd3739a1adec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad693484-f565-418b-acbc-dd3739a1adec-istio-envoy\") pod \"ad693484-f565-418b-acbc-dd3739a1adec\" (UID: \"ad693484-f565-418b-acbc-dd3739a1adec\") " Nov 03 13:17:41.434755 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:41.434713 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad693484-f565-418b-acbc-dd3739a1adec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad693484-f565-418b-acbc-dd3739a1adec-istio-data\") pod \"ad693484-f565-418b-acbc-dd3739a1adec\" (UID: \"ad693484-f565-418b-acbc-dd3739a1adec\") " Nov 03 13:17:41.434755 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:41.434732 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad693484-f565-418b-acbc-dd3739a1adec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad693484-f565-418b-acbc-dd3739a1adec-workload-certs\") pod \"ad693484-f565-418b-acbc-dd3739a1adec\" (UID: \"ad693484-f565-418b-acbc-dd3739a1adec\") " Nov 03 13:17:41.434981 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:41.434764 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad693484-f565-418b-acbc-dd3739a1adec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad693484-f565-418b-acbc-dd3739a1adec-istio-podinfo\") pod \"ad693484-f565-418b-acbc-dd3739a1adec\" (UID: \"ad693484-f565-418b-acbc-dd3739a1adec\") " Nov 03 13:17:41.434981 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:41.434805 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad693484-f565-418b-acbc-dd3739a1adec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad693484-f565-418b-acbc-dd3739a1adec-credential-socket\") pod \"ad693484-f565-418b-acbc-dd3739a1adec\" (UID: \"ad693484-f565-418b-acbc-dd3739a1adec\") " Nov 03 13:17:41.434981 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:41.434844 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad693484-f565-418b-acbc-dd3739a1adec-istio-token\" (UniqueName: \"kubernetes.io/projected/ad693484-f565-418b-acbc-dd3739a1adec-istio-token\") pod \"ad693484-f565-418b-acbc-dd3739a1adec\" (UID: \"ad693484-f565-418b-acbc-dd3739a1adec\") " Nov 03 13:17:41.434981 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:41.434873 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad693484-f565-418b-acbc-dd3739a1adec-kube-api-access-8pwgz\" (UniqueName: \"kubernetes.io/projected/ad693484-f565-418b-acbc-dd3739a1adec-kube-api-access-8pwgz\") pod \"ad693484-f565-418b-acbc-dd3739a1adec\" (UID: \"ad693484-f565-418b-acbc-dd3739a1adec\") " Nov 03 13:17:41.435180 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:41.434972 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad693484-f565-418b-acbc-dd3739a1adec-istiod-ca-cert" pod "ad693484-f565-418b-acbc-dd3739a1adec" (UID: "ad693484-f565-418b-acbc-dd3739a1adec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:17:41.435180 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:41.435025 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad693484-f565-418b-acbc-dd3739a1adec-istio-data" pod "ad693484-f565-418b-acbc-dd3739a1adec" (UID: "ad693484-f565-418b-acbc-dd3739a1adec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:17:41.435180 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:41.435045 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad693484-f565-418b-acbc-dd3739a1adec-workload-socket" pod "ad693484-f565-418b-acbc-dd3739a1adec" (UID: "ad693484-f565-418b-acbc-dd3739a1adec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:17:41.435180 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:41.435056 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad693484-f565-418b-acbc-dd3739a1adec-workload-certs" pod "ad693484-f565-418b-acbc-dd3739a1adec" (UID: "ad693484-f565-418b-acbc-dd3739a1adec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:17:41.435387 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:41.435356 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad693484-f565-418b-acbc-dd3739a1adec-credential-socket" pod "ad693484-f565-418b-acbc-dd3739a1adec" (UID: "ad693484-f565-418b-acbc-dd3739a1adec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:17:41.435538 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:41.435461 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad693484-f565-418b-acbc-dd3739a1adec-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:17:41.435538 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:41.435508 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad693484-f565-418b-acbc-dd3739a1adec-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:17:41.435538 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:41.435526 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad693484-f565-418b-acbc-dd3739a1adec-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:17:41.435736 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:41.435542 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad693484-f565-418b-acbc-dd3739a1adec-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:17:41.435736 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:41.435556 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad693484-f565-418b-acbc-dd3739a1adec-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:17:41.437273 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:41.437126 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad693484-f565-418b-acbc-dd3739a1adec-istio-envoy" pod "ad693484-f565-418b-acbc-dd3739a1adec" (UID: "ad693484-f565-418b-acbc-dd3739a1adec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:17:41.437273 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:41.437232 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad693484-f565-418b-acbc-dd3739a1adec-kube-api-access-8pwgz" pod "ad693484-f565-418b-acbc-dd3739a1adec" (UID: "ad693484-f565-418b-acbc-dd3739a1adec"). InnerVolumeSpecName "kube-api-access-8pwgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:17:41.437445 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:41.437424 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad693484-f565-418b-acbc-dd3739a1adec-istio-token" pod "ad693484-f565-418b-acbc-dd3739a1adec" (UID: "ad693484-f565-418b-acbc-dd3739a1adec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:17:41.437572 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:41.437553 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad693484-f565-418b-acbc-dd3739a1adec-istio-podinfo" pod "ad693484-f565-418b-acbc-dd3739a1adec" (UID: "ad693484-f565-418b-acbc-dd3739a1adec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:17:41.536527 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:41.536427 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad693484-f565-418b-acbc-dd3739a1adec-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:17:41.536527 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:41.536461 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad693484-f565-418b-acbc-dd3739a1adec-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:17:41.536527 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:41.536496 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad693484-f565-418b-acbc-dd3739a1adec-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:17:41.536527 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:41.536509 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pwgz\" (UniqueName: \"kubernetes.io/projected/ad693484-f565-418b-acbc-dd3739a1adec-kube-api-access-8pwgz\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:17:41.969496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:41.963462 5979 generic.go:356] "Generic (PLEG): container finished" podID="ad693484-f565-418b-acbc-dd3739a1adec" containerID="87c926e28778597ea4da599eb56fe1c1b283acac8f63325aed312e2811580a09" exitCode=0 Nov 03 13:17:41.969496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:41.963558 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb" event={"ID":"ad693484-f565-418b-acbc-dd3739a1adec","Type":"ContainerDied","Data":"87c926e28778597ea4da599eb56fe1c1b283acac8f63325aed312e2811580a09"} Nov 03 13:17:41.969496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:41.963609 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb" event={"ID":"ad693484-f565-418b-acbc-dd3739a1adec","Type":"ContainerDied","Data":"cb2123912b6862f22fba9ef3e688551b25b83d65ec75cfc0e055299589105f8b"} Nov 03 13:17:41.969496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:41.963639 5979 scope.go:122] "RemoveContainer" containerID="87c926e28778597ea4da599eb56fe1c1b283acac8f63325aed312e2811580a09" Nov 03 13:17:41.969496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:41.963802 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb" Nov 03 13:17:41.982363 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:41.982340 5979 scope.go:122] "RemoveContainer" containerID="691aca82b8868a5d414a9d5f626f259f2e3ac04c6529c65cb66fba10f959ecbb" Nov 03 13:17:41.991080 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:41.991060 5979 scope.go:122] "RemoveContainer" containerID="87c926e28778597ea4da599eb56fe1c1b283acac8f63325aed312e2811580a09" Nov 03 13:17:41.991369 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:17:41.991349 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87c926e28778597ea4da599eb56fe1c1b283acac8f63325aed312e2811580a09\": container with ID starting with 87c926e28778597ea4da599eb56fe1c1b283acac8f63325aed312e2811580a09 not found: ID does not exist" containerID="87c926e28778597ea4da599eb56fe1c1b283acac8f63325aed312e2811580a09" Nov 03 13:17:41.991419 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:41.991377 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87c926e28778597ea4da599eb56fe1c1b283acac8f63325aed312e2811580a09"} err="failed to get container status \"87c926e28778597ea4da599eb56fe1c1b283acac8f63325aed312e2811580a09\": rpc error: code = NotFound desc = could not find container \"87c926e28778597ea4da599eb56fe1c1b283acac8f63325aed312e2811580a09\": container with ID starting with 87c926e28778597ea4da599eb56fe1c1b283acac8f63325aed312e2811580a09 not found: ID does not exist" Nov 03 13:17:41.991419 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:41.991394 5979 scope.go:122] "RemoveContainer" containerID="691aca82b8868a5d414a9d5f626f259f2e3ac04c6529c65cb66fba10f959ecbb" Nov 03 13:17:41.991681 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:17:41.991656 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"691aca82b8868a5d414a9d5f626f259f2e3ac04c6529c65cb66fba10f959ecbb\": container with ID starting with 691aca82b8868a5d414a9d5f626f259f2e3ac04c6529c65cb66fba10f959ecbb not found: ID does not exist" containerID="691aca82b8868a5d414a9d5f626f259f2e3ac04c6529c65cb66fba10f959ecbb" Nov 03 13:17:41.991725 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:41.991692 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"691aca82b8868a5d414a9d5f626f259f2e3ac04c6529c65cb66fba10f959ecbb"} err="failed to get container status \"691aca82b8868a5d414a9d5f626f259f2e3ac04c6529c65cb66fba10f959ecbb\": rpc error: code = NotFound desc = could not find container \"691aca82b8868a5d414a9d5f626f259f2e3ac04c6529c65cb66fba10f959ecbb\": container with ID starting with 691aca82b8868a5d414a9d5f626f259f2e3ac04c6529c65cb66fba10f959ecbb not found: ID does not exist" Nov 03 13:17:42.129430 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:42.129341 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb"] Nov 03 13:17:42.137686 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:42.137659 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-11ca03ad-openshift-default-d8456d9b9-jfmnb"] Nov 03 13:17:42.892570 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:42.892539 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-34b13c-6bb577bd6f-r6tz2" Nov 03 13:17:42.974002 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:42.973974 5979 generic.go:356] "Generic (PLEG): container finished" podID="7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48" containerID="3b0bb741dc94ba17beaa92dee8f2d1ae54bb513ae3a6dbe9734b2ee5d01627a9" exitCode=0 Nov 03 13:17:42.974434 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:42.974051 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b" event={"ID":"7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48","Type":"ContainerDied","Data":"3b0bb741dc94ba17beaa92dee8f2d1ae54bb513ae3a6dbe9734b2ee5d01627a9"} Nov 03 13:17:42.974434 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:42.974410 5979 scope.go:122] "RemoveContainer" containerID="3b0bb741dc94ba17beaa92dee8f2d1ae54bb513ae3a6dbe9734b2ee5d01627a9" Nov 03 13:17:42.975798 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:42.975774 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:17:42.975870 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:42.975842 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:17:42.975919 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:42.975872 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:17:43.998060 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:43.998028 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b" event={"ID":"7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48","Type":"ContainerStarted","Data":"3a45e1eb1ce8fc3d09630da3da69fcedea2efa4460dfac8c37d65d275ec2f47b"} Nov 03 13:17:44.085494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:44.084881 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b" Nov 03 13:17:44.085494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:44.085009 5979 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 03 13:17:44.085494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:44.085051 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b" podUID="7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 03 13:17:44.114654 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:44.114623 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad693484-f565-418b-acbc-dd3739a1adec" path="/var/lib/kubelet/pods/ad693484-f565-418b-acbc-dd3739a1adec/volumes" Nov 03 13:17:45.085500 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:45.085218 5979 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 03 13:17:45.085500 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:45.085281 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b" podUID="7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 03 13:17:46.083933 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:46.083883 5979 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 03 13:17:46.084237 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:46.084208 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b" podUID="7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 03 13:17:46.786603 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:46.786567 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-42c707-79c696bfc5-65rpj"] Nov 03 13:17:46.787940 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:46.787913 5979 cpu_manager.go:414] "containerMap: removing container" podUID="ad693484-f565-418b-acbc-dd3739a1adec" containerName="istio-proxy" Nov 03 13:17:46.788099 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:46.788084 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad693484-f565-418b-acbc-dd3739a1adec" containerName="istio-proxy" Nov 03 13:17:46.788228 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:46.788217 5979 cpu_manager.go:414] "containerMap: removing container" podUID="ad693484-f565-418b-acbc-dd3739a1adec" containerName="istio-proxy" Nov 03 13:17:46.788309 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:46.788299 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad693484-f565-418b-acbc-dd3739a1adec" containerName="istio-proxy" Nov 03 13:17:46.788553 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:46.788538 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad693484-f565-418b-acbc-dd3739a1adec" containerName="istio-proxy" Nov 03 13:17:46.788669 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:46.788658 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad693484-f565-418b-acbc-dd3739a1adec" containerName="istio-proxy" Nov 03 13:17:46.806555 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:46.806521 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-42c707-79c696bfc5-65rpj" Nov 03 13:17:46.845772 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:46.845739 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-42c707-79c696bfc5-65rpj"] Nov 03 13:17:46.846682 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:46.846556 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a748a184-2a39-4997-bbd4-337399775ae4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-42c707-79c696bfc5-65rpj\" (UID: \"a748a184-2a39-4997-bbd4-337399775ae4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-42c707-79c696bfc5-65rpj" Nov 03 13:17:46.846947 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:46.846929 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n695\" (UniqueName: \"kubernetes.io/projected/a748a184-2a39-4997-bbd4-337399775ae4-kube-api-access-8n695\") pod \"payload-pre-processing-e2e-worker-w0-42c707-79c696bfc5-65rpj\" (UID: \"a748a184-2a39-4997-bbd4-337399775ae4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-42c707-79c696bfc5-65rpj" Nov 03 13:17:46.878503 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:46.877873 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-42c707\"" Nov 03 13:17:46.899822 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:46.899789 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-42c707-dockercfg-dp5sf\"" Nov 03 13:17:46.942647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:46.942611 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-42c707-69d5b98f4f-jwf86"] Nov 03 13:17:46.965496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:46.948425 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n695\" (UniqueName: \"kubernetes.io/projected/a748a184-2a39-4997-bbd4-337399775ae4-kube-api-access-8n695\") pod \"payload-pre-processing-e2e-worker-w0-42c707-79c696bfc5-65rpj\" (UID: \"a748a184-2a39-4997-bbd4-337399775ae4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-42c707-79c696bfc5-65rpj" Nov 03 13:17:46.965496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:46.948627 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a748a184-2a39-4997-bbd4-337399775ae4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-42c707-79c696bfc5-65rpj\" (UID: \"a748a184-2a39-4997-bbd4-337399775ae4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-42c707-79c696bfc5-65rpj" Nov 03 13:17:46.965496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:46.962510 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-42c707-69d5b98f4f-jwf86" Nov 03 13:17:46.966439 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:46.966411 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a748a184-2a39-4997-bbd4-337399775ae4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-42c707-79c696bfc5-65rpj\" (UID: \"a748a184-2a39-4997-bbd4-337399775ae4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-42c707-79c696bfc5-65rpj" Nov 03 13:17:46.996164 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:46.996129 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-42c707-69d5b98f4f-jwf86"] Nov 03 13:17:47.015884 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:47.015851 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n695\" (UniqueName: \"kubernetes.io/projected/a748a184-2a39-4997-bbd4-337399775ae4-kube-api-access-8n695\") pod \"payload-pre-processing-e2e-worker-w0-42c707-79c696bfc5-65rpj\" (UID: \"a748a184-2a39-4997-bbd4-337399775ae4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-42c707-79c696bfc5-65rpj" Nov 03 13:17:47.054622 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:47.054208 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25jl2\" (UniqueName: \"kubernetes.io/projected/3bc04196-6680-45dd-bb04-7a8a0f5aac29-kube-api-access-25jl2\") pod \"payload-processing-e2e-worker-w0-42c707-69d5b98f4f-jwf86\" (UID: \"3bc04196-6680-45dd-bb04-7a8a0f5aac29\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-42c707-69d5b98f4f-jwf86" Nov 03 13:17:47.054622 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:47.054386 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bc04196-6680-45dd-bb04-7a8a0f5aac29-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-42c707-69d5b98f4f-jwf86\" (UID: \"3bc04196-6680-45dd-bb04-7a8a0f5aac29\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-42c707-69d5b98f4f-jwf86" Nov 03 13:17:47.093829 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:47.093796 5979 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 03 13:17:47.094125 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:47.094097 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b" podUID="7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 03 13:17:47.122321 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:47.122295 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-42c707-79c696bfc5-65rpj" Nov 03 13:17:47.168494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:47.166713 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25jl2\" (UniqueName: \"kubernetes.io/projected/3bc04196-6680-45dd-bb04-7a8a0f5aac29-kube-api-access-25jl2\") pod \"payload-processing-e2e-worker-w0-42c707-69d5b98f4f-jwf86\" (UID: \"3bc04196-6680-45dd-bb04-7a8a0f5aac29\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-42c707-69d5b98f4f-jwf86" Nov 03 13:17:47.168494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:47.166856 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bc04196-6680-45dd-bb04-7a8a0f5aac29-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-42c707-69d5b98f4f-jwf86\" (UID: \"3bc04196-6680-45dd-bb04-7a8a0f5aac29\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-42c707-69d5b98f4f-jwf86" Nov 03 13:17:47.168494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:47.167752 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bc04196-6680-45dd-bb04-7a8a0f5aac29-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-42c707-69d5b98f4f-jwf86\" (UID: \"3bc04196-6680-45dd-bb04-7a8a0f5aac29\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-42c707-69d5b98f4f-jwf86" Nov 03 13:17:47.225312 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:47.225259 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25jl2\" (UniqueName: \"kubernetes.io/projected/3bc04196-6680-45dd-bb04-7a8a0f5aac29-kube-api-access-25jl2\") pod \"payload-processing-e2e-worker-w0-42c707-69d5b98f4f-jwf86\" (UID: \"3bc04196-6680-45dd-bb04-7a8a0f5aac29\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-42c707-69d5b98f4f-jwf86" Nov 03 13:17:47.287824 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:47.287578 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-42c707-69d5b98f4f-jwf86" Nov 03 13:17:47.530185 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:47.530147 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-42c707-79c696bfc5-65rpj"] Nov 03 13:17:47.538300 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:17:47.538266 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda748a184_2a39_4997_bbd4_337399775ae4.slice/crio-84b3889e99e3f88a20a6ce5613792feb1c58b364246078cb56fc45087117ecb0 WatchSource:0}: Error finding container 84b3889e99e3f88a20a6ce5613792feb1c58b364246078cb56fc45087117ecb0: Status 404 returned error can't find the container with id 84b3889e99e3f88a20a6ce5613792feb1c58b364246078cb56fc45087117ecb0 Nov 03 13:17:47.678644 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:47.678613 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-42c707-69d5b98f4f-jwf86"] Nov 03 13:17:47.680289 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:17:47.680252 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bc04196_6680_45dd_bb04_7a8a0f5aac29.slice/crio-0c62f5cb8b9cb65c2d349d89c0afc0b43e99590dd627618bba88fffc35506a8f WatchSource:0}: Error finding container 0c62f5cb8b9cb65c2d349d89c0afc0b43e99590dd627618bba88fffc35506a8f: Status 404 returned error can't find the container with id 0c62f5cb8b9cb65c2d349d89c0afc0b43e99590dd627618bba88fffc35506a8f Nov 03 13:17:48.023434 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:48.023396 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-42c707-79c696bfc5-65rpj" event={"ID":"a748a184-2a39-4997-bbd4-337399775ae4","Type":"ContainerStarted","Data":"ac1155c4426edf5418c844d00b403d3dad26089c2d744717994799338a5871c2"} Nov 03 13:17:48.023937 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:48.023496 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-42c707-79c696bfc5-65rpj" Nov 03 13:17:48.023937 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:48.023529 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-42c707-79c696bfc5-65rpj" event={"ID":"a748a184-2a39-4997-bbd4-337399775ae4","Type":"ContainerStarted","Data":"84b3889e99e3f88a20a6ce5613792feb1c58b364246078cb56fc45087117ecb0"} Nov 03 13:17:48.025036 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:48.025013 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-42c707-69d5b98f4f-jwf86" event={"ID":"3bc04196-6680-45dd-bb04-7a8a0f5aac29","Type":"ContainerStarted","Data":"4970ac7ad4ab9d129cbff0e9733f8a60e4bda6174b30390c77b0e55a5290354c"} Nov 03 13:17:48.025162 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:48.025049 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-42c707-69d5b98f4f-jwf86" event={"ID":"3bc04196-6680-45dd-bb04-7a8a0f5aac29","Type":"ContainerStarted","Data":"0c62f5cb8b9cb65c2d349d89c0afc0b43e99590dd627618bba88fffc35506a8f"} Nov 03 13:17:48.025162 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:48.025149 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-42c707-69d5b98f4f-jwf86" Nov 03 13:17:48.079649 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:48.079616 5979 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 03 13:17:48.079823 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:48.079671 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b" podUID="7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 03 13:17:48.131752 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:48.131678 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-42c707-79c696bfc5-65rpj" podStartSLOduration=2.131658938 podStartE2EDuration="2.131658938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:17:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:17:48.130160306 +0000 UTC m=+750.140848548" watchObservedRunningTime="2026-11-03 13:17:48.131658938 +0000 UTC m=+750.142347181" Nov 03 13:17:48.295297 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:48.295183 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-42c707-69d5b98f4f-jwf86" podStartSLOduration=2.295161337 podStartE2EDuration="2.295161337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:17:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:17:48.290898847 +0000 UTC m=+750.301587173" watchObservedRunningTime="2026-11-03 13:17:48.295161337 +0000 UTC m=+750.305849580" Nov 03 13:17:48.633261 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:48.633227 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b" Nov 03 13:17:49.079180 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:49.079101 5979 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Nov 03 13:17:49.079180 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:49.079158 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b" podUID="7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Nov 03 13:17:49.674644 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:49.674601 5979 patch_prober.go:29] interesting pod/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:17:49.674816 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:49.674681 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj" podUID="a6729077-e216-4dd9-a68d-69ce28c4c103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:17:50.082803 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:50.082726 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b" Nov 03 13:17:50.083893 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:50.083871 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b" Nov 03 13:17:51.044147 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:51.044111 5979 generic.go:356] "Generic (PLEG): container finished" podID="a6729077-e216-4dd9-a68d-69ce28c4c103" containerID="4523c2499303c3864c2b430ca1044c2ffec208f7c8d94ebf83c16da3cb6e5c5e" exitCode=0 Nov 03 13:17:51.044395 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:51.044183 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj" event={"ID":"a6729077-e216-4dd9-a68d-69ce28c4c103","Type":"ContainerDied","Data":"4523c2499303c3864c2b430ca1044c2ffec208f7c8d94ebf83c16da3cb6e5c5e"} Nov 03 13:17:51.044694 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:51.044676 5979 scope.go:122] "RemoveContainer" containerID="4523c2499303c3864c2b430ca1044c2ffec208f7c8d94ebf83c16da3cb6e5c5e" Nov 03 13:17:51.045819 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:51.045787 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:17:51.045922 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:51.045851 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:17:51.045922 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:51.045882 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:17:52.074368 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:52.074334 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj" event={"ID":"a6729077-e216-4dd9-a68d-69ce28c4c103","Type":"ContainerStarted","Data":"6f45e00950225ed37bdd646c0aac73b693265fd2fa7f93812e644f2152ff77c2"} Nov 03 13:17:52.169289 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:52.168844 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj" Nov 03 13:17:52.169289 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:52.168969 5979 patch_prober.go:29] interesting pod/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 03 13:17:52.169289 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:52.169010 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj" podUID="a6729077-e216-4dd9-a68d-69ce28c4c103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 03 13:17:53.167154 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:53.167113 5979 patch_prober.go:29] interesting pod/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 03 13:17:53.167765 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:53.167721 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj" podUID="a6729077-e216-4dd9-a68d-69ce28c4c103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 03 13:17:54.168498 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:54.167928 5979 patch_prober.go:29] interesting pod/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 03 13:17:54.168498 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:54.167995 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj" podUID="a6729077-e216-4dd9-a68d-69ce28c4c103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 03 13:17:55.164941 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:55.164898 5979 patch_prober.go:29] interesting pod/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 03 13:17:55.165238 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:55.165207 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj" podUID="a6729077-e216-4dd9-a68d-69ce28c4c103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 03 13:17:55.674105 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:55.674066 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:17:56.164952 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:56.164906 5979 patch_prober.go:29] interesting pod/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Nov 03 13:17:56.165399 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:56.165358 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj" podUID="a6729077-e216-4dd9-a68d-69ce28c4c103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Nov 03 13:17:57.286647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:57.286612 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj" Nov 03 13:17:57.287266 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:57.287234 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj" Nov 03 13:17:57.288908 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:57.288889 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj" Nov 03 13:17:59.034071 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:59.034041 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-42c707-79c696bfc5-65rpj" Nov 03 13:17:59.034486 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:17:59.034108 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-42c707-69d5b98f4f-jwf86" Nov 03 13:18:04.500251 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:18:04.500033 5979 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": read tcp 10.134.0.2:58508->10.134.0.57:15021: read: connection reset by peer" start-of-body= Nov 03 13:18:04.500251 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:18:04.500107 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m" podUID="d8a57a81-8b6f-4049-9c5a-9884e968fee7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": read tcp 10.134.0.2:58508->10.134.0.57:15021: read: connection reset by peer" Nov 03 13:18:05.188888 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:18:05.188854 5979 generic.go:356] "Generic (PLEG): container finished" podID="d8a57a81-8b6f-4049-9c5a-9884e968fee7" containerID="5b71848d06a9680f1862c89aedcf2da53f592a49762b808ecb20a5c7195c4795" exitCode=0 Nov 03 13:18:05.189069 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:18:05.188924 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m" event={"ID":"d8a57a81-8b6f-4049-9c5a-9884e968fee7","Type":"ContainerDied","Data":"5b71848d06a9680f1862c89aedcf2da53f592a49762b808ecb20a5c7195c4795"} Nov 03 13:18:05.189333 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:18:05.189318 5979 scope.go:122] "RemoveContainer" containerID="5b71848d06a9680f1862c89aedcf2da53f592a49762b808ecb20a5c7195c4795" Nov 03 13:18:05.191532 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:18:05.191455 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:18:05.191680 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:18:05.191655 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:18:05.191782 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:18:05.191761 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:18:06.196669 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:18:06.196635 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m" event={"ID":"d8a57a81-8b6f-4049-9c5a-9884e968fee7","Type":"ContainerStarted","Data":"805c85f533d3da1367d1a1bed99c666311e802eeffa0006313025278dd1f26f2"} Nov 03 13:18:06.883148 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:18:06.883112 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m" Nov 03 13:18:06.883346 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:18:06.883315 5979 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 03 13:18:06.883402 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:18:06.883365 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m" podUID="d8a57a81-8b6f-4049-9c5a-9884e968fee7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 03 13:18:07.233306 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:18:07.233217 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:18:07.883344 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:18:07.883306 5979 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 03 13:18:07.883557 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:18:07.883375 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m" podUID="d8a57a81-8b6f-4049-9c5a-9884e968fee7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 03 13:18:08.883112 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:18:08.883077 5979 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 03 13:18:08.883588 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:18:08.883134 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m" podUID="d8a57a81-8b6f-4049-9c5a-9884e968fee7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 03 13:18:09.883039 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:18:09.882999 5979 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 03 13:18:09.883221 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:18:09.883057 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m" podUID="d8a57a81-8b6f-4049-9c5a-9884e968fee7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 03 13:18:10.882737 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:18:10.882699 5979 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Nov 03 13:18:10.882915 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:18:10.882769 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m" podUID="d8a57a81-8b6f-4049-9c5a-9884e968fee7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Nov 03 13:18:11.886777 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:18:11.886747 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m" Nov 03 13:18:11.887210 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:18:11.886994 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m" Nov 03 13:18:11.887732 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:18:11.887711 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m" Nov 03 13:18:17.420170 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:18:17.420133 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:18:34.183394 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:18:34.183361 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:18:37.233763 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:18:37.233722 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:18:47.036699 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:18:47.036670 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:18:58.930245 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:18:58.930162 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:19:07.008462 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:07.008417 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-3c5dbc/premium-w4-3c5dbc-kserve-577d8db86-mc72z"] Nov 03 13:19:07.012520 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:07.012494 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3c5dbc/premium-w4-3c5dbc-kserve-577d8db86-mc72z" Nov 03 13:19:07.061958 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:07.061925 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3c5dbc\"/\"default-dockercfg-4g6gg\"" Nov 03 13:19:07.067665 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:07.067633 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64d52ec5-fb72-4616-8d20-8b90043af828-home\") pod \"premium-w4-3c5dbc-kserve-577d8db86-mc72z\" (UID: \"64d52ec5-fb72-4616-8d20-8b90043af828\") " pod="e2e-models-e2e-worker-w4-3c5dbc/premium-w4-3c5dbc-kserve-577d8db86-mc72z" Nov 03 13:19:07.067847 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:07.067688 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6t2v\" (UniqueName: \"kubernetes.io/projected/64d52ec5-fb72-4616-8d20-8b90043af828-kube-api-access-n6t2v\") pod \"premium-w4-3c5dbc-kserve-577d8db86-mc72z\" (UID: \"64d52ec5-fb72-4616-8d20-8b90043af828\") " pod="e2e-models-e2e-worker-w4-3c5dbc/premium-w4-3c5dbc-kserve-577d8db86-mc72z" Nov 03 13:19:07.067847 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:07.067748 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64d52ec5-fb72-4616-8d20-8b90043af828-dshm\") pod \"premium-w4-3c5dbc-kserve-577d8db86-mc72z\" (UID: \"64d52ec5-fb72-4616-8d20-8b90043af828\") " pod="e2e-models-e2e-worker-w4-3c5dbc/premium-w4-3c5dbc-kserve-577d8db86-mc72z" Nov 03 13:19:07.067967 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:07.067850 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64d52ec5-fb72-4616-8d20-8b90043af828-tmp-dir\") pod \"premium-w4-3c5dbc-kserve-577d8db86-mc72z\" (UID: \"64d52ec5-fb72-4616-8d20-8b90043af828\") " pod="e2e-models-e2e-worker-w4-3c5dbc/premium-w4-3c5dbc-kserve-577d8db86-mc72z" Nov 03 13:19:07.068013 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:07.067980 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64d52ec5-fb72-4616-8d20-8b90043af828-tls-certs\") pod \"premium-w4-3c5dbc-kserve-577d8db86-mc72z\" (UID: \"64d52ec5-fb72-4616-8d20-8b90043af828\") " pod="e2e-models-e2e-worker-w4-3c5dbc/premium-w4-3c5dbc-kserve-577d8db86-mc72z" Nov 03 13:19:07.068046 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:07.068014 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64d52ec5-fb72-4616-8d20-8b90043af828-model-cache\") pod \"premium-w4-3c5dbc-kserve-577d8db86-mc72z\" (UID: \"64d52ec5-fb72-4616-8d20-8b90043af828\") " pod="e2e-models-e2e-worker-w4-3c5dbc/premium-w4-3c5dbc-kserve-577d8db86-mc72z" Nov 03 13:19:07.108637 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:07.108606 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3c5dbc\"/\"premium-w4-3c5dbc-kserve-self-signed-certs\"" Nov 03 13:19:07.108852 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:07.108805 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-3c5dbc\"/\"openshift-service-ca.crt\"" Nov 03 13:19:07.108925 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:07.108913 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-3c5dbc\"/\"kube-root-ca.crt\"" Nov 03 13:19:07.110576 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:07.110550 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3c5dbc/premium-w4-3c5dbc-kserve-577d8db86-mc72z"] Nov 03 13:19:07.175140 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:07.168612 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64d52ec5-fb72-4616-8d20-8b90043af828-dshm\") pod \"premium-w4-3c5dbc-kserve-577d8db86-mc72z\" (UID: \"64d52ec5-fb72-4616-8d20-8b90043af828\") " pod="e2e-models-e2e-worker-w4-3c5dbc/premium-w4-3c5dbc-kserve-577d8db86-mc72z" Nov 03 13:19:07.175140 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:07.168699 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64d52ec5-fb72-4616-8d20-8b90043af828-tmp-dir\") pod \"premium-w4-3c5dbc-kserve-577d8db86-mc72z\" (UID: \"64d52ec5-fb72-4616-8d20-8b90043af828\") " pod="e2e-models-e2e-worker-w4-3c5dbc/premium-w4-3c5dbc-kserve-577d8db86-mc72z" Nov 03 13:19:07.175140 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:07.168796 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64d52ec5-fb72-4616-8d20-8b90043af828-tls-certs\") pod \"premium-w4-3c5dbc-kserve-577d8db86-mc72z\" (UID: \"64d52ec5-fb72-4616-8d20-8b90043af828\") " pod="e2e-models-e2e-worker-w4-3c5dbc/premium-w4-3c5dbc-kserve-577d8db86-mc72z" Nov 03 13:19:07.175140 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:07.168857 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64d52ec5-fb72-4616-8d20-8b90043af828-model-cache\") pod \"premium-w4-3c5dbc-kserve-577d8db86-mc72z\" (UID: \"64d52ec5-fb72-4616-8d20-8b90043af828\") " pod="e2e-models-e2e-worker-w4-3c5dbc/premium-w4-3c5dbc-kserve-577d8db86-mc72z" Nov 03 13:19:07.175140 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:07.168897 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64d52ec5-fb72-4616-8d20-8b90043af828-home\") pod \"premium-w4-3c5dbc-kserve-577d8db86-mc72z\" (UID: \"64d52ec5-fb72-4616-8d20-8b90043af828\") " pod="e2e-models-e2e-worker-w4-3c5dbc/premium-w4-3c5dbc-kserve-577d8db86-mc72z" Nov 03 13:19:07.175140 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:07.168956 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6t2v\" (UniqueName: \"kubernetes.io/projected/64d52ec5-fb72-4616-8d20-8b90043af828-kube-api-access-n6t2v\") pod \"premium-w4-3c5dbc-kserve-577d8db86-mc72z\" (UID: \"64d52ec5-fb72-4616-8d20-8b90043af828\") " pod="e2e-models-e2e-worker-w4-3c5dbc/premium-w4-3c5dbc-kserve-577d8db86-mc72z" Nov 03 13:19:07.175140 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:07.173389 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64d52ec5-fb72-4616-8d20-8b90043af828-model-cache\") pod \"premium-w4-3c5dbc-kserve-577d8db86-mc72z\" (UID: \"64d52ec5-fb72-4616-8d20-8b90043af828\") " pod="e2e-models-e2e-worker-w4-3c5dbc/premium-w4-3c5dbc-kserve-577d8db86-mc72z" Nov 03 13:19:07.175140 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:07.173678 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64d52ec5-fb72-4616-8d20-8b90043af828-tmp-dir\") pod \"premium-w4-3c5dbc-kserve-577d8db86-mc72z\" (UID: \"64d52ec5-fb72-4616-8d20-8b90043af828\") " pod="e2e-models-e2e-worker-w4-3c5dbc/premium-w4-3c5dbc-kserve-577d8db86-mc72z" Nov 03 13:19:07.175140 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:07.174441 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64d52ec5-fb72-4616-8d20-8b90043af828-home\") pod \"premium-w4-3c5dbc-kserve-577d8db86-mc72z\" (UID: \"64d52ec5-fb72-4616-8d20-8b90043af828\") " pod="e2e-models-e2e-worker-w4-3c5dbc/premium-w4-3c5dbc-kserve-577d8db86-mc72z" Nov 03 13:19:07.180450 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:07.179926 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64d52ec5-fb72-4616-8d20-8b90043af828-dshm\") pod \"premium-w4-3c5dbc-kserve-577d8db86-mc72z\" (UID: \"64d52ec5-fb72-4616-8d20-8b90043af828\") " pod="e2e-models-e2e-worker-w4-3c5dbc/premium-w4-3c5dbc-kserve-577d8db86-mc72z" Nov 03 13:19:07.182430 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:07.182399 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64d52ec5-fb72-4616-8d20-8b90043af828-tls-certs\") pod \"premium-w4-3c5dbc-kserve-577d8db86-mc72z\" (UID: \"64d52ec5-fb72-4616-8d20-8b90043af828\") " pod="e2e-models-e2e-worker-w4-3c5dbc/premium-w4-3c5dbc-kserve-577d8db86-mc72z" Nov 03 13:19:07.228149 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:07.228118 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6t2v\" (UniqueName: \"kubernetes.io/projected/64d52ec5-fb72-4616-8d20-8b90043af828-kube-api-access-n6t2v\") pod \"premium-w4-3c5dbc-kserve-577d8db86-mc72z\" (UID: \"64d52ec5-fb72-4616-8d20-8b90043af828\") " pod="e2e-models-e2e-worker-w4-3c5dbc/premium-w4-3c5dbc-kserve-577d8db86-mc72z" Nov 03 13:19:07.235988 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:19:07.235958 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:19:07.325541 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:07.325448 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3c5dbc/premium-w4-3c5dbc-kserve-577d8db86-mc72z" Nov 03 13:19:07.727338 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:07.727303 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3c5dbc/premium-w4-3c5dbc-kserve-577d8db86-mc72z"] Nov 03 13:19:07.734551 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:19:07.734520 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod64d52ec5_fb72_4616_8d20_8b90043af828.slice/crio-5dd1a6c1f8a24f8ac6b784bc52834e02ef996fcfa4117ebf76ab8d20456fc836 WatchSource:0}: Error finding container 5dd1a6c1f8a24f8ac6b784bc52834e02ef996fcfa4117ebf76ab8d20456fc836: Status 404 returned error can't find the container with id 5dd1a6c1f8a24f8ac6b784bc52834e02ef996fcfa4117ebf76ab8d20456fc836 Nov 03 13:19:08.545950 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:08.545911 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3c5dbc/premium-w4-3c5dbc-kserve-577d8db86-mc72z" event={"ID":"64d52ec5-fb72-4616-8d20-8b90043af828","Type":"ContainerStarted","Data":"161695f2a055d41eddba2c4f40a2b949bfe9883c11fa05945c528a97f0a75828"} Nov 03 13:19:08.546350 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:08.545961 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3c5dbc/premium-w4-3c5dbc-kserve-577d8db86-mc72z" event={"ID":"64d52ec5-fb72-4616-8d20-8b90043af828","Type":"ContainerStarted","Data":"5dd1a6c1f8a24f8ac6b784bc52834e02ef996fcfa4117ebf76ab8d20456fc836"} Nov 03 13:19:08.655146 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:08.655090 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-3c5dbc/premium-w4-3c5dbc-kserve-577d8db86-mc72z" podStartSLOduration=2.655074243 podStartE2EDuration="2.655074243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:19:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:19:08.653573409 +0000 UTC m=+830.664261652" watchObservedRunningTime="2026-11-03 13:19:08.655074243 +0000 UTC m=+830.665762486" Nov 03 13:19:09.326311 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:09.326276 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3c5dbc/premium-w4-3c5dbc-kserve-577d8db86-mc72z" Nov 03 13:19:12.585270 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:12.585233 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:19:17.326488 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:17.326447 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-3c5dbc/premium-w4-3c5dbc-kserve-577d8db86-mc72z" Nov 03 13:19:17.339303 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:17.339274 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-3c5dbc/premium-w4-3c5dbc-kserve-577d8db86-mc72z" Nov 03 13:19:17.612865 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:17.612776 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3c5dbc/premium-w4-3c5dbc-kserve-577d8db86-mc72z" Nov 03 13:19:30.341395 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:30.341362 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-5bb463/premium-w3-5bb463-kserve-5f6ff6795c-jxmsr"] Nov 03 13:19:30.345373 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:30.345346 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5bb463/premium-w3-5bb463-kserve-5f6ff6795c-jxmsr" Nov 03 13:19:30.394440 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:30.394403 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-5bb463\"/\"openshift-service-ca.crt\"" Nov 03 13:19:30.418639 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:30.418602 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5bb463\"/\"premium-w3-5bb463-kserve-self-signed-certs\"" Nov 03 13:19:30.437032 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:30.436992 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fvfp\" (UniqueName: \"kubernetes.io/projected/8ee26f55-5c0c-4bd8-ad9b-fb8c03641698-kube-api-access-9fvfp\") pod \"premium-w3-5bb463-kserve-5f6ff6795c-jxmsr\" (UID: \"8ee26f55-5c0c-4bd8-ad9b-fb8c03641698\") " pod="e2e-models-e2e-worker-w3-5bb463/premium-w3-5bb463-kserve-5f6ff6795c-jxmsr" Nov 03 13:19:30.437032 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:30.437033 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ee26f55-5c0c-4bd8-ad9b-fb8c03641698-home\") pod \"premium-w3-5bb463-kserve-5f6ff6795c-jxmsr\" (UID: \"8ee26f55-5c0c-4bd8-ad9b-fb8c03641698\") " pod="e2e-models-e2e-worker-w3-5bb463/premium-w3-5bb463-kserve-5f6ff6795c-jxmsr" Nov 03 13:19:30.437257 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:30.437066 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ee26f55-5c0c-4bd8-ad9b-fb8c03641698-tmp-dir\") pod \"premium-w3-5bb463-kserve-5f6ff6795c-jxmsr\" (UID: \"8ee26f55-5c0c-4bd8-ad9b-fb8c03641698\") " pod="e2e-models-e2e-worker-w3-5bb463/premium-w3-5bb463-kserve-5f6ff6795c-jxmsr" Nov 03 13:19:30.437257 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:30.437210 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ee26f55-5c0c-4bd8-ad9b-fb8c03641698-tls-certs\") pod \"premium-w3-5bb463-kserve-5f6ff6795c-jxmsr\" (UID: \"8ee26f55-5c0c-4bd8-ad9b-fb8c03641698\") " pod="e2e-models-e2e-worker-w3-5bb463/premium-w3-5bb463-kserve-5f6ff6795c-jxmsr" Nov 03 13:19:30.437362 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:30.437264 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ee26f55-5c0c-4bd8-ad9b-fb8c03641698-dshm\") pod \"premium-w3-5bb463-kserve-5f6ff6795c-jxmsr\" (UID: \"8ee26f55-5c0c-4bd8-ad9b-fb8c03641698\") " pod="e2e-models-e2e-worker-w3-5bb463/premium-w3-5bb463-kserve-5f6ff6795c-jxmsr" Nov 03 13:19:30.437362 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:30.437353 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ee26f55-5c0c-4bd8-ad9b-fb8c03641698-model-cache\") pod \"premium-w3-5bb463-kserve-5f6ff6795c-jxmsr\" (UID: \"8ee26f55-5c0c-4bd8-ad9b-fb8c03641698\") " pod="e2e-models-e2e-worker-w3-5bb463/premium-w3-5bb463-kserve-5f6ff6795c-jxmsr" Nov 03 13:19:30.441156 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:30.441135 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-5bb463\"/\"kube-root-ca.crt\"" Nov 03 13:19:30.442968 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:30.442945 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5bb463/premium-w3-5bb463-kserve-5f6ff6795c-jxmsr"] Nov 03 13:19:30.502549 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:30.502510 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5bb463\"/\"default-dockercfg-bhrh5\"" Nov 03 13:19:30.538294 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:30.538249 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ee26f55-5c0c-4bd8-ad9b-fb8c03641698-tls-certs\") pod \"premium-w3-5bb463-kserve-5f6ff6795c-jxmsr\" (UID: \"8ee26f55-5c0c-4bd8-ad9b-fb8c03641698\") " pod="e2e-models-e2e-worker-w3-5bb463/premium-w3-5bb463-kserve-5f6ff6795c-jxmsr" Nov 03 13:19:30.538533 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:30.538323 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ee26f55-5c0c-4bd8-ad9b-fb8c03641698-dshm\") pod \"premium-w3-5bb463-kserve-5f6ff6795c-jxmsr\" (UID: \"8ee26f55-5c0c-4bd8-ad9b-fb8c03641698\") " pod="e2e-models-e2e-worker-w3-5bb463/premium-w3-5bb463-kserve-5f6ff6795c-jxmsr" Nov 03 13:19:30.538533 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:30.538367 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ee26f55-5c0c-4bd8-ad9b-fb8c03641698-model-cache\") pod \"premium-w3-5bb463-kserve-5f6ff6795c-jxmsr\" (UID: \"8ee26f55-5c0c-4bd8-ad9b-fb8c03641698\") " pod="e2e-models-e2e-worker-w3-5bb463/premium-w3-5bb463-kserve-5f6ff6795c-jxmsr" Nov 03 13:19:30.538533 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:30.538500 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fvfp\" (UniqueName: \"kubernetes.io/projected/8ee26f55-5c0c-4bd8-ad9b-fb8c03641698-kube-api-access-9fvfp\") pod \"premium-w3-5bb463-kserve-5f6ff6795c-jxmsr\" (UID: \"8ee26f55-5c0c-4bd8-ad9b-fb8c03641698\") " pod="e2e-models-e2e-worker-w3-5bb463/premium-w3-5bb463-kserve-5f6ff6795c-jxmsr" Nov 03 13:19:30.538724 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:30.538539 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ee26f55-5c0c-4bd8-ad9b-fb8c03641698-home\") pod \"premium-w3-5bb463-kserve-5f6ff6795c-jxmsr\" (UID: \"8ee26f55-5c0c-4bd8-ad9b-fb8c03641698\") " pod="e2e-models-e2e-worker-w3-5bb463/premium-w3-5bb463-kserve-5f6ff6795c-jxmsr" Nov 03 13:19:30.538724 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:30.538591 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ee26f55-5c0c-4bd8-ad9b-fb8c03641698-tmp-dir\") pod \"premium-w3-5bb463-kserve-5f6ff6795c-jxmsr\" (UID: \"8ee26f55-5c0c-4bd8-ad9b-fb8c03641698\") " pod="e2e-models-e2e-worker-w3-5bb463/premium-w3-5bb463-kserve-5f6ff6795c-jxmsr" Nov 03 13:19:30.538941 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:30.538916 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ee26f55-5c0c-4bd8-ad9b-fb8c03641698-model-cache\") pod \"premium-w3-5bb463-kserve-5f6ff6795c-jxmsr\" (UID: \"8ee26f55-5c0c-4bd8-ad9b-fb8c03641698\") " pod="e2e-models-e2e-worker-w3-5bb463/premium-w3-5bb463-kserve-5f6ff6795c-jxmsr" Nov 03 13:19:30.539035 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:30.538952 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ee26f55-5c0c-4bd8-ad9b-fb8c03641698-tmp-dir\") pod \"premium-w3-5bb463-kserve-5f6ff6795c-jxmsr\" (UID: \"8ee26f55-5c0c-4bd8-ad9b-fb8c03641698\") " pod="e2e-models-e2e-worker-w3-5bb463/premium-w3-5bb463-kserve-5f6ff6795c-jxmsr" Nov 03 13:19:30.539035 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:30.538990 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ee26f55-5c0c-4bd8-ad9b-fb8c03641698-home\") pod \"premium-w3-5bb463-kserve-5f6ff6795c-jxmsr\" (UID: \"8ee26f55-5c0c-4bd8-ad9b-fb8c03641698\") " pod="e2e-models-e2e-worker-w3-5bb463/premium-w3-5bb463-kserve-5f6ff6795c-jxmsr" Nov 03 13:19:30.540654 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:30.540637 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ee26f55-5c0c-4bd8-ad9b-fb8c03641698-dshm\") pod \"premium-w3-5bb463-kserve-5f6ff6795c-jxmsr\" (UID: \"8ee26f55-5c0c-4bd8-ad9b-fb8c03641698\") " pod="e2e-models-e2e-worker-w3-5bb463/premium-w3-5bb463-kserve-5f6ff6795c-jxmsr" Nov 03 13:19:30.541012 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:30.540992 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ee26f55-5c0c-4bd8-ad9b-fb8c03641698-tls-certs\") pod \"premium-w3-5bb463-kserve-5f6ff6795c-jxmsr\" (UID: \"8ee26f55-5c0c-4bd8-ad9b-fb8c03641698\") " pod="e2e-models-e2e-worker-w3-5bb463/premium-w3-5bb463-kserve-5f6ff6795c-jxmsr" Nov 03 13:19:30.593341 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:30.593260 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fvfp\" (UniqueName: \"kubernetes.io/projected/8ee26f55-5c0c-4bd8-ad9b-fb8c03641698-kube-api-access-9fvfp\") pod \"premium-w3-5bb463-kserve-5f6ff6795c-jxmsr\" (UID: \"8ee26f55-5c0c-4bd8-ad9b-fb8c03641698\") " pod="e2e-models-e2e-worker-w3-5bb463/premium-w3-5bb463-kserve-5f6ff6795c-jxmsr" Nov 03 13:19:30.656971 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:30.656929 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5bb463/premium-w3-5bb463-kserve-5f6ff6795c-jxmsr" Nov 03 13:19:31.059789 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:31.059748 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5bb463/premium-w3-5bb463-kserve-5f6ff6795c-jxmsr"] Nov 03 13:19:31.067744 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:19:31.067713 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8ee26f55_5c0c_4bd8_ad9b_fb8c03641698.slice/crio-22623d2a1cc1a758a4387af732b8ec448c13884b57608e57a084490b95d67c20 WatchSource:0}: Error finding container 22623d2a1cc1a758a4387af732b8ec448c13884b57608e57a084490b95d67c20: Status 404 returned error can't find the container with id 22623d2a1cc1a758a4387af732b8ec448c13884b57608e57a084490b95d67c20 Nov 03 13:19:31.678749 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:31.678704 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5bb463/premium-w3-5bb463-kserve-5f6ff6795c-jxmsr" event={"ID":"8ee26f55-5c0c-4bd8-ad9b-fb8c03641698","Type":"ContainerStarted","Data":"7874929dd2fd364bf24c1a522a68cf1174b153b65c56d3b7a377566e71e3f23b"} Nov 03 13:19:31.679145 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:31.678755 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5bb463/premium-w3-5bb463-kserve-5f6ff6795c-jxmsr" event={"ID":"8ee26f55-5c0c-4bd8-ad9b-fb8c03641698","Type":"ContainerStarted","Data":"22623d2a1cc1a758a4387af732b8ec448c13884b57608e57a084490b95d67c20"} Nov 03 13:19:31.804214 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:31.804155 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-5bb463/premium-w3-5bb463-kserve-5f6ff6795c-jxmsr" podStartSLOduration=1.804140214 podStartE2EDuration="1.804140214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:19:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:19:31.802056755 +0000 UTC m=+853.812744998" watchObservedRunningTime="2026-11-03 13:19:31.804140214 +0000 UTC m=+853.814828456" Nov 03 13:19:32.657305 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:32.657270 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5bb463/premium-w3-5bb463-kserve-5f6ff6795c-jxmsr" Nov 03 13:19:37.234249 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:19:37.234213 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:19:37.989529 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:37.989494 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:19:40.657573 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:40.657544 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-5bb463/premium-w3-5bb463-kserve-5f6ff6795c-jxmsr" Nov 03 13:19:40.670245 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:40.670222 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-5bb463/premium-w3-5bb463-kserve-5f6ff6795c-jxmsr" Nov 03 13:19:40.737941 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:40.737912 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5bb463/premium-w3-5bb463-kserve-5f6ff6795c-jxmsr" Nov 03 13:19:49.750356 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:49.750322 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60178b32-5jrcz"] Nov 03 13:19:49.754407 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:49.754386 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60178b32-5jrcz" Nov 03 13:19:49.807772 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:49.807735 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60178b32-5jrcz"] Nov 03 13:19:49.851392 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:49.851354 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:19:49.851392 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:49.851373 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 13:19:49.851392 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:49.851387 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 13:19:49.851726 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:49.851457 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:19:49.858872 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:49.858837 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fa694a6-90c3-4124-a81d-1b71a84f003a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60178b32-5jrcz\" (UID: \"3fa694a6-90c3-4124-a81d-1b71a84f003a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60178b32-5jrcz" Nov 03 13:19:49.858965 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:49.858917 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xstpv\" (UniqueName: \"kubernetes.io/projected/3fa694a6-90c3-4124-a81d-1b71a84f003a-kube-api-access-xstpv\") pod \"maas-api-revoke-sub-60178b32-5jrcz\" (UID: \"3fa694a6-90c3-4124-a81d-1b71a84f003a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60178b32-5jrcz" Nov 03 13:19:49.859077 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:49.859055 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fa694a6-90c3-4124-a81d-1b71a84f003a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60178b32-5jrcz\" (UID: \"3fa694a6-90c3-4124-a81d-1b71a84f003a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60178b32-5jrcz" Nov 03 13:19:49.960189 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:49.960152 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fa694a6-90c3-4124-a81d-1b71a84f003a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60178b32-5jrcz\" (UID: \"3fa694a6-90c3-4124-a81d-1b71a84f003a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60178b32-5jrcz" Nov 03 13:19:49.960368 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:49.960210 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fa694a6-90c3-4124-a81d-1b71a84f003a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60178b32-5jrcz\" (UID: \"3fa694a6-90c3-4124-a81d-1b71a84f003a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60178b32-5jrcz" Nov 03 13:19:49.960368 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:49.960250 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xstpv\" (UniqueName: \"kubernetes.io/projected/3fa694a6-90c3-4124-a81d-1b71a84f003a-kube-api-access-xstpv\") pod \"maas-api-revoke-sub-60178b32-5jrcz\" (UID: \"3fa694a6-90c3-4124-a81d-1b71a84f003a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60178b32-5jrcz" Nov 03 13:19:49.960944 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:49.960923 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fa694a6-90c3-4124-a81d-1b71a84f003a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60178b32-5jrcz\" (UID: \"3fa694a6-90c3-4124-a81d-1b71a84f003a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60178b32-5jrcz" Nov 03 13:19:49.962476 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:49.962450 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fa694a6-90c3-4124-a81d-1b71a84f003a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60178b32-5jrcz\" (UID: \"3fa694a6-90c3-4124-a81d-1b71a84f003a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60178b32-5jrcz" Nov 03 13:19:50.013733 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:50.013663 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xstpv\" (UniqueName: \"kubernetes.io/projected/3fa694a6-90c3-4124-a81d-1b71a84f003a-kube-api-access-xstpv\") pod \"maas-api-revoke-sub-60178b32-5jrcz\" (UID: \"3fa694a6-90c3-4124-a81d-1b71a84f003a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60178b32-5jrcz" Nov 03 13:19:50.068035 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:50.067991 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60178b32-5jrcz" Nov 03 13:19:50.446488 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:50.446450 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60178b32-5jrcz"] Nov 03 13:19:50.453684 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:19:50.453658 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fa694a6_90c3_4124_a81d_1b71a84f003a.slice/crio-82768abe6af9ce65bddf9ca3fc95d1988a4beab37172157d382e594f922b2138 WatchSource:0}: Error finding container 82768abe6af9ce65bddf9ca3fc95d1988a4beab37172157d382e594f922b2138: Status 404 returned error can't find the container with id 82768abe6af9ce65bddf9ca3fc95d1988a4beab37172157d382e594f922b2138 Nov 03 13:19:50.783924 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:50.783885 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60178b32-5jrcz" event={"ID":"3fa694a6-90c3-4124-a81d-1b71a84f003a","Type":"ContainerStarted","Data":"51b3ad4623534410f3f945a318e3b202ef2570630e7fdd9d055cc26806ce0299"} Nov 03 13:19:50.784312 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:50.783949 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60178b32-5jrcz" event={"ID":"3fa694a6-90c3-4124-a81d-1b71a84f003a","Type":"ContainerStarted","Data":"82768abe6af9ce65bddf9ca3fc95d1988a4beab37172157d382e594f922b2138"} Nov 03 13:19:50.892038 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:50.891990 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60178b32-5jrcz" podStartSLOduration=1.8919759040000002 podStartE2EDuration="1.891975904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:19:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:19:50.89105621 +0000 UTC m=+872.901744452" watchObservedRunningTime="2026-11-03 13:19:50.891975904 +0000 UTC m=+872.902664142" Nov 03 13:19:51.791727 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:51.791686 5979 generic.go:356] "Generic (PLEG): container finished" podID="3fa694a6-90c3-4124-a81d-1b71a84f003a" containerID="51b3ad4623534410f3f945a318e3b202ef2570630e7fdd9d055cc26806ce0299" exitCode=0 Nov 03 13:19:51.792135 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:51.791776 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60178b32-5jrcz" event={"ID":"3fa694a6-90c3-4124-a81d-1b71a84f003a","Type":"ContainerDied","Data":"51b3ad4623534410f3f945a318e3b202ef2570630e7fdd9d055cc26806ce0299"} Nov 03 13:19:52.937891 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:52.937867 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60178b32-5jrcz" Nov 03 13:19:53.093913 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:53.093837 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fa694a6-90c3-4124-a81d-1b71a84f003a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fa694a6-90c3-4124-a81d-1b71a84f003a-maas-api-openshift-ca\") pod \"3fa694a6-90c3-4124-a81d-1b71a84f003a\" (UID: \"3fa694a6-90c3-4124-a81d-1b71a84f003a\") " Nov 03 13:19:53.094048 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:53.093934 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fa694a6-90c3-4124-a81d-1b71a84f003a-kube-api-access-xstpv\" (UniqueName: \"kubernetes.io/projected/3fa694a6-90c3-4124-a81d-1b71a84f003a-kube-api-access-xstpv\") pod \"3fa694a6-90c3-4124-a81d-1b71a84f003a\" (UID: \"3fa694a6-90c3-4124-a81d-1b71a84f003a\") " Nov 03 13:19:53.094048 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:53.094035 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fa694a6-90c3-4124-a81d-1b71a84f003a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fa694a6-90c3-4124-a81d-1b71a84f003a-maas-api-xks-ca\") pod \"3fa694a6-90c3-4124-a81d-1b71a84f003a\" (UID: \"3fa694a6-90c3-4124-a81d-1b71a84f003a\") " Nov 03 13:19:53.094239 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:53.094213 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fa694a6-90c3-4124-a81d-1b71a84f003a-maas-api-openshift-ca" pod "3fa694a6-90c3-4124-a81d-1b71a84f003a" (UID: "3fa694a6-90c3-4124-a81d-1b71a84f003a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:19:53.094406 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:53.094389 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fa694a6-90c3-4124-a81d-1b71a84f003a-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:19:53.096006 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:53.095977 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fa694a6-90c3-4124-a81d-1b71a84f003a-maas-api-xks-ca" pod "3fa694a6-90c3-4124-a81d-1b71a84f003a" (UID: "3fa694a6-90c3-4124-a81d-1b71a84f003a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:19:53.096006 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:53.095979 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fa694a6-90c3-4124-a81d-1b71a84f003a-kube-api-access-xstpv" pod "3fa694a6-90c3-4124-a81d-1b71a84f003a" (UID: "3fa694a6-90c3-4124-a81d-1b71a84f003a"). InnerVolumeSpecName "kube-api-access-xstpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:19:53.196094 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:53.196059 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xstpv\" (UniqueName: \"kubernetes.io/projected/3fa694a6-90c3-4124-a81d-1b71a84f003a-kube-api-access-xstpv\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:19:53.196094 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:53.196096 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fa694a6-90c3-4124-a81d-1b71a84f003a-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:19:53.804392 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:53.804367 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60178b32-5jrcz" Nov 03 13:19:53.804632 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:53.804392 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60178b32-5jrcz" event={"ID":"3fa694a6-90c3-4124-a81d-1b71a84f003a","Type":"ContainerDied","Data":"82768abe6af9ce65bddf9ca3fc95d1988a4beab37172157d382e594f922b2138"} Nov 03 13:19:53.804632 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:19:53.804443 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82768abe6af9ce65bddf9ca3fc95d1988a4beab37172157d382e594f922b2138" Nov 03 13:20:01.979963 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:01.979926 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:20:07.233656 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:20:07.233615 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:20:07.735099 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:07.735066 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93aefb28-r8bjv"] Nov 03 13:20:07.735567 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:07.735554 5979 cpu_manager.go:414] "containerMap: removing container" podUID="3fa694a6-90c3-4124-a81d-1b71a84f003a" containerName="revoke-keys" Nov 03 13:20:07.735620 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:07.735568 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fa694a6-90c3-4124-a81d-1b71a84f003a" containerName="revoke-keys" Nov 03 13:20:07.735662 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:07.735655 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fa694a6-90c3-4124-a81d-1b71a84f003a" containerName="revoke-keys" Nov 03 13:20:07.738964 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:07.738936 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93aefb28-r8bjv" Nov 03 13:20:07.759852 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:07.759821 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjpn8\" (UniqueName: \"kubernetes.io/projected/8f1800fb-b1b8-4113-a50d-01fa71e064e0-kube-api-access-vjpn8\") pod \"maas-api-revoke-sub-93aefb28-r8bjv\" (UID: \"8f1800fb-b1b8-4113-a50d-01fa71e064e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93aefb28-r8bjv" Nov 03 13:20:07.759978 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:07.759944 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f1800fb-b1b8-4113-a50d-01fa71e064e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93aefb28-r8bjv\" (UID: \"8f1800fb-b1b8-4113-a50d-01fa71e064e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93aefb28-r8bjv" Nov 03 13:20:07.760023 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:07.760003 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f1800fb-b1b8-4113-a50d-01fa71e064e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93aefb28-r8bjv\" (UID: \"8f1800fb-b1b8-4113-a50d-01fa71e064e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93aefb28-r8bjv" Nov 03 13:20:07.789820 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:07.789790 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:20:07.789820 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:07.789798 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 13:20:07.850738 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:07.841832 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:20:07.850738 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:07.841907 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 13:20:07.850738 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:07.847927 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93aefb28-r8bjv"] Nov 03 13:20:07.861455 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:07.861424 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f1800fb-b1b8-4113-a50d-01fa71e064e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93aefb28-r8bjv\" (UID: \"8f1800fb-b1b8-4113-a50d-01fa71e064e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93aefb28-r8bjv" Nov 03 13:20:07.861604 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:07.861495 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f1800fb-b1b8-4113-a50d-01fa71e064e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93aefb28-r8bjv\" (UID: \"8f1800fb-b1b8-4113-a50d-01fa71e064e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93aefb28-r8bjv" Nov 03 13:20:07.861604 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:07.861570 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjpn8\" (UniqueName: \"kubernetes.io/projected/8f1800fb-b1b8-4113-a50d-01fa71e064e0-kube-api-access-vjpn8\") pod \"maas-api-revoke-sub-93aefb28-r8bjv\" (UID: \"8f1800fb-b1b8-4113-a50d-01fa71e064e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93aefb28-r8bjv" Nov 03 13:20:07.862235 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:07.862216 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f1800fb-b1b8-4113-a50d-01fa71e064e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93aefb28-r8bjv\" (UID: \"8f1800fb-b1b8-4113-a50d-01fa71e064e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93aefb28-r8bjv" Nov 03 13:20:07.863879 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:07.863854 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f1800fb-b1b8-4113-a50d-01fa71e064e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93aefb28-r8bjv\" (UID: \"8f1800fb-b1b8-4113-a50d-01fa71e064e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93aefb28-r8bjv" Nov 03 13:20:07.916270 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:07.916238 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjpn8\" (UniqueName: \"kubernetes.io/projected/8f1800fb-b1b8-4113-a50d-01fa71e064e0-kube-api-access-vjpn8\") pod \"maas-api-revoke-sub-93aefb28-r8bjv\" (UID: \"8f1800fb-b1b8-4113-a50d-01fa71e064e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93aefb28-r8bjv" Nov 03 13:20:08.052030 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:08.051943 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93aefb28-r8bjv" Nov 03 13:20:08.429543 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:08.429508 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93aefb28-r8bjv"] Nov 03 13:20:08.436521 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:20:08.436493 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f1800fb_b1b8_4113_a50d_01fa71e064e0.slice/crio-77c40bdc2e0793a11e16278b312d6e032d65c081ac5bd260d4ef99a20f43a44b WatchSource:0}: Error finding container 77c40bdc2e0793a11e16278b312d6e032d65c081ac5bd260d4ef99a20f43a44b: Status 404 returned error can't find the container with id 77c40bdc2e0793a11e16278b312d6e032d65c081ac5bd260d4ef99a20f43a44b Nov 03 13:20:08.894796 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:08.894760 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93aefb28-r8bjv" event={"ID":"8f1800fb-b1b8-4113-a50d-01fa71e064e0","Type":"ContainerStarted","Data":"13f26f6159be9f1afdba03b9ca1a53290a3e754a9d2567e51451c4b605a62760"} Nov 03 13:20:08.894969 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:08.894823 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93aefb28-r8bjv" event={"ID":"8f1800fb-b1b8-4113-a50d-01fa71e064e0","Type":"ContainerStarted","Data":"77c40bdc2e0793a11e16278b312d6e032d65c081ac5bd260d4ef99a20f43a44b"} Nov 03 13:20:09.002733 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:09.002679 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93aefb28-r8bjv" podStartSLOduration=2.002662769 podStartE2EDuration="2.002662769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:20:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:20:09.001646233 +0000 UTC m=+891.012334474" watchObservedRunningTime="2026-11-03 13:20:09.002662769 +0000 UTC m=+891.013351011" Nov 03 13:20:09.902063 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:09.902031 5979 generic.go:356] "Generic (PLEG): container finished" podID="8f1800fb-b1b8-4113-a50d-01fa71e064e0" containerID="13f26f6159be9f1afdba03b9ca1a53290a3e754a9d2567e51451c4b605a62760" exitCode=0 Nov 03 13:20:09.902455 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:09.902070 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93aefb28-r8bjv" event={"ID":"8f1800fb-b1b8-4113-a50d-01fa71e064e0","Type":"ContainerDied","Data":"13f26f6159be9f1afdba03b9ca1a53290a3e754a9d2567e51451c4b605a62760"} Nov 03 13:20:11.054983 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:11.054956 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93aefb28-r8bjv" Nov 03 13:20:11.095680 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:11.095591 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f1800fb-b1b8-4113-a50d-01fa71e064e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f1800fb-b1b8-4113-a50d-01fa71e064e0-maas-api-openshift-ca\") pod \"8f1800fb-b1b8-4113-a50d-01fa71e064e0\" (UID: \"8f1800fb-b1b8-4113-a50d-01fa71e064e0\") " Nov 03 13:20:11.095680 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:11.095650 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f1800fb-b1b8-4113-a50d-01fa71e064e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f1800fb-b1b8-4113-a50d-01fa71e064e0-maas-api-xks-ca\") pod \"8f1800fb-b1b8-4113-a50d-01fa71e064e0\" (UID: \"8f1800fb-b1b8-4113-a50d-01fa71e064e0\") " Nov 03 13:20:11.095954 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:11.095690 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f1800fb-b1b8-4113-a50d-01fa71e064e0-kube-api-access-vjpn8\" (UniqueName: \"kubernetes.io/projected/8f1800fb-b1b8-4113-a50d-01fa71e064e0-kube-api-access-vjpn8\") pod \"8f1800fb-b1b8-4113-a50d-01fa71e064e0\" (UID: \"8f1800fb-b1b8-4113-a50d-01fa71e064e0\") " Nov 03 13:20:11.097395 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:11.097113 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f1800fb-b1b8-4113-a50d-01fa71e064e0-maas-api-openshift-ca" pod "8f1800fb-b1b8-4113-a50d-01fa71e064e0" (UID: "8f1800fb-b1b8-4113-a50d-01fa71e064e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:20:11.121959 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:11.121671 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f1800fb-b1b8-4113-a50d-01fa71e064e0-kube-api-access-vjpn8" pod "8f1800fb-b1b8-4113-a50d-01fa71e064e0" (UID: "8f1800fb-b1b8-4113-a50d-01fa71e064e0"). InnerVolumeSpecName "kube-api-access-vjpn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:20:11.121959 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:11.121760 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f1800fb-b1b8-4113-a50d-01fa71e064e0-maas-api-xks-ca" pod "8f1800fb-b1b8-4113-a50d-01fa71e064e0" (UID: "8f1800fb-b1b8-4113-a50d-01fa71e064e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:20:11.197341 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:11.197252 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f1800fb-b1b8-4113-a50d-01fa71e064e0-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:20:11.197341 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:11.197294 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f1800fb-b1b8-4113-a50d-01fa71e064e0-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:20:11.197341 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:11.197313 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjpn8\" (UniqueName: \"kubernetes.io/projected/8f1800fb-b1b8-4113-a50d-01fa71e064e0-kube-api-access-vjpn8\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:20:11.914358 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:11.914329 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93aefb28-r8bjv" Nov 03 13:20:11.914556 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:11.914333 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93aefb28-r8bjv" event={"ID":"8f1800fb-b1b8-4113-a50d-01fa71e064e0","Type":"ContainerDied","Data":"77c40bdc2e0793a11e16278b312d6e032d65c081ac5bd260d4ef99a20f43a44b"} Nov 03 13:20:11.914556 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:11.914436 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77c40bdc2e0793a11e16278b312d6e032d65c081ac5bd260d4ef99a20f43a44b" Nov 03 13:20:20.894904 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:20.894872 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:20:27.561578 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:27.561539 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9f5wz"] Nov 03 13:20:27.561977 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:27.561777 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9f5wz" podUID="1a14eed4-c891-453a-af30-354574069bc0" containerName="istio-proxy" containerID="cri-o://34a92327e9ddcfaa0aa13e6a764eff86bf05f3f38ac84ebd02a764796377a2d1" gracePeriod=30 Nov 03 13:20:27.608395 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:27.608363 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz"] Nov 03 13:20:27.608665 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:27.608634 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz" podUID="832a1db8-0f65-48a3-b561-30b77a5bc2bd" containerName="istio-proxy" containerID="cri-o://9d351475ba6fcc26c7d5073b1ea9c7f866dc80a852bf57b2b9c6cc402ae79c90" gracePeriod=30 Nov 03 13:20:29.088910 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:29.088833 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0922e14c-lhk8f"] Nov 03 13:20:29.089328 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:29.089315 5979 cpu_manager.go:414] "containerMap: removing container" podUID="8f1800fb-b1b8-4113-a50d-01fa71e064e0" containerName="revoke-keys" Nov 03 13:20:29.089372 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:29.089329 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f1800fb-b1b8-4113-a50d-01fa71e064e0" containerName="revoke-keys" Nov 03 13:20:29.089414 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:29.089405 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f1800fb-b1b8-4113-a50d-01fa71e064e0" containerName="revoke-keys" Nov 03 13:20:29.093922 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:29.093902 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0922e14c-lhk8f" Nov 03 13:20:29.145850 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:29.145810 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0922e14c-lhk8f"] Nov 03 13:20:29.189902 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:29.189866 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:20:29.190095 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:29.190009 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 13:20:29.190095 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:29.190031 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:20:29.190095 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:29.190067 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 13:20:29.208861 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:29.208824 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/463dd10d-31ba-4f68-af24-f68b13a04095-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0922e14c-lhk8f\" (UID: \"463dd10d-31ba-4f68-af24-f68b13a04095\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0922e14c-lhk8f" Nov 03 13:20:29.209032 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:29.208877 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/463dd10d-31ba-4f68-af24-f68b13a04095-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0922e14c-lhk8f\" (UID: \"463dd10d-31ba-4f68-af24-f68b13a04095\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0922e14c-lhk8f" Nov 03 13:20:29.209077 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:29.209038 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbqmp\" (UniqueName: \"kubernetes.io/projected/463dd10d-31ba-4f68-af24-f68b13a04095-kube-api-access-mbqmp\") pod \"maas-api-revoke-sub-0922e14c-lhk8f\" (UID: \"463dd10d-31ba-4f68-af24-f68b13a04095\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0922e14c-lhk8f" Nov 03 13:20:29.310675 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:29.310638 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbqmp\" (UniqueName: \"kubernetes.io/projected/463dd10d-31ba-4f68-af24-f68b13a04095-kube-api-access-mbqmp\") pod \"maas-api-revoke-sub-0922e14c-lhk8f\" (UID: \"463dd10d-31ba-4f68-af24-f68b13a04095\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0922e14c-lhk8f" Nov 03 13:20:29.310871 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:29.310707 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/463dd10d-31ba-4f68-af24-f68b13a04095-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0922e14c-lhk8f\" (UID: \"463dd10d-31ba-4f68-af24-f68b13a04095\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0922e14c-lhk8f" Nov 03 13:20:29.310871 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:29.310735 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/463dd10d-31ba-4f68-af24-f68b13a04095-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0922e14c-lhk8f\" (UID: \"463dd10d-31ba-4f68-af24-f68b13a04095\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0922e14c-lhk8f" Nov 03 13:20:29.311383 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:29.311361 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/463dd10d-31ba-4f68-af24-f68b13a04095-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0922e14c-lhk8f\" (UID: \"463dd10d-31ba-4f68-af24-f68b13a04095\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0922e14c-lhk8f" Nov 03 13:20:29.313078 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:29.313057 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/463dd10d-31ba-4f68-af24-f68b13a04095-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0922e14c-lhk8f\" (UID: \"463dd10d-31ba-4f68-af24-f68b13a04095\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0922e14c-lhk8f" Nov 03 13:20:29.364347 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:29.364275 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbqmp\" (UniqueName: \"kubernetes.io/projected/463dd10d-31ba-4f68-af24-f68b13a04095-kube-api-access-mbqmp\") pod \"maas-api-revoke-sub-0922e14c-lhk8f\" (UID: \"463dd10d-31ba-4f68-af24-f68b13a04095\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0922e14c-lhk8f" Nov 03 13:20:29.407154 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:29.407118 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0922e14c-lhk8f" Nov 03 13:20:29.778734 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:29.778702 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0922e14c-lhk8f"] Nov 03 13:20:29.786337 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:20:29.786312 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod463dd10d_31ba_4f68_af24_f68b13a04095.slice/crio-aeb7bee5cb000b873df379e9c8f73f56a8982cfa3726f6ead27024e562f986e7 WatchSource:0}: Error finding container aeb7bee5cb000b873df379e9c8f73f56a8982cfa3726f6ead27024e562f986e7: Status 404 returned error can't find the container with id aeb7bee5cb000b873df379e9c8f73f56a8982cfa3726f6ead27024e562f986e7 Nov 03 13:20:30.020362 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:30.020324 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0922e14c-lhk8f" event={"ID":"463dd10d-31ba-4f68-af24-f68b13a04095","Type":"ContainerStarted","Data":"aeb7bee5cb000b873df379e9c8f73f56a8982cfa3726f6ead27024e562f986e7"} Nov 03 13:20:31.027893 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:31.027860 5979 generic.go:356] "Generic (PLEG): container finished" podID="463dd10d-31ba-4f68-af24-f68b13a04095" containerID="f03a37fa604ee9e3e87066f2faed6236739b03d6a166a2dd066f1ec55a002be4" exitCode=0 Nov 03 13:20:31.028281 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:31.027945 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0922e14c-lhk8f" event={"ID":"463dd10d-31ba-4f68-af24-f68b13a04095","Type":"ContainerDied","Data":"f03a37fa604ee9e3e87066f2faed6236739b03d6a166a2dd066f1ec55a002be4"} Nov 03 13:20:31.207333 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:31.207292 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e9b2751-h8fbf"] Nov 03 13:20:31.211090 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:31.211066 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9b2751-h8fbf" Nov 03 13:20:31.264577 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:31.264541 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e9b2751-h8fbf"] Nov 03 13:20:31.333492 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:31.333386 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/289c69a7-8c3a-4376-8634-fe80d013f1bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e9b2751-h8fbf\" (UID: \"289c69a7-8c3a-4376-8634-fe80d013f1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9b2751-h8fbf" Nov 03 13:20:31.333492 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:31.333426 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/289c69a7-8c3a-4376-8634-fe80d013f1bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e9b2751-h8fbf\" (UID: \"289c69a7-8c3a-4376-8634-fe80d013f1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9b2751-h8fbf" Nov 03 13:20:31.333492 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:31.333449 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmjwt\" (UniqueName: \"kubernetes.io/projected/289c69a7-8c3a-4376-8634-fe80d013f1bf-kube-api-access-vmjwt\") pod \"maas-api-revoke-sub-8e9b2751-h8fbf\" (UID: \"289c69a7-8c3a-4376-8634-fe80d013f1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9b2751-h8fbf" Nov 03 13:20:31.434175 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:31.434134 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/289c69a7-8c3a-4376-8634-fe80d013f1bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e9b2751-h8fbf\" (UID: \"289c69a7-8c3a-4376-8634-fe80d013f1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9b2751-h8fbf" Nov 03 13:20:31.434175 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:31.434182 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/289c69a7-8c3a-4376-8634-fe80d013f1bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e9b2751-h8fbf\" (UID: \"289c69a7-8c3a-4376-8634-fe80d013f1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9b2751-h8fbf" Nov 03 13:20:31.434453 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:31.434215 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmjwt\" (UniqueName: \"kubernetes.io/projected/289c69a7-8c3a-4376-8634-fe80d013f1bf-kube-api-access-vmjwt\") pod \"maas-api-revoke-sub-8e9b2751-h8fbf\" (UID: \"289c69a7-8c3a-4376-8634-fe80d013f1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9b2751-h8fbf" Nov 03 13:20:31.434948 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:31.434920 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/289c69a7-8c3a-4376-8634-fe80d013f1bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e9b2751-h8fbf\" (UID: \"289c69a7-8c3a-4376-8634-fe80d013f1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9b2751-h8fbf" Nov 03 13:20:31.436863 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:31.436842 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/289c69a7-8c3a-4376-8634-fe80d013f1bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e9b2751-h8fbf\" (UID: \"289c69a7-8c3a-4376-8634-fe80d013f1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9b2751-h8fbf" Nov 03 13:20:31.488465 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:31.488433 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmjwt\" (UniqueName: \"kubernetes.io/projected/289c69a7-8c3a-4376-8634-fe80d013f1bf-kube-api-access-vmjwt\") pod \"maas-api-revoke-sub-8e9b2751-h8fbf\" (UID: \"289c69a7-8c3a-4376-8634-fe80d013f1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9b2751-h8fbf" Nov 03 13:20:31.523920 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:31.523878 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9b2751-h8fbf" Nov 03 13:20:31.895150 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:31.895113 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e9b2751-h8fbf"] Nov 03 13:20:31.902143 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:20:31.902115 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod289c69a7_8c3a_4376_8634_fe80d013f1bf.slice/crio-b2d7e149802236fcfa26fa435a7cb2cf3963d8a3785a6a19460a0414cc4ceabf WatchSource:0}: Error finding container b2d7e149802236fcfa26fa435a7cb2cf3963d8a3785a6a19460a0414cc4ceabf: Status 404 returned error can't find the container with id b2d7e149802236fcfa26fa435a7cb2cf3963d8a3785a6a19460a0414cc4ceabf Nov 03 13:20:32.035028 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:32.034993 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9b2751-h8fbf" event={"ID":"289c69a7-8c3a-4376-8634-fe80d013f1bf","Type":"ContainerStarted","Data":"b2d7e149802236fcfa26fa435a7cb2cf3963d8a3785a6a19460a0414cc4ceabf"} Nov 03 13:20:32.167136 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:32.167111 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0922e14c-lhk8f" Nov 03 13:20:32.343896 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:32.343854 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/463dd10d-31ba-4f68-af24-f68b13a04095-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/463dd10d-31ba-4f68-af24-f68b13a04095-maas-api-xks-ca\") pod \"463dd10d-31ba-4f68-af24-f68b13a04095\" (UID: \"463dd10d-31ba-4f68-af24-f68b13a04095\") " Nov 03 13:20:32.344110 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:32.343933 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/463dd10d-31ba-4f68-af24-f68b13a04095-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/463dd10d-31ba-4f68-af24-f68b13a04095-maas-api-openshift-ca\") pod \"463dd10d-31ba-4f68-af24-f68b13a04095\" (UID: \"463dd10d-31ba-4f68-af24-f68b13a04095\") " Nov 03 13:20:32.344110 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:32.343969 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/463dd10d-31ba-4f68-af24-f68b13a04095-kube-api-access-mbqmp\" (UniqueName: \"kubernetes.io/projected/463dd10d-31ba-4f68-af24-f68b13a04095-kube-api-access-mbqmp\") pod \"463dd10d-31ba-4f68-af24-f68b13a04095\" (UID: \"463dd10d-31ba-4f68-af24-f68b13a04095\") " Nov 03 13:20:32.344348 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:32.344324 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/463dd10d-31ba-4f68-af24-f68b13a04095-maas-api-openshift-ca" pod "463dd10d-31ba-4f68-af24-f68b13a04095" (UID: "463dd10d-31ba-4f68-af24-f68b13a04095"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:20:32.346358 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:32.346334 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/463dd10d-31ba-4f68-af24-f68b13a04095-maas-api-xks-ca" pod "463dd10d-31ba-4f68-af24-f68b13a04095" (UID: "463dd10d-31ba-4f68-af24-f68b13a04095"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:20:32.346504 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:32.346390 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/463dd10d-31ba-4f68-af24-f68b13a04095-kube-api-access-mbqmp" pod "463dd10d-31ba-4f68-af24-f68b13a04095" (UID: "463dd10d-31ba-4f68-af24-f68b13a04095"). InnerVolumeSpecName "kube-api-access-mbqmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:20:32.445400 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:32.445367 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbqmp\" (UniqueName: \"kubernetes.io/projected/463dd10d-31ba-4f68-af24-f68b13a04095-kube-api-access-mbqmp\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:20:32.445527 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:32.445404 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/463dd10d-31ba-4f68-af24-f68b13a04095-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:20:32.445527 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:32.445420 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/463dd10d-31ba-4f68-af24-f68b13a04095-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:20:32.862928 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:32.862903 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9f5wz" Nov 03 13:20:32.887682 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:32.887654 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz" Nov 03 13:20:33.042023 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.041937 5979 generic.go:356] "Generic (PLEG): container finished" podID="832a1db8-0f65-48a3-b561-30b77a5bc2bd" containerID="9d351475ba6fcc26c7d5073b1ea9c7f866dc80a852bf57b2b9c6cc402ae79c90" exitCode=0 Nov 03 13:20:33.042023 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.042000 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz" Nov 03 13:20:33.042525 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.042020 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz" event={"ID":"832a1db8-0f65-48a3-b561-30b77a5bc2bd","Type":"ContainerDied","Data":"9d351475ba6fcc26c7d5073b1ea9c7f866dc80a852bf57b2b9c6cc402ae79c90"} Nov 03 13:20:33.042525 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.042073 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz" event={"ID":"832a1db8-0f65-48a3-b561-30b77a5bc2bd","Type":"ContainerDied","Data":"9e0d29253ff808529edcb29eeabf170a770dbb16ddcfd7b99938dbdd82602796"} Nov 03 13:20:33.042525 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.042097 5979 scope.go:122] "RemoveContainer" containerID="9d351475ba6fcc26c7d5073b1ea9c7f866dc80a852bf57b2b9c6cc402ae79c90" Nov 03 13:20:33.043540 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.043510 5979 generic.go:356] "Generic (PLEG): container finished" podID="1a14eed4-c891-453a-af30-354574069bc0" containerID="34a92327e9ddcfaa0aa13e6a764eff86bf05f3f38ac84ebd02a764796377a2d1" exitCode=0 Nov 03 13:20:33.043661 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.043576 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9f5wz" Nov 03 13:20:33.043661 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.043605 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9f5wz" event={"ID":"1a14eed4-c891-453a-af30-354574069bc0","Type":"ContainerDied","Data":"34a92327e9ddcfaa0aa13e6a764eff86bf05f3f38ac84ebd02a764796377a2d1"} Nov 03 13:20:33.043769 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.043659 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9f5wz" event={"ID":"1a14eed4-c891-453a-af30-354574069bc0","Type":"ContainerDied","Data":"ab00879295e2b7771f6c0d70dfc642874df828772a3097e285797b63e236ef52"} Nov 03 13:20:33.044935 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.044917 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0922e14c-lhk8f" event={"ID":"463dd10d-31ba-4f68-af24-f68b13a04095","Type":"ContainerDied","Data":"aeb7bee5cb000b873df379e9c8f73f56a8982cfa3726f6ead27024e562f986e7"} Nov 03 13:20:33.044935 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.044929 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0922e14c-lhk8f" Nov 03 13:20:33.045090 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.044956 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aeb7bee5cb000b873df379e9c8f73f56a8982cfa3726f6ead27024e562f986e7" Nov 03 13:20:33.046441 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.046418 5979 generic.go:356] "Generic (PLEG): container finished" podID="289c69a7-8c3a-4376-8634-fe80d013f1bf" containerID="f84d8766f31a15e389a9087abc4a13116ed63f6e70dfe3efa8af8ca4b0a0da30" exitCode=0 Nov 03 13:20:33.046564 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.046498 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9b2751-h8fbf" event={"ID":"289c69a7-8c3a-4376-8634-fe80d013f1bf","Type":"ContainerDied","Data":"f84d8766f31a15e389a9087abc4a13116ed63f6e70dfe3efa8af8ca4b0a0da30"} Nov 03 13:20:33.050804 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.050782 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a14eed4-c891-453a-af30-354574069bc0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a14eed4-c891-453a-af30-354574069bc0-istio-data\") pod \"1a14eed4-c891-453a-af30-354574069bc0\" (UID: \"1a14eed4-c891-453a-af30-354574069bc0\") " Nov 03 13:20:33.050893 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.050821 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/832a1db8-0f65-48a3-b561-30b77a5bc2bd-istio-token\" (UniqueName: \"kubernetes.io/projected/832a1db8-0f65-48a3-b561-30b77a5bc2bd-istio-token\") pod \"832a1db8-0f65-48a3-b561-30b77a5bc2bd\" (UID: \"832a1db8-0f65-48a3-b561-30b77a5bc2bd\") " Nov 03 13:20:33.050893 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.050855 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a14eed4-c891-453a-af30-354574069bc0-istio-token\" (UniqueName: \"kubernetes.io/projected/1a14eed4-c891-453a-af30-354574069bc0-istio-token\") pod \"1a14eed4-c891-453a-af30-354574069bc0\" (UID: \"1a14eed4-c891-453a-af30-354574069bc0\") " Nov 03 13:20:33.050893 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.050884 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/832a1db8-0f65-48a3-b561-30b77a5bc2bd-kube-api-access-lrpt7\" (UniqueName: \"kubernetes.io/projected/832a1db8-0f65-48a3-b561-30b77a5bc2bd-kube-api-access-lrpt7\") pod \"832a1db8-0f65-48a3-b561-30b77a5bc2bd\" (UID: \"832a1db8-0f65-48a3-b561-30b77a5bc2bd\") " Nov 03 13:20:33.051053 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.050908 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/832a1db8-0f65-48a3-b561-30b77a5bc2bd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/832a1db8-0f65-48a3-b561-30b77a5bc2bd-credential-socket\") pod \"832a1db8-0f65-48a3-b561-30b77a5bc2bd\" (UID: \"832a1db8-0f65-48a3-b561-30b77a5bc2bd\") " Nov 03 13:20:33.051053 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.050960 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/832a1db8-0f65-48a3-b561-30b77a5bc2bd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/832a1db8-0f65-48a3-b561-30b77a5bc2bd-istio-podinfo\") pod \"832a1db8-0f65-48a3-b561-30b77a5bc2bd\" (UID: \"832a1db8-0f65-48a3-b561-30b77a5bc2bd\") " Nov 03 13:20:33.051053 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.050988 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a14eed4-c891-453a-af30-354574069bc0-kube-api-access-nv2rm\" (UniqueName: \"kubernetes.io/projected/1a14eed4-c891-453a-af30-354574069bc0-kube-api-access-nv2rm\") pod \"1a14eed4-c891-453a-af30-354574069bc0\" (UID: \"1a14eed4-c891-453a-af30-354574069bc0\") " Nov 03 13:20:33.051053 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.051016 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a14eed4-c891-453a-af30-354574069bc0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a14eed4-c891-453a-af30-354574069bc0-istio-podinfo\") pod \"1a14eed4-c891-453a-af30-354574069bc0\" (UID: \"1a14eed4-c891-453a-af30-354574069bc0\") " Nov 03 13:20:33.051053 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.051050 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/832a1db8-0f65-48a3-b561-30b77a5bc2bd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/832a1db8-0f65-48a3-b561-30b77a5bc2bd-workload-certs\") pod \"832a1db8-0f65-48a3-b561-30b77a5bc2bd\" (UID: \"832a1db8-0f65-48a3-b561-30b77a5bc2bd\") " Nov 03 13:20:33.051326 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.051063 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a14eed4-c891-453a-af30-354574069bc0-istio-data" pod "1a14eed4-c891-453a-af30-354574069bc0" (UID: "1a14eed4-c891-453a-af30-354574069bc0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:20:33.051326 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.051092 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/832a1db8-0f65-48a3-b561-30b77a5bc2bd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/832a1db8-0f65-48a3-b561-30b77a5bc2bd-workload-socket\") pod \"832a1db8-0f65-48a3-b561-30b77a5bc2bd\" (UID: \"832a1db8-0f65-48a3-b561-30b77a5bc2bd\") " Nov 03 13:20:33.051326 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.051118 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/832a1db8-0f65-48a3-b561-30b77a5bc2bd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/832a1db8-0f65-48a3-b561-30b77a5bc2bd-istio-envoy\") pod \"832a1db8-0f65-48a3-b561-30b77a5bc2bd\" (UID: \"832a1db8-0f65-48a3-b561-30b77a5bc2bd\") " Nov 03 13:20:33.051326 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.051145 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a14eed4-c891-453a-af30-354574069bc0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a14eed4-c891-453a-af30-354574069bc0-istio-envoy\") pod \"1a14eed4-c891-453a-af30-354574069bc0\" (UID: \"1a14eed4-c891-453a-af30-354574069bc0\") " Nov 03 13:20:33.051326 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.051178 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/832a1db8-0f65-48a3-b561-30b77a5bc2bd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/832a1db8-0f65-48a3-b561-30b77a5bc2bd-istio-data\") pod \"832a1db8-0f65-48a3-b561-30b77a5bc2bd\" (UID: \"832a1db8-0f65-48a3-b561-30b77a5bc2bd\") " Nov 03 13:20:33.051326 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.051214 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/832a1db8-0f65-48a3-b561-30b77a5bc2bd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/832a1db8-0f65-48a3-b561-30b77a5bc2bd-istiod-ca-cert\") pod \"832a1db8-0f65-48a3-b561-30b77a5bc2bd\" (UID: \"832a1db8-0f65-48a3-b561-30b77a5bc2bd\") " Nov 03 13:20:33.051326 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.051270 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a14eed4-c891-453a-af30-354574069bc0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a14eed4-c891-453a-af30-354574069bc0-credential-socket\") pod \"1a14eed4-c891-453a-af30-354574069bc0\" (UID: \"1a14eed4-c891-453a-af30-354574069bc0\") " Nov 03 13:20:33.051326 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.051300 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a14eed4-c891-453a-af30-354574069bc0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a14eed4-c891-453a-af30-354574069bc0-istiod-ca-cert\") pod \"1a14eed4-c891-453a-af30-354574069bc0\" (UID: \"1a14eed4-c891-453a-af30-354574069bc0\") " Nov 03 13:20:33.051753 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.051421 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a14eed4-c891-453a-af30-354574069bc0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a14eed4-c891-453a-af30-354574069bc0-workload-socket\") pod \"1a14eed4-c891-453a-af30-354574069bc0\" (UID: \"1a14eed4-c891-453a-af30-354574069bc0\") " Nov 03 13:20:33.051753 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.051543 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/832a1db8-0f65-48a3-b561-30b77a5bc2bd-workload-socket" pod "832a1db8-0f65-48a3-b561-30b77a5bc2bd" (UID: "832a1db8-0f65-48a3-b561-30b77a5bc2bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:20:33.051753 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.051565 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/832a1db8-0f65-48a3-b561-30b77a5bc2bd-credential-socket" pod "832a1db8-0f65-48a3-b561-30b77a5bc2bd" (UID: "832a1db8-0f65-48a3-b561-30b77a5bc2bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:20:33.051753 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.051587 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a14eed4-c891-453a-af30-354574069bc0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a14eed4-c891-453a-af30-354574069bc0-workload-certs\") pod \"1a14eed4-c891-453a-af30-354574069bc0\" (UID: \"1a14eed4-c891-453a-af30-354574069bc0\") " Nov 03 13:20:33.053510 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.052059 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/832a1db8-0f65-48a3-b561-30b77a5bc2bd-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:20:33.053510 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.052084 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/832a1db8-0f65-48a3-b561-30b77a5bc2bd-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:20:33.053510 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.052099 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a14eed4-c891-453a-af30-354574069bc0-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:20:33.053510 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.052323 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a14eed4-c891-453a-af30-354574069bc0-workload-certs" pod "1a14eed4-c891-453a-af30-354574069bc0" (UID: "1a14eed4-c891-453a-af30-354574069bc0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:20:33.054591 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.054556 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/832a1db8-0f65-48a3-b561-30b77a5bc2bd-istio-envoy" pod "832a1db8-0f65-48a3-b561-30b77a5bc2bd" (UID: "832a1db8-0f65-48a3-b561-30b77a5bc2bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:20:33.054821 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.054799 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/832a1db8-0f65-48a3-b561-30b77a5bc2bd-istio-podinfo" pod "832a1db8-0f65-48a3-b561-30b77a5bc2bd" (UID: "832a1db8-0f65-48a3-b561-30b77a5bc2bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:20:33.055147 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.055128 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a14eed4-c891-453a-af30-354574069bc0-kube-api-access-nv2rm" pod "1a14eed4-c891-453a-af30-354574069bc0" (UID: "1a14eed4-c891-453a-af30-354574069bc0"). InnerVolumeSpecName "kube-api-access-nv2rm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:20:33.055843 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.055816 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/832a1db8-0f65-48a3-b561-30b77a5bc2bd-workload-certs" pod "832a1db8-0f65-48a3-b561-30b77a5bc2bd" (UID: "832a1db8-0f65-48a3-b561-30b77a5bc2bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:20:33.056015 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.055997 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/832a1db8-0f65-48a3-b561-30b77a5bc2bd-kube-api-access-lrpt7" pod "832a1db8-0f65-48a3-b561-30b77a5bc2bd" (UID: "832a1db8-0f65-48a3-b561-30b77a5bc2bd"). InnerVolumeSpecName "kube-api-access-lrpt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:20:33.056130 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.056110 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a14eed4-c891-453a-af30-354574069bc0-istio-podinfo" pod "1a14eed4-c891-453a-af30-354574069bc0" (UID: "1a14eed4-c891-453a-af30-354574069bc0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:20:33.056296 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.056149 5979 scope.go:122] "RemoveContainer" containerID="9d351475ba6fcc26c7d5073b1ea9c7f866dc80a852bf57b2b9c6cc402ae79c90" Nov 03 13:20:33.056364 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.056346 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a14eed4-c891-453a-af30-354574069bc0-credential-socket" pod "1a14eed4-c891-453a-af30-354574069bc0" (UID: "1a14eed4-c891-453a-af30-354574069bc0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:20:33.056566 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.056545 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/832a1db8-0f65-48a3-b561-30b77a5bc2bd-istio-data" pod "832a1db8-0f65-48a3-b561-30b77a5bc2bd" (UID: "832a1db8-0f65-48a3-b561-30b77a5bc2bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:20:33.056695 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.056589 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a14eed4-c891-453a-af30-354574069bc0-istiod-ca-cert" pod "1a14eed4-c891-453a-af30-354574069bc0" (UID: "1a14eed4-c891-453a-af30-354574069bc0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:20:33.056876 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.056811 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a14eed4-c891-453a-af30-354574069bc0-istio-envoy" pod "1a14eed4-c891-453a-af30-354574069bc0" (UID: "1a14eed4-c891-453a-af30-354574069bc0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:20:33.057055 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.056910 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a14eed4-c891-453a-af30-354574069bc0-workload-socket" pod "1a14eed4-c891-453a-af30-354574069bc0" (UID: "1a14eed4-c891-453a-af30-354574069bc0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:20:33.057055 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.056971 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/832a1db8-0f65-48a3-b561-30b77a5bc2bd-istiod-ca-cert" pod "832a1db8-0f65-48a3-b561-30b77a5bc2bd" (UID: "832a1db8-0f65-48a3-b561-30b77a5bc2bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:20:33.057156 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.057057 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a14eed4-c891-453a-af30-354574069bc0-istio-token" pod "1a14eed4-c891-453a-af30-354574069bc0" (UID: "1a14eed4-c891-453a-af30-354574069bc0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:20:33.057200 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.057127 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/832a1db8-0f65-48a3-b561-30b77a5bc2bd-istio-token" pod "832a1db8-0f65-48a3-b561-30b77a5bc2bd" (UID: "832a1db8-0f65-48a3-b561-30b77a5bc2bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:20:33.064932 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:20:33.058704 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d351475ba6fcc26c7d5073b1ea9c7f866dc80a852bf57b2b9c6cc402ae79c90\": container with ID starting with 9d351475ba6fcc26c7d5073b1ea9c7f866dc80a852bf57b2b9c6cc402ae79c90 not found: ID does not exist" containerID="9d351475ba6fcc26c7d5073b1ea9c7f866dc80a852bf57b2b9c6cc402ae79c90" Nov 03 13:20:33.064932 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.058756 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d351475ba6fcc26c7d5073b1ea9c7f866dc80a852bf57b2b9c6cc402ae79c90"} err="failed to get container status \"9d351475ba6fcc26c7d5073b1ea9c7f866dc80a852bf57b2b9c6cc402ae79c90\": rpc error: code = NotFound desc = could not find container \"9d351475ba6fcc26c7d5073b1ea9c7f866dc80a852bf57b2b9c6cc402ae79c90\": container with ID starting with 9d351475ba6fcc26c7d5073b1ea9c7f866dc80a852bf57b2b9c6cc402ae79c90 not found: ID does not exist" Nov 03 13:20:33.064932 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.058799 5979 scope.go:122] "RemoveContainer" containerID="34a92327e9ddcfaa0aa13e6a764eff86bf05f3f38ac84ebd02a764796377a2d1" Nov 03 13:20:33.074244 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.074216 5979 scope.go:122] "RemoveContainer" containerID="34a92327e9ddcfaa0aa13e6a764eff86bf05f3f38ac84ebd02a764796377a2d1" Nov 03 13:20:33.074537 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:20:33.074515 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34a92327e9ddcfaa0aa13e6a764eff86bf05f3f38ac84ebd02a764796377a2d1\": container with ID starting with 34a92327e9ddcfaa0aa13e6a764eff86bf05f3f38ac84ebd02a764796377a2d1 not found: ID does not exist" containerID="34a92327e9ddcfaa0aa13e6a764eff86bf05f3f38ac84ebd02a764796377a2d1" Nov 03 13:20:33.074613 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.074545 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34a92327e9ddcfaa0aa13e6a764eff86bf05f3f38ac84ebd02a764796377a2d1"} err="failed to get container status \"34a92327e9ddcfaa0aa13e6a764eff86bf05f3f38ac84ebd02a764796377a2d1\": rpc error: code = NotFound desc = could not find container \"34a92327e9ddcfaa0aa13e6a764eff86bf05f3f38ac84ebd02a764796377a2d1\": container with ID starting with 34a92327e9ddcfaa0aa13e6a764eff86bf05f3f38ac84ebd02a764796377a2d1 not found: ID does not exist" Nov 03 13:20:33.153230 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.153200 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a14eed4-c891-453a-af30-354574069bc0-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:20:33.153230 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.153229 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/832a1db8-0f65-48a3-b561-30b77a5bc2bd-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:20:33.153425 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.153244 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a14eed4-c891-453a-af30-354574069bc0-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:20:33.153425 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.153254 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrpt7\" (UniqueName: \"kubernetes.io/projected/832a1db8-0f65-48a3-b561-30b77a5bc2bd-kube-api-access-lrpt7\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:20:33.153425 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.153265 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/832a1db8-0f65-48a3-b561-30b77a5bc2bd-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:20:33.153425 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.153273 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv2rm\" (UniqueName: \"kubernetes.io/projected/1a14eed4-c891-453a-af30-354574069bc0-kube-api-access-nv2rm\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:20:33.153425 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.153281 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a14eed4-c891-453a-af30-354574069bc0-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:20:33.153425 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.153291 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/832a1db8-0f65-48a3-b561-30b77a5bc2bd-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:20:33.153425 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.153299 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/832a1db8-0f65-48a3-b561-30b77a5bc2bd-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:20:33.153425 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.153312 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a14eed4-c891-453a-af30-354574069bc0-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:20:33.153425 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.153322 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/832a1db8-0f65-48a3-b561-30b77a5bc2bd-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:20:33.153425 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.153330 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/832a1db8-0f65-48a3-b561-30b77a5bc2bd-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:20:33.153425 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.153338 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a14eed4-c891-453a-af30-354574069bc0-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:20:33.153425 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.153347 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a14eed4-c891-453a-af30-354574069bc0-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:20:33.153425 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.153355 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a14eed4-c891-453a-af30-354574069bc0-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:20:33.468583 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.468550 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz"] Nov 03 13:20:33.469485 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.469437 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-jmzwz"] Nov 03 13:20:33.567289 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.567254 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9f5wz"] Nov 03 13:20:33.570283 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:33.570256 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-9f5wz"] Nov 03 13:20:34.118429 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:34.118394 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a14eed4-c891-453a-af30-354574069bc0" path="/var/lib/kubelet/pods/1a14eed4-c891-453a-af30-354574069bc0/volumes" Nov 03 13:20:34.119037 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:34.119016 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="832a1db8-0f65-48a3-b561-30b77a5bc2bd" path="/var/lib/kubelet/pods/832a1db8-0f65-48a3-b561-30b77a5bc2bd/volumes" Nov 03 13:20:34.195953 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:34.195927 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9b2751-h8fbf" Nov 03 13:20:34.271037 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:34.270997 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/289c69a7-8c3a-4376-8634-fe80d013f1bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/289c69a7-8c3a-4376-8634-fe80d013f1bf-maas-api-openshift-ca\") pod \"289c69a7-8c3a-4376-8634-fe80d013f1bf\" (UID: \"289c69a7-8c3a-4376-8634-fe80d013f1bf\") " Nov 03 13:20:34.271192 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:34.271154 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/289c69a7-8c3a-4376-8634-fe80d013f1bf-kube-api-access-vmjwt\" (UniqueName: \"kubernetes.io/projected/289c69a7-8c3a-4376-8634-fe80d013f1bf-kube-api-access-vmjwt\") pod \"289c69a7-8c3a-4376-8634-fe80d013f1bf\" (UID: \"289c69a7-8c3a-4376-8634-fe80d013f1bf\") " Nov 03 13:20:34.271192 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:34.271177 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/289c69a7-8c3a-4376-8634-fe80d013f1bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/289c69a7-8c3a-4376-8634-fe80d013f1bf-maas-api-xks-ca\") pod \"289c69a7-8c3a-4376-8634-fe80d013f1bf\" (UID: \"289c69a7-8c3a-4376-8634-fe80d013f1bf\") " Nov 03 13:20:34.271370 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:34.271342 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/289c69a7-8c3a-4376-8634-fe80d013f1bf-maas-api-openshift-ca" pod "289c69a7-8c3a-4376-8634-fe80d013f1bf" (UID: "289c69a7-8c3a-4376-8634-fe80d013f1bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:20:34.271515 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:34.271490 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/289c69a7-8c3a-4376-8634-fe80d013f1bf-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:20:34.273272 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:34.273245 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/289c69a7-8c3a-4376-8634-fe80d013f1bf-kube-api-access-vmjwt" pod "289c69a7-8c3a-4376-8634-fe80d013f1bf" (UID: "289c69a7-8c3a-4376-8634-fe80d013f1bf"). InnerVolumeSpecName "kube-api-access-vmjwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:20:34.273376 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:34.273261 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/289c69a7-8c3a-4376-8634-fe80d013f1bf-maas-api-xks-ca" pod "289c69a7-8c3a-4376-8634-fe80d013f1bf" (UID: "289c69a7-8c3a-4376-8634-fe80d013f1bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:20:34.372564 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:34.372465 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmjwt\" (UniqueName: \"kubernetes.io/projected/289c69a7-8c3a-4376-8634-fe80d013f1bf-kube-api-access-vmjwt\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:20:34.372564 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:34.372522 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/289c69a7-8c3a-4376-8634-fe80d013f1bf-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:20:35.062814 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:35.062780 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9b2751-h8fbf" Nov 03 13:20:35.062964 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:35.062816 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e9b2751-h8fbf" event={"ID":"289c69a7-8c3a-4376-8634-fe80d013f1bf","Type":"ContainerDied","Data":"b2d7e149802236fcfa26fa435a7cb2cf3963d8a3785a6a19460a0414cc4ceabf"} Nov 03 13:20:35.062964 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:35.062869 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2d7e149802236fcfa26fa435a7cb2cf3963d8a3785a6a19460a0414cc4ceabf" Nov 03 13:20:37.233525 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:20:37.233461 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:20:44.824878 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:44.824841 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb46c11f-jq7sn"] Nov 03 13:20:44.825349 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:44.825323 5979 cpu_manager.go:414] "containerMap: removing container" podUID="289c69a7-8c3a-4376-8634-fe80d013f1bf" containerName="revoke-keys" Nov 03 13:20:44.825349 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:44.825334 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="289c69a7-8c3a-4376-8634-fe80d013f1bf" containerName="revoke-keys" Nov 03 13:20:44.825349 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:44.825348 5979 cpu_manager.go:414] "containerMap: removing container" podUID="463dd10d-31ba-4f68-af24-f68b13a04095" containerName="revoke-keys" Nov 03 13:20:44.825449 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:44.825353 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="463dd10d-31ba-4f68-af24-f68b13a04095" containerName="revoke-keys" Nov 03 13:20:44.825449 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:44.825367 5979 cpu_manager.go:414] "containerMap: removing container" podUID="832a1db8-0f65-48a3-b561-30b77a5bc2bd" containerName="istio-proxy" Nov 03 13:20:44.825449 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:44.825373 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="832a1db8-0f65-48a3-b561-30b77a5bc2bd" containerName="istio-proxy" Nov 03 13:20:44.825449 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:44.825379 5979 cpu_manager.go:414] "containerMap: removing container" podUID="1a14eed4-c891-453a-af30-354574069bc0" containerName="istio-proxy" Nov 03 13:20:44.825449 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:44.825384 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a14eed4-c891-453a-af30-354574069bc0" containerName="istio-proxy" Nov 03 13:20:44.825449 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:44.825449 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="832a1db8-0f65-48a3-b561-30b77a5bc2bd" containerName="istio-proxy" Nov 03 13:20:44.825669 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:44.825458 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a14eed4-c891-453a-af30-354574069bc0" containerName="istio-proxy" Nov 03 13:20:44.825669 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:44.825485 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="289c69a7-8c3a-4376-8634-fe80d013f1bf" containerName="revoke-keys" Nov 03 13:20:44.825669 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:44.825493 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="463dd10d-31ba-4f68-af24-f68b13a04095" containerName="revoke-keys" Nov 03 13:20:44.828782 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:44.828757 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb46c11f-jq7sn" Nov 03 13:20:44.889377 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:44.889334 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8efac707-fd2a-473e-afce-7e3aa9caeaa9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb46c11f-jq7sn\" (UID: \"8efac707-fd2a-473e-afce-7e3aa9caeaa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb46c11f-jq7sn" Nov 03 13:20:44.889565 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:44.889393 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8efac707-fd2a-473e-afce-7e3aa9caeaa9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb46c11f-jq7sn\" (UID: \"8efac707-fd2a-473e-afce-7e3aa9caeaa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb46c11f-jq7sn" Nov 03 13:20:44.889639 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:44.889612 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttlrv\" (UniqueName: \"kubernetes.io/projected/8efac707-fd2a-473e-afce-7e3aa9caeaa9-kube-api-access-ttlrv\") pod \"maas-api-revoke-sub-eb46c11f-jq7sn\" (UID: \"8efac707-fd2a-473e-afce-7e3aa9caeaa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb46c11f-jq7sn" Nov 03 13:20:44.925504 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:44.925445 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:20:44.925665 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:44.925503 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:20:44.925665 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:44.925516 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 13:20:44.925665 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:44.925532 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 13:20:44.927466 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:44.927450 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb46c11f-jq7sn"] Nov 03 13:20:44.990687 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:44.990651 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttlrv\" (UniqueName: \"kubernetes.io/projected/8efac707-fd2a-473e-afce-7e3aa9caeaa9-kube-api-access-ttlrv\") pod \"maas-api-revoke-sub-eb46c11f-jq7sn\" (UID: \"8efac707-fd2a-473e-afce-7e3aa9caeaa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb46c11f-jq7sn" Nov 03 13:20:44.990883 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:44.990713 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8efac707-fd2a-473e-afce-7e3aa9caeaa9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb46c11f-jq7sn\" (UID: \"8efac707-fd2a-473e-afce-7e3aa9caeaa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb46c11f-jq7sn" Nov 03 13:20:44.990883 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:44.990745 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8efac707-fd2a-473e-afce-7e3aa9caeaa9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb46c11f-jq7sn\" (UID: \"8efac707-fd2a-473e-afce-7e3aa9caeaa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb46c11f-jq7sn" Nov 03 13:20:44.991415 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:44.991392 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8efac707-fd2a-473e-afce-7e3aa9caeaa9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb46c11f-jq7sn\" (UID: \"8efac707-fd2a-473e-afce-7e3aa9caeaa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb46c11f-jq7sn" Nov 03 13:20:44.993040 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:44.993018 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8efac707-fd2a-473e-afce-7e3aa9caeaa9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb46c11f-jq7sn\" (UID: \"8efac707-fd2a-473e-afce-7e3aa9caeaa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb46c11f-jq7sn" Nov 03 13:20:45.044911 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:45.044876 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttlrv\" (UniqueName: \"kubernetes.io/projected/8efac707-fd2a-473e-afce-7e3aa9caeaa9-kube-api-access-ttlrv\") pod \"maas-api-revoke-sub-eb46c11f-jq7sn\" (UID: \"8efac707-fd2a-473e-afce-7e3aa9caeaa9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb46c11f-jq7sn" Nov 03 13:20:45.142395 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:45.142363 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb46c11f-jq7sn" Nov 03 13:20:45.514379 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:45.514302 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb46c11f-jq7sn"] Nov 03 13:20:45.523261 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:20:45.523232 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8efac707_fd2a_473e_afce_7e3aa9caeaa9.slice/crio-cbecc979210e28aefb1064b708af07cc12c525ee0664d3ea7f721c2e8cad3d5c WatchSource:0}: Error finding container cbecc979210e28aefb1064b708af07cc12c525ee0664d3ea7f721c2e8cad3d5c: Status 404 returned error can't find the container with id cbecc979210e28aefb1064b708af07cc12c525ee0664d3ea7f721c2e8cad3d5c Nov 03 13:20:46.116699 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:46.116626 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:20:46.137455 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:46.137418 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb46c11f-jq7sn" event={"ID":"8efac707-fd2a-473e-afce-7e3aa9caeaa9","Type":"ContainerStarted","Data":"62b02f1874dd3e55cb02585b3552ec012e3c8335c43db15a46ef8e906ae8f6c9"} Nov 03 13:20:46.137663 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:46.137493 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb46c11f-jq7sn" event={"ID":"8efac707-fd2a-473e-afce-7e3aa9caeaa9","Type":"ContainerStarted","Data":"cbecc979210e28aefb1064b708af07cc12c525ee0664d3ea7f721c2e8cad3d5c"} Nov 03 13:20:46.244805 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:46.244742 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb46c11f-jq7sn" podStartSLOduration=2.244723112 podStartE2EDuration="2.244723112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:20:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:20:46.243876854 +0000 UTC m=+928.254565097" watchObservedRunningTime="2026-11-03 13:20:46.244723112 +0000 UTC m=+928.255411362" Nov 03 13:20:47.145619 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:47.145586 5979 generic.go:356] "Generic (PLEG): container finished" podID="8efac707-fd2a-473e-afce-7e3aa9caeaa9" containerID="62b02f1874dd3e55cb02585b3552ec012e3c8335c43db15a46ef8e906ae8f6c9" exitCode=0 Nov 03 13:20:47.146062 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:47.145633 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb46c11f-jq7sn" event={"ID":"8efac707-fd2a-473e-afce-7e3aa9caeaa9","Type":"ContainerDied","Data":"62b02f1874dd3e55cb02585b3552ec012e3c8335c43db15a46ef8e906ae8f6c9"} Nov 03 13:20:48.300818 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:48.300791 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb46c11f-jq7sn" Nov 03 13:20:48.434288 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:48.434193 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8efac707-fd2a-473e-afce-7e3aa9caeaa9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8efac707-fd2a-473e-afce-7e3aa9caeaa9-maas-api-xks-ca\") pod \"8efac707-fd2a-473e-afce-7e3aa9caeaa9\" (UID: \"8efac707-fd2a-473e-afce-7e3aa9caeaa9\") " Nov 03 13:20:48.434458 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:48.434300 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8efac707-fd2a-473e-afce-7e3aa9caeaa9-kube-api-access-ttlrv\" (UniqueName: \"kubernetes.io/projected/8efac707-fd2a-473e-afce-7e3aa9caeaa9-kube-api-access-ttlrv\") pod \"8efac707-fd2a-473e-afce-7e3aa9caeaa9\" (UID: \"8efac707-fd2a-473e-afce-7e3aa9caeaa9\") " Nov 03 13:20:48.434458 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:48.434345 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8efac707-fd2a-473e-afce-7e3aa9caeaa9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8efac707-fd2a-473e-afce-7e3aa9caeaa9-maas-api-openshift-ca\") pod \"8efac707-fd2a-473e-afce-7e3aa9caeaa9\" (UID: \"8efac707-fd2a-473e-afce-7e3aa9caeaa9\") " Nov 03 13:20:48.434958 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:48.434921 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8efac707-fd2a-473e-afce-7e3aa9caeaa9-maas-api-openshift-ca" pod "8efac707-fd2a-473e-afce-7e3aa9caeaa9" (UID: "8efac707-fd2a-473e-afce-7e3aa9caeaa9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:20:48.436519 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:48.436490 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8efac707-fd2a-473e-afce-7e3aa9caeaa9-maas-api-xks-ca" pod "8efac707-fd2a-473e-afce-7e3aa9caeaa9" (UID: "8efac707-fd2a-473e-afce-7e3aa9caeaa9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:20:48.436631 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:48.436586 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8efac707-fd2a-473e-afce-7e3aa9caeaa9-kube-api-access-ttlrv" pod "8efac707-fd2a-473e-afce-7e3aa9caeaa9" (UID: "8efac707-fd2a-473e-afce-7e3aa9caeaa9"). InnerVolumeSpecName "kube-api-access-ttlrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:20:48.535495 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:48.535437 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttlrv\" (UniqueName: \"kubernetes.io/projected/8efac707-fd2a-473e-afce-7e3aa9caeaa9-kube-api-access-ttlrv\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:20:48.535495 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:48.535481 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8efac707-fd2a-473e-afce-7e3aa9caeaa9-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:20:48.535495 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:48.535496 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8efac707-fd2a-473e-afce-7e3aa9caeaa9-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:20:49.166643 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:49.166609 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb46c11f-jq7sn" Nov 03 13:20:49.166643 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:49.166633 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb46c11f-jq7sn" event={"ID":"8efac707-fd2a-473e-afce-7e3aa9caeaa9","Type":"ContainerDied","Data":"cbecc979210e28aefb1064b708af07cc12c525ee0664d3ea7f721c2e8cad3d5c"} Nov 03 13:20:49.166857 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:49.166679 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbecc979210e28aefb1064b708af07cc12c525ee0664d3ea7f721c2e8cad3d5c" Nov 03 13:20:51.470552 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:51.470516 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe460bed-7q2pg"] Nov 03 13:20:51.471029 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:51.471015 5979 cpu_manager.go:414] "containerMap: removing container" podUID="8efac707-fd2a-473e-afce-7e3aa9caeaa9" containerName="revoke-keys" Nov 03 13:20:51.471079 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:51.471031 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8efac707-fd2a-473e-afce-7e3aa9caeaa9" containerName="revoke-keys" Nov 03 13:20:51.471118 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:51.471105 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="8efac707-fd2a-473e-afce-7e3aa9caeaa9" containerName="revoke-keys" Nov 03 13:20:51.474341 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:51.474324 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe460bed-7q2pg" Nov 03 13:20:51.525246 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:51.525208 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 13:20:51.563294 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:51.563254 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/057489b7-ac12-4d95-9bc1-bf100a735dd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe460bed-7q2pg\" (UID: \"057489b7-ac12-4d95-9bc1-bf100a735dd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe460bed-7q2pg" Nov 03 13:20:51.563492 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:51.563313 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxmsf\" (UniqueName: \"kubernetes.io/projected/057489b7-ac12-4d95-9bc1-bf100a735dd2-kube-api-access-kxmsf\") pod \"maas-api-revoke-sub-fe460bed-7q2pg\" (UID: \"057489b7-ac12-4d95-9bc1-bf100a735dd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe460bed-7q2pg" Nov 03 13:20:51.563492 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:51.563418 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/057489b7-ac12-4d95-9bc1-bf100a735dd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe460bed-7q2pg\" (UID: \"057489b7-ac12-4d95-9bc1-bf100a735dd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe460bed-7q2pg" Nov 03 13:20:51.571729 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:51.571693 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:20:51.571729 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:51.571722 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:20:51.571729 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:51.571704 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 13:20:51.573654 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:51.573634 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe460bed-7q2pg"] Nov 03 13:20:51.664383 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:51.664332 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/057489b7-ac12-4d95-9bc1-bf100a735dd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe460bed-7q2pg\" (UID: \"057489b7-ac12-4d95-9bc1-bf100a735dd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe460bed-7q2pg" Nov 03 13:20:51.664646 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:51.664501 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/057489b7-ac12-4d95-9bc1-bf100a735dd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe460bed-7q2pg\" (UID: \"057489b7-ac12-4d95-9bc1-bf100a735dd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe460bed-7q2pg" Nov 03 13:20:51.664646 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:51.664555 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxmsf\" (UniqueName: \"kubernetes.io/projected/057489b7-ac12-4d95-9bc1-bf100a735dd2-kube-api-access-kxmsf\") pod \"maas-api-revoke-sub-fe460bed-7q2pg\" (UID: \"057489b7-ac12-4d95-9bc1-bf100a735dd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe460bed-7q2pg" Nov 03 13:20:51.665209 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:51.665189 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/057489b7-ac12-4d95-9bc1-bf100a735dd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe460bed-7q2pg\" (UID: \"057489b7-ac12-4d95-9bc1-bf100a735dd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe460bed-7q2pg" Nov 03 13:20:51.666720 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:51.666704 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/057489b7-ac12-4d95-9bc1-bf100a735dd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe460bed-7q2pg\" (UID: \"057489b7-ac12-4d95-9bc1-bf100a735dd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe460bed-7q2pg" Nov 03 13:20:51.718161 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:51.718129 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxmsf\" (UniqueName: \"kubernetes.io/projected/057489b7-ac12-4d95-9bc1-bf100a735dd2-kube-api-access-kxmsf\") pod \"maas-api-revoke-sub-fe460bed-7q2pg\" (UID: \"057489b7-ac12-4d95-9bc1-bf100a735dd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe460bed-7q2pg" Nov 03 13:20:51.787766 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:51.787683 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe460bed-7q2pg" Nov 03 13:20:52.158531 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:52.158499 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe460bed-7q2pg"] Nov 03 13:20:52.166002 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:20:52.165974 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod057489b7_ac12_4d95_9bc1_bf100a735dd2.slice/crio-203b1ffda52f125624ff187fa9587676a543b392ebcbd9b8bfec22b6dd9d16d8 WatchSource:0}: Error finding container 203b1ffda52f125624ff187fa9587676a543b392ebcbd9b8bfec22b6dd9d16d8: Status 404 returned error can't find the container with id 203b1ffda52f125624ff187fa9587676a543b392ebcbd9b8bfec22b6dd9d16d8 Nov 03 13:20:52.186333 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:52.186307 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe460bed-7q2pg" event={"ID":"057489b7-ac12-4d95-9bc1-bf100a735dd2","Type":"ContainerStarted","Data":"203b1ffda52f125624ff187fa9587676a543b392ebcbd9b8bfec22b6dd9d16d8"} Nov 03 13:20:53.193348 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:53.193252 5979 generic.go:356] "Generic (PLEG): container finished" podID="057489b7-ac12-4d95-9bc1-bf100a735dd2" containerID="e9d0324e2eba7d820e1d4ea1c6fee494e9b6928346b6c0d26aa9c1f9c44b7c0e" exitCode=0 Nov 03 13:20:53.193348 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:53.193294 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe460bed-7q2pg" event={"ID":"057489b7-ac12-4d95-9bc1-bf100a735dd2","Type":"ContainerDied","Data":"e9d0324e2eba7d820e1d4ea1c6fee494e9b6928346b6c0d26aa9c1f9c44b7c0e"} Nov 03 13:20:54.329972 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:54.329946 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe460bed-7q2pg" Nov 03 13:20:54.392304 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:54.392280 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/057489b7-ac12-4d95-9bc1-bf100a735dd2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/057489b7-ac12-4d95-9bc1-bf100a735dd2-maas-api-openshift-ca\") pod \"057489b7-ac12-4d95-9bc1-bf100a735dd2\" (UID: \"057489b7-ac12-4d95-9bc1-bf100a735dd2\") " Nov 03 13:20:54.392497 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:54.392329 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/057489b7-ac12-4d95-9bc1-bf100a735dd2-kube-api-access-kxmsf\" (UniqueName: \"kubernetes.io/projected/057489b7-ac12-4d95-9bc1-bf100a735dd2-kube-api-access-kxmsf\") pod \"057489b7-ac12-4d95-9bc1-bf100a735dd2\" (UID: \"057489b7-ac12-4d95-9bc1-bf100a735dd2\") " Nov 03 13:20:54.392497 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:54.392412 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/057489b7-ac12-4d95-9bc1-bf100a735dd2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/057489b7-ac12-4d95-9bc1-bf100a735dd2-maas-api-xks-ca\") pod \"057489b7-ac12-4d95-9bc1-bf100a735dd2\" (UID: \"057489b7-ac12-4d95-9bc1-bf100a735dd2\") " Nov 03 13:20:54.392660 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:54.392636 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/057489b7-ac12-4d95-9bc1-bf100a735dd2-maas-api-openshift-ca" pod "057489b7-ac12-4d95-9bc1-bf100a735dd2" (UID: "057489b7-ac12-4d95-9bc1-bf100a735dd2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:20:54.392796 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:54.392780 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/057489b7-ac12-4d95-9bc1-bf100a735dd2-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:20:54.394466 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:54.394437 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/057489b7-ac12-4d95-9bc1-bf100a735dd2-maas-api-xks-ca" pod "057489b7-ac12-4d95-9bc1-bf100a735dd2" (UID: "057489b7-ac12-4d95-9bc1-bf100a735dd2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:20:54.394580 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:54.394542 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/057489b7-ac12-4d95-9bc1-bf100a735dd2-kube-api-access-kxmsf" pod "057489b7-ac12-4d95-9bc1-bf100a735dd2" (UID: "057489b7-ac12-4d95-9bc1-bf100a735dd2"). InnerVolumeSpecName "kube-api-access-kxmsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:20:54.494224 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:54.494186 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxmsf\" (UniqueName: \"kubernetes.io/projected/057489b7-ac12-4d95-9bc1-bf100a735dd2-kube-api-access-kxmsf\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:20:54.494224 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:54.494217 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/057489b7-ac12-4d95-9bc1-bf100a735dd2-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:20:55.206195 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:55.206165 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe460bed-7q2pg" Nov 03 13:20:55.206195 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:55.206188 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe460bed-7q2pg" event={"ID":"057489b7-ac12-4d95-9bc1-bf100a735dd2","Type":"ContainerDied","Data":"203b1ffda52f125624ff187fa9587676a543b392ebcbd9b8bfec22b6dd9d16d8"} Nov 03 13:20:55.206403 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:20:55.206231 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="203b1ffda52f125624ff187fa9587676a543b392ebcbd9b8bfec22b6dd9d16d8" Nov 03 13:21:01.283939 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:01.283896 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07596b90-87n7l"] Nov 03 13:21:01.284396 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:01.284382 5979 cpu_manager.go:414] "containerMap: removing container" podUID="057489b7-ac12-4d95-9bc1-bf100a735dd2" containerName="revoke-keys" Nov 03 13:21:01.284440 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:01.284396 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="057489b7-ac12-4d95-9bc1-bf100a735dd2" containerName="revoke-keys" Nov 03 13:21:01.284525 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:01.284512 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="057489b7-ac12-4d95-9bc1-bf100a735dd2" containerName="revoke-keys" Nov 03 13:21:01.287928 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:01.287904 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07596b90-87n7l" Nov 03 13:21:01.341379 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:01.341345 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:21:01.387955 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:01.387898 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:21:01.387955 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:01.387911 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 13:21:01.387955 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:01.387957 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 13:21:01.390078 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:01.390058 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07596b90-87n7l"] Nov 03 13:21:01.467504 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:01.467452 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fgr5\" (UniqueName: \"kubernetes.io/projected/6bc1e136-292c-425b-b481-3836bb343d3b-kube-api-access-5fgr5\") pod \"maas-api-revoke-sub-07596b90-87n7l\" (UID: \"6bc1e136-292c-425b-b481-3836bb343d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07596b90-87n7l" Nov 03 13:21:01.467703 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:01.467681 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bc1e136-292c-425b-b481-3836bb343d3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07596b90-87n7l\" (UID: \"6bc1e136-292c-425b-b481-3836bb343d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07596b90-87n7l" Nov 03 13:21:01.467803 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:01.467785 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bc1e136-292c-425b-b481-3836bb343d3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07596b90-87n7l\" (UID: \"6bc1e136-292c-425b-b481-3836bb343d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07596b90-87n7l" Nov 03 13:21:01.568439 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:01.568344 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bc1e136-292c-425b-b481-3836bb343d3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07596b90-87n7l\" (UID: \"6bc1e136-292c-425b-b481-3836bb343d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07596b90-87n7l" Nov 03 13:21:01.568439 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:01.568406 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bc1e136-292c-425b-b481-3836bb343d3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07596b90-87n7l\" (UID: \"6bc1e136-292c-425b-b481-3836bb343d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07596b90-87n7l" Nov 03 13:21:01.568651 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:01.568449 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fgr5\" (UniqueName: \"kubernetes.io/projected/6bc1e136-292c-425b-b481-3836bb343d3b-kube-api-access-5fgr5\") pod \"maas-api-revoke-sub-07596b90-87n7l\" (UID: \"6bc1e136-292c-425b-b481-3836bb343d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07596b90-87n7l" Nov 03 13:21:01.569143 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:01.569116 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bc1e136-292c-425b-b481-3836bb343d3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07596b90-87n7l\" (UID: \"6bc1e136-292c-425b-b481-3836bb343d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07596b90-87n7l" Nov 03 13:21:01.570726 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:01.570701 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bc1e136-292c-425b-b481-3836bb343d3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07596b90-87n7l\" (UID: \"6bc1e136-292c-425b-b481-3836bb343d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07596b90-87n7l" Nov 03 13:21:01.643658 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:01.643623 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fgr5\" (UniqueName: \"kubernetes.io/projected/6bc1e136-292c-425b-b481-3836bb343d3b-kube-api-access-5fgr5\") pod \"maas-api-revoke-sub-07596b90-87n7l\" (UID: \"6bc1e136-292c-425b-b481-3836bb343d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07596b90-87n7l" Nov 03 13:21:01.900950 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:01.900912 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07596b90-87n7l" Nov 03 13:21:02.269973 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:02.269889 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07596b90-87n7l"] Nov 03 13:21:02.277159 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:21:02.277134 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bc1e136_292c_425b_b481_3836bb343d3b.slice/crio-9c539dd26badeee3195d5eeba974b91d71f76f33667ba50a0244790b50222e4d WatchSource:0}: Error finding container 9c539dd26badeee3195d5eeba974b91d71f76f33667ba50a0244790b50222e4d: Status 404 returned error can't find the container with id 9c539dd26badeee3195d5eeba974b91d71f76f33667ba50a0244790b50222e4d Nov 03 13:21:02.279958 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:02.279936 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0494db61-rrv4c"] Nov 03 13:21:02.285363 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:02.285346 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0494db61-rrv4c" Nov 03 13:21:02.374136 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:02.374098 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0494db61-rrv4c"] Nov 03 13:21:02.378091 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:02.377933 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b68be428-2283-4a4f-a55e-b9bb113d8d3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0494db61-rrv4c\" (UID: \"b68be428-2283-4a4f-a55e-b9bb113d8d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0494db61-rrv4c" Nov 03 13:21:02.378259 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:02.378042 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b68be428-2283-4a4f-a55e-b9bb113d8d3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0494db61-rrv4c\" (UID: \"b68be428-2283-4a4f-a55e-b9bb113d8d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0494db61-rrv4c" Nov 03 13:21:02.378364 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:02.378277 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjnn9\" (UniqueName: \"kubernetes.io/projected/b68be428-2283-4a4f-a55e-b9bb113d8d3b-kube-api-access-sjnn9\") pod \"maas-api-revoke-sub-0494db61-rrv4c\" (UID: \"b68be428-2283-4a4f-a55e-b9bb113d8d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0494db61-rrv4c" Nov 03 13:21:02.484899 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:02.479141 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b68be428-2283-4a4f-a55e-b9bb113d8d3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0494db61-rrv4c\" (UID: \"b68be428-2283-4a4f-a55e-b9bb113d8d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0494db61-rrv4c" Nov 03 13:21:02.484899 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:02.479246 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b68be428-2283-4a4f-a55e-b9bb113d8d3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0494db61-rrv4c\" (UID: \"b68be428-2283-4a4f-a55e-b9bb113d8d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0494db61-rrv4c" Nov 03 13:21:02.484899 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:02.480160 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b68be428-2283-4a4f-a55e-b9bb113d8d3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0494db61-rrv4c\" (UID: \"b68be428-2283-4a4f-a55e-b9bb113d8d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0494db61-rrv4c" Nov 03 13:21:02.484899 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:02.480210 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjnn9\" (UniqueName: \"kubernetes.io/projected/b68be428-2283-4a4f-a55e-b9bb113d8d3b-kube-api-access-sjnn9\") pod \"maas-api-revoke-sub-0494db61-rrv4c\" (UID: \"b68be428-2283-4a4f-a55e-b9bb113d8d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0494db61-rrv4c" Nov 03 13:21:02.486324 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:02.486286 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b68be428-2283-4a4f-a55e-b9bb113d8d3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0494db61-rrv4c\" (UID: \"b68be428-2283-4a4f-a55e-b9bb113d8d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0494db61-rrv4c" Nov 03 13:21:02.534238 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:02.534169 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjnn9\" (UniqueName: \"kubernetes.io/projected/b68be428-2283-4a4f-a55e-b9bb113d8d3b-kube-api-access-sjnn9\") pod \"maas-api-revoke-sub-0494db61-rrv4c\" (UID: \"b68be428-2283-4a4f-a55e-b9bb113d8d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0494db61-rrv4c" Nov 03 13:21:02.599113 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:02.599083 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0494db61-rrv4c" Nov 03 13:21:02.990510 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:02.989780 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0494db61-rrv4c"] Nov 03 13:21:03.008496 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:21:03.008434 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb68be428_2283_4a4f_a55e_b9bb113d8d3b.slice/crio-cd34d92836ae838e63a5d1794c89dd1b5ffc76806b02f9cbe8a3df6893584bb3 WatchSource:0}: Error finding container cd34d92836ae838e63a5d1794c89dd1b5ffc76806b02f9cbe8a3df6893584bb3: Status 404 returned error can't find the container with id cd34d92836ae838e63a5d1794c89dd1b5ffc76806b02f9cbe8a3df6893584bb3 Nov 03 13:21:03.255902 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:03.255806 5979 generic.go:356] "Generic (PLEG): container finished" podID="6bc1e136-292c-425b-b481-3836bb343d3b" containerID="e20760b43fbcedf017ffa7a649d2f9f2c72b9871568347b376fe05a638ede239" exitCode=0 Nov 03 13:21:03.255902 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:03.255876 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07596b90-87n7l" event={"ID":"6bc1e136-292c-425b-b481-3836bb343d3b","Type":"ContainerDied","Data":"e20760b43fbcedf017ffa7a649d2f9f2c72b9871568347b376fe05a638ede239"} Nov 03 13:21:03.256131 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:03.255961 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07596b90-87n7l" event={"ID":"6bc1e136-292c-425b-b481-3836bb343d3b","Type":"ContainerStarted","Data":"9c539dd26badeee3195d5eeba974b91d71f76f33667ba50a0244790b50222e4d"} Nov 03 13:21:03.257416 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:03.257396 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0494db61-rrv4c" event={"ID":"b68be428-2283-4a4f-a55e-b9bb113d8d3b","Type":"ContainerStarted","Data":"cd34d92836ae838e63a5d1794c89dd1b5ffc76806b02f9cbe8a3df6893584bb3"} Nov 03 13:21:04.265841 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:04.265800 5979 generic.go:356] "Generic (PLEG): container finished" podID="b68be428-2283-4a4f-a55e-b9bb113d8d3b" containerID="1ee0fdd8d860a17473e0eb3a92f970068cf649a5cbe3655f76ccc52af381ef72" exitCode=0 Nov 03 13:21:04.266326 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:04.265872 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0494db61-rrv4c" event={"ID":"b68be428-2283-4a4f-a55e-b9bb113d8d3b","Type":"ContainerDied","Data":"1ee0fdd8d860a17473e0eb3a92f970068cf649a5cbe3655f76ccc52af381ef72"} Nov 03 13:21:04.408677 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:04.408646 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07596b90-87n7l" Nov 03 13:21:04.507043 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:04.507007 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bc1e136-292c-425b-b481-3836bb343d3b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bc1e136-292c-425b-b481-3836bb343d3b-maas-api-openshift-ca\") pod \"6bc1e136-292c-425b-b481-3836bb343d3b\" (UID: \"6bc1e136-292c-425b-b481-3836bb343d3b\") " Nov 03 13:21:04.507193 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:04.507124 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bc1e136-292c-425b-b481-3836bb343d3b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bc1e136-292c-425b-b481-3836bb343d3b-maas-api-xks-ca\") pod \"6bc1e136-292c-425b-b481-3836bb343d3b\" (UID: \"6bc1e136-292c-425b-b481-3836bb343d3b\") " Nov 03 13:21:04.507229 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:04.507220 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bc1e136-292c-425b-b481-3836bb343d3b-kube-api-access-5fgr5\" (UniqueName: \"kubernetes.io/projected/6bc1e136-292c-425b-b481-3836bb343d3b-kube-api-access-5fgr5\") pod \"6bc1e136-292c-425b-b481-3836bb343d3b\" (UID: \"6bc1e136-292c-425b-b481-3836bb343d3b\") " Nov 03 13:21:04.507404 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:04.507380 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bc1e136-292c-425b-b481-3836bb343d3b-maas-api-openshift-ca" pod "6bc1e136-292c-425b-b481-3836bb343d3b" (UID: "6bc1e136-292c-425b-b481-3836bb343d3b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:21:04.507571 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:04.507548 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bc1e136-292c-425b-b481-3836bb343d3b-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:21:04.509150 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:04.509125 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bc1e136-292c-425b-b481-3836bb343d3b-maas-api-xks-ca" pod "6bc1e136-292c-425b-b481-3836bb343d3b" (UID: "6bc1e136-292c-425b-b481-3836bb343d3b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:21:04.509325 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:04.509304 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bc1e136-292c-425b-b481-3836bb343d3b-kube-api-access-5fgr5" pod "6bc1e136-292c-425b-b481-3836bb343d3b" (UID: "6bc1e136-292c-425b-b481-3836bb343d3b"). InnerVolumeSpecName "kube-api-access-5fgr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:21:04.608464 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:04.608432 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bc1e136-292c-425b-b481-3836bb343d3b-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:21:04.608464 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:04.608462 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fgr5\" (UniqueName: \"kubernetes.io/projected/6bc1e136-292c-425b-b481-3836bb343d3b-kube-api-access-5fgr5\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:21:05.273190 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:05.273161 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07596b90-87n7l" Nov 03 13:21:05.273609 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:05.273194 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07596b90-87n7l" event={"ID":"6bc1e136-292c-425b-b481-3836bb343d3b","Type":"ContainerDied","Data":"9c539dd26badeee3195d5eeba974b91d71f76f33667ba50a0244790b50222e4d"} Nov 03 13:21:05.273609 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:05.273241 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c539dd26badeee3195d5eeba974b91d71f76f33667ba50a0244790b50222e4d" Nov 03 13:21:05.413486 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:05.413439 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0494db61-rrv4c" Nov 03 13:21:05.517523 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:05.517461 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b68be428-2283-4a4f-a55e-b9bb113d8d3b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b68be428-2283-4a4f-a55e-b9bb113d8d3b-maas-api-openshift-ca\") pod \"b68be428-2283-4a4f-a55e-b9bb113d8d3b\" (UID: \"b68be428-2283-4a4f-a55e-b9bb113d8d3b\") " Nov 03 13:21:05.517717 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:05.517587 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b68be428-2283-4a4f-a55e-b9bb113d8d3b-kube-api-access-sjnn9\" (UniqueName: \"kubernetes.io/projected/b68be428-2283-4a4f-a55e-b9bb113d8d3b-kube-api-access-sjnn9\") pod \"b68be428-2283-4a4f-a55e-b9bb113d8d3b\" (UID: \"b68be428-2283-4a4f-a55e-b9bb113d8d3b\") " Nov 03 13:21:05.517717 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:05.517640 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b68be428-2283-4a4f-a55e-b9bb113d8d3b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b68be428-2283-4a4f-a55e-b9bb113d8d3b-maas-api-xks-ca\") pod \"b68be428-2283-4a4f-a55e-b9bb113d8d3b\" (UID: \"b68be428-2283-4a4f-a55e-b9bb113d8d3b\") " Nov 03 13:21:05.517833 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:05.517804 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b68be428-2283-4a4f-a55e-b9bb113d8d3b-maas-api-openshift-ca" pod "b68be428-2283-4a4f-a55e-b9bb113d8d3b" (UID: "b68be428-2283-4a4f-a55e-b9bb113d8d3b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:21:05.518038 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:05.518021 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b68be428-2283-4a4f-a55e-b9bb113d8d3b-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:21:05.520084 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:05.520053 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b68be428-2283-4a4f-a55e-b9bb113d8d3b-kube-api-access-sjnn9" pod "b68be428-2283-4a4f-a55e-b9bb113d8d3b" (UID: "b68be428-2283-4a4f-a55e-b9bb113d8d3b"). InnerVolumeSpecName "kube-api-access-sjnn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:21:05.520308 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:05.520285 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b68be428-2283-4a4f-a55e-b9bb113d8d3b-maas-api-xks-ca" pod "b68be428-2283-4a4f-a55e-b9bb113d8d3b" (UID: "b68be428-2283-4a4f-a55e-b9bb113d8d3b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:21:05.618843 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:05.618811 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjnn9\" (UniqueName: \"kubernetes.io/projected/b68be428-2283-4a4f-a55e-b9bb113d8d3b-kube-api-access-sjnn9\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:21:05.618843 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:05.618838 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b68be428-2283-4a4f-a55e-b9bb113d8d3b-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:21:06.280111 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:06.280078 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0494db61-rrv4c" Nov 03 13:21:06.280500 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:06.280115 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0494db61-rrv4c" event={"ID":"b68be428-2283-4a4f-a55e-b9bb113d8d3b","Type":"ContainerDied","Data":"cd34d92836ae838e63a5d1794c89dd1b5ffc76806b02f9cbe8a3df6893584bb3"} Nov 03 13:21:06.280500 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:06.280166 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd34d92836ae838e63a5d1794c89dd1b5ffc76806b02f9cbe8a3df6893584bb3" Nov 03 13:21:06.768553 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:06.768513 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7dfbfc6544-jz7cm"] Nov 03 13:21:06.769037 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:06.769018 5979 cpu_manager.go:414] "containerMap: removing container" podUID="6bc1e136-292c-425b-b481-3836bb343d3b" containerName="revoke-keys" Nov 03 13:21:06.769138 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:06.769038 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bc1e136-292c-425b-b481-3836bb343d3b" containerName="revoke-keys" Nov 03 13:21:06.769138 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:06.769109 5979 cpu_manager.go:414] "containerMap: removing container" podUID="b68be428-2283-4a4f-a55e-b9bb113d8d3b" containerName="revoke-keys" Nov 03 13:21:06.769138 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:06.769117 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b68be428-2283-4a4f-a55e-b9bb113d8d3b" containerName="revoke-keys" Nov 03 13:21:06.769290 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:06.769217 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="b68be428-2283-4a4f-a55e-b9bb113d8d3b" containerName="revoke-keys" Nov 03 13:21:06.769290 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:06.769233 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bc1e136-292c-425b-b481-3836bb343d3b" containerName="revoke-keys" Nov 03 13:21:06.772588 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:06.772565 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7dfbfc6544-jz7cm" Nov 03 13:21:06.821590 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:06.821554 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 13:21:06.821775 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:06.821616 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 13:21:06.821775 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:06.821678 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-f765a3aeab00\"" Nov 03 13:21:06.821775 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:06.821684 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-6d0760cb6b9f\"" Nov 03 13:21:06.824543 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:06.824524 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-l89p8\"" Nov 03 13:21:06.839329 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:06.839288 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/61d2af57-4003-4364-b4f4-769565d27b65-provider-credentials\") pod \"payload-processing-external-model-7dfbfc6544-jz7cm\" (UID: \"61d2af57-4003-4364-b4f4-769565d27b65\") " pod="models-as-a-service/payload-processing-external-model-7dfbfc6544-jz7cm" Nov 03 13:21:06.839513 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:06.839342 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/61d2af57-4003-4364-b4f4-769565d27b65-routing-overlay-volume\") pod \"payload-processing-external-model-7dfbfc6544-jz7cm\" (UID: \"61d2af57-4003-4364-b4f4-769565d27b65\") " pod="models-as-a-service/payload-processing-external-model-7dfbfc6544-jz7cm" Nov 03 13:21:06.839893 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:06.839548 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61d2af57-4003-4364-b4f4-769565d27b65-plugins-config-volume\") pod \"payload-processing-external-model-7dfbfc6544-jz7cm\" (UID: \"61d2af57-4003-4364-b4f4-769565d27b65\") " pod="models-as-a-service/payload-processing-external-model-7dfbfc6544-jz7cm" Nov 03 13:21:06.839893 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:06.839613 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tczmb\" (UniqueName: \"kubernetes.io/projected/61d2af57-4003-4364-b4f4-769565d27b65-kube-api-access-tczmb\") pod \"payload-processing-external-model-7dfbfc6544-jz7cm\" (UID: \"61d2af57-4003-4364-b4f4-769565d27b65\") " pod="models-as-a-service/payload-processing-external-model-7dfbfc6544-jz7cm" Nov 03 13:21:06.870521 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:06.870459 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7dfbfc6544-jz7cm"] Nov 03 13:21:06.921753 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:06.921716 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 13:21:06.921933 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:06.921780 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 13:21:06.940684 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:06.940648 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/61d2af57-4003-4364-b4f4-769565d27b65-routing-overlay-volume\") pod \"payload-processing-external-model-7dfbfc6544-jz7cm\" (UID: \"61d2af57-4003-4364-b4f4-769565d27b65\") " pod="models-as-a-service/payload-processing-external-model-7dfbfc6544-jz7cm" Nov 03 13:21:06.940858 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:06.940776 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61d2af57-4003-4364-b4f4-769565d27b65-plugins-config-volume\") pod \"payload-processing-external-model-7dfbfc6544-jz7cm\" (UID: \"61d2af57-4003-4364-b4f4-769565d27b65\") " pod="models-as-a-service/payload-processing-external-model-7dfbfc6544-jz7cm" Nov 03 13:21:06.940858 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:06.940835 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tczmb\" (UniqueName: \"kubernetes.io/projected/61d2af57-4003-4364-b4f4-769565d27b65-kube-api-access-tczmb\") pod \"payload-processing-external-model-7dfbfc6544-jz7cm\" (UID: \"61d2af57-4003-4364-b4f4-769565d27b65\") " pod="models-as-a-service/payload-processing-external-model-7dfbfc6544-jz7cm" Nov 03 13:21:06.940982 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:06.940908 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/61d2af57-4003-4364-b4f4-769565d27b65-provider-credentials\") pod \"payload-processing-external-model-7dfbfc6544-jz7cm\" (UID: \"61d2af57-4003-4364-b4f4-769565d27b65\") " pod="models-as-a-service/payload-processing-external-model-7dfbfc6544-jz7cm" Nov 03 13:21:06.941465 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:06.941444 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/61d2af57-4003-4364-b4f4-769565d27b65-routing-overlay-volume\") pod \"payload-processing-external-model-7dfbfc6544-jz7cm\" (UID: \"61d2af57-4003-4364-b4f4-769565d27b65\") " pod="models-as-a-service/payload-processing-external-model-7dfbfc6544-jz7cm" Nov 03 13:21:06.941465 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:06.941461 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61d2af57-4003-4364-b4f4-769565d27b65-plugins-config-volume\") pod \"payload-processing-external-model-7dfbfc6544-jz7cm\" (UID: \"61d2af57-4003-4364-b4f4-769565d27b65\") " pod="models-as-a-service/payload-processing-external-model-7dfbfc6544-jz7cm" Nov 03 13:21:06.943322 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:06.943296 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/61d2af57-4003-4364-b4f4-769565d27b65-provider-credentials\") pod \"payload-processing-external-model-7dfbfc6544-jz7cm\" (UID: \"61d2af57-4003-4364-b4f4-769565d27b65\") " pod="models-as-a-service/payload-processing-external-model-7dfbfc6544-jz7cm" Nov 03 13:21:06.995146 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:06.995111 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tczmb\" (UniqueName: \"kubernetes.io/projected/61d2af57-4003-4364-b4f4-769565d27b65-kube-api-access-tczmb\") pod \"payload-processing-external-model-7dfbfc6544-jz7cm\" (UID: \"61d2af57-4003-4364-b4f4-769565d27b65\") " pod="models-as-a-service/payload-processing-external-model-7dfbfc6544-jz7cm" Nov 03 13:21:07.085803 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:07.085703 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7dfbfc6544-jz7cm" Nov 03 13:21:07.234657 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:21:07.234622 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:21:07.462528 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:07.462487 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7dfbfc6544-jz7cm"] Nov 03 13:21:07.469125 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:21:07.469094 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61d2af57_4003_4364_b4f4_769565d27b65.slice/crio-eb05b8470a6bd9f7b05b7bce6da653bba1bfbf4c34f16bd474978417637e49c5 WatchSource:0}: Error finding container eb05b8470a6bd9f7b05b7bce6da653bba1bfbf4c34f16bd474978417637e49c5: Status 404 returned error can't find the container with id eb05b8470a6bd9f7b05b7bce6da653bba1bfbf4c34f16bd474978417637e49c5 Nov 03 13:21:08.295782 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:08.295738 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7dfbfc6544-jz7cm" event={"ID":"61d2af57-4003-4364-b4f4-769565d27b65","Type":"ContainerStarted","Data":"e31650c713af37198b81d35bb65f3d480e0ae15a37d30650debd01f487357874"} Nov 03 13:21:08.296004 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:08.295810 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7dfbfc6544-jz7cm" event={"ID":"61d2af57-4003-4364-b4f4-769565d27b65","Type":"ContainerStarted","Data":"eb05b8470a6bd9f7b05b7bce6da653bba1bfbf4c34f16bd474978417637e49c5"} Nov 03 13:21:08.296099 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:08.296024 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7dfbfc6544-jz7cm" Nov 03 13:21:08.407280 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:08.407201 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7dfbfc6544-jz7cm" podStartSLOduration=2.407182392 podStartE2EDuration="2.407182392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:21:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:21:08.4051055 +0000 UTC m=+950.415793748" watchObservedRunningTime="2026-11-03 13:21:08.407182392 +0000 UTC m=+950.417870635" Nov 03 13:21:09.124770 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:09.124740 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn"] Nov 03 13:21:09.128435 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:09.128412 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn" Nov 03 13:21:09.162865 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:09.162833 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f156e9bb-8f04-4805-b8ff-ca1561fb598c-istio-data\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn\" (UID: \"f156e9bb-8f04-4805-b8ff-ca1561fb598c\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn" Nov 03 13:21:09.163014 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:09.162994 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f156e9bb-8f04-4805-b8ff-ca1561fb598c-credential-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn\" (UID: \"f156e9bb-8f04-4805-b8ff-ca1561fb598c\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn" Nov 03 13:21:09.163071 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:09.163052 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f156e9bb-8f04-4805-b8ff-ca1561fb598c-istiod-ca-cert\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn\" (UID: \"f156e9bb-8f04-4805-b8ff-ca1561fb598c\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn" Nov 03 13:21:09.163129 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:09.163084 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f156e9bb-8f04-4805-b8ff-ca1561fb598c-workload-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn\" (UID: \"f156e9bb-8f04-4805-b8ff-ca1561fb598c\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn" Nov 03 13:21:09.163129 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:09.163114 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f156e9bb-8f04-4805-b8ff-ca1561fb598c-istio-podinfo\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn\" (UID: \"f156e9bb-8f04-4805-b8ff-ca1561fb598c\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn" Nov 03 13:21:09.163287 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:09.163246 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f156e9bb-8f04-4805-b8ff-ca1561fb598c-workload-certs\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn\" (UID: \"f156e9bb-8f04-4805-b8ff-ca1561fb598c\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn" Nov 03 13:21:09.163415 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:09.163347 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9zpd\" (UniqueName: \"kubernetes.io/projected/f156e9bb-8f04-4805-b8ff-ca1561fb598c-kube-api-access-r9zpd\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn\" (UID: \"f156e9bb-8f04-4805-b8ff-ca1561fb598c\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn" Nov 03 13:21:09.163521 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:09.163416 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f156e9bb-8f04-4805-b8ff-ca1561fb598c-istio-token\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn\" (UID: \"f156e9bb-8f04-4805-b8ff-ca1561fb598c\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn" Nov 03 13:21:09.163602 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:09.163520 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f156e9bb-8f04-4805-b8ff-ca1561fb598c-istio-envoy\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn\" (UID: \"f156e9bb-8f04-4805-b8ff-ca1561fb598c\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn" Nov 03 13:21:09.177276 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:09.177248 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-42c707-openshift-default-dockercfg-7t7mv\"" Nov 03 13:21:09.227177 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:09.227140 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn"] Nov 03 13:21:09.264439 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:09.264402 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f156e9bb-8f04-4805-b8ff-ca1561fb598c-istio-data\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn\" (UID: \"f156e9bb-8f04-4805-b8ff-ca1561fb598c\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn" Nov 03 13:21:09.264621 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:09.264514 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f156e9bb-8f04-4805-b8ff-ca1561fb598c-credential-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn\" (UID: \"f156e9bb-8f04-4805-b8ff-ca1561fb598c\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn" Nov 03 13:21:09.264621 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:09.264545 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f156e9bb-8f04-4805-b8ff-ca1561fb598c-istiod-ca-cert\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn\" (UID: \"f156e9bb-8f04-4805-b8ff-ca1561fb598c\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn" Nov 03 13:21:09.264621 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:09.264561 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f156e9bb-8f04-4805-b8ff-ca1561fb598c-workload-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn\" (UID: \"f156e9bb-8f04-4805-b8ff-ca1561fb598c\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn" Nov 03 13:21:09.264621 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:09.264580 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f156e9bb-8f04-4805-b8ff-ca1561fb598c-istio-podinfo\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn\" (UID: \"f156e9bb-8f04-4805-b8ff-ca1561fb598c\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn" Nov 03 13:21:09.264811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:09.264714 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f156e9bb-8f04-4805-b8ff-ca1561fb598c-workload-certs\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn\" (UID: \"f156e9bb-8f04-4805-b8ff-ca1561fb598c\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn" Nov 03 13:21:09.264811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:09.264785 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9zpd\" (UniqueName: \"kubernetes.io/projected/f156e9bb-8f04-4805-b8ff-ca1561fb598c-kube-api-access-r9zpd\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn\" (UID: \"f156e9bb-8f04-4805-b8ff-ca1561fb598c\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn" Nov 03 13:21:09.264894 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:09.264849 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f156e9bb-8f04-4805-b8ff-ca1561fb598c-istio-token\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn\" (UID: \"f156e9bb-8f04-4805-b8ff-ca1561fb598c\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn" Nov 03 13:21:09.264894 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:09.264880 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f156e9bb-8f04-4805-b8ff-ca1561fb598c-istio-envoy\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn\" (UID: \"f156e9bb-8f04-4805-b8ff-ca1561fb598c\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn" Nov 03 13:21:09.265000 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:09.264898 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f156e9bb-8f04-4805-b8ff-ca1561fb598c-istio-data\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn\" (UID: \"f156e9bb-8f04-4805-b8ff-ca1561fb598c\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn" Nov 03 13:21:09.265064 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:09.264997 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f156e9bb-8f04-4805-b8ff-ca1561fb598c-credential-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn\" (UID: \"f156e9bb-8f04-4805-b8ff-ca1561fb598c\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn" Nov 03 13:21:09.265064 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:09.265030 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f156e9bb-8f04-4805-b8ff-ca1561fb598c-workload-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn\" (UID: \"f156e9bb-8f04-4805-b8ff-ca1561fb598c\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn" Nov 03 13:21:09.265175 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:09.265132 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f156e9bb-8f04-4805-b8ff-ca1561fb598c-workload-certs\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn\" (UID: \"f156e9bb-8f04-4805-b8ff-ca1561fb598c\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn" Nov 03 13:21:09.265276 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:09.265244 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f156e9bb-8f04-4805-b8ff-ca1561fb598c-istiod-ca-cert\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn\" (UID: \"f156e9bb-8f04-4805-b8ff-ca1561fb598c\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn" Nov 03 13:21:09.267014 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:09.266993 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f156e9bb-8f04-4805-b8ff-ca1561fb598c-istio-envoy\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn\" (UID: \"f156e9bb-8f04-4805-b8ff-ca1561fb598c\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn" Nov 03 13:21:09.267121 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:09.267023 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f156e9bb-8f04-4805-b8ff-ca1561fb598c-istio-podinfo\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn\" (UID: \"f156e9bb-8f04-4805-b8ff-ca1561fb598c\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn" Nov 03 13:21:09.318752 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:09.318722 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f156e9bb-8f04-4805-b8ff-ca1561fb598c-istio-token\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn\" (UID: \"f156e9bb-8f04-4805-b8ff-ca1561fb598c\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn" Nov 03 13:21:09.318931 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:09.318844 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9zpd\" (UniqueName: \"kubernetes.io/projected/f156e9bb-8f04-4805-b8ff-ca1561fb598c-kube-api-access-r9zpd\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn\" (UID: \"f156e9bb-8f04-4805-b8ff-ca1561fb598c\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn" Nov 03 13:21:09.450411 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:09.450377 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn" Nov 03 13:21:09.875815 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:09.875736 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn"] Nov 03 13:21:09.883044 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:21:09.883014 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf156e9bb_8f04_4805_b8ff_ca1561fb598c.slice/crio-aaf3165c67b38e7f134f22612698c539116224341a4eea752a561fd1b3aec482 WatchSource:0}: Error finding container aaf3165c67b38e7f134f22612698c539116224341a4eea752a561fd1b3aec482: Status 404 returned error can't find the container with id aaf3165c67b38e7f134f22612698c539116224341a4eea752a561fd1b3aec482 Nov 03 13:21:09.885417 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:09.885385 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:21:09.885546 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:09.885495 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:21:09.885546 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:09.885532 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:21:10.309777 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:10.309738 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn" event={"ID":"f156e9bb-8f04-4805-b8ff-ca1561fb598c","Type":"ContainerStarted","Data":"bef503c247cd6c60e93a5a66dc1ce2f2cf2aee3c879196084c9f5277d53556f1"} Nov 03 13:21:10.310181 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:10.309792 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn" event={"ID":"f156e9bb-8f04-4805-b8ff-ca1561fb598c","Type":"ContainerStarted","Data":"aaf3165c67b38e7f134f22612698c539116224341a4eea752a561fd1b3aec482"} Nov 03 13:21:10.458493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:10.455216 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn" Nov 03 13:21:10.458493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:10.455397 5979 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 13:21:10.458493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:10.455442 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn" podUID="f156e9bb-8f04-4805-b8ff-ca1561fb598c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 13:21:10.517437 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:10.517377 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn" podStartSLOduration=1.517359336 podStartE2EDuration="1.517359336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:21:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:21:10.468939811 +0000 UTC m=+952.479628053" watchObservedRunningTime="2026-11-03 13:21:10.517359336 +0000 UTC m=+952.528047582" Nov 03 13:21:11.461496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:11.455838 5979 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 13:21:11.461496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:11.455896 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn" podUID="f156e9bb-8f04-4805-b8ff-ca1561fb598c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 13:21:12.456497 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:12.450888 5979 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 13:21:12.456497 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:12.450957 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn" podUID="f156e9bb-8f04-4805-b8ff-ca1561fb598c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 13:21:13.468682 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:13.468644 5979 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 13:21:13.469179 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:13.468711 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn" podUID="f156e9bb-8f04-4805-b8ff-ca1561fb598c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 13:21:14.458891 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:14.458851 5979 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 13:21:14.459197 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:14.459165 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn" podUID="f156e9bb-8f04-4805-b8ff-ca1561fb598c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 13:21:15.458883 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:15.458845 5979 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Nov 03 13:21:15.459490 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:15.459438 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn" podUID="f156e9bb-8f04-4805-b8ff-ca1561fb598c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Nov 03 13:21:16.470684 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:16.470645 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn" Nov 03 13:21:16.471432 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:16.471412 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn" Nov 03 13:21:16.472406 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:16.472388 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn" Nov 03 13:21:18.606964 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:18.606881 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:21:19.305878 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:19.305851 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-7dfbfc6544-jz7cm" Nov 03 13:21:19.332240 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:19.332207 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-42c707/e2e-conflict-a-f2a72750-kserve-5d75f8cf5c-mjjqx"] Nov 03 13:21:19.337909 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:19.337881 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-a-f2a72750-kserve-5d75f8cf5c-mjjqx" Nov 03 13:21:19.378686 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:19.378656 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc4c9\" (UniqueName: \"kubernetes.io/projected/9cd1a0a5-82e9-481b-a0d2-5d51b21bada1-kube-api-access-xc4c9\") pod \"e2e-conflict-a-f2a72750-kserve-5d75f8cf5c-mjjqx\" (UID: \"9cd1a0a5-82e9-481b-a0d2-5d51b21bada1\") " pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-a-f2a72750-kserve-5d75f8cf5c-mjjqx" Nov 03 13:21:19.378888 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:19.378695 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cd1a0a5-82e9-481b-a0d2-5d51b21bada1-dshm\") pod \"e2e-conflict-a-f2a72750-kserve-5d75f8cf5c-mjjqx\" (UID: \"9cd1a0a5-82e9-481b-a0d2-5d51b21bada1\") " pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-a-f2a72750-kserve-5d75f8cf5c-mjjqx" Nov 03 13:21:19.378888 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:19.378796 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cd1a0a5-82e9-481b-a0d2-5d51b21bada1-tls-certs\") pod \"e2e-conflict-a-f2a72750-kserve-5d75f8cf5c-mjjqx\" (UID: \"9cd1a0a5-82e9-481b-a0d2-5d51b21bada1\") " pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-a-f2a72750-kserve-5d75f8cf5c-mjjqx" Nov 03 13:21:19.378888 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:19.378845 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cd1a0a5-82e9-481b-a0d2-5d51b21bada1-tmp-dir\") pod \"e2e-conflict-a-f2a72750-kserve-5d75f8cf5c-mjjqx\" (UID: \"9cd1a0a5-82e9-481b-a0d2-5d51b21bada1\") " pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-a-f2a72750-kserve-5d75f8cf5c-mjjqx" Nov 03 13:21:19.379075 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:19.378960 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cd1a0a5-82e9-481b-a0d2-5d51b21bada1-home\") pod \"e2e-conflict-a-f2a72750-kserve-5d75f8cf5c-mjjqx\" (UID: \"9cd1a0a5-82e9-481b-a0d2-5d51b21bada1\") " pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-a-f2a72750-kserve-5d75f8cf5c-mjjqx" Nov 03 13:21:19.379137 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:19.379083 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cd1a0a5-82e9-481b-a0d2-5d51b21bada1-model-cache\") pod \"e2e-conflict-a-f2a72750-kserve-5d75f8cf5c-mjjqx\" (UID: \"9cd1a0a5-82e9-481b-a0d2-5d51b21bada1\") " pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-a-f2a72750-kserve-5d75f8cf5c-mjjqx" Nov 03 13:21:19.391577 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:19.391546 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-42c707\"/\"kube-root-ca.crt\"" Nov 03 13:21:19.394252 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:19.394231 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-42c707\"/\"openshift-service-ca.crt\"" Nov 03 13:21:19.416439 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:19.416413 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-42c707\"/\"default-dockercfg-mzpk2\"" Nov 03 13:21:19.420257 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:19.420228 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-42c707/e2e-conflict-a-f2a72750-kserve-5d75f8cf5c-mjjqx"] Nov 03 13:21:19.422890 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:19.422868 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-42c707\"/\"e2e-conflict-a-f2a72750-kserve-self-signed-certs\"" Nov 03 13:21:19.492531 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:19.480215 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc4c9\" (UniqueName: \"kubernetes.io/projected/9cd1a0a5-82e9-481b-a0d2-5d51b21bada1-kube-api-access-xc4c9\") pod \"e2e-conflict-a-f2a72750-kserve-5d75f8cf5c-mjjqx\" (UID: \"9cd1a0a5-82e9-481b-a0d2-5d51b21bada1\") " pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-a-f2a72750-kserve-5d75f8cf5c-mjjqx" Nov 03 13:21:19.523910 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:19.515565 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cd1a0a5-82e9-481b-a0d2-5d51b21bada1-dshm\") pod \"e2e-conflict-a-f2a72750-kserve-5d75f8cf5c-mjjqx\" (UID: \"9cd1a0a5-82e9-481b-a0d2-5d51b21bada1\") " pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-a-f2a72750-kserve-5d75f8cf5c-mjjqx" Nov 03 13:21:19.523910 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:19.515646 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cd1a0a5-82e9-481b-a0d2-5d51b21bada1-tls-certs\") pod \"e2e-conflict-a-f2a72750-kserve-5d75f8cf5c-mjjqx\" (UID: \"9cd1a0a5-82e9-481b-a0d2-5d51b21bada1\") " pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-a-f2a72750-kserve-5d75f8cf5c-mjjqx" Nov 03 13:21:19.523910 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:19.515691 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cd1a0a5-82e9-481b-a0d2-5d51b21bada1-tmp-dir\") pod \"e2e-conflict-a-f2a72750-kserve-5d75f8cf5c-mjjqx\" (UID: \"9cd1a0a5-82e9-481b-a0d2-5d51b21bada1\") " pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-a-f2a72750-kserve-5d75f8cf5c-mjjqx" Nov 03 13:21:19.523910 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:19.515784 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cd1a0a5-82e9-481b-a0d2-5d51b21bada1-home\") pod \"e2e-conflict-a-f2a72750-kserve-5d75f8cf5c-mjjqx\" (UID: \"9cd1a0a5-82e9-481b-a0d2-5d51b21bada1\") " pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-a-f2a72750-kserve-5d75f8cf5c-mjjqx" Nov 03 13:21:19.523910 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:19.515848 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cd1a0a5-82e9-481b-a0d2-5d51b21bada1-model-cache\") pod \"e2e-conflict-a-f2a72750-kserve-5d75f8cf5c-mjjqx\" (UID: \"9cd1a0a5-82e9-481b-a0d2-5d51b21bada1\") " pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-a-f2a72750-kserve-5d75f8cf5c-mjjqx" Nov 03 13:21:19.523910 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:19.516329 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cd1a0a5-82e9-481b-a0d2-5d51b21bada1-model-cache\") pod \"e2e-conflict-a-f2a72750-kserve-5d75f8cf5c-mjjqx\" (UID: \"9cd1a0a5-82e9-481b-a0d2-5d51b21bada1\") " pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-a-f2a72750-kserve-5d75f8cf5c-mjjqx" Nov 03 13:21:19.523910 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:19.516980 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cd1a0a5-82e9-481b-a0d2-5d51b21bada1-tmp-dir\") pod \"e2e-conflict-a-f2a72750-kserve-5d75f8cf5c-mjjqx\" (UID: \"9cd1a0a5-82e9-481b-a0d2-5d51b21bada1\") " pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-a-f2a72750-kserve-5d75f8cf5c-mjjqx" Nov 03 13:21:19.523910 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:19.517351 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cd1a0a5-82e9-481b-a0d2-5d51b21bada1-home\") pod \"e2e-conflict-a-f2a72750-kserve-5d75f8cf5c-mjjqx\" (UID: \"9cd1a0a5-82e9-481b-a0d2-5d51b21bada1\") " pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-a-f2a72750-kserve-5d75f8cf5c-mjjqx" Nov 03 13:21:19.545494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:19.539817 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cd1a0a5-82e9-481b-a0d2-5d51b21bada1-dshm\") pod \"e2e-conflict-a-f2a72750-kserve-5d75f8cf5c-mjjqx\" (UID: \"9cd1a0a5-82e9-481b-a0d2-5d51b21bada1\") " pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-a-f2a72750-kserve-5d75f8cf5c-mjjqx" Nov 03 13:21:19.558426 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:19.557872 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cd1a0a5-82e9-481b-a0d2-5d51b21bada1-tls-certs\") pod \"e2e-conflict-a-f2a72750-kserve-5d75f8cf5c-mjjqx\" (UID: \"9cd1a0a5-82e9-481b-a0d2-5d51b21bada1\") " pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-a-f2a72750-kserve-5d75f8cf5c-mjjqx" Nov 03 13:21:19.570407 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:19.570373 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc4c9\" (UniqueName: \"kubernetes.io/projected/9cd1a0a5-82e9-481b-a0d2-5d51b21bada1-kube-api-access-xc4c9\") pod \"e2e-conflict-a-f2a72750-kserve-5d75f8cf5c-mjjqx\" (UID: \"9cd1a0a5-82e9-481b-a0d2-5d51b21bada1\") " pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-a-f2a72750-kserve-5d75f8cf5c-mjjqx" Nov 03 13:21:19.652682 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:19.652632 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-a-f2a72750-kserve-5d75f8cf5c-mjjqx" Nov 03 13:21:20.068719 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:20.068680 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-42c707/e2e-conflict-a-f2a72750-kserve-5d75f8cf5c-mjjqx"] Nov 03 13:21:20.075374 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:21:20.075336 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9cd1a0a5_82e9_481b_a0d2_5d51b21bada1.slice/crio-309c980608fe58772c2bfeacef27c8c6a7350438846b808a69271623f653050b WatchSource:0}: Error finding container 309c980608fe58772c2bfeacef27c8c6a7350438846b808a69271623f653050b: Status 404 returned error can't find the container with id 309c980608fe58772c2bfeacef27c8c6a7350438846b808a69271623f653050b Nov 03 13:21:20.372855 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:20.372749 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-a-f2a72750-kserve-5d75f8cf5c-mjjqx" event={"ID":"9cd1a0a5-82e9-481b-a0d2-5d51b21bada1","Type":"ContainerStarted","Data":"8de160e677a37781142866f14aeebc2c97f7c87e437e4d59a12cdf54d2f28fa7"} Nov 03 13:21:20.372855 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:20.372822 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-a-f2a72750-kserve-5d75f8cf5c-mjjqx" event={"ID":"9cd1a0a5-82e9-481b-a0d2-5d51b21bada1","Type":"ContainerStarted","Data":"309c980608fe58772c2bfeacef27c8c6a7350438846b808a69271623f653050b"} Nov 03 13:21:20.479987 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:20.479928 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-a-f2a72750-kserve-5d75f8cf5c-mjjqx" podStartSLOduration=1.479903649 podStartE2EDuration="1.479903649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:21:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:21:20.47940928 +0000 UTC m=+962.490097541" watchObservedRunningTime="2026-11-03 13:21:20.479903649 +0000 UTC m=+962.490591892" Nov 03 13:21:20.653251 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:20.653214 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-a-f2a72750-kserve-5d75f8cf5c-mjjqx" Nov 03 13:21:21.338644 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:21.338603 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-dkw27 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:21:21.338821 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:21.338660 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dkw27" podUID="f313510b-02d8-4c68-accd-920343219917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:21:25.843997 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:25.843955 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5d66dc6b6c-tnc8j"] Nov 03 13:21:25.848013 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:25.847991 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5d66dc6b6c-tnc8j" Nov 03 13:21:25.948008 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:25.947970 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5d66dc6b6c-tnc8j"] Nov 03 13:21:25.998562 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:25.998521 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce8ed64a-b642-4e26-aeb1-ab8d483b51f1-plugins-config-volume\") pod \"payload-processing-external-model-5d66dc6b6c-tnc8j\" (UID: \"ce8ed64a-b642-4e26-aeb1-ab8d483b51f1\") " pod="models-as-a-service/payload-processing-external-model-5d66dc6b6c-tnc8j" Nov 03 13:21:25.998738 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:25.998568 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ce8ed64a-b642-4e26-aeb1-ab8d483b51f1-routing-overlay-volume\") pod \"payload-processing-external-model-5d66dc6b6c-tnc8j\" (UID: \"ce8ed64a-b642-4e26-aeb1-ab8d483b51f1\") " pod="models-as-a-service/payload-processing-external-model-5d66dc6b6c-tnc8j" Nov 03 13:21:25.998738 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:25.998664 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ce8ed64a-b642-4e26-aeb1-ab8d483b51f1-provider-credentials\") pod \"payload-processing-external-model-5d66dc6b6c-tnc8j\" (UID: \"ce8ed64a-b642-4e26-aeb1-ab8d483b51f1\") " pod="models-as-a-service/payload-processing-external-model-5d66dc6b6c-tnc8j" Nov 03 13:21:25.998738 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:25.998735 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sw62\" (UniqueName: \"kubernetes.io/projected/ce8ed64a-b642-4e26-aeb1-ab8d483b51f1-kube-api-access-9sw62\") pod \"payload-processing-external-model-5d66dc6b6c-tnc8j\" (UID: \"ce8ed64a-b642-4e26-aeb1-ab8d483b51f1\") " pod="models-as-a-service/payload-processing-external-model-5d66dc6b6c-tnc8j" Nov 03 13:21:26.099729 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:26.099637 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce8ed64a-b642-4e26-aeb1-ab8d483b51f1-plugins-config-volume\") pod \"payload-processing-external-model-5d66dc6b6c-tnc8j\" (UID: \"ce8ed64a-b642-4e26-aeb1-ab8d483b51f1\") " pod="models-as-a-service/payload-processing-external-model-5d66dc6b6c-tnc8j" Nov 03 13:21:26.099729 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:26.099682 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ce8ed64a-b642-4e26-aeb1-ab8d483b51f1-routing-overlay-volume\") pod \"payload-processing-external-model-5d66dc6b6c-tnc8j\" (UID: \"ce8ed64a-b642-4e26-aeb1-ab8d483b51f1\") " pod="models-as-a-service/payload-processing-external-model-5d66dc6b6c-tnc8j" Nov 03 13:21:26.099919 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:26.099781 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ce8ed64a-b642-4e26-aeb1-ab8d483b51f1-provider-credentials\") pod \"payload-processing-external-model-5d66dc6b6c-tnc8j\" (UID: \"ce8ed64a-b642-4e26-aeb1-ab8d483b51f1\") " pod="models-as-a-service/payload-processing-external-model-5d66dc6b6c-tnc8j" Nov 03 13:21:26.099919 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:26.099830 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sw62\" (UniqueName: \"kubernetes.io/projected/ce8ed64a-b642-4e26-aeb1-ab8d483b51f1-kube-api-access-9sw62\") pod \"payload-processing-external-model-5d66dc6b6c-tnc8j\" (UID: \"ce8ed64a-b642-4e26-aeb1-ab8d483b51f1\") " pod="models-as-a-service/payload-processing-external-model-5d66dc6b6c-tnc8j" Nov 03 13:21:26.100412 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:26.100386 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ce8ed64a-b642-4e26-aeb1-ab8d483b51f1-routing-overlay-volume\") pod \"payload-processing-external-model-5d66dc6b6c-tnc8j\" (UID: \"ce8ed64a-b642-4e26-aeb1-ab8d483b51f1\") " pod="models-as-a-service/payload-processing-external-model-5d66dc6b6c-tnc8j" Nov 03 13:21:26.100611 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:26.100591 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce8ed64a-b642-4e26-aeb1-ab8d483b51f1-plugins-config-volume\") pod \"payload-processing-external-model-5d66dc6b6c-tnc8j\" (UID: \"ce8ed64a-b642-4e26-aeb1-ab8d483b51f1\") " pod="models-as-a-service/payload-processing-external-model-5d66dc6b6c-tnc8j" Nov 03 13:21:26.102257 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:26.102237 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ce8ed64a-b642-4e26-aeb1-ab8d483b51f1-provider-credentials\") pod \"payload-processing-external-model-5d66dc6b6c-tnc8j\" (UID: \"ce8ed64a-b642-4e26-aeb1-ab8d483b51f1\") " pod="models-as-a-service/payload-processing-external-model-5d66dc6b6c-tnc8j" Nov 03 13:21:26.154822 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:26.154787 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sw62\" (UniqueName: \"kubernetes.io/projected/ce8ed64a-b642-4e26-aeb1-ab8d483b51f1-kube-api-access-9sw62\") pod \"payload-processing-external-model-5d66dc6b6c-tnc8j\" (UID: \"ce8ed64a-b642-4e26-aeb1-ab8d483b51f1\") " pod="models-as-a-service/payload-processing-external-model-5d66dc6b6c-tnc8j" Nov 03 13:21:26.161620 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:26.161596 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5d66dc6b6c-tnc8j" Nov 03 13:21:26.534288 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:26.534254 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5d66dc6b6c-tnc8j"] Nov 03 13:21:27.415930 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:27.415887 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5d66dc6b6c-tnc8j" event={"ID":"ce8ed64a-b642-4e26-aeb1-ab8d483b51f1","Type":"ContainerStarted","Data":"5e42a0a2d751118aa110a7534e59d51a9a8691ef5b867336c241bfff0a70bb4f"} Nov 03 13:21:27.416322 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:27.416049 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5d66dc6b6c-tnc8j" Nov 03 13:21:27.416322 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:27.416091 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5d66dc6b6c-tnc8j" event={"ID":"ce8ed64a-b642-4e26-aeb1-ab8d483b51f1","Type":"ContainerStarted","Data":"6144a89f909ad3fc0ef66296f07a29a573038b790a79337b9c8066707eb32f2e"} Nov 03 13:21:27.535131 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:27.535074 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5d66dc6b6c-tnc8j" podStartSLOduration=2.535060169 podStartE2EDuration="2.535060169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:21:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:21:27.532534812 +0000 UTC m=+969.543223053" watchObservedRunningTime="2026-11-03 13:21:27.535060169 +0000 UTC m=+969.545748412" Nov 03 13:21:28.031437 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.031402 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq"] Nov 03 13:21:28.037952 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.037921 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq" Nov 03 13:21:28.051007 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.050979 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mksft"] Nov 03 13:21:28.054876 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.054853 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mksft" Nov 03 13:21:28.122057 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.122026 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j9rbq\" (UID: \"ad7c3ee7-2be7-4124-9a89-1be00bbc1b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq" Nov 03 13:21:28.122225 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.122074 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j9rbq\" (UID: \"ad7c3ee7-2be7-4124-9a89-1be00bbc1b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq" Nov 03 13:21:28.122225 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.122089 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j9rbq\" (UID: \"ad7c3ee7-2be7-4124-9a89-1be00bbc1b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq" Nov 03 13:21:28.122306 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.122229 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j9rbq\" (UID: \"ad7c3ee7-2be7-4124-9a89-1be00bbc1b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq" Nov 03 13:21:28.122306 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.122264 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j9rbq\" (UID: \"ad7c3ee7-2be7-4124-9a89-1be00bbc1b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq" Nov 03 13:21:28.122382 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.122365 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp9vl\" (UniqueName: \"kubernetes.io/projected/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-kube-api-access-qp9vl\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j9rbq\" (UID: \"ad7c3ee7-2be7-4124-9a89-1be00bbc1b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq" Nov 03 13:21:28.122418 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.122405 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j9rbq\" (UID: \"ad7c3ee7-2be7-4124-9a89-1be00bbc1b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq" Nov 03 13:21:28.122459 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.122437 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j9rbq\" (UID: \"ad7c3ee7-2be7-4124-9a89-1be00bbc1b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq" Nov 03 13:21:28.122459 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.122454 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j9rbq\" (UID: \"ad7c3ee7-2be7-4124-9a89-1be00bbc1b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq" Nov 03 13:21:28.133884 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.133844 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq"] Nov 03 13:21:28.141951 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.141921 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mksft"] Nov 03 13:21:28.223393 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.223351 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp9vl\" (UniqueName: \"kubernetes.io/projected/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-kube-api-access-qp9vl\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j9rbq\" (UID: \"ad7c3ee7-2be7-4124-9a89-1be00bbc1b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq" Nov 03 13:21:28.223630 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.223400 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42cb885d-1072-4057-ab7d-71fe9f166f7d-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-mksft\" (UID: \"42cb885d-1072-4057-ab7d-71fe9f166f7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mksft" Nov 03 13:21:28.223630 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.223426 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j9rbq\" (UID: \"ad7c3ee7-2be7-4124-9a89-1be00bbc1b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq" Nov 03 13:21:28.223630 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.223456 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j9rbq\" (UID: \"ad7c3ee7-2be7-4124-9a89-1be00bbc1b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq" Nov 03 13:21:28.223630 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.223505 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42cb885d-1072-4057-ab7d-71fe9f166f7d-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-mksft\" (UID: \"42cb885d-1072-4057-ab7d-71fe9f166f7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mksft" Nov 03 13:21:28.223862 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.223646 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j9rbq\" (UID: \"ad7c3ee7-2be7-4124-9a89-1be00bbc1b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq" Nov 03 13:21:28.223862 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.223741 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42cb885d-1072-4057-ab7d-71fe9f166f7d-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-mksft\" (UID: \"42cb885d-1072-4057-ab7d-71fe9f166f7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mksft" Nov 03 13:21:28.223862 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.223765 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j9rbq\" (UID: \"ad7c3ee7-2be7-4124-9a89-1be00bbc1b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq" Nov 03 13:21:28.223862 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.223787 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcl6b\" (UniqueName: \"kubernetes.io/projected/42cb885d-1072-4057-ab7d-71fe9f166f7d-kube-api-access-lcl6b\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-mksft\" (UID: \"42cb885d-1072-4057-ab7d-71fe9f166f7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mksft" Nov 03 13:21:28.224088 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.223928 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j9rbq\" (UID: \"ad7c3ee7-2be7-4124-9a89-1be00bbc1b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq" Nov 03 13:21:28.224088 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.223962 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j9rbq\" (UID: \"ad7c3ee7-2be7-4124-9a89-1be00bbc1b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq" Nov 03 13:21:28.224088 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.224021 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42cb885d-1072-4057-ab7d-71fe9f166f7d-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-mksft\" (UID: \"42cb885d-1072-4057-ab7d-71fe9f166f7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mksft" Nov 03 13:21:28.224088 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.223962 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j9rbq\" (UID: \"ad7c3ee7-2be7-4124-9a89-1be00bbc1b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq" Nov 03 13:21:28.224088 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.224050 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42cb885d-1072-4057-ab7d-71fe9f166f7d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-mksft\" (UID: \"42cb885d-1072-4057-ab7d-71fe9f166f7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mksft" Nov 03 13:21:28.224351 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.224123 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j9rbq\" (UID: \"ad7c3ee7-2be7-4124-9a89-1be00bbc1b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq" Nov 03 13:21:28.224351 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.224208 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j9rbq\" (UID: \"ad7c3ee7-2be7-4124-9a89-1be00bbc1b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq" Nov 03 13:21:28.224351 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.224247 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42cb885d-1072-4057-ab7d-71fe9f166f7d-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-mksft\" (UID: \"42cb885d-1072-4057-ab7d-71fe9f166f7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mksft" Nov 03 13:21:28.224351 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.224332 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42cb885d-1072-4057-ab7d-71fe9f166f7d-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-mksft\" (UID: \"42cb885d-1072-4057-ab7d-71fe9f166f7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mksft" Nov 03 13:21:28.224576 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.224378 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j9rbq\" (UID: \"ad7c3ee7-2be7-4124-9a89-1be00bbc1b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq" Nov 03 13:21:28.224576 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.224391 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j9rbq\" (UID: \"ad7c3ee7-2be7-4124-9a89-1be00bbc1b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq" Nov 03 13:21:28.224576 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.224441 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42cb885d-1072-4057-ab7d-71fe9f166f7d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-mksft\" (UID: \"42cb885d-1072-4057-ab7d-71fe9f166f7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mksft" Nov 03 13:21:28.224576 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.224554 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j9rbq\" (UID: \"ad7c3ee7-2be7-4124-9a89-1be00bbc1b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq" Nov 03 13:21:28.224810 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.224781 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j9rbq\" (UID: \"ad7c3ee7-2be7-4124-9a89-1be00bbc1b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq" Nov 03 13:21:28.226345 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.226320 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j9rbq\" (UID: \"ad7c3ee7-2be7-4124-9a89-1be00bbc1b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq" Nov 03 13:21:28.226650 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.226632 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j9rbq\" (UID: \"ad7c3ee7-2be7-4124-9a89-1be00bbc1b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq" Nov 03 13:21:28.279695 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.279655 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j9rbq\" (UID: \"ad7c3ee7-2be7-4124-9a89-1be00bbc1b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq" Nov 03 13:21:28.279851 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.279768 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp9vl\" (UniqueName: \"kubernetes.io/projected/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-kube-api-access-qp9vl\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j9rbq\" (UID: \"ad7c3ee7-2be7-4124-9a89-1be00bbc1b21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq" Nov 03 13:21:28.325843 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.325761 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42cb885d-1072-4057-ab7d-71fe9f166f7d-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-mksft\" (UID: \"42cb885d-1072-4057-ab7d-71fe9f166f7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mksft" Nov 03 13:21:28.325843 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.325812 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42cb885d-1072-4057-ab7d-71fe9f166f7d-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-mksft\" (UID: \"42cb885d-1072-4057-ab7d-71fe9f166f7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mksft" Nov 03 13:21:28.325843 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.325842 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42cb885d-1072-4057-ab7d-71fe9f166f7d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-mksft\" (UID: \"42cb885d-1072-4057-ab7d-71fe9f166f7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mksft" Nov 03 13:21:28.326066 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.325911 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42cb885d-1072-4057-ab7d-71fe9f166f7d-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-mksft\" (UID: \"42cb885d-1072-4057-ab7d-71fe9f166f7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mksft" Nov 03 13:21:28.326066 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.325939 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42cb885d-1072-4057-ab7d-71fe9f166f7d-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-mksft\" (UID: \"42cb885d-1072-4057-ab7d-71fe9f166f7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mksft" Nov 03 13:21:28.326066 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.326001 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42cb885d-1072-4057-ab7d-71fe9f166f7d-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-mksft\" (UID: \"42cb885d-1072-4057-ab7d-71fe9f166f7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mksft" Nov 03 13:21:28.326066 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.326023 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcl6b\" (UniqueName: \"kubernetes.io/projected/42cb885d-1072-4057-ab7d-71fe9f166f7d-kube-api-access-lcl6b\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-mksft\" (UID: \"42cb885d-1072-4057-ab7d-71fe9f166f7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mksft" Nov 03 13:21:28.326256 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.326096 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42cb885d-1072-4057-ab7d-71fe9f166f7d-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-mksft\" (UID: \"42cb885d-1072-4057-ab7d-71fe9f166f7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mksft" Nov 03 13:21:28.326256 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.326122 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42cb885d-1072-4057-ab7d-71fe9f166f7d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-mksft\" (UID: \"42cb885d-1072-4057-ab7d-71fe9f166f7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mksft" Nov 03 13:21:28.326256 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.326202 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42cb885d-1072-4057-ab7d-71fe9f166f7d-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-mksft\" (UID: \"42cb885d-1072-4057-ab7d-71fe9f166f7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mksft" Nov 03 13:21:28.326256 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.326235 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42cb885d-1072-4057-ab7d-71fe9f166f7d-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-mksft\" (UID: \"42cb885d-1072-4057-ab7d-71fe9f166f7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mksft" Nov 03 13:21:28.326450 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.326287 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42cb885d-1072-4057-ab7d-71fe9f166f7d-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-mksft\" (UID: \"42cb885d-1072-4057-ab7d-71fe9f166f7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mksft" Nov 03 13:21:28.326450 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.326401 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42cb885d-1072-4057-ab7d-71fe9f166f7d-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-mksft\" (UID: \"42cb885d-1072-4057-ab7d-71fe9f166f7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mksft" Nov 03 13:21:28.326788 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.326770 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42cb885d-1072-4057-ab7d-71fe9f166f7d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-mksft\" (UID: \"42cb885d-1072-4057-ab7d-71fe9f166f7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mksft" Nov 03 13:21:28.328390 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.328369 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42cb885d-1072-4057-ab7d-71fe9f166f7d-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-mksft\" (UID: \"42cb885d-1072-4057-ab7d-71fe9f166f7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mksft" Nov 03 13:21:28.328536 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.328519 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42cb885d-1072-4057-ab7d-71fe9f166f7d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-mksft\" (UID: \"42cb885d-1072-4057-ab7d-71fe9f166f7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mksft" Nov 03 13:21:28.354484 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.354447 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq" Nov 03 13:21:28.380399 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.380371 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42cb885d-1072-4057-ab7d-71fe9f166f7d-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-mksft\" (UID: \"42cb885d-1072-4057-ab7d-71fe9f166f7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mksft" Nov 03 13:21:28.380567 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.380405 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcl6b\" (UniqueName: \"kubernetes.io/projected/42cb885d-1072-4057-ab7d-71fe9f166f7d-kube-api-access-lcl6b\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-mksft\" (UID: \"42cb885d-1072-4057-ab7d-71fe9f166f7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mksft" Nov 03 13:21:28.666883 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.666849 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mksft" Nov 03 13:21:28.779835 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.779794 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq"] Nov 03 13:21:28.785591 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:21:28.785564 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad7c3ee7_2be7_4124_9a89_1be00bbc1b21.slice/crio-e555f8164b9b3d9d3d58b8a3d9ef16949b8df5cb7046bddfbc82366ff452696d WatchSource:0}: Error finding container e555f8164b9b3d9d3d58b8a3d9ef16949b8df5cb7046bddfbc82366ff452696d: Status 404 returned error can't find the container with id e555f8164b9b3d9d3d58b8a3d9ef16949b8df5cb7046bddfbc82366ff452696d Nov 03 13:21:28.787978 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.787937 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:21:28.788078 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.788014 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:21:28.788078 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:28.788058 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:21:29.050379 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:29.050292 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mksft"] Nov 03 13:21:29.057009 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:21:29.056979 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42cb885d_1072_4057_ab7d_71fe9f166f7d.slice/crio-82d8c8a1fa4d6121a24d97cbdd4c87a5cba3ffe4335812576e169aabe93550b5 WatchSource:0}: Error finding container 82d8c8a1fa4d6121a24d97cbdd4c87a5cba3ffe4335812576e169aabe93550b5: Status 404 returned error can't find the container with id 82d8c8a1fa4d6121a24d97cbdd4c87a5cba3ffe4335812576e169aabe93550b5 Nov 03 13:21:29.059403 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:29.059368 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:21:29.059535 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:29.059448 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:21:29.059535 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:29.059506 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:21:29.431648 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:29.431613 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mksft" event={"ID":"42cb885d-1072-4057-ab7d-71fe9f166f7d","Type":"ContainerStarted","Data":"d6319f06d469cad4eeae33f3859af5dc985efb88b2f5b185684c96f3d55ab0d2"} Nov 03 13:21:29.431818 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:29.431669 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mksft" event={"ID":"42cb885d-1072-4057-ab7d-71fe9f166f7d","Type":"ContainerStarted","Data":"82d8c8a1fa4d6121a24d97cbdd4c87a5cba3ffe4335812576e169aabe93550b5"} Nov 03 13:21:29.433384 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:29.433360 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq" event={"ID":"ad7c3ee7-2be7-4124-9a89-1be00bbc1b21","Type":"ContainerStarted","Data":"10149c7f0295a4527780a4427491469fccef3771d539f615733ae686a7328110"} Nov 03 13:21:29.433525 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:29.433396 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq" event={"ID":"ad7c3ee7-2be7-4124-9a89-1be00bbc1b21","Type":"ContainerStarted","Data":"e555f8164b9b3d9d3d58b8a3d9ef16949b8df5cb7046bddfbc82366ff452696d"} Nov 03 13:21:29.556108 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:29.555814 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mksft" podStartSLOduration=1.5557912379999999 podStartE2EDuration="1.555791238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:21:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:21:29.54115885 +0000 UTC m=+971.551847091" watchObservedRunningTime="2026-11-03 13:21:29.555791238 +0000 UTC m=+971.566479484" Nov 03 13:21:29.657509 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:29.655592 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq" podStartSLOduration=2.6555729340000003 podStartE2EDuration="2.655572934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:21:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:21:29.654024439 +0000 UTC m=+971.664712706" watchObservedRunningTime="2026-11-03 13:21:29.655572934 +0000 UTC m=+971.666261179" Nov 03 13:21:29.661508 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:29.659870 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-a-f2a72750-kserve-5d75f8cf5c-mjjqx" Nov 03 13:21:29.671503 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:29.670067 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mksft" Nov 03 13:21:29.712163 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:29.712083 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-a-f2a72750-kserve-5d75f8cf5c-mjjqx" Nov 03 13:21:30.378498 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:30.376556 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq" Nov 03 13:21:30.382492 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:30.381601 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 13:21:30.382492 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:30.381663 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq" podUID="ad7c3ee7-2be7-4124-9a89-1be00bbc1b21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 13:21:30.500256 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:30.500220 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-a-f2a72750-kserve-5d75f8cf5c-mjjqx" Nov 03 13:21:30.700498 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:30.700442 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-mksft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 13:21:30.700988 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:30.700517 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mksft" podUID="42cb885d-1072-4057-ab7d-71fe9f166f7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 13:21:31.355497 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:31.354950 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 13:21:31.355497 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:31.355015 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq" podUID="ad7c3ee7-2be7-4124-9a89-1be00bbc1b21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 13:21:31.698959 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:31.698913 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-mksft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 13:21:31.699130 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:31.698990 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mksft" podUID="42cb885d-1072-4057-ab7d-71fe9f166f7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 13:21:32.357508 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:32.356960 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 13:21:32.357508 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:32.357025 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq" podUID="ad7c3ee7-2be7-4124-9a89-1be00bbc1b21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 13:21:32.675496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:32.673946 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-mksft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 13:21:32.675496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:32.674010 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mksft" podUID="42cb885d-1072-4057-ab7d-71fe9f166f7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 13:21:33.360496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:33.356923 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 13:21:33.360496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:33.356986 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq" podUID="ad7c3ee7-2be7-4124-9a89-1be00bbc1b21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 13:21:33.668000 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:33.667963 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-mksft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 13:21:33.668312 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:33.668282 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mksft" podUID="42cb885d-1072-4057-ab7d-71fe9f166f7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 13:21:34.356911 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:34.356871 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 13:21:34.357212 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:34.357176 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq" podUID="ad7c3ee7-2be7-4124-9a89-1be00bbc1b21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 13:21:34.668912 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:34.668868 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-mksft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 13:21:34.669553 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:34.669511 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mksft" podUID="42cb885d-1072-4057-ab7d-71fe9f166f7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 13:21:35.355054 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:35.355015 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 13:21:35.355340 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:35.355309 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq" podUID="ad7c3ee7-2be7-4124-9a89-1be00bbc1b21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 13:21:35.667368 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:35.667330 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-mksft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 13:21:35.667704 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:35.667671 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mksft" podUID="42cb885d-1072-4057-ab7d-71fe9f166f7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 13:21:36.355704 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:36.355655 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Nov 03 13:21:36.356241 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:36.355728 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq" podUID="ad7c3ee7-2be7-4124-9a89-1be00bbc1b21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Nov 03 13:21:36.667428 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:36.667381 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-mksft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Nov 03 13:21:36.667655 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:36.667450 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mksft" podUID="42cb885d-1072-4057-ab7d-71fe9f166f7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Nov 03 13:21:37.250024 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:21:37.249976 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:21:38.356004 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:38.355956 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:21:38.356449 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:38.356045 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq" podUID="ad7c3ee7-2be7-4124-9a89-1be00bbc1b21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:21:38.426894 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:38.426858 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5d66dc6b6c-tnc8j" podUID="ce8ed64a-b642-4e26-aeb1-ab8d483b51f1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:21:38.668689 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:38.668648 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-mksft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:21:38.668902 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:38.668716 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mksft" podUID="42cb885d-1072-4057-ab7d-71fe9f166f7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:21:39.398681 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:39.398639 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:21:39.399333 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:39.399294 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq" podUID="ad7c3ee7-2be7-4124-9a89-1be00bbc1b21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:21:39.710678 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:39.710583 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-mksft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:21:39.710678 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:39.710634 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mksft" podUID="42cb885d-1072-4057-ab7d-71fe9f166f7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:21:40.229197 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:40.229153 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq" Nov 03 13:21:40.229655 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:40.229629 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq" Nov 03 13:21:40.230324 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:40.230307 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq" Nov 03 13:21:40.711380 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:40.711329 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-mksft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:21:40.712040 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:40.711382 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mksft" podUID="42cb885d-1072-4057-ab7d-71fe9f166f7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:21:40.796665 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:40.796628 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mksft" Nov 03 13:21:40.796953 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:40.796924 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mksft" Nov 03 13:21:40.797622 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:40.797605 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mksft" Nov 03 13:21:43.462825 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:43.462785 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:21:48.425403 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:48.425366 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5d66dc6b6c-tnc8j" podUID="ce8ed64a-b642-4e26-aeb1-ab8d483b51f1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:21:58.425020 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:21:58.424981 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5d66dc6b6c-tnc8j" podUID="ce8ed64a-b642-4e26-aeb1-ab8d483b51f1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:22:07.235533 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:22:07.235489 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:22:08.425279 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:08.425238 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5d66dc6b6c-tnc8j" podUID="ce8ed64a-b642-4e26-aeb1-ab8d483b51f1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:22:10.154502 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:10.154456 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:22:18.059190 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:18.059156 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4"] Nov 03 13:22:18.063557 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:18.063530 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4" Nov 03 13:22:18.151357 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:18.151322 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4"] Nov 03 13:22:18.191168 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:18.191132 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a4f947e-dd99-4332-95dc-c41c040733b7-istio-token\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4\" (UID: \"0a4f947e-dd99-4332-95dc-c41c040733b7\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4" Nov 03 13:22:18.191427 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:18.191191 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a4f947e-dd99-4332-95dc-c41c040733b7-credential-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4\" (UID: \"0a4f947e-dd99-4332-95dc-c41c040733b7\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4" Nov 03 13:22:18.191427 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:18.191376 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a4f947e-dd99-4332-95dc-c41c040733b7-istiod-ca-cert\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4\" (UID: \"0a4f947e-dd99-4332-95dc-c41c040733b7\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4" Nov 03 13:22:18.191585 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:18.191429 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a4f947e-dd99-4332-95dc-c41c040733b7-workload-certs\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4\" (UID: \"0a4f947e-dd99-4332-95dc-c41c040733b7\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4" Nov 03 13:22:18.191585 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:18.191507 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a4f947e-dd99-4332-95dc-c41c040733b7-istio-envoy\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4\" (UID: \"0a4f947e-dd99-4332-95dc-c41c040733b7\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4" Nov 03 13:22:18.191585 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:18.191559 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a4f947e-dd99-4332-95dc-c41c040733b7-workload-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4\" (UID: \"0a4f947e-dd99-4332-95dc-c41c040733b7\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4" Nov 03 13:22:18.191690 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:18.191589 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a4f947e-dd99-4332-95dc-c41c040733b7-istio-data\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4\" (UID: \"0a4f947e-dd99-4332-95dc-c41c040733b7\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4" Nov 03 13:22:18.191690 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:18.191616 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a4f947e-dd99-4332-95dc-c41c040733b7-istio-podinfo\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4\" (UID: \"0a4f947e-dd99-4332-95dc-c41c040733b7\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4" Nov 03 13:22:18.191690 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:18.191650 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vlpc\" (UniqueName: \"kubernetes.io/projected/0a4f947e-dd99-4332-95dc-c41c040733b7-kube-api-access-7vlpc\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4\" (UID: \"0a4f947e-dd99-4332-95dc-c41c040733b7\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4" Nov 03 13:22:18.277997 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:18.277962 5979 scope.go:122] "RemoveContainer" containerID="295b3b803b0f5fa9b75257eff4ed7415f1f5d4903570220a47694717d6490fdb" Nov 03 13:22:18.292242 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:18.292214 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a4f947e-dd99-4332-95dc-c41c040733b7-istiod-ca-cert\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4\" (UID: \"0a4f947e-dd99-4332-95dc-c41c040733b7\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4" Nov 03 13:22:18.292368 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:18.292256 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a4f947e-dd99-4332-95dc-c41c040733b7-workload-certs\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4\" (UID: \"0a4f947e-dd99-4332-95dc-c41c040733b7\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4" Nov 03 13:22:18.292368 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:18.292288 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a4f947e-dd99-4332-95dc-c41c040733b7-istio-envoy\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4\" (UID: \"0a4f947e-dd99-4332-95dc-c41c040733b7\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4" Nov 03 13:22:18.292368 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:18.292326 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a4f947e-dd99-4332-95dc-c41c040733b7-workload-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4\" (UID: \"0a4f947e-dd99-4332-95dc-c41c040733b7\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4" Nov 03 13:22:18.292555 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:18.292367 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a4f947e-dd99-4332-95dc-c41c040733b7-istio-data\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4\" (UID: \"0a4f947e-dd99-4332-95dc-c41c040733b7\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4" Nov 03 13:22:18.292555 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:18.292407 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a4f947e-dd99-4332-95dc-c41c040733b7-istio-podinfo\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4\" (UID: \"0a4f947e-dd99-4332-95dc-c41c040733b7\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4" Nov 03 13:22:18.292555 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:18.292458 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vlpc\" (UniqueName: \"kubernetes.io/projected/0a4f947e-dd99-4332-95dc-c41c040733b7-kube-api-access-7vlpc\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4\" (UID: \"0a4f947e-dd99-4332-95dc-c41c040733b7\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4" Nov 03 13:22:18.292555 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:18.292518 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a4f947e-dd99-4332-95dc-c41c040733b7-istio-token\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4\" (UID: \"0a4f947e-dd99-4332-95dc-c41c040733b7\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4" Nov 03 13:22:18.292766 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:18.292564 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a4f947e-dd99-4332-95dc-c41c040733b7-credential-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4\" (UID: \"0a4f947e-dd99-4332-95dc-c41c040733b7\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4" Nov 03 13:22:18.292822 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:18.292775 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a4f947e-dd99-4332-95dc-c41c040733b7-workload-certs\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4\" (UID: \"0a4f947e-dd99-4332-95dc-c41c040733b7\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4" Nov 03 13:22:18.292822 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:18.292802 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a4f947e-dd99-4332-95dc-c41c040733b7-workload-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4\" (UID: \"0a4f947e-dd99-4332-95dc-c41c040733b7\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4" Nov 03 13:22:18.292928 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:18.292885 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a4f947e-dd99-4332-95dc-c41c040733b7-credential-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4\" (UID: \"0a4f947e-dd99-4332-95dc-c41c040733b7\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4" Nov 03 13:22:18.293101 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:18.293076 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a4f947e-dd99-4332-95dc-c41c040733b7-istiod-ca-cert\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4\" (UID: \"0a4f947e-dd99-4332-95dc-c41c040733b7\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4" Nov 03 13:22:18.293241 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:18.293217 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a4f947e-dd99-4332-95dc-c41c040733b7-istio-data\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4\" (UID: \"0a4f947e-dd99-4332-95dc-c41c040733b7\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4" Nov 03 13:22:18.294680 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:18.294661 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a4f947e-dd99-4332-95dc-c41c040733b7-istio-envoy\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4\" (UID: \"0a4f947e-dd99-4332-95dc-c41c040733b7\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4" Nov 03 13:22:18.295984 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:18.295963 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a4f947e-dd99-4332-95dc-c41c040733b7-istio-podinfo\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4\" (UID: \"0a4f947e-dd99-4332-95dc-c41c040733b7\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4" Nov 03 13:22:18.346320 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:18.346249 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a4f947e-dd99-4332-95dc-c41c040733b7-istio-token\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4\" (UID: \"0a4f947e-dd99-4332-95dc-c41c040733b7\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4" Nov 03 13:22:18.346507 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:18.346322 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vlpc\" (UniqueName: \"kubernetes.io/projected/0a4f947e-dd99-4332-95dc-c41c040733b7-kube-api-access-7vlpc\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4\" (UID: \"0a4f947e-dd99-4332-95dc-c41c040733b7\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4" Nov 03 13:22:18.377639 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:18.377609 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4" Nov 03 13:22:18.425752 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:18.425718 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5d66dc6b6c-tnc8j" podUID="ce8ed64a-b642-4e26-aeb1-ab8d483b51f1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:22:18.811430 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:18.811398 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4"] Nov 03 13:22:18.818058 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:22:18.818028 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a4f947e_dd99_4332_95dc_c41c040733b7.slice/crio-745cb3f242784008e96da88ac43b8a545cda62a317620c9f1fd5ebe409ca0ff4 WatchSource:0}: Error finding container 745cb3f242784008e96da88ac43b8a545cda62a317620c9f1fd5ebe409ca0ff4: Status 404 returned error can't find the container with id 745cb3f242784008e96da88ac43b8a545cda62a317620c9f1fd5ebe409ca0ff4 Nov 03 13:22:18.820550 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:18.820517 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:22:18.820652 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:18.820585 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:22:18.820652 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:18.820615 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:22:19.829330 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:19.829296 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4" event={"ID":"0a4f947e-dd99-4332-95dc-c41c040733b7","Type":"ContainerStarted","Data":"b59c381b93fa9c8d3329dc5260151cf8162f092a87241d3305f9d4acf0ca40cf"} Nov 03 13:22:19.829902 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:19.829349 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4" event={"ID":"0a4f947e-dd99-4332-95dc-c41c040733b7","Type":"ContainerStarted","Data":"745cb3f242784008e96da88ac43b8a545cda62a317620c9f1fd5ebe409ca0ff4"} Nov 03 13:22:19.985590 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:19.985521 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4" podStartSLOduration=1.98550309 podStartE2EDuration="1.98550309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:22:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:22:19.98423995 +0000 UTC m=+1021.994928192" watchObservedRunningTime="2026-11-03 13:22:19.98550309 +0000 UTC m=+1021.996191329" Nov 03 13:22:20.378826 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:20.378787 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4" Nov 03 13:22:20.380583 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:20.380556 5979 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 13:22:20.380654 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:20.380608 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4" podUID="0a4f947e-dd99-4332-95dc-c41c040733b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 13:22:21.379103 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:21.379066 5979 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 13:22:21.379535 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:21.379125 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4" podUID="0a4f947e-dd99-4332-95dc-c41c040733b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 13:22:22.378537 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:22.378503 5979 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 13:22:22.378718 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:22.378557 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4" podUID="0a4f947e-dd99-4332-95dc-c41c040733b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 13:22:23.379064 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:23.379019 5979 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 13:22:23.379498 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:23.379085 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4" podUID="0a4f947e-dd99-4332-95dc-c41c040733b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 13:22:24.379081 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:24.379044 5979 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 13:22:24.379526 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:24.379103 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4" podUID="0a4f947e-dd99-4332-95dc-c41c040733b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 13:22:25.378260 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:25.378219 5979 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Nov 03 13:22:25.378440 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:25.378275 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4" podUID="0a4f947e-dd99-4332-95dc-c41c040733b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Nov 03 13:22:26.383174 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:26.383144 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4" Nov 03 13:22:26.383610 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:26.383572 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4" Nov 03 13:22:26.384254 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:26.384234 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4" Nov 03 13:22:28.425552 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:28.425502 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5d66dc6b6c-tnc8j" podUID="ce8ed64a-b642-4e26-aeb1-ab8d483b51f1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:22:36.459213 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:36.459174 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:22:37.295248 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:22:37.295211 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:22:38.114023 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:38.113988 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5d66dc6b6c-tnc8j" podUID="ce8ed64a-b642-4e26-aeb1-ab8d483b51f1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:22:48.114142 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:48.114106 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5d66dc6b6c-tnc8j" podUID="ce8ed64a-b642-4e26-aeb1-ab8d483b51f1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:22:48.176645 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:22:48.176613 5979 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-6d0760cb6b9f: secret "agc-cred-6d0760cb6b9f" not found Nov 03 13:22:48.176871 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:22:48.176852 5979 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7dfbfc6544-jz7cm: secret "agc-cred-6d0760cb6b9f" not found Nov 03 13:22:48.176975 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:22:48.176950 5979 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/61d2af57-4003-4364-b4f4-769565d27b65-provider-credentials podName:61d2af57-4003-4364-b4f4-769565d27b65 nodeName:}" failed. No retries permitted until 2026-11-03 13:22:48.676929306 +0000 UTC m=+1050.687617546 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/61d2af57-4003-4364-b4f4-769565d27b65-provider-credentials") pod "payload-processing-external-model-7dfbfc6544-jz7cm" (UID: "61d2af57-4003-4364-b4f4-769565d27b65") : secret "agc-cred-6d0760cb6b9f" not found Nov 03 13:22:48.681993 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:22:48.681960 5979 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-6d0760cb6b9f: secret "agc-cred-6d0760cb6b9f" not found Nov 03 13:22:48.681993 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:22:48.681993 5979 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7dfbfc6544-jz7cm: secret "agc-cred-6d0760cb6b9f" not found Nov 03 13:22:48.682217 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:22:48.682060 5979 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/61d2af57-4003-4364-b4f4-769565d27b65-provider-credentials podName:61d2af57-4003-4364-b4f4-769565d27b65 nodeName:}" failed. No retries permitted until 2026-11-03 13:22:49.682039381 +0000 UTC m=+1051.692727606 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/61d2af57-4003-4364-b4f4-769565d27b65-provider-credentials") pod "payload-processing-external-model-7dfbfc6544-jz7cm" (UID: "61d2af57-4003-4364-b4f4-769565d27b65") : secret "agc-cred-6d0760cb6b9f" not found Nov 03 13:22:49.694819 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:22:49.694783 5979 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-6d0760cb6b9f: secret "agc-cred-6d0760cb6b9f" not found Nov 03 13:22:49.694819 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:22:49.694814 5979 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7dfbfc6544-jz7cm: secret "agc-cred-6d0760cb6b9f" not found Nov 03 13:22:49.695249 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:22:49.694877 5979 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/61d2af57-4003-4364-b4f4-769565d27b65-provider-credentials podName:61d2af57-4003-4364-b4f4-769565d27b65 nodeName:}" failed. No retries permitted until 2026-11-03 13:22:51.694861049 +0000 UTC m=+1053.705549269 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/61d2af57-4003-4364-b4f4-769565d27b65-provider-credentials") pod "payload-processing-external-model-7dfbfc6544-jz7cm" (UID: "61d2af57-4003-4364-b4f4-769565d27b65") : secret "agc-cred-6d0760cb6b9f" not found Nov 03 13:22:51.718393 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:22:51.718358 5979 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-6d0760cb6b9f: secret "agc-cred-6d0760cb6b9f" not found Nov 03 13:22:51.718393 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:22:51.718389 5979 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7dfbfc6544-jz7cm: secret "agc-cred-6d0760cb6b9f" not found Nov 03 13:22:51.718949 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:22:51.718451 5979 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/61d2af57-4003-4364-b4f4-769565d27b65-provider-credentials podName:61d2af57-4003-4364-b4f4-769565d27b65 nodeName:}" failed. No retries permitted until 2026-11-03 13:22:55.71842819 +0000 UTC m=+1057.729116412 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/61d2af57-4003-4364-b4f4-769565d27b65-provider-credentials") pod "payload-processing-external-model-7dfbfc6544-jz7cm" (UID: "61d2af57-4003-4364-b4f4-769565d27b65") : secret "agc-cred-6d0760cb6b9f" not found Nov 03 13:22:55.786794 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:22:55.786755 5979 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-f765a3aeab00: secret "agc-cred-f765a3aeab00" not found Nov 03 13:22:55.786794 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:22:55.786786 5979 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7dfbfc6544-jz7cm: secret "agc-cred-f765a3aeab00" not found Nov 03 13:22:55.787227 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:22:55.786844 5979 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/61d2af57-4003-4364-b4f4-769565d27b65-provider-credentials podName:61d2af57-4003-4364-b4f4-769565d27b65 nodeName:}" failed. No retries permitted until 2026-11-03 13:23:03.786829827 +0000 UTC m=+1065.797518047 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/61d2af57-4003-4364-b4f4-769565d27b65-provider-credentials") pod "payload-processing-external-model-7dfbfc6544-jz7cm" (UID: "61d2af57-4003-4364-b4f4-769565d27b65") : secret "agc-cred-f765a3aeab00" not found Nov 03 13:22:57.078979 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:57.078922 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5d66dc6b6c-tnc8j"] Nov 03 13:22:57.080388 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:57.080353 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5d66dc6b6c-tnc8j" podUID="ce8ed64a-b642-4e26-aeb1-ab8d483b51f1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:22:57.109816 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:57.109787 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-84b78b6f97-6vs8s"] Nov 03 13:22:57.114458 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:57.114437 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-84b78b6f97-6vs8s" Nov 03 13:22:57.183014 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:57.182982 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-84b78b6f97-6vs8s"] Nov 03 13:22:57.308620 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:57.308588 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ec671a2-5c40-4140-839a-8399ec4a5b92-plugins-config-volume\") pod \"payload-processing-external-model-84b78b6f97-6vs8s\" (UID: \"8ec671a2-5c40-4140-839a-8399ec4a5b92\") " pod="models-as-a-service/payload-processing-external-model-84b78b6f97-6vs8s" Nov 03 13:22:57.308795 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:57.308646 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8ec671a2-5c40-4140-839a-8399ec4a5b92-provider-credentials\") pod \"payload-processing-external-model-84b78b6f97-6vs8s\" (UID: \"8ec671a2-5c40-4140-839a-8399ec4a5b92\") " pod="models-as-a-service/payload-processing-external-model-84b78b6f97-6vs8s" Nov 03 13:22:57.308852 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:57.308800 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8ec671a2-5c40-4140-839a-8399ec4a5b92-routing-overlay-volume\") pod \"payload-processing-external-model-84b78b6f97-6vs8s\" (UID: \"8ec671a2-5c40-4140-839a-8399ec4a5b92\") " pod="models-as-a-service/payload-processing-external-model-84b78b6f97-6vs8s" Nov 03 13:22:57.308945 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:57.308929 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh2j5\" (UniqueName: \"kubernetes.io/projected/8ec671a2-5c40-4140-839a-8399ec4a5b92-kube-api-access-lh2j5\") pod \"payload-processing-external-model-84b78b6f97-6vs8s\" (UID: \"8ec671a2-5c40-4140-839a-8399ec4a5b92\") " pod="models-as-a-service/payload-processing-external-model-84b78b6f97-6vs8s" Nov 03 13:22:57.410344 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:57.410311 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ec671a2-5c40-4140-839a-8399ec4a5b92-plugins-config-volume\") pod \"payload-processing-external-model-84b78b6f97-6vs8s\" (UID: \"8ec671a2-5c40-4140-839a-8399ec4a5b92\") " pod="models-as-a-service/payload-processing-external-model-84b78b6f97-6vs8s" Nov 03 13:22:57.410548 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:57.410365 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8ec671a2-5c40-4140-839a-8399ec4a5b92-provider-credentials\") pod \"payload-processing-external-model-84b78b6f97-6vs8s\" (UID: \"8ec671a2-5c40-4140-839a-8399ec4a5b92\") " pod="models-as-a-service/payload-processing-external-model-84b78b6f97-6vs8s" Nov 03 13:22:57.410548 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:57.410424 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8ec671a2-5c40-4140-839a-8399ec4a5b92-routing-overlay-volume\") pod \"payload-processing-external-model-84b78b6f97-6vs8s\" (UID: \"8ec671a2-5c40-4140-839a-8399ec4a5b92\") " pod="models-as-a-service/payload-processing-external-model-84b78b6f97-6vs8s" Nov 03 13:22:57.410672 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:57.410569 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh2j5\" (UniqueName: \"kubernetes.io/projected/8ec671a2-5c40-4140-839a-8399ec4a5b92-kube-api-access-lh2j5\") pod \"payload-processing-external-model-84b78b6f97-6vs8s\" (UID: \"8ec671a2-5c40-4140-839a-8399ec4a5b92\") " pod="models-as-a-service/payload-processing-external-model-84b78b6f97-6vs8s" Nov 03 13:22:57.411122 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:57.411099 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ec671a2-5c40-4140-839a-8399ec4a5b92-plugins-config-volume\") pod \"payload-processing-external-model-84b78b6f97-6vs8s\" (UID: \"8ec671a2-5c40-4140-839a-8399ec4a5b92\") " pod="models-as-a-service/payload-processing-external-model-84b78b6f97-6vs8s" Nov 03 13:22:57.411122 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:57.411113 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8ec671a2-5c40-4140-839a-8399ec4a5b92-routing-overlay-volume\") pod \"payload-processing-external-model-84b78b6f97-6vs8s\" (UID: \"8ec671a2-5c40-4140-839a-8399ec4a5b92\") " pod="models-as-a-service/payload-processing-external-model-84b78b6f97-6vs8s" Nov 03 13:22:57.412860 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:57.412841 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8ec671a2-5c40-4140-839a-8399ec4a5b92-provider-credentials\") pod \"payload-processing-external-model-84b78b6f97-6vs8s\" (UID: \"8ec671a2-5c40-4140-839a-8399ec4a5b92\") " pod="models-as-a-service/payload-processing-external-model-84b78b6f97-6vs8s" Nov 03 13:22:57.472886 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:57.472854 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh2j5\" (UniqueName: \"kubernetes.io/projected/8ec671a2-5c40-4140-839a-8399ec4a5b92-kube-api-access-lh2j5\") pod \"payload-processing-external-model-84b78b6f97-6vs8s\" (UID: \"8ec671a2-5c40-4140-839a-8399ec4a5b92\") " pod="models-as-a-service/payload-processing-external-model-84b78b6f97-6vs8s" Nov 03 13:22:57.728030 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:57.727940 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-84b78b6f97-6vs8s" Nov 03 13:22:58.100989 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:58.100895 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-84b78b6f97-6vs8s"] Nov 03 13:22:58.107847 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:22:58.107814 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ec671a2_5c40_4140_839a_8399ec4a5b92.slice/crio-1ce8b0d680921b9799eabf3392a2bb358eef787fb40316d4a40f36ecbe38479f WatchSource:0}: Error finding container 1ce8b0d680921b9799eabf3392a2bb358eef787fb40316d4a40f36ecbe38479f: Status 404 returned error can't find the container with id 1ce8b0d680921b9799eabf3392a2bb358eef787fb40316d4a40f36ecbe38479f Nov 03 13:22:59.104994 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:59.104958 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-84b78b6f97-6vs8s" event={"ID":"8ec671a2-5c40-4140-839a-8399ec4a5b92","Type":"ContainerStarted","Data":"8bb6fe1938ee907cf7b68abbe9fec9ed4f83670230ab1434f834a0694fb45a79"} Nov 03 13:22:59.105389 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:59.105044 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-84b78b6f97-6vs8s" event={"ID":"8ec671a2-5c40-4140-839a-8399ec4a5b92","Type":"ContainerStarted","Data":"1ce8b0d680921b9799eabf3392a2bb358eef787fb40316d4a40f36ecbe38479f"} Nov 03 13:22:59.105389 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:59.105067 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-84b78b6f97-6vs8s" Nov 03 13:22:59.213522 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:22:59.213448 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-84b78b6f97-6vs8s" podStartSLOduration=2.213433509 podStartE2EDuration="2.213433509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:22:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:22:59.212196582 +0000 UTC m=+1061.222884824" watchObservedRunningTime="2026-11-03 13:22:59.213433509 +0000 UTC m=+1061.224121750" Nov 03 13:23:02.079395 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:02.079357 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5d66dc6b6c-tnc8j" podUID="ce8ed64a-b642-4e26-aeb1-ab8d483b51f1" containerName="payload-processing" containerID="cri-o://5e42a0a2d751118aa110a7534e59d51a9a8691ef5b867336c241bfff0a70bb4f" gracePeriod=25 Nov 03 13:23:02.344243 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:02.344175 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5d66dc6b6c-tnc8j" Nov 03 13:23:02.372093 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:02.372063 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce8ed64a-b642-4e26-aeb1-ab8d483b51f1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ce8ed64a-b642-4e26-aeb1-ab8d483b51f1-provider-credentials\") pod \"ce8ed64a-b642-4e26-aeb1-ab8d483b51f1\" (UID: \"ce8ed64a-b642-4e26-aeb1-ab8d483b51f1\") " Nov 03 13:23:02.372255 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:02.372242 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce8ed64a-b642-4e26-aeb1-ab8d483b51f1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce8ed64a-b642-4e26-aeb1-ab8d483b51f1-plugins-config-volume\") pod \"ce8ed64a-b642-4e26-aeb1-ab8d483b51f1\" (UID: \"ce8ed64a-b642-4e26-aeb1-ab8d483b51f1\") " Nov 03 13:23:02.372297 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:02.372288 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce8ed64a-b642-4e26-aeb1-ab8d483b51f1-kube-api-access-9sw62\" (UniqueName: \"kubernetes.io/projected/ce8ed64a-b642-4e26-aeb1-ab8d483b51f1-kube-api-access-9sw62\") pod \"ce8ed64a-b642-4e26-aeb1-ab8d483b51f1\" (UID: \"ce8ed64a-b642-4e26-aeb1-ab8d483b51f1\") " Nov 03 13:23:02.372344 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:02.372325 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce8ed64a-b642-4e26-aeb1-ab8d483b51f1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ce8ed64a-b642-4e26-aeb1-ab8d483b51f1-routing-overlay-volume\") pod \"ce8ed64a-b642-4e26-aeb1-ab8d483b51f1\" (UID: \"ce8ed64a-b642-4e26-aeb1-ab8d483b51f1\") " Nov 03 13:23:02.372749 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:02.372720 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce8ed64a-b642-4e26-aeb1-ab8d483b51f1-plugins-config-volume" pod "ce8ed64a-b642-4e26-aeb1-ab8d483b51f1" (UID: "ce8ed64a-b642-4e26-aeb1-ab8d483b51f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:23:02.372875 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:02.372772 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce8ed64a-b642-4e26-aeb1-ab8d483b51f1-routing-overlay-volume" pod "ce8ed64a-b642-4e26-aeb1-ab8d483b51f1" (UID: "ce8ed64a-b642-4e26-aeb1-ab8d483b51f1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:23:02.374300 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:02.374276 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce8ed64a-b642-4e26-aeb1-ab8d483b51f1-provider-credentials" pod "ce8ed64a-b642-4e26-aeb1-ab8d483b51f1" (UID: "ce8ed64a-b642-4e26-aeb1-ab8d483b51f1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:23:02.374300 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:02.374291 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce8ed64a-b642-4e26-aeb1-ab8d483b51f1-kube-api-access-9sw62" pod "ce8ed64a-b642-4e26-aeb1-ab8d483b51f1" (UID: "ce8ed64a-b642-4e26-aeb1-ab8d483b51f1"). InnerVolumeSpecName "kube-api-access-9sw62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:23:02.474175 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:02.474136 5979 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ce8ed64a-b642-4e26-aeb1-ab8d483b51f1-provider-credentials\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:23:02.474175 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:02.474164 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce8ed64a-b642-4e26-aeb1-ab8d483b51f1-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:23:02.474175 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:02.474175 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sw62\" (UniqueName: \"kubernetes.io/projected/ce8ed64a-b642-4e26-aeb1-ab8d483b51f1-kube-api-access-9sw62\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:23:02.474175 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:02.474184 5979 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ce8ed64a-b642-4e26-aeb1-ab8d483b51f1-routing-overlay-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:23:03.068989 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:03.068013 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-3c5dbc/distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd"] Nov 03 13:23:03.069329 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:03.069307 5979 cpu_manager.go:414] "containerMap: removing container" podUID="ce8ed64a-b642-4e26-aeb1-ab8d483b51f1" containerName="payload-processing" Nov 03 13:23:03.069329 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:03.069329 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce8ed64a-b642-4e26-aeb1-ab8d483b51f1" containerName="payload-processing" Nov 03 13:23:03.069570 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:03.069556 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce8ed64a-b642-4e26-aeb1-ab8d483b51f1" containerName="payload-processing" Nov 03 13:23:03.074637 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:03.074609 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd" Nov 03 13:23:03.126748 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:03.126678 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3c5dbc/distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd"] Nov 03 13:23:03.133542 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:03.133509 5979 generic.go:356] "Generic (PLEG): container finished" podID="ce8ed64a-b642-4e26-aeb1-ab8d483b51f1" containerID="5e42a0a2d751118aa110a7534e59d51a9a8691ef5b867336c241bfff0a70bb4f" exitCode=0 Nov 03 13:23:03.133721 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:03.133637 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5d66dc6b6c-tnc8j" Nov 03 13:23:03.133721 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:03.133638 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5d66dc6b6c-tnc8j" event={"ID":"ce8ed64a-b642-4e26-aeb1-ab8d483b51f1","Type":"ContainerDied","Data":"5e42a0a2d751118aa110a7534e59d51a9a8691ef5b867336c241bfff0a70bb4f"} Nov 03 13:23:03.133721 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:03.133702 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5d66dc6b6c-tnc8j" event={"ID":"ce8ed64a-b642-4e26-aeb1-ab8d483b51f1","Type":"ContainerDied","Data":"6144a89f909ad3fc0ef66296f07a29a573038b790a79337b9c8066707eb32f2e"} Nov 03 13:23:03.133905 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:03.133731 5979 scope.go:122] "RemoveContainer" containerID="5e42a0a2d751118aa110a7534e59d51a9a8691ef5b867336c241bfff0a70bb4f" Nov 03 13:23:03.146430 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:03.146403 5979 scope.go:122] "RemoveContainer" containerID="5e42a0a2d751118aa110a7534e59d51a9a8691ef5b867336c241bfff0a70bb4f" Nov 03 13:23:03.146788 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:23:03.146767 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e42a0a2d751118aa110a7534e59d51a9a8691ef5b867336c241bfff0a70bb4f\": container with ID starting with 5e42a0a2d751118aa110a7534e59d51a9a8691ef5b867336c241bfff0a70bb4f not found: ID does not exist" containerID="5e42a0a2d751118aa110a7534e59d51a9a8691ef5b867336c241bfff0a70bb4f" Nov 03 13:23:03.146868 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:03.146799 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e42a0a2d751118aa110a7534e59d51a9a8691ef5b867336c241bfff0a70bb4f"} err="failed to get container status \"5e42a0a2d751118aa110a7534e59d51a9a8691ef5b867336c241bfff0a70bb4f\": rpc error: code = NotFound desc = could not find container \"5e42a0a2d751118aa110a7534e59d51a9a8691ef5b867336c241bfff0a70bb4f\": container with ID starting with 5e42a0a2d751118aa110a7534e59d51a9a8691ef5b867336c241bfff0a70bb4f not found: ID does not exist" Nov 03 13:23:03.169665 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:03.169630 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3c5dbc\"/\"distinct-2-w4-3c5dbc-kserve-self-signed-certs\"" Nov 03 13:23:03.199300 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:03.199262 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87shq\" (UniqueName: \"kubernetes.io/projected/a4b4df54-0aac-419d-8bb2-3258267bc088-kube-api-access-87shq\") pod \"distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd\" (UID: \"a4b4df54-0aac-419d-8bb2-3258267bc088\") " pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd" Nov 03 13:23:03.199452 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:03.199323 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4b4df54-0aac-419d-8bb2-3258267bc088-model-cache\") pod \"distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd\" (UID: \"a4b4df54-0aac-419d-8bb2-3258267bc088\") " pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd" Nov 03 13:23:03.199452 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:03.199353 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4b4df54-0aac-419d-8bb2-3258267bc088-dshm\") pod \"distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd\" (UID: \"a4b4df54-0aac-419d-8bb2-3258267bc088\") " pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd" Nov 03 13:23:03.199592 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:03.199522 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4b4df54-0aac-419d-8bb2-3258267bc088-home\") pod \"distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd\" (UID: \"a4b4df54-0aac-419d-8bb2-3258267bc088\") " pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd" Nov 03 13:23:03.199592 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:03.199574 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4b4df54-0aac-419d-8bb2-3258267bc088-tls-certs\") pod \"distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd\" (UID: \"a4b4df54-0aac-419d-8bb2-3258267bc088\") " pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd" Nov 03 13:23:03.199713 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:03.199697 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4b4df54-0aac-419d-8bb2-3258267bc088-tmp-dir\") pod \"distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd\" (UID: \"a4b4df54-0aac-419d-8bb2-3258267bc088\") " pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd" Nov 03 13:23:03.300660 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:03.300614 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4b4df54-0aac-419d-8bb2-3258267bc088-model-cache\") pod \"distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd\" (UID: \"a4b4df54-0aac-419d-8bb2-3258267bc088\") " pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd" Nov 03 13:23:03.300660 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:03.300663 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4b4df54-0aac-419d-8bb2-3258267bc088-dshm\") pod \"distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd\" (UID: \"a4b4df54-0aac-419d-8bb2-3258267bc088\") " pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd" Nov 03 13:23:03.300927 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:03.300701 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4b4df54-0aac-419d-8bb2-3258267bc088-home\") pod \"distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd\" (UID: \"a4b4df54-0aac-419d-8bb2-3258267bc088\") " pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd" Nov 03 13:23:03.300927 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:03.300746 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4b4df54-0aac-419d-8bb2-3258267bc088-tls-certs\") pod \"distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd\" (UID: \"a4b4df54-0aac-419d-8bb2-3258267bc088\") " pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd" Nov 03 13:23:03.300927 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:03.300841 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4b4df54-0aac-419d-8bb2-3258267bc088-tmp-dir\") pod \"distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd\" (UID: \"a4b4df54-0aac-419d-8bb2-3258267bc088\") " pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd" Nov 03 13:23:03.301105 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:03.300976 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87shq\" (UniqueName: \"kubernetes.io/projected/a4b4df54-0aac-419d-8bb2-3258267bc088-kube-api-access-87shq\") pod \"distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd\" (UID: \"a4b4df54-0aac-419d-8bb2-3258267bc088\") " pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd" Nov 03 13:23:03.301105 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:03.301072 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4b4df54-0aac-419d-8bb2-3258267bc088-model-cache\") pod \"distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd\" (UID: \"a4b4df54-0aac-419d-8bb2-3258267bc088\") " pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd" Nov 03 13:23:03.301211 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:03.301144 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4b4df54-0aac-419d-8bb2-3258267bc088-home\") pod \"distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd\" (UID: \"a4b4df54-0aac-419d-8bb2-3258267bc088\") " pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd" Nov 03 13:23:03.301262 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:03.301214 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4b4df54-0aac-419d-8bb2-3258267bc088-tmp-dir\") pod \"distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd\" (UID: \"a4b4df54-0aac-419d-8bb2-3258267bc088\") " pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd" Nov 03 13:23:03.302982 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:03.302959 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4b4df54-0aac-419d-8bb2-3258267bc088-dshm\") pod \"distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd\" (UID: \"a4b4df54-0aac-419d-8bb2-3258267bc088\") " pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd" Nov 03 13:23:03.303259 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:03.303241 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4b4df54-0aac-419d-8bb2-3258267bc088-tls-certs\") pod \"distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd\" (UID: \"a4b4df54-0aac-419d-8bb2-3258267bc088\") " pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd" Nov 03 13:23:03.313234 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:03.313209 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5d66dc6b6c-tnc8j"] Nov 03 13:23:03.316761 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:03.316738 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5d66dc6b6c-tnc8j"] Nov 03 13:23:03.355546 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:03.355461 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87shq\" (UniqueName: \"kubernetes.io/projected/a4b4df54-0aac-419d-8bb2-3258267bc088-kube-api-access-87shq\") pod \"distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd\" (UID: \"a4b4df54-0aac-419d-8bb2-3258267bc088\") " pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd" Nov 03 13:23:03.388634 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:03.388600 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd" Nov 03 13:23:03.809169 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:23:03.809132 5979 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-f765a3aeab00: secret "agc-cred-f765a3aeab00" not found Nov 03 13:23:03.809169 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:23:03.809161 5979 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7dfbfc6544-jz7cm: secret "agc-cred-f765a3aeab00" not found Nov 03 13:23:03.809401 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:23:03.809212 5979 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/61d2af57-4003-4364-b4f4-769565d27b65-provider-credentials podName:61d2af57-4003-4364-b4f4-769565d27b65 nodeName:}" failed. No retries permitted until 2026-11-03 13:23:19.809198546 +0000 UTC m=+1081.819886784 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/61d2af57-4003-4364-b4f4-769565d27b65-provider-credentials") pod "payload-processing-external-model-7dfbfc6544-jz7cm" (UID: "61d2af57-4003-4364-b4f4-769565d27b65") : secret "agc-cred-f765a3aeab00" not found Nov 03 13:23:03.838940 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:03.838905 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3c5dbc/distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd"] Nov 03 13:23:03.845750 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:23:03.845717 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda4b4df54_0aac_419d_8bb2_3258267bc088.slice/crio-e2b4e6437bbf2f3ae387c04570c3ba824a4abc75f30cd9b28bade1a5caca5d14 WatchSource:0}: Error finding container e2b4e6437bbf2f3ae387c04570c3ba824a4abc75f30cd9b28bade1a5caca5d14: Status 404 returned error can't find the container with id e2b4e6437bbf2f3ae387c04570c3ba824a4abc75f30cd9b28bade1a5caca5d14 Nov 03 13:23:04.114944 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:04.114865 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce8ed64a-b642-4e26-aeb1-ab8d483b51f1" path="/var/lib/kubelet/pods/ce8ed64a-b642-4e26-aeb1-ab8d483b51f1/volumes" Nov 03 13:23:04.142545 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:04.142503 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd" event={"ID":"a4b4df54-0aac-419d-8bb2-3258267bc088","Type":"ContainerStarted","Data":"7e0b9b76152fbf6f744744032118cefd3ac91dc36b339256ad93539ede57dd44"} Nov 03 13:23:04.142918 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:04.142554 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd" event={"ID":"a4b4df54-0aac-419d-8bb2-3258267bc088","Type":"ContainerStarted","Data":"e2b4e6437bbf2f3ae387c04570c3ba824a4abc75f30cd9b28bade1a5caca5d14"} Nov 03 13:23:04.251274 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:04.251205 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd" podStartSLOduration=1.251187619 podStartE2EDuration="1.251187619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:23:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:23:04.250585026 +0000 UTC m=+1066.261273280" watchObservedRunningTime="2026-11-03 13:23:04.251187619 +0000 UTC m=+1066.261875864" Nov 03 13:23:04.399783 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:04.399750 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd" Nov 03 13:23:07.243191 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:23:07.243153 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:23:10.115276 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:10.115233 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-84b78b6f97-6vs8s" podUID="8ec671a2-5c40-4140-839a-8399ec4a5b92" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:23:13.389445 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:13.389412 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd" Nov 03 13:23:13.401985 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:13.401961 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd" Nov 03 13:23:14.265414 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:14.265381 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd" Nov 03 13:23:15.483988 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:15.483960 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:23:18.302173 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:18.302138 5979 scope.go:122] "RemoveContainer" containerID="0ae3ad017e58ac65a197849a32b9636b04504c99126da636f4e7f3df76072014" Nov 03 13:23:18.311943 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:18.311921 5979 scope.go:122] "RemoveContainer" containerID="e36e7c73e854b5db34ac463164d4436736bdb11672c45c67f7f6a7f90b9359b0" Nov 03 13:23:18.323436 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:18.322699 5979 scope.go:122] "RemoveContainer" containerID="37729346ca6aa0a0934ea94b239a713906fad7b5986bd47f7e1954b06c86d0d9" Nov 03 13:23:18.332465 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:18.332446 5979 scope.go:122] "RemoveContainer" containerID="191520a43514af6d7723138623a1fbd03798e4d38f114119a9c6d2833a74b319" Nov 03 13:23:18.341113 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:18.341090 5979 scope.go:122] "RemoveContainer" containerID="48b415cb9202ec7e64f3da0c0404d0d311035fac5f6b1fd7fa60a38d77baadcc" Nov 03 13:23:19.846653 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:23:19.846620 5979 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-f765a3aeab00: secret "agc-cred-f765a3aeab00" not found Nov 03 13:23:19.846653 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:23:19.846645 5979 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7dfbfc6544-jz7cm: secret "agc-cred-f765a3aeab00" not found Nov 03 13:23:19.847082 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:23:19.846697 5979 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/61d2af57-4003-4364-b4f4-769565d27b65-provider-credentials podName:61d2af57-4003-4364-b4f4-769565d27b65 nodeName:}" failed. No retries permitted until 2026-11-03 13:23:51.846679559 +0000 UTC m=+1113.857367796 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/61d2af57-4003-4364-b4f4-769565d27b65-provider-credentials") pod "payload-processing-external-model-7dfbfc6544-jz7cm" (UID: "61d2af57-4003-4364-b4f4-769565d27b65") : secret "agc-cred-f765a3aeab00" not found Nov 03 13:23:20.113766 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:20.113675 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-84b78b6f97-6vs8s" podUID="8ec671a2-5c40-4140-839a-8399ec4a5b92" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:23:30.112987 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:30.112951 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-84b78b6f97-6vs8s" podUID="8ec671a2-5c40-4140-839a-8399ec4a5b92" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:23:37.061579 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:37.061539 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fcf7766-ptb4c"] Nov 03 13:23:37.065686 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:37.065663 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fcf7766-ptb4c" Nov 03 13:23:37.098599 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:37.098567 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a32d8b0-e28d-4f9a-9bad-461cd71473fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fcf7766-ptb4c\" (UID: \"7a32d8b0-e28d-4f9a-9bad-461cd71473fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fcf7766-ptb4c" Nov 03 13:23:37.098740 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:37.098602 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a32d8b0-e28d-4f9a-9bad-461cd71473fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fcf7766-ptb4c\" (UID: \"7a32d8b0-e28d-4f9a-9bad-461cd71473fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fcf7766-ptb4c" Nov 03 13:23:37.098740 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:37.098630 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpqtv\" (UniqueName: \"kubernetes.io/projected/7a32d8b0-e28d-4f9a-9bad-461cd71473fa-kube-api-access-fpqtv\") pod \"maas-api-revoke-sub-5fcf7766-ptb4c\" (UID: \"7a32d8b0-e28d-4f9a-9bad-461cd71473fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fcf7766-ptb4c" Nov 03 13:23:37.119399 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:37.119368 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fcf7766-ptb4c"] Nov 03 13:23:37.162196 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:37.162161 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:23:37.162196 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:37.162184 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 13:23:37.162196 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:37.162202 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 13:23:37.162462 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:37.162248 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:23:37.200127 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:37.200087 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a32d8b0-e28d-4f9a-9bad-461cd71473fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fcf7766-ptb4c\" (UID: \"7a32d8b0-e28d-4f9a-9bad-461cd71473fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fcf7766-ptb4c" Nov 03 13:23:37.200328 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:37.200136 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a32d8b0-e28d-4f9a-9bad-461cd71473fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fcf7766-ptb4c\" (UID: \"7a32d8b0-e28d-4f9a-9bad-461cd71473fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fcf7766-ptb4c" Nov 03 13:23:37.200328 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:37.200165 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpqtv\" (UniqueName: \"kubernetes.io/projected/7a32d8b0-e28d-4f9a-9bad-461cd71473fa-kube-api-access-fpqtv\") pod \"maas-api-revoke-sub-5fcf7766-ptb4c\" (UID: \"7a32d8b0-e28d-4f9a-9bad-461cd71473fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fcf7766-ptb4c" Nov 03 13:23:37.200821 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:37.200797 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a32d8b0-e28d-4f9a-9bad-461cd71473fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fcf7766-ptb4c\" (UID: \"7a32d8b0-e28d-4f9a-9bad-461cd71473fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fcf7766-ptb4c" Nov 03 13:23:37.202595 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:37.202569 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a32d8b0-e28d-4f9a-9bad-461cd71473fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fcf7766-ptb4c\" (UID: \"7a32d8b0-e28d-4f9a-9bad-461cd71473fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fcf7766-ptb4c" Nov 03 13:23:37.235055 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:23:37.235017 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:23:37.254123 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:37.254079 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpqtv\" (UniqueName: \"kubernetes.io/projected/7a32d8b0-e28d-4f9a-9bad-461cd71473fa-kube-api-access-fpqtv\") pod \"maas-api-revoke-sub-5fcf7766-ptb4c\" (UID: \"7a32d8b0-e28d-4f9a-9bad-461cd71473fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fcf7766-ptb4c" Nov 03 13:23:37.379409 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:37.379319 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fcf7766-ptb4c" Nov 03 13:23:37.753549 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:37.753518 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fcf7766-ptb4c"] Nov 03 13:23:37.760176 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:23:37.760135 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a32d8b0_e28d_4f9a_9bad_461cd71473fa.slice/crio-949f09c00e2d720fd2522af995cbeb6f8a654d2e2926a41372b0c8fb926c3860 WatchSource:0}: Error finding container 949f09c00e2d720fd2522af995cbeb6f8a654d2e2926a41372b0c8fb926c3860: Status 404 returned error can't find the container with id 949f09c00e2d720fd2522af995cbeb6f8a654d2e2926a41372b0c8fb926c3860 Nov 03 13:23:38.408210 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:38.408175 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fcf7766-ptb4c" event={"ID":"7a32d8b0-e28d-4f9a-9bad-461cd71473fa","Type":"ContainerStarted","Data":"a0c8051819b8aa5e8529d622d95453fc39b8a0e3cc6ff8c6270a3da4c1596244"} Nov 03 13:23:38.408623 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:38.408231 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fcf7766-ptb4c" event={"ID":"7a32d8b0-e28d-4f9a-9bad-461cd71473fa","Type":"ContainerStarted","Data":"949f09c00e2d720fd2522af995cbeb6f8a654d2e2926a41372b0c8fb926c3860"} Nov 03 13:23:39.414957 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:39.414924 5979 generic.go:356] "Generic (PLEG): container finished" podID="7a32d8b0-e28d-4f9a-9bad-461cd71473fa" containerID="a0c8051819b8aa5e8529d622d95453fc39b8a0e3cc6ff8c6270a3da4c1596244" exitCode=0 Nov 03 13:23:39.415379 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:39.415008 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fcf7766-ptb4c" event={"ID":"7a32d8b0-e28d-4f9a-9bad-461cd71473fa","Type":"ContainerDied","Data":"a0c8051819b8aa5e8529d622d95453fc39b8a0e3cc6ff8c6270a3da4c1596244"} Nov 03 13:23:40.113789 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:40.113750 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-84b78b6f97-6vs8s" podUID="8ec671a2-5c40-4140-839a-8399ec4a5b92" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:23:40.567223 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:40.567199 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fcf7766-ptb4c" Nov 03 13:23:40.644590 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:40.644555 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a32d8b0-e28d-4f9a-9bad-461cd71473fa-kube-api-access-fpqtv\" (UniqueName: \"kubernetes.io/projected/7a32d8b0-e28d-4f9a-9bad-461cd71473fa-kube-api-access-fpqtv\") pod \"7a32d8b0-e28d-4f9a-9bad-461cd71473fa\" (UID: \"7a32d8b0-e28d-4f9a-9bad-461cd71473fa\") " Nov 03 13:23:40.644780 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:40.644642 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a32d8b0-e28d-4f9a-9bad-461cd71473fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a32d8b0-e28d-4f9a-9bad-461cd71473fa-maas-api-openshift-ca\") pod \"7a32d8b0-e28d-4f9a-9bad-461cd71473fa\" (UID: \"7a32d8b0-e28d-4f9a-9bad-461cd71473fa\") " Nov 03 13:23:40.644780 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:40.644663 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a32d8b0-e28d-4f9a-9bad-461cd71473fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a32d8b0-e28d-4f9a-9bad-461cd71473fa-maas-api-xks-ca\") pod \"7a32d8b0-e28d-4f9a-9bad-461cd71473fa\" (UID: \"7a32d8b0-e28d-4f9a-9bad-461cd71473fa\") " Nov 03 13:23:40.644960 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:40.644936 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a32d8b0-e28d-4f9a-9bad-461cd71473fa-maas-api-openshift-ca" pod "7a32d8b0-e28d-4f9a-9bad-461cd71473fa" (UID: "7a32d8b0-e28d-4f9a-9bad-461cd71473fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:23:40.645242 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:40.645223 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a32d8b0-e28d-4f9a-9bad-461cd71473fa-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:23:40.646693 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:40.646673 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a32d8b0-e28d-4f9a-9bad-461cd71473fa-kube-api-access-fpqtv" pod "7a32d8b0-e28d-4f9a-9bad-461cd71473fa" (UID: "7a32d8b0-e28d-4f9a-9bad-461cd71473fa"). InnerVolumeSpecName "kube-api-access-fpqtv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:23:40.646798 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:40.646777 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a32d8b0-e28d-4f9a-9bad-461cd71473fa-maas-api-xks-ca" pod "7a32d8b0-e28d-4f9a-9bad-461cd71473fa" (UID: "7a32d8b0-e28d-4f9a-9bad-461cd71473fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:23:40.746091 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:40.746009 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a32d8b0-e28d-4f9a-9bad-461cd71473fa-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:23:40.746091 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:40.746038 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpqtv\" (UniqueName: \"kubernetes.io/projected/7a32d8b0-e28d-4f9a-9bad-461cd71473fa-kube-api-access-fpqtv\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:23:41.438006 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:41.437972 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fcf7766-ptb4c" Nov 03 13:23:41.438006 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:41.437994 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fcf7766-ptb4c" event={"ID":"7a32d8b0-e28d-4f9a-9bad-461cd71473fa","Type":"ContainerDied","Data":"949f09c00e2d720fd2522af995cbeb6f8a654d2e2926a41372b0c8fb926c3860"} Nov 03 13:23:41.438231 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:41.438074 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="949f09c00e2d720fd2522af995cbeb6f8a654d2e2926a41372b0c8fb926c3860" Nov 03 13:23:43.141344 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:43.141309 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:23:50.113438 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:50.113399 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-84b78b6f97-6vs8s" podUID="8ec671a2-5c40-4140-839a-8399ec4a5b92" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:23:51.897375 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:23:51.897343 5979 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-f765a3aeab00: secret "agc-cred-f765a3aeab00" not found Nov 03 13:23:51.897375 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:23:51.897367 5979 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7dfbfc6544-jz7cm: secret "agc-cred-f765a3aeab00" not found Nov 03 13:23:51.897816 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:23:51.897427 5979 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/61d2af57-4003-4364-b4f4-769565d27b65-provider-credentials podName:61d2af57-4003-4364-b4f4-769565d27b65 nodeName:}" failed. No retries permitted until 2026-11-03 13:24:55.897411465 +0000 UTC m=+1177.908099701 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/61d2af57-4003-4364-b4f4-769565d27b65-provider-credentials") pod "payload-processing-external-model-7dfbfc6544-jz7cm" (UID: "61d2af57-4003-4364-b4f4-769565d27b65") : secret "agc-cred-f765a3aeab00" not found Nov 03 13:23:53.671716 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:53.671680 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21564fc7-b4kkc"] Nov 03 13:23:53.672213 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:53.672199 5979 cpu_manager.go:414] "containerMap: removing container" podUID="7a32d8b0-e28d-4f9a-9bad-461cd71473fa" containerName="revoke-keys" Nov 03 13:23:53.672263 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:53.672215 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a32d8b0-e28d-4f9a-9bad-461cd71473fa" containerName="revoke-keys" Nov 03 13:23:53.672345 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:53.672335 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a32d8b0-e28d-4f9a-9bad-461cd71473fa" containerName="revoke-keys" Nov 03 13:23:53.676018 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:53.675997 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21564fc7-b4kkc" Nov 03 13:23:53.725900 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:53.725448 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:23:53.725900 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:53.725463 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:23:53.772023 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:53.771984 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 13:23:53.772023 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:53.771985 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 13:23:53.774515 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:53.774493 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21564fc7-b4kkc"] Nov 03 13:23:53.819637 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:53.819601 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56b539c3-1509-49a0-9a52-009dc1c0f3f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21564fc7-b4kkc\" (UID: \"56b539c3-1509-49a0-9a52-009dc1c0f3f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21564fc7-b4kkc" Nov 03 13:23:53.819794 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:53.819744 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56b539c3-1509-49a0-9a52-009dc1c0f3f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21564fc7-b4kkc\" (UID: \"56b539c3-1509-49a0-9a52-009dc1c0f3f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21564fc7-b4kkc" Nov 03 13:23:53.819848 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:53.819808 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4psdh\" (UniqueName: \"kubernetes.io/projected/56b539c3-1509-49a0-9a52-009dc1c0f3f7-kube-api-access-4psdh\") pod \"maas-api-revoke-sub-21564fc7-b4kkc\" (UID: \"56b539c3-1509-49a0-9a52-009dc1c0f3f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21564fc7-b4kkc" Nov 03 13:23:53.921292 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:53.921255 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56b539c3-1509-49a0-9a52-009dc1c0f3f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21564fc7-b4kkc\" (UID: \"56b539c3-1509-49a0-9a52-009dc1c0f3f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21564fc7-b4kkc" Nov 03 13:23:53.921524 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:53.921369 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56b539c3-1509-49a0-9a52-009dc1c0f3f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21564fc7-b4kkc\" (UID: \"56b539c3-1509-49a0-9a52-009dc1c0f3f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21564fc7-b4kkc" Nov 03 13:23:53.921524 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:53.921442 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4psdh\" (UniqueName: \"kubernetes.io/projected/56b539c3-1509-49a0-9a52-009dc1c0f3f7-kube-api-access-4psdh\") pod \"maas-api-revoke-sub-21564fc7-b4kkc\" (UID: \"56b539c3-1509-49a0-9a52-009dc1c0f3f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21564fc7-b4kkc" Nov 03 13:23:53.922091 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:53.922033 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56b539c3-1509-49a0-9a52-009dc1c0f3f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21564fc7-b4kkc\" (UID: \"56b539c3-1509-49a0-9a52-009dc1c0f3f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21564fc7-b4kkc" Nov 03 13:23:53.923853 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:53.923829 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56b539c3-1509-49a0-9a52-009dc1c0f3f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21564fc7-b4kkc\" (UID: \"56b539c3-1509-49a0-9a52-009dc1c0f3f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21564fc7-b4kkc" Nov 03 13:23:53.975651 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:53.975619 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4psdh\" (UniqueName: \"kubernetes.io/projected/56b539c3-1509-49a0-9a52-009dc1c0f3f7-kube-api-access-4psdh\") pod \"maas-api-revoke-sub-21564fc7-b4kkc\" (UID: \"56b539c3-1509-49a0-9a52-009dc1c0f3f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21564fc7-b4kkc" Nov 03 13:23:53.989320 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:53.989293 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21564fc7-b4kkc" Nov 03 13:23:54.408759 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:54.408725 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21564fc7-b4kkc"] Nov 03 13:23:54.565995 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:54.565957 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21564fc7-b4kkc" event={"ID":"56b539c3-1509-49a0-9a52-009dc1c0f3f7","Type":"ContainerStarted","Data":"1db94662e956120c97ebba7914d8fd568da2dbab9bfc7f8665a99b8f15020b88"} Nov 03 13:23:55.573991 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:55.573955 5979 generic.go:356] "Generic (PLEG): container finished" podID="56b539c3-1509-49a0-9a52-009dc1c0f3f7" containerID="e24901793e81dd52699a4f7934a6d11fd6d7bdfb051e59bb0ce92aa3533d3b5c" exitCode=0 Nov 03 13:23:55.574431 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:55.574043 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21564fc7-b4kkc" event={"ID":"56b539c3-1509-49a0-9a52-009dc1c0f3f7","Type":"ContainerDied","Data":"e24901793e81dd52699a4f7934a6d11fd6d7bdfb051e59bb0ce92aa3533d3b5c"} Nov 03 13:23:56.715374 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:56.715351 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21564fc7-b4kkc" Nov 03 13:23:56.858715 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:56.858633 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56b539c3-1509-49a0-9a52-009dc1c0f3f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56b539c3-1509-49a0-9a52-009dc1c0f3f7-maas-api-openshift-ca\") pod \"56b539c3-1509-49a0-9a52-009dc1c0f3f7\" (UID: \"56b539c3-1509-49a0-9a52-009dc1c0f3f7\") " Nov 03 13:23:56.858889 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:56.858783 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56b539c3-1509-49a0-9a52-009dc1c0f3f7-kube-api-access-4psdh\" (UniqueName: \"kubernetes.io/projected/56b539c3-1509-49a0-9a52-009dc1c0f3f7-kube-api-access-4psdh\") pod \"56b539c3-1509-49a0-9a52-009dc1c0f3f7\" (UID: \"56b539c3-1509-49a0-9a52-009dc1c0f3f7\") " Nov 03 13:23:56.858889 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:56.858868 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56b539c3-1509-49a0-9a52-009dc1c0f3f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56b539c3-1509-49a0-9a52-009dc1c0f3f7-maas-api-xks-ca\") pod \"56b539c3-1509-49a0-9a52-009dc1c0f3f7\" (UID: \"56b539c3-1509-49a0-9a52-009dc1c0f3f7\") " Nov 03 13:23:56.859065 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:56.859040 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56b539c3-1509-49a0-9a52-009dc1c0f3f7-maas-api-openshift-ca" pod "56b539c3-1509-49a0-9a52-009dc1c0f3f7" (UID: "56b539c3-1509-49a0-9a52-009dc1c0f3f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:23:56.859377 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:56.859348 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56b539c3-1509-49a0-9a52-009dc1c0f3f7-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:23:56.860985 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:56.860955 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56b539c3-1509-49a0-9a52-009dc1c0f3f7-maas-api-xks-ca" pod "56b539c3-1509-49a0-9a52-009dc1c0f3f7" (UID: "56b539c3-1509-49a0-9a52-009dc1c0f3f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:23:56.861106 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:56.861039 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56b539c3-1509-49a0-9a52-009dc1c0f3f7-kube-api-access-4psdh" pod "56b539c3-1509-49a0-9a52-009dc1c0f3f7" (UID: "56b539c3-1509-49a0-9a52-009dc1c0f3f7"). InnerVolumeSpecName "kube-api-access-4psdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:23:56.960367 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:56.960337 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4psdh\" (UniqueName: \"kubernetes.io/projected/56b539c3-1509-49a0-9a52-009dc1c0f3f7-kube-api-access-4psdh\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:23:56.960367 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:56.960366 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56b539c3-1509-49a0-9a52-009dc1c0f3f7-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:23:57.586621 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:57.586584 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21564fc7-b4kkc" Nov 03 13:23:57.586621 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:57.586612 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21564fc7-b4kkc" event={"ID":"56b539c3-1509-49a0-9a52-009dc1c0f3f7","Type":"ContainerDied","Data":"1db94662e956120c97ebba7914d8fd568da2dbab9bfc7f8665a99b8f15020b88"} Nov 03 13:23:57.586837 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:23:57.586661 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1db94662e956120c97ebba7914d8fd568da2dbab9bfc7f8665a99b8f15020b88" Nov 03 13:24:00.113320 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:00.113267 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-84b78b6f97-6vs8s" podUID="8ec671a2-5c40-4140-839a-8399ec4a5b92" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:24:07.234286 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:24:07.234245 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:24:08.112905 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:08.112869 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-84b78b6f97-6vs8s" podUID="8ec671a2-5c40-4140-839a-8399ec4a5b92" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:24:09.970029 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:09.969993 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a0494c9-pvzls"] Nov 03 13:24:09.970714 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:09.970690 5979 cpu_manager.go:414] "containerMap: removing container" podUID="56b539c3-1509-49a0-9a52-009dc1c0f3f7" containerName="revoke-keys" Nov 03 13:24:09.970811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:09.970715 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56b539c3-1509-49a0-9a52-009dc1c0f3f7" containerName="revoke-keys" Nov 03 13:24:09.970912 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:09.970898 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="56b539c3-1509-49a0-9a52-009dc1c0f3f7" containerName="revoke-keys" Nov 03 13:24:09.974285 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:09.974266 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a0494c9-pvzls" Nov 03 13:24:10.024079 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:10.024043 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0604ce99-cf1e-4a2c-a1f4-b6363a0fa865-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a0494c9-pvzls\" (UID: \"0604ce99-cf1e-4a2c-a1f4-b6363a0fa865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a0494c9-pvzls" Nov 03 13:24:10.024079 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:10.024087 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0604ce99-cf1e-4a2c-a1f4-b6363a0fa865-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a0494c9-pvzls\" (UID: \"0604ce99-cf1e-4a2c-a1f4-b6363a0fa865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a0494c9-pvzls" Nov 03 13:24:10.024301 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:10.024279 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmqvt\" (UniqueName: \"kubernetes.io/projected/0604ce99-cf1e-4a2c-a1f4-b6363a0fa865-kube-api-access-dmqvt\") pod \"maas-api-revoke-sub-8a0494c9-pvzls\" (UID: \"0604ce99-cf1e-4a2c-a1f4-b6363a0fa865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a0494c9-pvzls" Nov 03 13:24:10.029317 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:10.029293 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a0494c9-pvzls"] Nov 03 13:24:10.072902 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:10.072853 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:24:10.072902 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:10.072887 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 13:24:10.073189 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:10.072955 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 13:24:10.073189 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:10.072963 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:24:10.124933 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:10.124902 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmqvt\" (UniqueName: \"kubernetes.io/projected/0604ce99-cf1e-4a2c-a1f4-b6363a0fa865-kube-api-access-dmqvt\") pod \"maas-api-revoke-sub-8a0494c9-pvzls\" (UID: \"0604ce99-cf1e-4a2c-a1f4-b6363a0fa865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a0494c9-pvzls" Nov 03 13:24:10.125116 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:10.124989 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0604ce99-cf1e-4a2c-a1f4-b6363a0fa865-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a0494c9-pvzls\" (UID: \"0604ce99-cf1e-4a2c-a1f4-b6363a0fa865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a0494c9-pvzls" Nov 03 13:24:10.125116 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:10.125012 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0604ce99-cf1e-4a2c-a1f4-b6363a0fa865-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a0494c9-pvzls\" (UID: \"0604ce99-cf1e-4a2c-a1f4-b6363a0fa865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a0494c9-pvzls" Nov 03 13:24:10.125721 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:10.125700 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0604ce99-cf1e-4a2c-a1f4-b6363a0fa865-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a0494c9-pvzls\" (UID: \"0604ce99-cf1e-4a2c-a1f4-b6363a0fa865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a0494c9-pvzls" Nov 03 13:24:10.127301 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:10.127276 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0604ce99-cf1e-4a2c-a1f4-b6363a0fa865-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a0494c9-pvzls\" (UID: \"0604ce99-cf1e-4a2c-a1f4-b6363a0fa865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a0494c9-pvzls" Nov 03 13:24:10.178680 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:10.178651 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmqvt\" (UniqueName: \"kubernetes.io/projected/0604ce99-cf1e-4a2c-a1f4-b6363a0fa865-kube-api-access-dmqvt\") pod \"maas-api-revoke-sub-8a0494c9-pvzls\" (UID: \"0604ce99-cf1e-4a2c-a1f4-b6363a0fa865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a0494c9-pvzls" Nov 03 13:24:10.287774 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:10.287686 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a0494c9-pvzls" Nov 03 13:24:10.508941 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:10.508907 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8841c477-xzd8t"] Nov 03 13:24:10.513108 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:10.513088 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8841c477-xzd8t" Nov 03 13:24:10.530321 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:10.530296 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/353ec922-1cbb-465a-aefa-60803be0959e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8841c477-xzd8t\" (UID: \"353ec922-1cbb-465a-aefa-60803be0959e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8841c477-xzd8t" Nov 03 13:24:10.530441 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:10.530425 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlrg4\" (UniqueName: \"kubernetes.io/projected/353ec922-1cbb-465a-aefa-60803be0959e-kube-api-access-zlrg4\") pod \"maas-api-revoke-sub-8841c477-xzd8t\" (UID: \"353ec922-1cbb-465a-aefa-60803be0959e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8841c477-xzd8t" Nov 03 13:24:10.530563 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:10.530545 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/353ec922-1cbb-465a-aefa-60803be0959e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8841c477-xzd8t\" (UID: \"353ec922-1cbb-465a-aefa-60803be0959e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8841c477-xzd8t" Nov 03 13:24:10.579636 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:10.579464 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8841c477-xzd8t"] Nov 03 13:24:10.631819 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:10.631785 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/353ec922-1cbb-465a-aefa-60803be0959e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8841c477-xzd8t\" (UID: \"353ec922-1cbb-465a-aefa-60803be0959e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8841c477-xzd8t" Nov 03 13:24:10.631991 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:10.631871 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlrg4\" (UniqueName: \"kubernetes.io/projected/353ec922-1cbb-465a-aefa-60803be0959e-kube-api-access-zlrg4\") pod \"maas-api-revoke-sub-8841c477-xzd8t\" (UID: \"353ec922-1cbb-465a-aefa-60803be0959e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8841c477-xzd8t" Nov 03 13:24:10.631991 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:10.631934 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/353ec922-1cbb-465a-aefa-60803be0959e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8841c477-xzd8t\" (UID: \"353ec922-1cbb-465a-aefa-60803be0959e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8841c477-xzd8t" Nov 03 13:24:10.632425 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:10.632402 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/353ec922-1cbb-465a-aefa-60803be0959e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8841c477-xzd8t\" (UID: \"353ec922-1cbb-465a-aefa-60803be0959e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8841c477-xzd8t" Nov 03 13:24:10.634173 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:10.634153 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/353ec922-1cbb-465a-aefa-60803be0959e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8841c477-xzd8t\" (UID: \"353ec922-1cbb-465a-aefa-60803be0959e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8841c477-xzd8t" Nov 03 13:24:10.683961 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:10.683932 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a0494c9-pvzls"] Nov 03 13:24:10.686316 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:10.686290 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlrg4\" (UniqueName: \"kubernetes.io/projected/353ec922-1cbb-465a-aefa-60803be0959e-kube-api-access-zlrg4\") pod \"maas-api-revoke-sub-8841c477-xzd8t\" (UID: \"353ec922-1cbb-465a-aefa-60803be0959e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8841c477-xzd8t" Nov 03 13:24:10.691937 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:24:10.691909 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0604ce99_cf1e_4a2c_a1f4_b6363a0fa865.slice/crio-b099634028cc0bcf9ad54380d7b8873767cf84103a5f561bf293d579f4ccd593 WatchSource:0}: Error finding container b099634028cc0bcf9ad54380d7b8873767cf84103a5f561bf293d579f4ccd593: Status 404 returned error can't find the container with id b099634028cc0bcf9ad54380d7b8873767cf84103a5f561bf293d579f4ccd593 Nov 03 13:24:10.827376 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:10.827342 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8841c477-xzd8t" Nov 03 13:24:11.200181 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:11.200143 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8841c477-xzd8t"] Nov 03 13:24:11.207080 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:24:11.207055 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod353ec922_1cbb_465a_aefa_60803be0959e.slice/crio-c2a5996a911e3dbc67934a3e75dce2a882767b8111a641fc5c87385a20c1b378 WatchSource:0}: Error finding container c2a5996a911e3dbc67934a3e75dce2a882767b8111a641fc5c87385a20c1b378: Status 404 returned error can't find the container with id c2a5996a911e3dbc67934a3e75dce2a882767b8111a641fc5c87385a20c1b378 Nov 03 13:24:11.676558 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:11.676524 5979 generic.go:356] "Generic (PLEG): container finished" podID="0604ce99-cf1e-4a2c-a1f4-b6363a0fa865" containerID="f624f368d24560733813bc4f4a5e7ff2345915d65c55b32448ccba40df88a211" exitCode=0 Nov 03 13:24:11.676730 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:11.676602 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a0494c9-pvzls" event={"ID":"0604ce99-cf1e-4a2c-a1f4-b6363a0fa865","Type":"ContainerDied","Data":"f624f368d24560733813bc4f4a5e7ff2345915d65c55b32448ccba40df88a211"} Nov 03 13:24:11.676730 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:11.676655 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a0494c9-pvzls" event={"ID":"0604ce99-cf1e-4a2c-a1f4-b6363a0fa865","Type":"ContainerStarted","Data":"b099634028cc0bcf9ad54380d7b8873767cf84103a5f561bf293d579f4ccd593"} Nov 03 13:24:11.678240 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:11.678211 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8841c477-xzd8t" event={"ID":"353ec922-1cbb-465a-aefa-60803be0959e","Type":"ContainerStarted","Data":"88cf2463e3b603591a6c0d94d849071003887187674cee56a829f61fbb2ac24d"} Nov 03 13:24:11.678373 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:11.678255 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8841c477-xzd8t" event={"ID":"353ec922-1cbb-465a-aefa-60803be0959e","Type":"ContainerStarted","Data":"c2a5996a911e3dbc67934a3e75dce2a882767b8111a641fc5c87385a20c1b378"} Nov 03 13:24:11.894074 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:11.894024 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8841c477-xzd8t" podStartSLOduration=1.89401014 podStartE2EDuration="1.89401014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:24:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:24:11.891737163 +0000 UTC m=+1133.902425395" watchObservedRunningTime="2026-11-03 13:24:11.89401014 +0000 UTC m=+1133.904698381" Nov 03 13:24:12.209533 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:12.209420 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-42c707/e2e-conflict-a-f2a72750-kserve-5d75f8cf5c-mjjqx"] Nov 03 13:24:12.686024 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:12.685994 5979 generic.go:356] "Generic (PLEG): container finished" podID="353ec922-1cbb-465a-aefa-60803be0959e" containerID="88cf2463e3b603591a6c0d94d849071003887187674cee56a829f61fbb2ac24d" exitCode=0 Nov 03 13:24:12.686203 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:12.686049 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8841c477-xzd8t" event={"ID":"353ec922-1cbb-465a-aefa-60803be0959e","Type":"ContainerDied","Data":"88cf2463e3b603591a6c0d94d849071003887187674cee56a829f61fbb2ac24d"} Nov 03 13:24:12.840773 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:12.840748 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a0494c9-pvzls" Nov 03 13:24:12.976599 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:12.976519 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0604ce99-cf1e-4a2c-a1f4-b6363a0fa865-kube-api-access-dmqvt\" (UniqueName: \"kubernetes.io/projected/0604ce99-cf1e-4a2c-a1f4-b6363a0fa865-kube-api-access-dmqvt\") pod \"0604ce99-cf1e-4a2c-a1f4-b6363a0fa865\" (UID: \"0604ce99-cf1e-4a2c-a1f4-b6363a0fa865\") " Nov 03 13:24:12.976599 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:12.976577 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0604ce99-cf1e-4a2c-a1f4-b6363a0fa865-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0604ce99-cf1e-4a2c-a1f4-b6363a0fa865-maas-api-xks-ca\") pod \"0604ce99-cf1e-4a2c-a1f4-b6363a0fa865\" (UID: \"0604ce99-cf1e-4a2c-a1f4-b6363a0fa865\") " Nov 03 13:24:12.976829 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:12.976809 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0604ce99-cf1e-4a2c-a1f4-b6363a0fa865-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0604ce99-cf1e-4a2c-a1f4-b6363a0fa865-maas-api-openshift-ca\") pod \"0604ce99-cf1e-4a2c-a1f4-b6363a0fa865\" (UID: \"0604ce99-cf1e-4a2c-a1f4-b6363a0fa865\") " Nov 03 13:24:12.977236 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:12.977196 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0604ce99-cf1e-4a2c-a1f4-b6363a0fa865-maas-api-openshift-ca" pod "0604ce99-cf1e-4a2c-a1f4-b6363a0fa865" (UID: "0604ce99-cf1e-4a2c-a1f4-b6363a0fa865"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:24:12.978713 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:12.978687 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0604ce99-cf1e-4a2c-a1f4-b6363a0fa865-kube-api-access-dmqvt" pod "0604ce99-cf1e-4a2c-a1f4-b6363a0fa865" (UID: "0604ce99-cf1e-4a2c-a1f4-b6363a0fa865"). InnerVolumeSpecName "kube-api-access-dmqvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:24:12.979109 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:12.979095 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0604ce99-cf1e-4a2c-a1f4-b6363a0fa865-maas-api-xks-ca" pod "0604ce99-cf1e-4a2c-a1f4-b6363a0fa865" (UID: "0604ce99-cf1e-4a2c-a1f4-b6363a0fa865"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:24:13.078209 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:13.078173 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0604ce99-cf1e-4a2c-a1f4-b6363a0fa865-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:24:13.078209 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:13.078204 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmqvt\" (UniqueName: \"kubernetes.io/projected/0604ce99-cf1e-4a2c-a1f4-b6363a0fa865-kube-api-access-dmqvt\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:24:13.078209 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:13.078217 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0604ce99-cf1e-4a2c-a1f4-b6363a0fa865-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:24:13.693950 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:13.693913 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a0494c9-pvzls" Nov 03 13:24:13.693950 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:13.693928 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a0494c9-pvzls" event={"ID":"0604ce99-cf1e-4a2c-a1f4-b6363a0fa865","Type":"ContainerDied","Data":"b099634028cc0bcf9ad54380d7b8873767cf84103a5f561bf293d579f4ccd593"} Nov 03 13:24:13.694439 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:13.693984 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b099634028cc0bcf9ad54380d7b8873767cf84103a5f561bf293d579f4ccd593" Nov 03 13:24:13.838006 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:13.837982 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8841c477-xzd8t" Nov 03 13:24:13.887724 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:13.887688 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/353ec922-1cbb-465a-aefa-60803be0959e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/353ec922-1cbb-465a-aefa-60803be0959e-maas-api-xks-ca\") pod \"353ec922-1cbb-465a-aefa-60803be0959e\" (UID: \"353ec922-1cbb-465a-aefa-60803be0959e\") " Nov 03 13:24:13.887923 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:13.887777 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/353ec922-1cbb-465a-aefa-60803be0959e-kube-api-access-zlrg4\" (UniqueName: \"kubernetes.io/projected/353ec922-1cbb-465a-aefa-60803be0959e-kube-api-access-zlrg4\") pod \"353ec922-1cbb-465a-aefa-60803be0959e\" (UID: \"353ec922-1cbb-465a-aefa-60803be0959e\") " Nov 03 13:24:13.887998 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:13.887941 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/353ec922-1cbb-465a-aefa-60803be0959e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/353ec922-1cbb-465a-aefa-60803be0959e-maas-api-openshift-ca\") pod \"353ec922-1cbb-465a-aefa-60803be0959e\" (UID: \"353ec922-1cbb-465a-aefa-60803be0959e\") " Nov 03 13:24:13.888316 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:13.888288 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/353ec922-1cbb-465a-aefa-60803be0959e-maas-api-openshift-ca" pod "353ec922-1cbb-465a-aefa-60803be0959e" (UID: "353ec922-1cbb-465a-aefa-60803be0959e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:24:13.889873 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:13.889851 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/353ec922-1cbb-465a-aefa-60803be0959e-kube-api-access-zlrg4" pod "353ec922-1cbb-465a-aefa-60803be0959e" (UID: "353ec922-1cbb-465a-aefa-60803be0959e"). InnerVolumeSpecName "kube-api-access-zlrg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:24:13.889986 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:13.889934 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/353ec922-1cbb-465a-aefa-60803be0959e-maas-api-xks-ca" pod "353ec922-1cbb-465a-aefa-60803be0959e" (UID: "353ec922-1cbb-465a-aefa-60803be0959e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:24:13.989256 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:13.989183 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/353ec922-1cbb-465a-aefa-60803be0959e-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:24:13.989256 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:13.989212 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/353ec922-1cbb-465a-aefa-60803be0959e-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:24:13.989256 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:13.989225 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlrg4\" (UniqueName: \"kubernetes.io/projected/353ec922-1cbb-465a-aefa-60803be0959e-kube-api-access-zlrg4\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:24:14.716341 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:14.716311 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8841c477-xzd8t" Nov 03 13:24:14.716961 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:14.716340 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8841c477-xzd8t" event={"ID":"353ec922-1cbb-465a-aefa-60803be0959e","Type":"ContainerDied","Data":"c2a5996a911e3dbc67934a3e75dce2a882767b8111a641fc5c87385a20c1b378"} Nov 03 13:24:14.716961 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:14.716388 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2a5996a911e3dbc67934a3e75dce2a882767b8111a641fc5c87385a20c1b378" Nov 03 13:24:18.167802 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:18.166605 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-84b78b6f97-6vs8s" podUID="8ec671a2-5c40-4140-839a-8399ec4a5b92" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:24:18.536073 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:18.535981 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:24:20.228673 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:20.228635 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh"] Nov 03 13:24:20.229450 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:20.229420 5979 cpu_manager.go:414] "containerMap: removing container" podUID="353ec922-1cbb-465a-aefa-60803be0959e" containerName="revoke-keys" Nov 03 13:24:20.229450 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:20.229447 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="353ec922-1cbb-465a-aefa-60803be0959e" containerName="revoke-keys" Nov 03 13:24:20.229677 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:20.229485 5979 cpu_manager.go:414] "containerMap: removing container" podUID="0604ce99-cf1e-4a2c-a1f4-b6363a0fa865" containerName="revoke-keys" Nov 03 13:24:20.229677 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:20.229495 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0604ce99-cf1e-4a2c-a1f4-b6363a0fa865" containerName="revoke-keys" Nov 03 13:24:20.229677 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:20.229621 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="353ec922-1cbb-465a-aefa-60803be0959e" containerName="revoke-keys" Nov 03 13:24:20.229677 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:20.229640 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="0604ce99-cf1e-4a2c-a1f4-b6363a0fa865" containerName="revoke-keys" Nov 03 13:24:20.233274 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:20.233252 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh" Nov 03 13:24:20.332246 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:20.332214 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh"] Nov 03 13:24:20.433316 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:20.433275 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/821c73f7-cf2f-414c-8aea-1aff84e017af-credential-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh\" (UID: \"821c73f7-cf2f-414c-8aea-1aff84e017af\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh" Nov 03 13:24:20.433613 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:20.433587 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/821c73f7-cf2f-414c-8aea-1aff84e017af-istio-token\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh\" (UID: \"821c73f7-cf2f-414c-8aea-1aff84e017af\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh" Nov 03 13:24:20.433768 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:20.433752 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/821c73f7-cf2f-414c-8aea-1aff84e017af-workload-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh\" (UID: \"821c73f7-cf2f-414c-8aea-1aff84e017af\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh" Nov 03 13:24:20.433904 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:20.433886 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/821c73f7-cf2f-414c-8aea-1aff84e017af-istio-data\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh\" (UID: \"821c73f7-cf2f-414c-8aea-1aff84e017af\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh" Nov 03 13:24:20.434034 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:20.434018 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/821c73f7-cf2f-414c-8aea-1aff84e017af-istio-envoy\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh\" (UID: \"821c73f7-cf2f-414c-8aea-1aff84e017af\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh" Nov 03 13:24:20.434159 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:20.434144 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/821c73f7-cf2f-414c-8aea-1aff84e017af-istio-podinfo\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh\" (UID: \"821c73f7-cf2f-414c-8aea-1aff84e017af\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh" Nov 03 13:24:20.434267 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:20.434254 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/821c73f7-cf2f-414c-8aea-1aff84e017af-workload-certs\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh\" (UID: \"821c73f7-cf2f-414c-8aea-1aff84e017af\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh" Nov 03 13:24:20.434526 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:20.434504 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/821c73f7-cf2f-414c-8aea-1aff84e017af-istiod-ca-cert\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh\" (UID: \"821c73f7-cf2f-414c-8aea-1aff84e017af\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh" Nov 03 13:24:20.434662 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:20.434645 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm5hk\" (UniqueName: \"kubernetes.io/projected/821c73f7-cf2f-414c-8aea-1aff84e017af-kube-api-access-cm5hk\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh\" (UID: \"821c73f7-cf2f-414c-8aea-1aff84e017af\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh" Nov 03 13:24:20.536410 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:20.536331 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/821c73f7-cf2f-414c-8aea-1aff84e017af-credential-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh\" (UID: \"821c73f7-cf2f-414c-8aea-1aff84e017af\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh" Nov 03 13:24:20.536410 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:20.536372 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/821c73f7-cf2f-414c-8aea-1aff84e017af-istio-token\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh\" (UID: \"821c73f7-cf2f-414c-8aea-1aff84e017af\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh" Nov 03 13:24:20.536410 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:20.536397 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/821c73f7-cf2f-414c-8aea-1aff84e017af-workload-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh\" (UID: \"821c73f7-cf2f-414c-8aea-1aff84e017af\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh" Nov 03 13:24:20.536644 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:20.536431 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/821c73f7-cf2f-414c-8aea-1aff84e017af-istio-data\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh\" (UID: \"821c73f7-cf2f-414c-8aea-1aff84e017af\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh" Nov 03 13:24:20.536644 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:20.536460 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/821c73f7-cf2f-414c-8aea-1aff84e017af-istio-envoy\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh\" (UID: \"821c73f7-cf2f-414c-8aea-1aff84e017af\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh" Nov 03 13:24:20.536644 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:20.536521 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/821c73f7-cf2f-414c-8aea-1aff84e017af-istio-podinfo\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh\" (UID: \"821c73f7-cf2f-414c-8aea-1aff84e017af\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh" Nov 03 13:24:20.536644 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:20.536543 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/821c73f7-cf2f-414c-8aea-1aff84e017af-workload-certs\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh\" (UID: \"821c73f7-cf2f-414c-8aea-1aff84e017af\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh" Nov 03 13:24:20.536644 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:20.536623 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/821c73f7-cf2f-414c-8aea-1aff84e017af-istiod-ca-cert\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh\" (UID: \"821c73f7-cf2f-414c-8aea-1aff84e017af\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh" Nov 03 13:24:20.536803 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:20.536652 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm5hk\" (UniqueName: \"kubernetes.io/projected/821c73f7-cf2f-414c-8aea-1aff84e017af-kube-api-access-cm5hk\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh\" (UID: \"821c73f7-cf2f-414c-8aea-1aff84e017af\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh" Nov 03 13:24:20.537508 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:20.537385 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/821c73f7-cf2f-414c-8aea-1aff84e017af-istio-data\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh\" (UID: \"821c73f7-cf2f-414c-8aea-1aff84e017af\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh" Nov 03 13:24:20.537710 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:20.537638 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/821c73f7-cf2f-414c-8aea-1aff84e017af-credential-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh\" (UID: \"821c73f7-cf2f-414c-8aea-1aff84e017af\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh" Nov 03 13:24:20.537710 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:20.537441 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/821c73f7-cf2f-414c-8aea-1aff84e017af-workload-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh\" (UID: \"821c73f7-cf2f-414c-8aea-1aff84e017af\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh" Nov 03 13:24:20.537710 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:20.537419 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/821c73f7-cf2f-414c-8aea-1aff84e017af-workload-certs\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh\" (UID: \"821c73f7-cf2f-414c-8aea-1aff84e017af\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh" Nov 03 13:24:20.538049 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:20.538022 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/821c73f7-cf2f-414c-8aea-1aff84e017af-istiod-ca-cert\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh\" (UID: \"821c73f7-cf2f-414c-8aea-1aff84e017af\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh" Nov 03 13:24:20.540679 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:20.540651 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/821c73f7-cf2f-414c-8aea-1aff84e017af-istio-envoy\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh\" (UID: \"821c73f7-cf2f-414c-8aea-1aff84e017af\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh" Nov 03 13:24:20.540879 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:20.540863 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/821c73f7-cf2f-414c-8aea-1aff84e017af-istio-podinfo\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh\" (UID: \"821c73f7-cf2f-414c-8aea-1aff84e017af\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh" Nov 03 13:24:20.597277 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:20.597196 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm5hk\" (UniqueName: \"kubernetes.io/projected/821c73f7-cf2f-414c-8aea-1aff84e017af-kube-api-access-cm5hk\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh\" (UID: \"821c73f7-cf2f-414c-8aea-1aff84e017af\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh" Nov 03 13:24:20.613833 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:20.613764 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/821c73f7-cf2f-414c-8aea-1aff84e017af-istio-token\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh\" (UID: \"821c73f7-cf2f-414c-8aea-1aff84e017af\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh" Nov 03 13:24:20.866282 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:20.866200 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh" Nov 03 13:24:21.289791 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:21.289755 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e321db9b-gthpp"] Nov 03 13:24:21.294847 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:21.294826 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e321db9b-gthpp" Nov 03 13:24:21.301099 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:21.301075 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh"] Nov 03 13:24:21.307924 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:24:21.307893 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod821c73f7_cf2f_414c_8aea_1aff84e017af.slice/crio-85f6599716f9fbac7b9f2e78dbaf7cd6091655d2588f8a0b9b4da3f5d926413b WatchSource:0}: Error finding container 85f6599716f9fbac7b9f2e78dbaf7cd6091655d2588f8a0b9b4da3f5d926413b: Status 404 returned error can't find the container with id 85f6599716f9fbac7b9f2e78dbaf7cd6091655d2588f8a0b9b4da3f5d926413b Nov 03 13:24:21.310204 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:21.310177 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:24:21.310271 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:21.310235 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:24:21.310271 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:21.310261 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:24:21.346795 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:21.346763 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsxj7\" (UniqueName: \"kubernetes.io/projected/4f32e29d-157b-4baf-a2bf-01fe1102fac5-kube-api-access-wsxj7\") pod \"maas-api-revoke-sub-e321db9b-gthpp\" (UID: \"4f32e29d-157b-4baf-a2bf-01fe1102fac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e321db9b-gthpp" Nov 03 13:24:21.346928 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:21.346909 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f32e29d-157b-4baf-a2bf-01fe1102fac5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e321db9b-gthpp\" (UID: \"4f32e29d-157b-4baf-a2bf-01fe1102fac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e321db9b-gthpp" Nov 03 13:24:21.347007 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:21.346976 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f32e29d-157b-4baf-a2bf-01fe1102fac5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e321db9b-gthpp\" (UID: \"4f32e29d-157b-4baf-a2bf-01fe1102fac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e321db9b-gthpp" Nov 03 13:24:21.350734 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:21.350694 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 13:24:21.350734 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:21.350702 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:24:21.350906 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:21.350814 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 13:24:21.350906 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:21.350841 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:24:21.354262 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:21.354240 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e321db9b-gthpp"] Nov 03 13:24:21.449348 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:21.448810 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f32e29d-157b-4baf-a2bf-01fe1102fac5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e321db9b-gthpp\" (UID: \"4f32e29d-157b-4baf-a2bf-01fe1102fac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e321db9b-gthpp" Nov 03 13:24:21.449348 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:21.448868 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f32e29d-157b-4baf-a2bf-01fe1102fac5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e321db9b-gthpp\" (UID: \"4f32e29d-157b-4baf-a2bf-01fe1102fac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e321db9b-gthpp" Nov 03 13:24:21.449348 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:21.449079 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsxj7\" (UniqueName: \"kubernetes.io/projected/4f32e29d-157b-4baf-a2bf-01fe1102fac5-kube-api-access-wsxj7\") pod \"maas-api-revoke-sub-e321db9b-gthpp\" (UID: \"4f32e29d-157b-4baf-a2bf-01fe1102fac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e321db9b-gthpp" Nov 03 13:24:21.449994 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:21.449969 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f32e29d-157b-4baf-a2bf-01fe1102fac5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e321db9b-gthpp\" (UID: \"4f32e29d-157b-4baf-a2bf-01fe1102fac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e321db9b-gthpp" Nov 03 13:24:21.452431 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:21.452412 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f32e29d-157b-4baf-a2bf-01fe1102fac5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e321db9b-gthpp\" (UID: \"4f32e29d-157b-4baf-a2bf-01fe1102fac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e321db9b-gthpp" Nov 03 13:24:21.471063 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:21.471034 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-84b78b6f97-6vs8s"] Nov 03 13:24:21.472386 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:21.472350 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-84b78b6f97-6vs8s" podUID="8ec671a2-5c40-4140-839a-8399ec4a5b92" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:24:21.510347 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:21.510314 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7c48b6b6bf-2prt4"] Nov 03 13:24:21.514576 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:21.514543 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7c48b6b6bf-2prt4" Nov 03 13:24:21.514927 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:21.514906 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsxj7\" (UniqueName: \"kubernetes.io/projected/4f32e29d-157b-4baf-a2bf-01fe1102fac5-kube-api-access-wsxj7\") pod \"maas-api-revoke-sub-e321db9b-gthpp\" (UID: \"4f32e29d-157b-4baf-a2bf-01fe1102fac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e321db9b-gthpp" Nov 03 13:24:21.550769 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:21.550684 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl6gd\" (UniqueName: \"kubernetes.io/projected/b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4-kube-api-access-xl6gd\") pod \"payload-processing-external-model-7c48b6b6bf-2prt4\" (UID: \"b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4\") " pod="models-as-a-service/payload-processing-external-model-7c48b6b6bf-2prt4" Nov 03 13:24:21.550769 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:21.550756 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4-provider-credentials\") pod \"payload-processing-external-model-7c48b6b6bf-2prt4\" (UID: \"b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4\") " pod="models-as-a-service/payload-processing-external-model-7c48b6b6bf-2prt4" Nov 03 13:24:21.550998 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:21.550869 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4-routing-overlay-volume\") pod \"payload-processing-external-model-7c48b6b6bf-2prt4\" (UID: \"b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4\") " pod="models-as-a-service/payload-processing-external-model-7c48b6b6bf-2prt4" Nov 03 13:24:21.550998 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:21.550890 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4-plugins-config-volume\") pod \"payload-processing-external-model-7c48b6b6bf-2prt4\" (UID: \"b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4\") " pod="models-as-a-service/payload-processing-external-model-7c48b6b6bf-2prt4" Nov 03 13:24:21.565540 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:21.565511 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-de9101746298\"" Nov 03 13:24:21.565711 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:21.565579 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-7d775f439052\"" Nov 03 13:24:21.569071 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:21.569044 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7c48b6b6bf-2prt4"] Nov 03 13:24:21.610161 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:21.610132 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e321db9b-gthpp" Nov 03 13:24:21.652460 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:21.652430 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4-routing-overlay-volume\") pod \"payload-processing-external-model-7c48b6b6bf-2prt4\" (UID: \"b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4\") " pod="models-as-a-service/payload-processing-external-model-7c48b6b6bf-2prt4" Nov 03 13:24:21.652460 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:21.652480 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4-plugins-config-volume\") pod \"payload-processing-external-model-7c48b6b6bf-2prt4\" (UID: \"b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4\") " pod="models-as-a-service/payload-processing-external-model-7c48b6b6bf-2prt4" Nov 03 13:24:21.652742 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:21.652651 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl6gd\" (UniqueName: \"kubernetes.io/projected/b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4-kube-api-access-xl6gd\") pod \"payload-processing-external-model-7c48b6b6bf-2prt4\" (UID: \"b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4\") " pod="models-as-a-service/payload-processing-external-model-7c48b6b6bf-2prt4" Nov 03 13:24:21.652742 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:21.652715 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4-provider-credentials\") pod \"payload-processing-external-model-7c48b6b6bf-2prt4\" (UID: \"b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4\") " pod="models-as-a-service/payload-processing-external-model-7c48b6b6bf-2prt4" Nov 03 13:24:21.653203 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:21.653179 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4-routing-overlay-volume\") pod \"payload-processing-external-model-7c48b6b6bf-2prt4\" (UID: \"b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4\") " pod="models-as-a-service/payload-processing-external-model-7c48b6b6bf-2prt4" Nov 03 13:24:21.653500 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:21.653446 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4-plugins-config-volume\") pod \"payload-processing-external-model-7c48b6b6bf-2prt4\" (UID: \"b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4\") " pod="models-as-a-service/payload-processing-external-model-7c48b6b6bf-2prt4" Nov 03 13:24:21.656799 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:21.656772 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4-provider-credentials\") pod \"payload-processing-external-model-7c48b6b6bf-2prt4\" (UID: \"b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4\") " pod="models-as-a-service/payload-processing-external-model-7c48b6b6bf-2prt4" Nov 03 13:24:21.709908 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:21.709876 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl6gd\" (UniqueName: \"kubernetes.io/projected/b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4-kube-api-access-xl6gd\") pod \"payload-processing-external-model-7c48b6b6bf-2prt4\" (UID: \"b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4\") " pod="models-as-a-service/payload-processing-external-model-7c48b6b6bf-2prt4" Nov 03 13:24:21.768763 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:21.768722 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh" event={"ID":"821c73f7-cf2f-414c-8aea-1aff84e017af","Type":"ContainerStarted","Data":"c51d402870f6fc59222f6ef744e7c8e0e2d58ffbab0a0eefc227009683c30dc8"} Nov 03 13:24:21.768902 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:21.768779 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh" event={"ID":"821c73f7-cf2f-414c-8aea-1aff84e017af","Type":"ContainerStarted","Data":"85f6599716f9fbac7b9f2e78dbaf7cd6091655d2588f8a0b9b4da3f5d926413b"} Nov 03 13:24:21.827881 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:21.827789 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7c48b6b6bf-2prt4" Nov 03 13:24:21.867327 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:21.867292 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh" Nov 03 13:24:21.880886 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:21.880806 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh" podStartSLOduration=1.880789609 podStartE2EDuration="1.880789609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:24:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:24:21.878576619 +0000 UTC m=+1143.889264856" watchObservedRunningTime="2026-11-03 13:24:21.880789609 +0000 UTC m=+1143.891477850" Nov 03 13:24:21.989497 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:21.989439 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e321db9b-gthpp"] Nov 03 13:24:22.002897 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:24:22.002862 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f32e29d_157b_4baf_a2bf_01fe1102fac5.slice/crio-50bb336657a2c5cd375b34f8725b1399cdda541d0a7f4184de9cc4ccf4a2c58c WatchSource:0}: Error finding container 50bb336657a2c5cd375b34f8725b1399cdda541d0a7f4184de9cc4ccf4a2c58c: Status 404 returned error can't find the container with id 50bb336657a2c5cd375b34f8725b1399cdda541d0a7f4184de9cc4ccf4a2c58c Nov 03 13:24:22.209756 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:22.209724 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7c48b6b6bf-2prt4"] Nov 03 13:24:22.216358 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:24:22.216326 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3d318e3_0c74_4f44_ba58_6cd1bd0f39f4.slice/crio-ddc09c507e0a3bca87a56bafe3bf6aa9006520aeae96a5f1db4f501a74f60f1d WatchSource:0}: Error finding container ddc09c507e0a3bca87a56bafe3bf6aa9006520aeae96a5f1db4f501a74f60f1d: Status 404 returned error can't find the container with id ddc09c507e0a3bca87a56bafe3bf6aa9006520aeae96a5f1db4f501a74f60f1d Nov 03 13:24:22.776982 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:22.776880 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e321db9b-gthpp" event={"ID":"4f32e29d-157b-4baf-a2bf-01fe1102fac5","Type":"ContainerStarted","Data":"35e639b60d996109594af2e74b5f956fe01570a669ee4bdfdb87b103dbe24d73"} Nov 03 13:24:22.776982 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:22.776944 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e321db9b-gthpp" event={"ID":"4f32e29d-157b-4baf-a2bf-01fe1102fac5","Type":"ContainerStarted","Data":"50bb336657a2c5cd375b34f8725b1399cdda541d0a7f4184de9cc4ccf4a2c58c"} Nov 03 13:24:22.778613 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:22.778584 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7c48b6b6bf-2prt4" event={"ID":"b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4","Type":"ContainerStarted","Data":"c84d1214c12d75e9e3aca0b2d37d5c2251a60225a386c6af6d862d411907bbef"} Nov 03 13:24:22.778744 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:22.778626 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7c48b6b6bf-2prt4" event={"ID":"b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4","Type":"ContainerStarted","Data":"ddc09c507e0a3bca87a56bafe3bf6aa9006520aeae96a5f1db4f501a74f60f1d"} Nov 03 13:24:22.778794 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:22.778763 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7c48b6b6bf-2prt4" Nov 03 13:24:22.870097 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:22.870052 5979 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 13:24:22.870280 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:22.870119 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh" podUID="821c73f7-cf2f-414c-8aea-1aff84e017af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 13:24:22.889347 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:22.889286 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e321db9b-gthpp" podStartSLOduration=1.889272753 podStartE2EDuration="1.889272753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:24:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:24:22.886111946 +0000 UTC m=+1144.896800188" watchObservedRunningTime="2026-11-03 13:24:22.889272753 +0000 UTC m=+1144.899960995" Nov 03 13:24:22.997047 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:22.996995 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7c48b6b6bf-2prt4" podStartSLOduration=1.996981816 podStartE2EDuration="1.996981816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:24:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:24:22.994319867 +0000 UTC m=+1145.005008110" watchObservedRunningTime="2026-11-03 13:24:22.996981816 +0000 UTC m=+1145.007670068" Nov 03 13:24:23.786276 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:23.786240 5979 generic.go:356] "Generic (PLEG): container finished" podID="4f32e29d-157b-4baf-a2bf-01fe1102fac5" containerID="35e639b60d996109594af2e74b5f956fe01570a669ee4bdfdb87b103dbe24d73" exitCode=0 Nov 03 13:24:23.786715 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:23.786330 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e321db9b-gthpp" event={"ID":"4f32e29d-157b-4baf-a2bf-01fe1102fac5","Type":"ContainerDied","Data":"35e639b60d996109594af2e74b5f956fe01570a669ee4bdfdb87b103dbe24d73"} Nov 03 13:24:23.868925 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:23.868886 5979 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 13:24:23.869080 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:23.868944 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh" podUID="821c73f7-cf2f-414c-8aea-1aff84e017af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 13:24:24.867066 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:24.867031 5979 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 13:24:24.869609 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:24.869567 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh" podUID="821c73f7-cf2f-414c-8aea-1aff84e017af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 13:24:24.988953 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:24.988922 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e321db9b-gthpp" Nov 03 13:24:25.108718 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:25.108627 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f32e29d-157b-4baf-a2bf-01fe1102fac5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f32e29d-157b-4baf-a2bf-01fe1102fac5-maas-api-xks-ca\") pod \"4f32e29d-157b-4baf-a2bf-01fe1102fac5\" (UID: \"4f32e29d-157b-4baf-a2bf-01fe1102fac5\") " Nov 03 13:24:25.108718 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:25.108685 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f32e29d-157b-4baf-a2bf-01fe1102fac5-kube-api-access-wsxj7\" (UniqueName: \"kubernetes.io/projected/4f32e29d-157b-4baf-a2bf-01fe1102fac5-kube-api-access-wsxj7\") pod \"4f32e29d-157b-4baf-a2bf-01fe1102fac5\" (UID: \"4f32e29d-157b-4baf-a2bf-01fe1102fac5\") " Nov 03 13:24:25.108995 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:25.108818 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f32e29d-157b-4baf-a2bf-01fe1102fac5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f32e29d-157b-4baf-a2bf-01fe1102fac5-maas-api-openshift-ca\") pod \"4f32e29d-157b-4baf-a2bf-01fe1102fac5\" (UID: \"4f32e29d-157b-4baf-a2bf-01fe1102fac5\") " Nov 03 13:24:25.109219 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:25.109191 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f32e29d-157b-4baf-a2bf-01fe1102fac5-maas-api-openshift-ca" pod "4f32e29d-157b-4baf-a2bf-01fe1102fac5" (UID: "4f32e29d-157b-4baf-a2bf-01fe1102fac5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:24:25.109412 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:25.109391 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f32e29d-157b-4baf-a2bf-01fe1102fac5-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:24:25.110979 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:25.110951 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f32e29d-157b-4baf-a2bf-01fe1102fac5-maas-api-xks-ca" pod "4f32e29d-157b-4baf-a2bf-01fe1102fac5" (UID: "4f32e29d-157b-4baf-a2bf-01fe1102fac5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:24:25.112026 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:25.111994 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f32e29d-157b-4baf-a2bf-01fe1102fac5-kube-api-access-wsxj7" pod "4f32e29d-157b-4baf-a2bf-01fe1102fac5" (UID: "4f32e29d-157b-4baf-a2bf-01fe1102fac5"). InnerVolumeSpecName "kube-api-access-wsxj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:24:25.211536 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:25.211450 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f32e29d-157b-4baf-a2bf-01fe1102fac5-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:24:25.211536 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:25.211491 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsxj7\" (UniqueName: \"kubernetes.io/projected/4f32e29d-157b-4baf-a2bf-01fe1102fac5-kube-api-access-wsxj7\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:24:25.800298 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:25.800261 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e321db9b-gthpp" Nov 03 13:24:25.800438 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:25.800289 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e321db9b-gthpp" event={"ID":"4f32e29d-157b-4baf-a2bf-01fe1102fac5","Type":"ContainerDied","Data":"50bb336657a2c5cd375b34f8725b1399cdda541d0a7f4184de9cc4ccf4a2c58c"} Nov 03 13:24:25.800438 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:25.800365 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50bb336657a2c5cd375b34f8725b1399cdda541d0a7f4184de9cc4ccf4a2c58c" Nov 03 13:24:25.868971 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:25.868936 5979 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 13:24:25.869420 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:25.869011 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh" podUID="821c73f7-cf2f-414c-8aea-1aff84e017af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 13:24:26.471715 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:26.471671 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-84b78b6f97-6vs8s" podUID="8ec671a2-5c40-4140-839a-8399ec4a5b92" containerName="payload-processing" containerID="cri-o://8bb6fe1938ee907cf7b68abbe9fec9ed4f83670230ab1434f834a0694fb45a79" gracePeriod=25 Nov 03 13:24:26.795937 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:26.795910 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-84b78b6f97-6vs8s" Nov 03 13:24:26.808211 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:26.808178 5979 generic.go:356] "Generic (PLEG): container finished" podID="8ec671a2-5c40-4140-839a-8399ec4a5b92" containerID="8bb6fe1938ee907cf7b68abbe9fec9ed4f83670230ab1434f834a0694fb45a79" exitCode=0 Nov 03 13:24:26.808328 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:26.808228 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-84b78b6f97-6vs8s" Nov 03 13:24:26.808328 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:26.808265 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-84b78b6f97-6vs8s" event={"ID":"8ec671a2-5c40-4140-839a-8399ec4a5b92","Type":"ContainerDied","Data":"8bb6fe1938ee907cf7b68abbe9fec9ed4f83670230ab1434f834a0694fb45a79"} Nov 03 13:24:26.808328 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:26.808322 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-84b78b6f97-6vs8s" event={"ID":"8ec671a2-5c40-4140-839a-8399ec4a5b92","Type":"ContainerDied","Data":"1ce8b0d680921b9799eabf3392a2bb358eef787fb40316d4a40f36ecbe38479f"} Nov 03 13:24:26.808519 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:26.808351 5979 scope.go:122] "RemoveContainer" containerID="8bb6fe1938ee907cf7b68abbe9fec9ed4f83670230ab1434f834a0694fb45a79" Nov 03 13:24:26.817667 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:26.817649 5979 scope.go:122] "RemoveContainer" containerID="8bb6fe1938ee907cf7b68abbe9fec9ed4f83670230ab1434f834a0694fb45a79" Nov 03 13:24:26.817894 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:24:26.817874 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bb6fe1938ee907cf7b68abbe9fec9ed4f83670230ab1434f834a0694fb45a79\": container with ID starting with 8bb6fe1938ee907cf7b68abbe9fec9ed4f83670230ab1434f834a0694fb45a79 not found: ID does not exist" containerID="8bb6fe1938ee907cf7b68abbe9fec9ed4f83670230ab1434f834a0694fb45a79" Nov 03 13:24:26.817954 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:26.817901 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bb6fe1938ee907cf7b68abbe9fec9ed4f83670230ab1434f834a0694fb45a79"} err="failed to get container status \"8bb6fe1938ee907cf7b68abbe9fec9ed4f83670230ab1434f834a0694fb45a79\": rpc error: code = NotFound desc = could not find container \"8bb6fe1938ee907cf7b68abbe9fec9ed4f83670230ab1434f834a0694fb45a79\": container with ID starting with 8bb6fe1938ee907cf7b68abbe9fec9ed4f83670230ab1434f834a0694fb45a79 not found: ID does not exist" Nov 03 13:24:26.832079 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:26.832053 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ec671a2-5c40-4140-839a-8399ec4a5b92-kube-api-access-lh2j5\" (UniqueName: \"kubernetes.io/projected/8ec671a2-5c40-4140-839a-8399ec4a5b92-kube-api-access-lh2j5\") pod \"8ec671a2-5c40-4140-839a-8399ec4a5b92\" (UID: \"8ec671a2-5c40-4140-839a-8399ec4a5b92\") " Nov 03 13:24:26.832185 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:26.832171 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ec671a2-5c40-4140-839a-8399ec4a5b92-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ec671a2-5c40-4140-839a-8399ec4a5b92-plugins-config-volume\") pod \"8ec671a2-5c40-4140-839a-8399ec4a5b92\" (UID: \"8ec671a2-5c40-4140-839a-8399ec4a5b92\") " Nov 03 13:24:26.832228 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:26.832209 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ec671a2-5c40-4140-839a-8399ec4a5b92-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8ec671a2-5c40-4140-839a-8399ec4a5b92-provider-credentials\") pod \"8ec671a2-5c40-4140-839a-8399ec4a5b92\" (UID: \"8ec671a2-5c40-4140-839a-8399ec4a5b92\") " Nov 03 13:24:26.832288 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:26.832276 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ec671a2-5c40-4140-839a-8399ec4a5b92-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8ec671a2-5c40-4140-839a-8399ec4a5b92-routing-overlay-volume\") pod \"8ec671a2-5c40-4140-839a-8399ec4a5b92\" (UID: \"8ec671a2-5c40-4140-839a-8399ec4a5b92\") " Nov 03 13:24:26.832639 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:26.832537 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ec671a2-5c40-4140-839a-8399ec4a5b92-plugins-config-volume" pod "8ec671a2-5c40-4140-839a-8399ec4a5b92" (UID: "8ec671a2-5c40-4140-839a-8399ec4a5b92"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:24:26.832757 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:26.832700 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ec671a2-5c40-4140-839a-8399ec4a5b92-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:24:26.832757 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:26.832695 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ec671a2-5c40-4140-839a-8399ec4a5b92-routing-overlay-volume" pod "8ec671a2-5c40-4140-839a-8399ec4a5b92" (UID: "8ec671a2-5c40-4140-839a-8399ec4a5b92"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:24:26.834123 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:26.834098 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ec671a2-5c40-4140-839a-8399ec4a5b92-kube-api-access-lh2j5" pod "8ec671a2-5c40-4140-839a-8399ec4a5b92" (UID: "8ec671a2-5c40-4140-839a-8399ec4a5b92"). InnerVolumeSpecName "kube-api-access-lh2j5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:24:26.834226 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:26.834206 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ec671a2-5c40-4140-839a-8399ec4a5b92-provider-credentials" pod "8ec671a2-5c40-4140-839a-8399ec4a5b92" (UID: "8ec671a2-5c40-4140-839a-8399ec4a5b92"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:24:26.869137 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:26.869091 5979 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 13:24:26.869630 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:26.869166 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh" podUID="821c73f7-cf2f-414c-8aea-1aff84e017af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 13:24:26.934038 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:26.933991 5979 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8ec671a2-5c40-4140-839a-8399ec4a5b92-provider-credentials\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:24:26.934038 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:26.934041 5979 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8ec671a2-5c40-4140-839a-8399ec4a5b92-routing-overlay-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:24:26.934229 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:26.934062 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh2j5\" (UniqueName: \"kubernetes.io/projected/8ec671a2-5c40-4140-839a-8399ec4a5b92-kube-api-access-lh2j5\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:24:27.224136 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:27.224100 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-84b78b6f97-6vs8s"] Nov 03 13:24:27.228419 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:27.228393 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-84b78b6f97-6vs8s"] Nov 03 13:24:27.230098 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:27.230070 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-a-f2a72750-kserve-5d75f8cf5c-mjjqx" podUID="9cd1a0a5-82e9-481b-a0d2-5d51b21bada1" containerName="main" containerID="cri-o://8de160e677a37781142866f14aeebc2c97f7c87e437e4d59a12cdf54d2f28fa7" gracePeriod=45 Nov 03 13:24:27.518883 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:27.518854 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-a-f2a72750-kserve-5d75f8cf5c-mjjqx" Nov 03 13:24:27.542195 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:27.542165 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cd1a0a5-82e9-481b-a0d2-5d51b21bada1-tls-certs\" (UniqueName: \"kubernetes.io/secret/9cd1a0a5-82e9-481b-a0d2-5d51b21bada1-tls-certs\") pod \"9cd1a0a5-82e9-481b-a0d2-5d51b21bada1\" (UID: \"9cd1a0a5-82e9-481b-a0d2-5d51b21bada1\") " Nov 03 13:24:27.542336 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:27.542236 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cd1a0a5-82e9-481b-a0d2-5d51b21bada1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cd1a0a5-82e9-481b-a0d2-5d51b21bada1-dshm\") pod \"9cd1a0a5-82e9-481b-a0d2-5d51b21bada1\" (UID: \"9cd1a0a5-82e9-481b-a0d2-5d51b21bada1\") " Nov 03 13:24:27.542381 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:27.542345 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cd1a0a5-82e9-481b-a0d2-5d51b21bada1-kube-api-access-xc4c9\" (UniqueName: \"kubernetes.io/projected/9cd1a0a5-82e9-481b-a0d2-5d51b21bada1-kube-api-access-xc4c9\") pod \"9cd1a0a5-82e9-481b-a0d2-5d51b21bada1\" (UID: \"9cd1a0a5-82e9-481b-a0d2-5d51b21bada1\") " Nov 03 13:24:27.542536 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:27.542513 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cd1a0a5-82e9-481b-a0d2-5d51b21bada1-home\" (UniqueName: \"kubernetes.io/empty-dir/9cd1a0a5-82e9-481b-a0d2-5d51b21bada1-home\") pod \"9cd1a0a5-82e9-481b-a0d2-5d51b21bada1\" (UID: \"9cd1a0a5-82e9-481b-a0d2-5d51b21bada1\") " Nov 03 13:24:27.542642 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:27.542553 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cd1a0a5-82e9-481b-a0d2-5d51b21bada1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cd1a0a5-82e9-481b-a0d2-5d51b21bada1-model-cache\") pod \"9cd1a0a5-82e9-481b-a0d2-5d51b21bada1\" (UID: \"9cd1a0a5-82e9-481b-a0d2-5d51b21bada1\") " Nov 03 13:24:27.542642 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:27.542606 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cd1a0a5-82e9-481b-a0d2-5d51b21bada1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cd1a0a5-82e9-481b-a0d2-5d51b21bada1-tmp-dir\") pod \"9cd1a0a5-82e9-481b-a0d2-5d51b21bada1\" (UID: \"9cd1a0a5-82e9-481b-a0d2-5d51b21bada1\") " Nov 03 13:24:27.542781 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:27.542757 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cd1a0a5-82e9-481b-a0d2-5d51b21bada1-home" pod "9cd1a0a5-82e9-481b-a0d2-5d51b21bada1" (UID: "9cd1a0a5-82e9-481b-a0d2-5d51b21bada1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:24:27.543032 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:27.543006 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cd1a0a5-82e9-481b-a0d2-5d51b21bada1-tmp-dir" pod "9cd1a0a5-82e9-481b-a0d2-5d51b21bada1" (UID: "9cd1a0a5-82e9-481b-a0d2-5d51b21bada1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:24:27.543270 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:27.543243 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cd1a0a5-82e9-481b-a0d2-5d51b21bada1-model-cache" pod "9cd1a0a5-82e9-481b-a0d2-5d51b21bada1" (UID: "9cd1a0a5-82e9-481b-a0d2-5d51b21bada1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:24:27.543270 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:27.543249 5979 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9cd1a0a5-82e9-481b-a0d2-5d51b21bada1-home\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:24:27.543440 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:27.543286 5979 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9cd1a0a5-82e9-481b-a0d2-5d51b21bada1-tmp-dir\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:24:27.544699 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:27.544680 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cd1a0a5-82e9-481b-a0d2-5d51b21bada1-kube-api-access-xc4c9" pod "9cd1a0a5-82e9-481b-a0d2-5d51b21bada1" (UID: "9cd1a0a5-82e9-481b-a0d2-5d51b21bada1"). InnerVolumeSpecName "kube-api-access-xc4c9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:24:27.544816 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:27.544801 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cd1a0a5-82e9-481b-a0d2-5d51b21bada1-dshm" pod "9cd1a0a5-82e9-481b-a0d2-5d51b21bada1" (UID: "9cd1a0a5-82e9-481b-a0d2-5d51b21bada1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:24:27.544873 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:27.544820 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cd1a0a5-82e9-481b-a0d2-5d51b21bada1-tls-certs" pod "9cd1a0a5-82e9-481b-a0d2-5d51b21bada1" (UID: "9cd1a0a5-82e9-481b-a0d2-5d51b21bada1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:24:27.644696 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:27.644666 5979 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cd1a0a5-82e9-481b-a0d2-5d51b21bada1-tls-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:24:27.644696 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:27.644694 5979 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9cd1a0a5-82e9-481b-a0d2-5d51b21bada1-dshm\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:24:27.644903 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:27.644710 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xc4c9\" (UniqueName: \"kubernetes.io/projected/9cd1a0a5-82e9-481b-a0d2-5d51b21bada1-kube-api-access-xc4c9\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:24:27.644903 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:27.644723 5979 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9cd1a0a5-82e9-481b-a0d2-5d51b21bada1-model-cache\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:24:27.818423 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:27.818393 5979 generic.go:356] "Generic (PLEG): container finished" podID="9cd1a0a5-82e9-481b-a0d2-5d51b21bada1" containerID="8de160e677a37781142866f14aeebc2c97f7c87e437e4d59a12cdf54d2f28fa7" exitCode=0 Nov 03 13:24:27.818654 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:27.818442 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-a-f2a72750-kserve-5d75f8cf5c-mjjqx" Nov 03 13:24:27.818654 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:27.818490 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-a-f2a72750-kserve-5d75f8cf5c-mjjqx" event={"ID":"9cd1a0a5-82e9-481b-a0d2-5d51b21bada1","Type":"ContainerDied","Data":"8de160e677a37781142866f14aeebc2c97f7c87e437e4d59a12cdf54d2f28fa7"} Nov 03 13:24:27.818654 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:27.818539 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-a-f2a72750-kserve-5d75f8cf5c-mjjqx" event={"ID":"9cd1a0a5-82e9-481b-a0d2-5d51b21bada1","Type":"ContainerDied","Data":"309c980608fe58772c2bfeacef27c8c6a7350438846b808a69271623f653050b"} Nov 03 13:24:27.818654 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:27.818561 5979 scope.go:122] "RemoveContainer" containerID="8de160e677a37781142866f14aeebc2c97f7c87e437e4d59a12cdf54d2f28fa7" Nov 03 13:24:27.830506 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:27.830459 5979 scope.go:122] "RemoveContainer" containerID="8de160e677a37781142866f14aeebc2c97f7c87e437e4d59a12cdf54d2f28fa7" Nov 03 13:24:27.830737 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:24:27.830713 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8de160e677a37781142866f14aeebc2c97f7c87e437e4d59a12cdf54d2f28fa7\": container with ID starting with 8de160e677a37781142866f14aeebc2c97f7c87e437e4d59a12cdf54d2f28fa7 not found: ID does not exist" containerID="8de160e677a37781142866f14aeebc2c97f7c87e437e4d59a12cdf54d2f28fa7" Nov 03 13:24:27.830818 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:27.830747 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8de160e677a37781142866f14aeebc2c97f7c87e437e4d59a12cdf54d2f28fa7"} err="failed to get container status \"8de160e677a37781142866f14aeebc2c97f7c87e437e4d59a12cdf54d2f28fa7\": rpc error: code = NotFound desc = could not find container \"8de160e677a37781142866f14aeebc2c97f7c87e437e4d59a12cdf54d2f28fa7\": container with ID starting with 8de160e677a37781142866f14aeebc2c97f7c87e437e4d59a12cdf54d2f28fa7 not found: ID does not exist" Nov 03 13:24:27.867032 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:27.866998 5979 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 13:24:27.867174 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:27.867057 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh" podUID="821c73f7-cf2f-414c-8aea-1aff84e017af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 13:24:27.973891 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:27.973806 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-42c707/e2e-conflict-a-f2a72750-kserve-5d75f8cf5c-mjjqx"] Nov 03 13:24:27.979045 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:27.979008 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-42c707/e2e-conflict-a-f2a72750-kserve-5d75f8cf5c-mjjqx"] Nov 03 13:24:28.117321 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:28.117287 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ec671a2-5c40-4140-839a-8399ec4a5b92" path="/var/lib/kubelet/pods/8ec671a2-5c40-4140-839a-8399ec4a5b92/volumes" Nov 03 13:24:28.117723 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:28.117708 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cd1a0a5-82e9-481b-a0d2-5d51b21bada1" path="/var/lib/kubelet/pods/9cd1a0a5-82e9-481b-a0d2-5d51b21bada1/volumes" Nov 03 13:24:28.441260 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:28.441220 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42c707/e2e-conflict-a-f2a72750-kserve-5d75f8cf5c-mjjqx" podUID="9cd1a0a5-82e9-481b-a0d2-5d51b21bada1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:24:28.870954 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:28.870886 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh" Nov 03 13:24:28.871197 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:28.871175 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh" Nov 03 13:24:28.871966 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:28.871947 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh" Nov 03 13:24:33.790315 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:33.790286 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-7c48b6b6bf-2prt4" Nov 03 13:24:33.949060 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:33.949022 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7dfbfc6544-jz7cm"] Nov 03 13:24:37.234173 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:24:37.234137 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:24:38.079008 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:38.078970 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1866534f-pcvnf"] Nov 03 13:24:38.079542 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:38.079527 5979 cpu_manager.go:414] "containerMap: removing container" podUID="8ec671a2-5c40-4140-839a-8399ec4a5b92" containerName="payload-processing" Nov 03 13:24:38.079600 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:38.079543 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ec671a2-5c40-4140-839a-8399ec4a5b92" containerName="payload-processing" Nov 03 13:24:38.079600 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:38.079566 5979 cpu_manager.go:414] "containerMap: removing container" podUID="9cd1a0a5-82e9-481b-a0d2-5d51b21bada1" containerName="main" Nov 03 13:24:38.079600 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:38.079570 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cd1a0a5-82e9-481b-a0d2-5d51b21bada1" containerName="main" Nov 03 13:24:38.079600 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:38.079579 5979 cpu_manager.go:414] "containerMap: removing container" podUID="4f32e29d-157b-4baf-a2bf-01fe1102fac5" containerName="revoke-keys" Nov 03 13:24:38.079600 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:38.079584 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f32e29d-157b-4baf-a2bf-01fe1102fac5" containerName="revoke-keys" Nov 03 13:24:38.079754 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:38.079663 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ec671a2-5c40-4140-839a-8399ec4a5b92" containerName="payload-processing" Nov 03 13:24:38.079754 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:38.079673 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cd1a0a5-82e9-481b-a0d2-5d51b21bada1" containerName="main" Nov 03 13:24:38.079754 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:38.079681 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f32e29d-157b-4baf-a2bf-01fe1102fac5" containerName="revoke-keys" Nov 03 13:24:38.082274 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:38.082252 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1866534f-pcvnf" Nov 03 13:24:38.137755 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:38.137721 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1866534f-pcvnf"] Nov 03 13:24:38.177145 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:38.177021 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5z2d\" (UniqueName: \"kubernetes.io/projected/d6ea1f5b-6ba3-4053-a5e5-28554fefccbb-kube-api-access-s5z2d\") pod \"maas-api-revoke-sub-1866534f-pcvnf\" (UID: \"d6ea1f5b-6ba3-4053-a5e5-28554fefccbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1866534f-pcvnf" Nov 03 13:24:38.177554 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:38.177528 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6ea1f5b-6ba3-4053-a5e5-28554fefccbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1866534f-pcvnf\" (UID: \"d6ea1f5b-6ba3-4053-a5e5-28554fefccbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1866534f-pcvnf" Nov 03 13:24:38.177752 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:38.177730 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6ea1f5b-6ba3-4053-a5e5-28554fefccbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1866534f-pcvnf\" (UID: \"d6ea1f5b-6ba3-4053-a5e5-28554fefccbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1866534f-pcvnf" Nov 03 13:24:38.180447 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:38.180430 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 13:24:38.180528 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:38.180430 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 13:24:38.180528 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:38.180438 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:24:38.180528 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:38.180495 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:24:38.279094 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:38.279049 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6ea1f5b-6ba3-4053-a5e5-28554fefccbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1866534f-pcvnf\" (UID: \"d6ea1f5b-6ba3-4053-a5e5-28554fefccbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1866534f-pcvnf" Nov 03 13:24:38.279494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:38.279146 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6ea1f5b-6ba3-4053-a5e5-28554fefccbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1866534f-pcvnf\" (UID: \"d6ea1f5b-6ba3-4053-a5e5-28554fefccbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1866534f-pcvnf" Nov 03 13:24:38.279494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:38.279223 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5z2d\" (UniqueName: \"kubernetes.io/projected/d6ea1f5b-6ba3-4053-a5e5-28554fefccbb-kube-api-access-s5z2d\") pod \"maas-api-revoke-sub-1866534f-pcvnf\" (UID: \"d6ea1f5b-6ba3-4053-a5e5-28554fefccbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1866534f-pcvnf" Nov 03 13:24:38.279847 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:38.279827 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6ea1f5b-6ba3-4053-a5e5-28554fefccbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1866534f-pcvnf\" (UID: \"d6ea1f5b-6ba3-4053-a5e5-28554fefccbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1866534f-pcvnf" Nov 03 13:24:38.281335 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:38.281305 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6ea1f5b-6ba3-4053-a5e5-28554fefccbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1866534f-pcvnf\" (UID: \"d6ea1f5b-6ba3-4053-a5e5-28554fefccbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1866534f-pcvnf" Nov 03 13:24:38.337544 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:38.337439 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5z2d\" (UniqueName: \"kubernetes.io/projected/d6ea1f5b-6ba3-4053-a5e5-28554fefccbb-kube-api-access-s5z2d\") pod \"maas-api-revoke-sub-1866534f-pcvnf\" (UID: \"d6ea1f5b-6ba3-4053-a5e5-28554fefccbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1866534f-pcvnf" Nov 03 13:24:38.397055 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:38.397029 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1866534f-pcvnf" Nov 03 13:24:38.767004 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:38.766971 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1866534f-pcvnf"] Nov 03 13:24:38.773907 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:24:38.773877 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6ea1f5b_6ba3_4053_a5e5_28554fefccbb.slice/crio-4f0e61567286ab1ecb442098a65ea154c622654d5eb596ed454228585e4c2ebd WatchSource:0}: Error finding container 4f0e61567286ab1ecb442098a65ea154c622654d5eb596ed454228585e4c2ebd: Status 404 returned error can't find the container with id 4f0e61567286ab1ecb442098a65ea154c622654d5eb596ed454228585e4c2ebd Nov 03 13:24:38.897889 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:38.897842 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1866534f-pcvnf" event={"ID":"d6ea1f5b-6ba3-4053-a5e5-28554fefccbb","Type":"ContainerStarted","Data":"4f0e61567286ab1ecb442098a65ea154c622654d5eb596ed454228585e4c2ebd"} Nov 03 13:24:38.949807 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:38.949773 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7dfbfc6544-jz7cm" podUID="61d2af57-4003-4364-b4f4-769565d27b65" containerName="payload-processing" containerID="cri-o://e31650c713af37198b81d35bb65f3d480e0ae15a37d30650debd01f487357874" gracePeriod=25 Nov 03 13:24:39.192413 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:39.192385 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7dfbfc6544-jz7cm" Nov 03 13:24:39.291993 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:39.291966 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61d2af57-4003-4364-b4f4-769565d27b65-kube-api-access-tczmb\" (UniqueName: \"kubernetes.io/projected/61d2af57-4003-4364-b4f4-769565d27b65-kube-api-access-tczmb\") pod \"61d2af57-4003-4364-b4f4-769565d27b65\" (UID: \"61d2af57-4003-4364-b4f4-769565d27b65\") " Nov 03 13:24:39.292374 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:39.292029 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61d2af57-4003-4364-b4f4-769565d27b65-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61d2af57-4003-4364-b4f4-769565d27b65-plugins-config-volume\") pod \"61d2af57-4003-4364-b4f4-769565d27b65\" (UID: \"61d2af57-4003-4364-b4f4-769565d27b65\") " Nov 03 13:24:39.292374 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:39.292143 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61d2af57-4003-4364-b4f4-769565d27b65-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/61d2af57-4003-4364-b4f4-769565d27b65-routing-overlay-volume\") pod \"61d2af57-4003-4364-b4f4-769565d27b65\" (UID: \"61d2af57-4003-4364-b4f4-769565d27b65\") " Nov 03 13:24:39.292374 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:39.292188 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61d2af57-4003-4364-b4f4-769565d27b65-provider-credentials\" (UniqueName: \"kubernetes.io/projected/61d2af57-4003-4364-b4f4-769565d27b65-provider-credentials\") pod \"61d2af57-4003-4364-b4f4-769565d27b65\" (UID: \"61d2af57-4003-4364-b4f4-769565d27b65\") " Nov 03 13:24:39.292584 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:39.292512 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61d2af57-4003-4364-b4f4-769565d27b65-plugins-config-volume" pod "61d2af57-4003-4364-b4f4-769565d27b65" (UID: "61d2af57-4003-4364-b4f4-769565d27b65"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:24:39.292584 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:39.292554 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61d2af57-4003-4364-b4f4-769565d27b65-routing-overlay-volume" pod "61d2af57-4003-4364-b4f4-769565d27b65" (UID: "61d2af57-4003-4364-b4f4-769565d27b65"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:24:39.292771 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:39.292751 5979 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/61d2af57-4003-4364-b4f4-769565d27b65-routing-overlay-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:24:39.292834 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:39.292775 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61d2af57-4003-4364-b4f4-769565d27b65-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:24:39.294329 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:39.294301 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61d2af57-4003-4364-b4f4-769565d27b65-kube-api-access-tczmb" pod "61d2af57-4003-4364-b4f4-769565d27b65" (UID: "61d2af57-4003-4364-b4f4-769565d27b65"). InnerVolumeSpecName "kube-api-access-tczmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:24:39.294329 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:39.294318 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61d2af57-4003-4364-b4f4-769565d27b65-provider-credentials" pod "61d2af57-4003-4364-b4f4-769565d27b65" (UID: "61d2af57-4003-4364-b4f4-769565d27b65"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:24:39.393629 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:39.393597 5979 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/61d2af57-4003-4364-b4f4-769565d27b65-provider-credentials\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:24:39.393629 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:39.393627 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tczmb\" (UniqueName: \"kubernetes.io/projected/61d2af57-4003-4364-b4f4-769565d27b65-kube-api-access-tczmb\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:24:39.905466 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:39.905433 5979 generic.go:356] "Generic (PLEG): container finished" podID="61d2af57-4003-4364-b4f4-769565d27b65" containerID="e31650c713af37198b81d35bb65f3d480e0ae15a37d30650debd01f487357874" exitCode=0 Nov 03 13:24:39.905690 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:39.905507 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7dfbfc6544-jz7cm" Nov 03 13:24:39.905690 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:39.905507 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7dfbfc6544-jz7cm" event={"ID":"61d2af57-4003-4364-b4f4-769565d27b65","Type":"ContainerDied","Data":"e31650c713af37198b81d35bb65f3d480e0ae15a37d30650debd01f487357874"} Nov 03 13:24:39.905690 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:39.905655 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7dfbfc6544-jz7cm" event={"ID":"61d2af57-4003-4364-b4f4-769565d27b65","Type":"ContainerDied","Data":"eb05b8470a6bd9f7b05b7bce6da653bba1bfbf4c34f16bd474978417637e49c5"} Nov 03 13:24:39.905862 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:39.905692 5979 scope.go:122] "RemoveContainer" containerID="e31650c713af37198b81d35bb65f3d480e0ae15a37d30650debd01f487357874" Nov 03 13:24:39.907236 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:39.907213 5979 generic.go:356] "Generic (PLEG): container finished" podID="d6ea1f5b-6ba3-4053-a5e5-28554fefccbb" containerID="f05588e47270503226b9fc1fca1ba41f9490bf8a6a7136c3709544f2470e5978" exitCode=0 Nov 03 13:24:39.907367 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:39.907342 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1866534f-pcvnf" event={"ID":"d6ea1f5b-6ba3-4053-a5e5-28554fefccbb","Type":"ContainerDied","Data":"f05588e47270503226b9fc1fca1ba41f9490bf8a6a7136c3709544f2470e5978"} Nov 03 13:24:39.915443 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:39.915428 5979 scope.go:122] "RemoveContainer" containerID="e31650c713af37198b81d35bb65f3d480e0ae15a37d30650debd01f487357874" Nov 03 13:24:39.915722 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:24:39.915707 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e31650c713af37198b81d35bb65f3d480e0ae15a37d30650debd01f487357874\": container with ID starting with e31650c713af37198b81d35bb65f3d480e0ae15a37d30650debd01f487357874 not found: ID does not exist" containerID="e31650c713af37198b81d35bb65f3d480e0ae15a37d30650debd01f487357874" Nov 03 13:24:39.915768 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:39.915728 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e31650c713af37198b81d35bb65f3d480e0ae15a37d30650debd01f487357874"} err="failed to get container status \"e31650c713af37198b81d35bb65f3d480e0ae15a37d30650debd01f487357874\": rpc error: code = NotFound desc = could not find container \"e31650c713af37198b81d35bb65f3d480e0ae15a37d30650debd01f487357874\": container with ID starting with e31650c713af37198b81d35bb65f3d480e0ae15a37d30650debd01f487357874 not found: ID does not exist" Nov 03 13:24:40.064261 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:40.064223 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7dfbfc6544-jz7cm"] Nov 03 13:24:40.115354 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:40.115324 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7dfbfc6544-jz7cm"] Nov 03 13:24:41.370051 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:41.370021 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1866534f-pcvnf" Nov 03 13:24:41.443166 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:41.443130 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6ea1f5b-6ba3-4053-a5e5-28554fefccbb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6ea1f5b-6ba3-4053-a5e5-28554fefccbb-maas-api-openshift-ca\") pod \"d6ea1f5b-6ba3-4053-a5e5-28554fefccbb\" (UID: \"d6ea1f5b-6ba3-4053-a5e5-28554fefccbb\") " Nov 03 13:24:41.443370 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:41.443253 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6ea1f5b-6ba3-4053-a5e5-28554fefccbb-kube-api-access-s5z2d\" (UniqueName: \"kubernetes.io/projected/d6ea1f5b-6ba3-4053-a5e5-28554fefccbb-kube-api-access-s5z2d\") pod \"d6ea1f5b-6ba3-4053-a5e5-28554fefccbb\" (UID: \"d6ea1f5b-6ba3-4053-a5e5-28554fefccbb\") " Nov 03 13:24:41.443370 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:41.443359 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6ea1f5b-6ba3-4053-a5e5-28554fefccbb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6ea1f5b-6ba3-4053-a5e5-28554fefccbb-maas-api-xks-ca\") pod \"d6ea1f5b-6ba3-4053-a5e5-28554fefccbb\" (UID: \"d6ea1f5b-6ba3-4053-a5e5-28554fefccbb\") " Nov 03 13:24:41.443574 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:41.443545 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6ea1f5b-6ba3-4053-a5e5-28554fefccbb-maas-api-openshift-ca" pod "d6ea1f5b-6ba3-4053-a5e5-28554fefccbb" (UID: "d6ea1f5b-6ba3-4053-a5e5-28554fefccbb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:24:41.443931 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:41.443904 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6ea1f5b-6ba3-4053-a5e5-28554fefccbb-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:24:41.461675 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:41.461642 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6ea1f5b-6ba3-4053-a5e5-28554fefccbb-kube-api-access-s5z2d" pod "d6ea1f5b-6ba3-4053-a5e5-28554fefccbb" (UID: "d6ea1f5b-6ba3-4053-a5e5-28554fefccbb"). InnerVolumeSpecName "kube-api-access-s5z2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:24:41.461798 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:41.461742 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6ea1f5b-6ba3-4053-a5e5-28554fefccbb-maas-api-xks-ca" pod "d6ea1f5b-6ba3-4053-a5e5-28554fefccbb" (UID: "d6ea1f5b-6ba3-4053-a5e5-28554fefccbb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:24:41.546882 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:41.546823 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6ea1f5b-6ba3-4053-a5e5-28554fefccbb-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:24:41.547199 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:41.547157 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5z2d\" (UniqueName: \"kubernetes.io/projected/d6ea1f5b-6ba3-4053-a5e5-28554fefccbb-kube-api-access-s5z2d\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:24:41.945514 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:41.944457 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1866534f-pcvnf" event={"ID":"d6ea1f5b-6ba3-4053-a5e5-28554fefccbb","Type":"ContainerDied","Data":"4f0e61567286ab1ecb442098a65ea154c622654d5eb596ed454228585e4c2ebd"} Nov 03 13:24:41.945514 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:41.944536 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f0e61567286ab1ecb442098a65ea154c622654d5eb596ed454228585e4c2ebd" Nov 03 13:24:41.945514 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:41.944586 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1866534f-pcvnf" Nov 03 13:24:42.122797 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:42.122758 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61d2af57-4003-4364-b4f4-769565d27b65" path="/var/lib/kubelet/pods/61d2af57-4003-4364-b4f4-769565d27b65/volumes" Nov 03 13:24:53.057131 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:53.057093 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60178b32-5jrcz"] Nov 03 13:24:53.103618 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:53.103571 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60178b32-5jrcz"] Nov 03 13:24:53.686333 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:53.686300 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cea9b59f-qkx4q"] Nov 03 13:24:53.686895 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:53.686882 5979 cpu_manager.go:414] "containerMap: removing container" podUID="d6ea1f5b-6ba3-4053-a5e5-28554fefccbb" containerName="revoke-keys" Nov 03 13:24:53.686954 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:53.686898 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6ea1f5b-6ba3-4053-a5e5-28554fefccbb" containerName="revoke-keys" Nov 03 13:24:53.686954 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:53.686927 5979 cpu_manager.go:414] "containerMap: removing container" podUID="61d2af57-4003-4364-b4f4-769565d27b65" containerName="payload-processing" Nov 03 13:24:53.686954 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:53.686933 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61d2af57-4003-4364-b4f4-769565d27b65" containerName="payload-processing" Nov 03 13:24:53.687051 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:53.687017 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="61d2af57-4003-4364-b4f4-769565d27b65" containerName="payload-processing" Nov 03 13:24:53.687051 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:53.687035 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6ea1f5b-6ba3-4053-a5e5-28554fefccbb" containerName="revoke-keys" Nov 03 13:24:53.691627 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:53.691608 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cea9b59f-qkx4q" Nov 03 13:24:53.743163 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:53.743134 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cea9b59f-qkx4q"] Nov 03 13:24:53.752765 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:53.752733 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/881d9604-a7b9-4fdc-83e9-e66011e961f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cea9b59f-qkx4q\" (UID: \"881d9604-a7b9-4fdc-83e9-e66011e961f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cea9b59f-qkx4q" Nov 03 13:24:53.752935 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:53.752912 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzxl9\" (UniqueName: \"kubernetes.io/projected/881d9604-a7b9-4fdc-83e9-e66011e961f9-kube-api-access-pzxl9\") pod \"maas-api-revoke-sub-cea9b59f-qkx4q\" (UID: \"881d9604-a7b9-4fdc-83e9-e66011e961f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cea9b59f-qkx4q" Nov 03 13:24:53.753040 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:53.753025 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/881d9604-a7b9-4fdc-83e9-e66011e961f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cea9b59f-qkx4q\" (UID: \"881d9604-a7b9-4fdc-83e9-e66011e961f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cea9b59f-qkx4q" Nov 03 13:24:53.786367 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:53.786334 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:24:53.786546 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:53.786394 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:24:53.786546 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:53.786413 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 13:24:53.787294 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:53.787239 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 13:24:53.854189 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:53.854152 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/881d9604-a7b9-4fdc-83e9-e66011e961f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cea9b59f-qkx4q\" (UID: \"881d9604-a7b9-4fdc-83e9-e66011e961f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cea9b59f-qkx4q" Nov 03 13:24:53.854349 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:53.854216 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzxl9\" (UniqueName: \"kubernetes.io/projected/881d9604-a7b9-4fdc-83e9-e66011e961f9-kube-api-access-pzxl9\") pod \"maas-api-revoke-sub-cea9b59f-qkx4q\" (UID: \"881d9604-a7b9-4fdc-83e9-e66011e961f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cea9b59f-qkx4q" Nov 03 13:24:53.854349 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:53.854267 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/881d9604-a7b9-4fdc-83e9-e66011e961f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cea9b59f-qkx4q\" (UID: \"881d9604-a7b9-4fdc-83e9-e66011e961f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cea9b59f-qkx4q" Nov 03 13:24:53.854914 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:53.854894 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/881d9604-a7b9-4fdc-83e9-e66011e961f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cea9b59f-qkx4q\" (UID: \"881d9604-a7b9-4fdc-83e9-e66011e961f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cea9b59f-qkx4q" Nov 03 13:24:53.856510 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:53.856488 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/881d9604-a7b9-4fdc-83e9-e66011e961f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cea9b59f-qkx4q\" (UID: \"881d9604-a7b9-4fdc-83e9-e66011e961f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cea9b59f-qkx4q" Nov 03 13:24:53.908328 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:53.908294 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzxl9\" (UniqueName: \"kubernetes.io/projected/881d9604-a7b9-4fdc-83e9-e66011e961f9-kube-api-access-pzxl9\") pod \"maas-api-revoke-sub-cea9b59f-qkx4q\" (UID: \"881d9604-a7b9-4fdc-83e9-e66011e961f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cea9b59f-qkx4q" Nov 03 13:24:54.005349 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:54.005264 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cea9b59f-qkx4q" Nov 03 13:24:54.116800 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:54.116762 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fa694a6-90c3-4124-a81d-1b71a84f003a" path="/var/lib/kubelet/pods/3fa694a6-90c3-4124-a81d-1b71a84f003a/volumes" Nov 03 13:24:54.378627 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:54.378549 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cea9b59f-qkx4q"] Nov 03 13:24:54.385531 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:24:54.385501 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod881d9604_a7b9_4fdc_83e9_e66011e961f9.slice/crio-d6e1fe2c9a325c6f7d30675af5bfd2d6e9ca89d0949949fcf362d011c9d8732d WatchSource:0}: Error finding container d6e1fe2c9a325c6f7d30675af5bfd2d6e9ca89d0949949fcf362d011c9d8732d: Status 404 returned error can't find the container with id d6e1fe2c9a325c6f7d30675af5bfd2d6e9ca89d0949949fcf362d011c9d8732d Nov 03 13:24:55.047622 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:55.047536 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cea9b59f-qkx4q" event={"ID":"881d9604-a7b9-4fdc-83e9-e66011e961f9","Type":"ContainerStarted","Data":"2dbf5b669b9dcc29da5e3afa79f1ce43d953d669b791f906c8c62da72d2af378"} Nov 03 13:24:55.047622 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:55.047587 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cea9b59f-qkx4q" event={"ID":"881d9604-a7b9-4fdc-83e9-e66011e961f9","Type":"ContainerStarted","Data":"d6e1fe2c9a325c6f7d30675af5bfd2d6e9ca89d0949949fcf362d011c9d8732d"} Nov 03 13:24:55.160055 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:55.160011 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cea9b59f-qkx4q" podStartSLOduration=2.159997194 podStartE2EDuration="2.159997194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:24:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:24:55.157014533 +0000 UTC m=+1177.167702776" watchObservedRunningTime="2026-11-03 13:24:55.159997194 +0000 UTC m=+1177.170685436" Nov 03 13:24:55.213518 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:55.213487 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17e79185-qsj8x"] Nov 03 13:24:55.217566 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:55.217543 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e79185-qsj8x" Nov 03 13:24:55.271279 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:55.271230 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chl77\" (UniqueName: \"kubernetes.io/projected/6e02c865-a839-42dd-b387-c5444bfe0a22-kube-api-access-chl77\") pod \"maas-api-revoke-sub-17e79185-qsj8x\" (UID: \"6e02c865-a839-42dd-b387-c5444bfe0a22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e79185-qsj8x" Nov 03 13:24:55.271511 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:55.271301 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e02c865-a839-42dd-b387-c5444bfe0a22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17e79185-qsj8x\" (UID: \"6e02c865-a839-42dd-b387-c5444bfe0a22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e79185-qsj8x" Nov 03 13:24:55.271582 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:55.271542 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e02c865-a839-42dd-b387-c5444bfe0a22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17e79185-qsj8x\" (UID: \"6e02c865-a839-42dd-b387-c5444bfe0a22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e79185-qsj8x" Nov 03 13:24:55.271898 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:55.271872 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17e79185-qsj8x"] Nov 03 13:24:55.372778 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:55.372695 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e02c865-a839-42dd-b387-c5444bfe0a22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17e79185-qsj8x\" (UID: \"6e02c865-a839-42dd-b387-c5444bfe0a22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e79185-qsj8x" Nov 03 13:24:55.372954 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:55.372849 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chl77\" (UniqueName: \"kubernetes.io/projected/6e02c865-a839-42dd-b387-c5444bfe0a22-kube-api-access-chl77\") pod \"maas-api-revoke-sub-17e79185-qsj8x\" (UID: \"6e02c865-a839-42dd-b387-c5444bfe0a22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e79185-qsj8x" Nov 03 13:24:55.372954 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:55.372896 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e02c865-a839-42dd-b387-c5444bfe0a22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17e79185-qsj8x\" (UID: \"6e02c865-a839-42dd-b387-c5444bfe0a22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e79185-qsj8x" Nov 03 13:24:55.373447 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:55.373424 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e02c865-a839-42dd-b387-c5444bfe0a22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17e79185-qsj8x\" (UID: \"6e02c865-a839-42dd-b387-c5444bfe0a22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e79185-qsj8x" Nov 03 13:24:55.375056 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:55.375034 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e02c865-a839-42dd-b387-c5444bfe0a22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17e79185-qsj8x\" (UID: \"6e02c865-a839-42dd-b387-c5444bfe0a22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e79185-qsj8x" Nov 03 13:24:55.429660 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:55.429632 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chl77\" (UniqueName: \"kubernetes.io/projected/6e02c865-a839-42dd-b387-c5444bfe0a22-kube-api-access-chl77\") pod \"maas-api-revoke-sub-17e79185-qsj8x\" (UID: \"6e02c865-a839-42dd-b387-c5444bfe0a22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e79185-qsj8x" Nov 03 13:24:55.530611 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:55.530575 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e79185-qsj8x" Nov 03 13:24:55.903650 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:55.903606 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17e79185-qsj8x"] Nov 03 13:24:55.910683 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:24:55.910651 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e02c865_a839_42dd_b387_c5444bfe0a22.slice/crio-06153e0164ae0be60c61293b0cdb7a244ec074973904961d7e2474b6565ed083 WatchSource:0}: Error finding container 06153e0164ae0be60c61293b0cdb7a244ec074973904961d7e2474b6565ed083: Status 404 returned error can't find the container with id 06153e0164ae0be60c61293b0cdb7a244ec074973904961d7e2474b6565ed083 Nov 03 13:24:56.055724 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:56.055684 5979 generic.go:356] "Generic (PLEG): container finished" podID="881d9604-a7b9-4fdc-83e9-e66011e961f9" containerID="2dbf5b669b9dcc29da5e3afa79f1ce43d953d669b791f906c8c62da72d2af378" exitCode=0 Nov 03 13:24:56.055876 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:56.055745 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cea9b59f-qkx4q" event={"ID":"881d9604-a7b9-4fdc-83e9-e66011e961f9","Type":"ContainerDied","Data":"2dbf5b669b9dcc29da5e3afa79f1ce43d953d669b791f906c8c62da72d2af378"} Nov 03 13:24:56.057112 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:56.057089 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e79185-qsj8x" event={"ID":"6e02c865-a839-42dd-b387-c5444bfe0a22","Type":"ContainerStarted","Data":"06153e0164ae0be60c61293b0cdb7a244ec074973904961d7e2474b6565ed083"} Nov 03 13:24:57.065024 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:57.064987 5979 generic.go:356] "Generic (PLEG): container finished" podID="6e02c865-a839-42dd-b387-c5444bfe0a22" containerID="63b9f7499141642d39b9c9bb582b347f48ec38c7fa641252f7a5782b5eece4e1" exitCode=0 Nov 03 13:24:57.065451 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:57.065067 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e79185-qsj8x" event={"ID":"6e02c865-a839-42dd-b387-c5444bfe0a22","Type":"ContainerDied","Data":"63b9f7499141642d39b9c9bb582b347f48ec38c7fa641252f7a5782b5eece4e1"} Nov 03 13:24:57.203849 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:57.203824 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cea9b59f-qkx4q" Nov 03 13:24:57.298659 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:57.298627 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/881d9604-a7b9-4fdc-83e9-e66011e961f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/881d9604-a7b9-4fdc-83e9-e66011e961f9-maas-api-xks-ca\") pod \"881d9604-a7b9-4fdc-83e9-e66011e961f9\" (UID: \"881d9604-a7b9-4fdc-83e9-e66011e961f9\") " Nov 03 13:24:57.298807 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:57.298726 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/881d9604-a7b9-4fdc-83e9-e66011e961f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/881d9604-a7b9-4fdc-83e9-e66011e961f9-maas-api-openshift-ca\") pod \"881d9604-a7b9-4fdc-83e9-e66011e961f9\" (UID: \"881d9604-a7b9-4fdc-83e9-e66011e961f9\") " Nov 03 13:24:57.298807 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:57.298766 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/881d9604-a7b9-4fdc-83e9-e66011e961f9-kube-api-access-pzxl9\" (UniqueName: \"kubernetes.io/projected/881d9604-a7b9-4fdc-83e9-e66011e961f9-kube-api-access-pzxl9\") pod \"881d9604-a7b9-4fdc-83e9-e66011e961f9\" (UID: \"881d9604-a7b9-4fdc-83e9-e66011e961f9\") " Nov 03 13:24:57.299141 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:57.299107 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/881d9604-a7b9-4fdc-83e9-e66011e961f9-maas-api-openshift-ca" pod "881d9604-a7b9-4fdc-83e9-e66011e961f9" (UID: "881d9604-a7b9-4fdc-83e9-e66011e961f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:24:57.299294 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:57.299277 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/881d9604-a7b9-4fdc-83e9-e66011e961f9-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:24:57.300889 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:57.300869 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/881d9604-a7b9-4fdc-83e9-e66011e961f9-maas-api-xks-ca" pod "881d9604-a7b9-4fdc-83e9-e66011e961f9" (UID: "881d9604-a7b9-4fdc-83e9-e66011e961f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:24:57.300973 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:57.300882 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/881d9604-a7b9-4fdc-83e9-e66011e961f9-kube-api-access-pzxl9" pod "881d9604-a7b9-4fdc-83e9-e66011e961f9" (UID: "881d9604-a7b9-4fdc-83e9-e66011e961f9"). InnerVolumeSpecName "kube-api-access-pzxl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:24:57.400928 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:57.400892 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzxl9\" (UniqueName: \"kubernetes.io/projected/881d9604-a7b9-4fdc-83e9-e66011e961f9-kube-api-access-pzxl9\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:24:57.400928 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:57.400925 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/881d9604-a7b9-4fdc-83e9-e66011e961f9-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:24:58.072516 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:58.072467 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cea9b59f-qkx4q" Nov 03 13:24:58.072928 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:58.072509 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cea9b59f-qkx4q" event={"ID":"881d9604-a7b9-4fdc-83e9-e66011e961f9","Type":"ContainerDied","Data":"d6e1fe2c9a325c6f7d30675af5bfd2d6e9ca89d0949949fcf362d011c9d8732d"} Nov 03 13:24:58.072928 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:58.072561 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6e1fe2c9a325c6f7d30675af5bfd2d6e9ca89d0949949fcf362d011c9d8732d" Nov 03 13:24:58.219665 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:58.219637 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e79185-qsj8x" Nov 03 13:24:58.310953 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:58.310917 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e02c865-a839-42dd-b387-c5444bfe0a22-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e02c865-a839-42dd-b387-c5444bfe0a22-maas-api-openshift-ca\") pod \"6e02c865-a839-42dd-b387-c5444bfe0a22\" (UID: \"6e02c865-a839-42dd-b387-c5444bfe0a22\") " Nov 03 13:24:58.311146 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:58.310973 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e02c865-a839-42dd-b387-c5444bfe0a22-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e02c865-a839-42dd-b387-c5444bfe0a22-maas-api-xks-ca\") pod \"6e02c865-a839-42dd-b387-c5444bfe0a22\" (UID: \"6e02c865-a839-42dd-b387-c5444bfe0a22\") " Nov 03 13:24:58.311146 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:58.311028 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e02c865-a839-42dd-b387-c5444bfe0a22-kube-api-access-chl77\" (UniqueName: \"kubernetes.io/projected/6e02c865-a839-42dd-b387-c5444bfe0a22-kube-api-access-chl77\") pod \"6e02c865-a839-42dd-b387-c5444bfe0a22\" (UID: \"6e02c865-a839-42dd-b387-c5444bfe0a22\") " Nov 03 13:24:58.311340 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:58.311318 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e02c865-a839-42dd-b387-c5444bfe0a22-maas-api-openshift-ca" pod "6e02c865-a839-42dd-b387-c5444bfe0a22" (UID: "6e02c865-a839-42dd-b387-c5444bfe0a22"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:24:58.311583 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:58.311564 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e02c865-a839-42dd-b387-c5444bfe0a22-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:24:58.313109 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:58.313078 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e02c865-a839-42dd-b387-c5444bfe0a22-maas-api-xks-ca" pod "6e02c865-a839-42dd-b387-c5444bfe0a22" (UID: "6e02c865-a839-42dd-b387-c5444bfe0a22"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:24:58.313208 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:58.313166 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e02c865-a839-42dd-b387-c5444bfe0a22-kube-api-access-chl77" pod "6e02c865-a839-42dd-b387-c5444bfe0a22" (UID: "6e02c865-a839-42dd-b387-c5444bfe0a22"). InnerVolumeSpecName "kube-api-access-chl77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:24:58.412548 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:58.412517 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chl77\" (UniqueName: \"kubernetes.io/projected/6e02c865-a839-42dd-b387-c5444bfe0a22-kube-api-access-chl77\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:24:58.412548 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:58.412547 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e02c865-a839-42dd-b387-c5444bfe0a22-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:24:59.080033 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:59.080001 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e79185-qsj8x" Nov 03 13:24:59.080445 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:59.080027 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e79185-qsj8x" event={"ID":"6e02c865-a839-42dd-b387-c5444bfe0a22","Type":"ContainerDied","Data":"06153e0164ae0be60c61293b0cdb7a244ec074973904961d7e2474b6565ed083"} Nov 03 13:24:59.080445 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:24:59.080110 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06153e0164ae0be60c61293b0cdb7a244ec074973904961d7e2474b6565ed083" Nov 03 13:25:00.005882 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:00.005848 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-5bb463/distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw"] Nov 03 13:25:00.006388 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:00.006375 5979 cpu_manager.go:414] "containerMap: removing container" podUID="6e02c865-a839-42dd-b387-c5444bfe0a22" containerName="revoke-keys" Nov 03 13:25:00.006443 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:00.006390 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e02c865-a839-42dd-b387-c5444bfe0a22" containerName="revoke-keys" Nov 03 13:25:00.006443 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:00.006401 5979 cpu_manager.go:414] "containerMap: removing container" podUID="881d9604-a7b9-4fdc-83e9-e66011e961f9" containerName="revoke-keys" Nov 03 13:25:00.006443 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:00.006406 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="881d9604-a7b9-4fdc-83e9-e66011e961f9" containerName="revoke-keys" Nov 03 13:25:00.006585 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:00.006575 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="881d9604-a7b9-4fdc-83e9-e66011e961f9" containerName="revoke-keys" Nov 03 13:25:00.006622 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:00.006586 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e02c865-a839-42dd-b387-c5444bfe0a22" containerName="revoke-keys" Nov 03 13:25:00.010293 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:00.010274 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5bb463/distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw" Nov 03 13:25:00.108190 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:00.108134 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5bb463\"/\"distinct-2-w3-5bb463-kserve-self-signed-certs\"" Nov 03 13:25:00.117164 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:00.117134 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5bb463/distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw"] Nov 03 13:25:00.133103 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:00.133072 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7c171f9-4569-431e-a891-5cae3328ce47-home\") pod \"distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw\" (UID: \"f7c171f9-4569-431e-a891-5cae3328ce47\") " pod="e2e-models-e2e-worker-w3-5bb463/distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw" Nov 03 13:25:00.133261 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:00.133121 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7c171f9-4569-431e-a891-5cae3328ce47-tls-certs\") pod \"distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw\" (UID: \"f7c171f9-4569-431e-a891-5cae3328ce47\") " pod="e2e-models-e2e-worker-w3-5bb463/distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw" Nov 03 13:25:00.133261 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:00.133240 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7c171f9-4569-431e-a891-5cae3328ce47-tmp-dir\") pod \"distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw\" (UID: \"f7c171f9-4569-431e-a891-5cae3328ce47\") " pod="e2e-models-e2e-worker-w3-5bb463/distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw" Nov 03 13:25:00.133349 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:00.133302 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g9rx\" (UniqueName: \"kubernetes.io/projected/f7c171f9-4569-431e-a891-5cae3328ce47-kube-api-access-9g9rx\") pod \"distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw\" (UID: \"f7c171f9-4569-431e-a891-5cae3328ce47\") " pod="e2e-models-e2e-worker-w3-5bb463/distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw" Nov 03 13:25:00.133405 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:00.133380 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7c171f9-4569-431e-a891-5cae3328ce47-dshm\") pod \"distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw\" (UID: \"f7c171f9-4569-431e-a891-5cae3328ce47\") " pod="e2e-models-e2e-worker-w3-5bb463/distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw" Nov 03 13:25:00.133438 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:00.133413 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7c171f9-4569-431e-a891-5cae3328ce47-model-cache\") pod \"distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw\" (UID: \"f7c171f9-4569-431e-a891-5cae3328ce47\") " pod="e2e-models-e2e-worker-w3-5bb463/distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw" Nov 03 13:25:00.234740 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:00.234703 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7c171f9-4569-431e-a891-5cae3328ce47-model-cache\") pod \"distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw\" (UID: \"f7c171f9-4569-431e-a891-5cae3328ce47\") " pod="e2e-models-e2e-worker-w3-5bb463/distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw" Nov 03 13:25:00.234933 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:00.234831 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7c171f9-4569-431e-a891-5cae3328ce47-home\") pod \"distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw\" (UID: \"f7c171f9-4569-431e-a891-5cae3328ce47\") " pod="e2e-models-e2e-worker-w3-5bb463/distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw" Nov 03 13:25:00.234933 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:00.234888 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7c171f9-4569-431e-a891-5cae3328ce47-tls-certs\") pod \"distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw\" (UID: \"f7c171f9-4569-431e-a891-5cae3328ce47\") " pod="e2e-models-e2e-worker-w3-5bb463/distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw" Nov 03 13:25:00.235073 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:00.234977 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7c171f9-4569-431e-a891-5cae3328ce47-tmp-dir\") pod \"distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw\" (UID: \"f7c171f9-4569-431e-a891-5cae3328ce47\") " pod="e2e-models-e2e-worker-w3-5bb463/distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw" Nov 03 13:25:00.235129 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:00.235084 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g9rx\" (UniqueName: \"kubernetes.io/projected/f7c171f9-4569-431e-a891-5cae3328ce47-kube-api-access-9g9rx\") pod \"distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw\" (UID: \"f7c171f9-4569-431e-a891-5cae3328ce47\") " pod="e2e-models-e2e-worker-w3-5bb463/distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw" Nov 03 13:25:00.235181 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:00.235156 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7c171f9-4569-431e-a891-5cae3328ce47-dshm\") pod \"distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw\" (UID: \"f7c171f9-4569-431e-a891-5cae3328ce47\") " pod="e2e-models-e2e-worker-w3-5bb463/distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw" Nov 03 13:25:00.235234 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:00.235178 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7c171f9-4569-431e-a891-5cae3328ce47-model-cache\") pod \"distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw\" (UID: \"f7c171f9-4569-431e-a891-5cae3328ce47\") " pod="e2e-models-e2e-worker-w3-5bb463/distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw" Nov 03 13:25:00.235591 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:00.235560 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7c171f9-4569-431e-a891-5cae3328ce47-tmp-dir\") pod \"distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw\" (UID: \"f7c171f9-4569-431e-a891-5cae3328ce47\") " pod="e2e-models-e2e-worker-w3-5bb463/distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw" Nov 03 13:25:00.235705 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:00.235612 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7c171f9-4569-431e-a891-5cae3328ce47-home\") pod \"distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw\" (UID: \"f7c171f9-4569-431e-a891-5cae3328ce47\") " pod="e2e-models-e2e-worker-w3-5bb463/distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw" Nov 03 13:25:00.237460 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:00.237441 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7c171f9-4569-431e-a891-5cae3328ce47-dshm\") pod \"distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw\" (UID: \"f7c171f9-4569-431e-a891-5cae3328ce47\") " pod="e2e-models-e2e-worker-w3-5bb463/distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw" Nov 03 13:25:00.237703 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:00.237681 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7c171f9-4569-431e-a891-5cae3328ce47-tls-certs\") pod \"distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw\" (UID: \"f7c171f9-4569-431e-a891-5cae3328ce47\") " pod="e2e-models-e2e-worker-w3-5bb463/distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw" Nov 03 13:25:00.289493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:00.289408 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g9rx\" (UniqueName: \"kubernetes.io/projected/f7c171f9-4569-431e-a891-5cae3328ce47-kube-api-access-9g9rx\") pod \"distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw\" (UID: \"f7c171f9-4569-431e-a891-5cae3328ce47\") " pod="e2e-models-e2e-worker-w3-5bb463/distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw" Nov 03 13:25:00.322512 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:00.322453 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5bb463/distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw" Nov 03 13:25:00.410829 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:00.410773 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:25:00.771792 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:00.771754 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5bb463/distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw"] Nov 03 13:25:00.778273 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:25:00.778247 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf7c171f9_4569_431e_a891_5cae3328ce47.slice/crio-b6e0d36f4ab41004c2ded0c880395833444566b2929954c0be1a488a43c28ee1 WatchSource:0}: Error finding container b6e0d36f4ab41004c2ded0c880395833444566b2929954c0be1a488a43c28ee1: Status 404 returned error can't find the container with id b6e0d36f4ab41004c2ded0c880395833444566b2929954c0be1a488a43c28ee1 Nov 03 13:25:01.132549 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:01.132226 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5bb463/distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw" event={"ID":"f7c171f9-4569-431e-a891-5cae3328ce47","Type":"ContainerStarted","Data":"55ca2f708f84e253317eb6c95027c22de16c5ee1e2fabbefd12f2809d16ee760"} Nov 03 13:25:01.132549 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:01.132288 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5bb463/distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw" event={"ID":"f7c171f9-4569-431e-a891-5cae3328ce47","Type":"ContainerStarted","Data":"b6e0d36f4ab41004c2ded0c880395833444566b2929954c0be1a488a43c28ee1"} Nov 03 13:25:01.244260 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:01.244196 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-5bb463/distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw" podStartSLOduration=2.244180042 podStartE2EDuration="2.244180042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:24:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:25:01.242718216 +0000 UTC m=+1183.253406454" watchObservedRunningTime="2026-11-03 13:25:01.244180042 +0000 UTC m=+1183.254868284" Nov 03 13:25:01.323329 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:01.323293 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5bb463/distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw" Nov 03 13:25:05.561717 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:05.561680 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-42c707/distinct-w0-42c707-kserve-64949d6cb8-kq4w9"] Nov 03 13:25:05.565341 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:05.565322 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-42c707/distinct-w0-42c707-kserve-64949d6cb8-kq4w9" Nov 03 13:25:05.656615 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:05.656580 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/430d8f95-204c-4243-b771-a03e30303e77-dshm\") pod \"distinct-w0-42c707-kserve-64949d6cb8-kq4w9\" (UID: \"430d8f95-204c-4243-b771-a03e30303e77\") " pod="e2e-models-e2e-worker-w0-42c707/distinct-w0-42c707-kserve-64949d6cb8-kq4w9" Nov 03 13:25:05.656760 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:05.656632 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/430d8f95-204c-4243-b771-a03e30303e77-home\") pod \"distinct-w0-42c707-kserve-64949d6cb8-kq4w9\" (UID: \"430d8f95-204c-4243-b771-a03e30303e77\") " pod="e2e-models-e2e-worker-w0-42c707/distinct-w0-42c707-kserve-64949d6cb8-kq4w9" Nov 03 13:25:05.656797 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:05.656766 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zchgp\" (UniqueName: \"kubernetes.io/projected/430d8f95-204c-4243-b771-a03e30303e77-kube-api-access-zchgp\") pod \"distinct-w0-42c707-kserve-64949d6cb8-kq4w9\" (UID: \"430d8f95-204c-4243-b771-a03e30303e77\") " pod="e2e-models-e2e-worker-w0-42c707/distinct-w0-42c707-kserve-64949d6cb8-kq4w9" Nov 03 13:25:05.656853 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:05.656802 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/430d8f95-204c-4243-b771-a03e30303e77-model-cache\") pod \"distinct-w0-42c707-kserve-64949d6cb8-kq4w9\" (UID: \"430d8f95-204c-4243-b771-a03e30303e77\") " pod="e2e-models-e2e-worker-w0-42c707/distinct-w0-42c707-kserve-64949d6cb8-kq4w9" Nov 03 13:25:05.656853 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:05.656823 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/430d8f95-204c-4243-b771-a03e30303e77-tls-certs\") pod \"distinct-w0-42c707-kserve-64949d6cb8-kq4w9\" (UID: \"430d8f95-204c-4243-b771-a03e30303e77\") " pod="e2e-models-e2e-worker-w0-42c707/distinct-w0-42c707-kserve-64949d6cb8-kq4w9" Nov 03 13:25:05.656965 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:05.656947 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/430d8f95-204c-4243-b771-a03e30303e77-tmp-dir\") pod \"distinct-w0-42c707-kserve-64949d6cb8-kq4w9\" (UID: \"430d8f95-204c-4243-b771-a03e30303e77\") " pod="e2e-models-e2e-worker-w0-42c707/distinct-w0-42c707-kserve-64949d6cb8-kq4w9" Nov 03 13:25:05.664885 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:05.664822 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-42c707\"/\"openshift-service-ca.crt\"" Nov 03 13:25:05.664885 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:05.664847 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-42c707\"/\"kube-root-ca.crt\"" Nov 03 13:25:05.664885 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:05.664864 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-42c707\"/\"distinct-w0-42c707-kserve-self-signed-certs\"" Nov 03 13:25:05.665102 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:05.664925 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-42c707\"/\"default-dockercfg-mzpk2\"" Nov 03 13:25:05.668520 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:05.668498 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-42c707/distinct-w0-42c707-kserve-64949d6cb8-kq4w9"] Nov 03 13:25:05.723137 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:05.723103 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9"] Nov 03 13:25:05.727822 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:05.727797 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9" Nov 03 13:25:05.766254 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:05.766216 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/430d8f95-204c-4243-b771-a03e30303e77-model-cache\") pod \"distinct-w0-42c707-kserve-64949d6cb8-kq4w9\" (UID: \"430d8f95-204c-4243-b771-a03e30303e77\") " pod="e2e-models-e2e-worker-w0-42c707/distinct-w0-42c707-kserve-64949d6cb8-kq4w9" Nov 03 13:25:05.766456 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:05.766265 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/430d8f95-204c-4243-b771-a03e30303e77-tls-certs\") pod \"distinct-w0-42c707-kserve-64949d6cb8-kq4w9\" (UID: \"430d8f95-204c-4243-b771-a03e30303e77\") " pod="e2e-models-e2e-worker-w0-42c707/distinct-w0-42c707-kserve-64949d6cb8-kq4w9" Nov 03 13:25:05.766456 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:05.766358 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/430d8f95-204c-4243-b771-a03e30303e77-tmp-dir\") pod \"distinct-w0-42c707-kserve-64949d6cb8-kq4w9\" (UID: \"430d8f95-204c-4243-b771-a03e30303e77\") " pod="e2e-models-e2e-worker-w0-42c707/distinct-w0-42c707-kserve-64949d6cb8-kq4w9" Nov 03 13:25:05.766456 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:05.766434 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/430d8f95-204c-4243-b771-a03e30303e77-dshm\") pod \"distinct-w0-42c707-kserve-64949d6cb8-kq4w9\" (UID: \"430d8f95-204c-4243-b771-a03e30303e77\") " pod="e2e-models-e2e-worker-w0-42c707/distinct-w0-42c707-kserve-64949d6cb8-kq4w9" Nov 03 13:25:05.766657 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:05.766507 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/430d8f95-204c-4243-b771-a03e30303e77-home\") pod \"distinct-w0-42c707-kserve-64949d6cb8-kq4w9\" (UID: \"430d8f95-204c-4243-b771-a03e30303e77\") " pod="e2e-models-e2e-worker-w0-42c707/distinct-w0-42c707-kserve-64949d6cb8-kq4w9" Nov 03 13:25:05.766657 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:05.766610 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zchgp\" (UniqueName: \"kubernetes.io/projected/430d8f95-204c-4243-b771-a03e30303e77-kube-api-access-zchgp\") pod \"distinct-w0-42c707-kserve-64949d6cb8-kq4w9\" (UID: \"430d8f95-204c-4243-b771-a03e30303e77\") " pod="e2e-models-e2e-worker-w0-42c707/distinct-w0-42c707-kserve-64949d6cb8-kq4w9" Nov 03 13:25:05.767376 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:05.767348 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/430d8f95-204c-4243-b771-a03e30303e77-model-cache\") pod \"distinct-w0-42c707-kserve-64949d6cb8-kq4w9\" (UID: \"430d8f95-204c-4243-b771-a03e30303e77\") " pod="e2e-models-e2e-worker-w0-42c707/distinct-w0-42c707-kserve-64949d6cb8-kq4w9" Nov 03 13:25:05.795658 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:05.783821 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9"] Nov 03 13:25:05.795658 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:05.784125 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/430d8f95-204c-4243-b771-a03e30303e77-tmp-dir\") pod \"distinct-w0-42c707-kserve-64949d6cb8-kq4w9\" (UID: \"430d8f95-204c-4243-b771-a03e30303e77\") " pod="e2e-models-e2e-worker-w0-42c707/distinct-w0-42c707-kserve-64949d6cb8-kq4w9" Nov 03 13:25:05.795658 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:05.784180 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/430d8f95-204c-4243-b771-a03e30303e77-home\") pod \"distinct-w0-42c707-kserve-64949d6cb8-kq4w9\" (UID: \"430d8f95-204c-4243-b771-a03e30303e77\") " pod="e2e-models-e2e-worker-w0-42c707/distinct-w0-42c707-kserve-64949d6cb8-kq4w9" Nov 03 13:25:05.819178 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:05.819108 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/430d8f95-204c-4243-b771-a03e30303e77-dshm\") pod \"distinct-w0-42c707-kserve-64949d6cb8-kq4w9\" (UID: \"430d8f95-204c-4243-b771-a03e30303e77\") " pod="e2e-models-e2e-worker-w0-42c707/distinct-w0-42c707-kserve-64949d6cb8-kq4w9" Nov 03 13:25:05.823982 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:05.823954 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/430d8f95-204c-4243-b771-a03e30303e77-tls-certs\") pod \"distinct-w0-42c707-kserve-64949d6cb8-kq4w9\" (UID: \"430d8f95-204c-4243-b771-a03e30303e77\") " pod="e2e-models-e2e-worker-w0-42c707/distinct-w0-42c707-kserve-64949d6cb8-kq4w9" Nov 03 13:25:05.838026 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:05.837998 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zchgp\" (UniqueName: \"kubernetes.io/projected/430d8f95-204c-4243-b771-a03e30303e77-kube-api-access-zchgp\") pod \"distinct-w0-42c707-kserve-64949d6cb8-kq4w9\" (UID: \"430d8f95-204c-4243-b771-a03e30303e77\") " pod="e2e-models-e2e-worker-w0-42c707/distinct-w0-42c707-kserve-64949d6cb8-kq4w9" Nov 03 13:25:05.867361 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:05.867328 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26360457-2ef0-4c64-9a1c-4d62a783dc49-istiod-ca-cert\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9\" (UID: \"26360457-2ef0-4c64-9a1c-4d62a783dc49\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9" Nov 03 13:25:05.867527 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:05.867393 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26360457-2ef0-4c64-9a1c-4d62a783dc49-istio-podinfo\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9\" (UID: \"26360457-2ef0-4c64-9a1c-4d62a783dc49\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9" Nov 03 13:25:05.867591 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:05.867567 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26360457-2ef0-4c64-9a1c-4d62a783dc49-workload-certs\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9\" (UID: \"26360457-2ef0-4c64-9a1c-4d62a783dc49\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9" Nov 03 13:25:05.867629 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:05.867610 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq9hl\" (UniqueName: \"kubernetes.io/projected/26360457-2ef0-4c64-9a1c-4d62a783dc49-kube-api-access-kq9hl\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9\" (UID: \"26360457-2ef0-4c64-9a1c-4d62a783dc49\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9" Nov 03 13:25:05.867712 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:05.867700 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26360457-2ef0-4c64-9a1c-4d62a783dc49-istio-data\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9\" (UID: \"26360457-2ef0-4c64-9a1c-4d62a783dc49\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9" Nov 03 13:25:05.867748 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:05.867727 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26360457-2ef0-4c64-9a1c-4d62a783dc49-credential-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9\" (UID: \"26360457-2ef0-4c64-9a1c-4d62a783dc49\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9" Nov 03 13:25:05.867789 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:05.867779 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26360457-2ef0-4c64-9a1c-4d62a783dc49-workload-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9\" (UID: \"26360457-2ef0-4c64-9a1c-4d62a783dc49\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9" Nov 03 13:25:05.867829 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:05.867816 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26360457-2ef0-4c64-9a1c-4d62a783dc49-istio-token\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9\" (UID: \"26360457-2ef0-4c64-9a1c-4d62a783dc49\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9" Nov 03 13:25:05.867864 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:05.867837 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26360457-2ef0-4c64-9a1c-4d62a783dc49-istio-envoy\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9\" (UID: \"26360457-2ef0-4c64-9a1c-4d62a783dc49\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9" Nov 03 13:25:05.877486 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:05.877443 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-42c707/distinct-w0-42c707-kserve-64949d6cb8-kq4w9" Nov 03 13:25:05.995275 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:05.990236 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26360457-2ef0-4c64-9a1c-4d62a783dc49-istio-podinfo\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9\" (UID: \"26360457-2ef0-4c64-9a1c-4d62a783dc49\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9" Nov 03 13:25:05.995275 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:05.990381 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26360457-2ef0-4c64-9a1c-4d62a783dc49-workload-certs\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9\" (UID: \"26360457-2ef0-4c64-9a1c-4d62a783dc49\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9" Nov 03 13:25:05.995275 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:05.990424 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq9hl\" (UniqueName: \"kubernetes.io/projected/26360457-2ef0-4c64-9a1c-4d62a783dc49-kube-api-access-kq9hl\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9\" (UID: \"26360457-2ef0-4c64-9a1c-4d62a783dc49\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9" Nov 03 13:25:05.995275 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:05.990490 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26360457-2ef0-4c64-9a1c-4d62a783dc49-istio-data\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9\" (UID: \"26360457-2ef0-4c64-9a1c-4d62a783dc49\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9" Nov 03 13:25:05.995275 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:05.990523 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26360457-2ef0-4c64-9a1c-4d62a783dc49-credential-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9\" (UID: \"26360457-2ef0-4c64-9a1c-4d62a783dc49\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9" Nov 03 13:25:05.995275 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:05.990565 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26360457-2ef0-4c64-9a1c-4d62a783dc49-workload-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9\" (UID: \"26360457-2ef0-4c64-9a1c-4d62a783dc49\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9" Nov 03 13:25:05.995275 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:05.990608 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26360457-2ef0-4c64-9a1c-4d62a783dc49-istio-token\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9\" (UID: \"26360457-2ef0-4c64-9a1c-4d62a783dc49\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9" Nov 03 13:25:05.995275 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:05.990642 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26360457-2ef0-4c64-9a1c-4d62a783dc49-istio-envoy\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9\" (UID: \"26360457-2ef0-4c64-9a1c-4d62a783dc49\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9" Nov 03 13:25:05.995275 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:05.990687 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26360457-2ef0-4c64-9a1c-4d62a783dc49-istiod-ca-cert\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9\" (UID: \"26360457-2ef0-4c64-9a1c-4d62a783dc49\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9" Nov 03 13:25:05.995275 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:05.991532 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26360457-2ef0-4c64-9a1c-4d62a783dc49-istiod-ca-cert\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9\" (UID: \"26360457-2ef0-4c64-9a1c-4d62a783dc49\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9" Nov 03 13:25:05.995275 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:05.993340 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26360457-2ef0-4c64-9a1c-4d62a783dc49-credential-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9\" (UID: \"26360457-2ef0-4c64-9a1c-4d62a783dc49\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9" Nov 03 13:25:05.995275 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:05.993594 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26360457-2ef0-4c64-9a1c-4d62a783dc49-workload-certs\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9\" (UID: \"26360457-2ef0-4c64-9a1c-4d62a783dc49\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9" Nov 03 13:25:05.999264 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:05.999132 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26360457-2ef0-4c64-9a1c-4d62a783dc49-workload-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9\" (UID: \"26360457-2ef0-4c64-9a1c-4d62a783dc49\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9" Nov 03 13:25:05.999264 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:05.999186 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26360457-2ef0-4c64-9a1c-4d62a783dc49-istio-data\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9\" (UID: \"26360457-2ef0-4c64-9a1c-4d62a783dc49\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9" Nov 03 13:25:06.004228 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:06.004156 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26360457-2ef0-4c64-9a1c-4d62a783dc49-istio-envoy\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9\" (UID: \"26360457-2ef0-4c64-9a1c-4d62a783dc49\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9" Nov 03 13:25:06.010397 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:06.004516 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26360457-2ef0-4c64-9a1c-4d62a783dc49-istio-podinfo\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9\" (UID: \"26360457-2ef0-4c64-9a1c-4d62a783dc49\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9" Nov 03 13:25:06.051644 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:06.051609 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq9hl\" (UniqueName: \"kubernetes.io/projected/26360457-2ef0-4c64-9a1c-4d62a783dc49-kube-api-access-kq9hl\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9\" (UID: \"26360457-2ef0-4c64-9a1c-4d62a783dc49\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9" Nov 03 13:25:06.059395 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:06.059371 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a821e75-8tt4c"] Nov 03 13:25:06.062813 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:06.062791 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26360457-2ef0-4c64-9a1c-4d62a783dc49-istio-token\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9\" (UID: \"26360457-2ef0-4c64-9a1c-4d62a783dc49\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9" Nov 03 13:25:06.064878 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:06.064855 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a821e75-8tt4c" Nov 03 13:25:06.114183 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:06.113981 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 13:25:06.114183 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:06.114063 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 13:25:06.114183 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:06.114126 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:25:06.114183 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:06.114147 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:25:06.118176 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:06.118150 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a821e75-8tt4c"] Nov 03 13:25:06.133326 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:06.133296 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9" Nov 03 13:25:06.193377 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:06.193343 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0746779-2ad5-4002-b404-a62c1e171da0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a821e75-8tt4c\" (UID: \"d0746779-2ad5-4002-b404-a62c1e171da0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a821e75-8tt4c" Nov 03 13:25:06.193683 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:06.193658 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0746779-2ad5-4002-b404-a62c1e171da0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a821e75-8tt4c\" (UID: \"d0746779-2ad5-4002-b404-a62c1e171da0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a821e75-8tt4c" Nov 03 13:25:06.193794 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:06.193725 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxt5b\" (UniqueName: \"kubernetes.io/projected/d0746779-2ad5-4002-b404-a62c1e171da0-kube-api-access-dxt5b\") pod \"maas-api-revoke-sub-9a821e75-8tt4c\" (UID: \"d0746779-2ad5-4002-b404-a62c1e171da0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a821e75-8tt4c" Nov 03 13:25:06.261011 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:06.260976 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-42c707/distinct-w0-42c707-kserve-64949d6cb8-kq4w9"] Nov 03 13:25:06.267044 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:25:06.267019 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod430d8f95_204c_4243_b771_a03e30303e77.slice/crio-9575b1bb9e48389d621cf183c3c9bd0ff112c07023da9358421c151b7c881f4b WatchSource:0}: Error finding container 9575b1bb9e48389d621cf183c3c9bd0ff112c07023da9358421c151b7c881f4b: Status 404 returned error can't find the container with id 9575b1bb9e48389d621cf183c3c9bd0ff112c07023da9358421c151b7c881f4b Nov 03 13:25:06.294783 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:06.294744 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0746779-2ad5-4002-b404-a62c1e171da0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a821e75-8tt4c\" (UID: \"d0746779-2ad5-4002-b404-a62c1e171da0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a821e75-8tt4c" Nov 03 13:25:06.294919 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:06.294812 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0746779-2ad5-4002-b404-a62c1e171da0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a821e75-8tt4c\" (UID: \"d0746779-2ad5-4002-b404-a62c1e171da0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a821e75-8tt4c" Nov 03 13:25:06.294919 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:06.294846 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxt5b\" (UniqueName: \"kubernetes.io/projected/d0746779-2ad5-4002-b404-a62c1e171da0-kube-api-access-dxt5b\") pod \"maas-api-revoke-sub-9a821e75-8tt4c\" (UID: \"d0746779-2ad5-4002-b404-a62c1e171da0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a821e75-8tt4c" Nov 03 13:25:06.295547 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:06.295518 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0746779-2ad5-4002-b404-a62c1e171da0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9a821e75-8tt4c\" (UID: \"d0746779-2ad5-4002-b404-a62c1e171da0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a821e75-8tt4c" Nov 03 13:25:06.297202 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:06.297180 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0746779-2ad5-4002-b404-a62c1e171da0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9a821e75-8tt4c\" (UID: \"d0746779-2ad5-4002-b404-a62c1e171da0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a821e75-8tt4c" Nov 03 13:25:06.349888 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:06.349807 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxt5b\" (UniqueName: \"kubernetes.io/projected/d0746779-2ad5-4002-b404-a62c1e171da0-kube-api-access-dxt5b\") pod \"maas-api-revoke-sub-9a821e75-8tt4c\" (UID: \"d0746779-2ad5-4002-b404-a62c1e171da0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a821e75-8tt4c" Nov 03 13:25:06.383188 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:06.383158 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a821e75-8tt4c" Nov 03 13:25:06.520808 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:06.520774 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9"] Nov 03 13:25:06.524559 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:25:06.524527 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26360457_2ef0_4c64_9a1c_4d62a783dc49.slice/crio-3100cc868accee62836bd6b0c9fd084052265266a77c0cdc388ee6126c68d5fa WatchSource:0}: Error finding container 3100cc868accee62836bd6b0c9fd084052265266a77c0cdc388ee6126c68d5fa: Status 404 returned error can't find the container with id 3100cc868accee62836bd6b0c9fd084052265266a77c0cdc388ee6126c68d5fa Nov 03 13:25:06.526934 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:06.526903 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:25:06.527012 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:06.526967 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:25:06.527012 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:06.526998 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:25:06.771697 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:06.771656 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a821e75-8tt4c"] Nov 03 13:25:06.779399 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:25:06.779372 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0746779_2ad5_4002_b404_a62c1e171da0.slice/crio-919a89ad8128238d1a5ab9d1b11bbc93396c6b98b9553a993122d24e1dfc8025 WatchSource:0}: Error finding container 919a89ad8128238d1a5ab9d1b11bbc93396c6b98b9553a993122d24e1dfc8025: Status 404 returned error can't find the container with id 919a89ad8128238d1a5ab9d1b11bbc93396c6b98b9553a993122d24e1dfc8025 Nov 03 13:25:07.194279 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:07.194240 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a821e75-8tt4c" event={"ID":"d0746779-2ad5-4002-b404-a62c1e171da0","Type":"ContainerStarted","Data":"a60acca48c729c78b87b02f02addbcd386cc86001ba1ff6b4dfee6a82c79f3ef"} Nov 03 13:25:07.194420 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:07.194294 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a821e75-8tt4c" event={"ID":"d0746779-2ad5-4002-b404-a62c1e171da0","Type":"ContainerStarted","Data":"919a89ad8128238d1a5ab9d1b11bbc93396c6b98b9553a993122d24e1dfc8025"} Nov 03 13:25:07.195826 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:07.195798 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9" event={"ID":"26360457-2ef0-4c64-9a1c-4d62a783dc49","Type":"ContainerStarted","Data":"494e19cb04491dbd8aea6b37136258d4636937646ec2d53aaee8af9e85fff96d"} Nov 03 13:25:07.195924 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:07.195849 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9" event={"ID":"26360457-2ef0-4c64-9a1c-4d62a783dc49","Type":"ContainerStarted","Data":"3100cc868accee62836bd6b0c9fd084052265266a77c0cdc388ee6126c68d5fa"} Nov 03 13:25:07.197357 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:07.197336 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-42c707/distinct-w0-42c707-kserve-64949d6cb8-kq4w9" event={"ID":"430d8f95-204c-4243-b771-a03e30303e77","Type":"ContainerStarted","Data":"a0f2959ef1981d75e9f92d0ecec6408978d72894f804638fce4da5f87cbc5676"} Nov 03 13:25:07.197441 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:07.197365 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-42c707/distinct-w0-42c707-kserve-64949d6cb8-kq4w9" event={"ID":"430d8f95-204c-4243-b771-a03e30303e77","Type":"ContainerStarted","Data":"9575b1bb9e48389d621cf183c3c9bd0ff112c07023da9358421c151b7c881f4b"} Nov 03 13:25:07.233947 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:25:07.233914 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:25:07.302102 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:07.302050 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a821e75-8tt4c" podStartSLOduration=1.302032762 podStartE2EDuration="1.302032762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:25:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:25:07.301360035 +0000 UTC m=+1189.312048276" watchObservedRunningTime="2026-11-03 13:25:07.302032762 +0000 UTC m=+1189.312721003" Nov 03 13:25:07.416080 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:07.416024 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-42c707/distinct-w0-42c707-kserve-64949d6cb8-kq4w9" podStartSLOduration=2.416009303 podStartE2EDuration="2.416009303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:25:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:25:07.412680457 +0000 UTC m=+1189.423368699" watchObservedRunningTime="2026-11-03 13:25:07.416009303 +0000 UTC m=+1189.426697568" Nov 03 13:25:07.530895 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:07.530799 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9" podStartSLOduration=2.530782024 podStartE2EDuration="2.530782024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:25:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:25:07.527970771 +0000 UTC m=+1189.538659012" watchObservedRunningTime="2026-11-03 13:25:07.530782024 +0000 UTC m=+1189.541470406" Nov 03 13:25:07.877776 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:07.877743 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-42c707/distinct-w0-42c707-kserve-64949d6cb8-kq4w9" Nov 03 13:25:08.134210 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:08.134184 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9" Nov 03 13:25:08.136082 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:08.136058 5979 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 03 13:25:08.136189 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:08.136103 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9" podUID="26360457-2ef0-4c64-9a1c-4d62a783dc49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 03 13:25:08.205157 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:08.205123 5979 generic.go:356] "Generic (PLEG): container finished" podID="d0746779-2ad5-4002-b404-a62c1e171da0" containerID="a60acca48c729c78b87b02f02addbcd386cc86001ba1ff6b4dfee6a82c79f3ef" exitCode=0 Nov 03 13:25:08.205331 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:08.205207 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a821e75-8tt4c" event={"ID":"d0746779-2ad5-4002-b404-a62c1e171da0","Type":"ContainerDied","Data":"a60acca48c729c78b87b02f02addbcd386cc86001ba1ff6b4dfee6a82c79f3ef"} Nov 03 13:25:09.134500 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:09.134429 5979 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 03 13:25:09.134899 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:09.134526 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9" podUID="26360457-2ef0-4c64-9a1c-4d62a783dc49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 03 13:25:09.385785 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:09.385716 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a821e75-8tt4c" Nov 03 13:25:09.470168 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:09.470139 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0746779-2ad5-4002-b404-a62c1e171da0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0746779-2ad5-4002-b404-a62c1e171da0-maas-api-xks-ca\") pod \"d0746779-2ad5-4002-b404-a62c1e171da0\" (UID: \"d0746779-2ad5-4002-b404-a62c1e171da0\") " Nov 03 13:25:09.470388 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:09.470280 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0746779-2ad5-4002-b404-a62c1e171da0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0746779-2ad5-4002-b404-a62c1e171da0-maas-api-openshift-ca\") pod \"d0746779-2ad5-4002-b404-a62c1e171da0\" (UID: \"d0746779-2ad5-4002-b404-a62c1e171da0\") " Nov 03 13:25:09.470388 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:09.470306 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0746779-2ad5-4002-b404-a62c1e171da0-kube-api-access-dxt5b\" (UniqueName: \"kubernetes.io/projected/d0746779-2ad5-4002-b404-a62c1e171da0-kube-api-access-dxt5b\") pod \"d0746779-2ad5-4002-b404-a62c1e171da0\" (UID: \"d0746779-2ad5-4002-b404-a62c1e171da0\") " Nov 03 13:25:09.470707 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:09.470675 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0746779-2ad5-4002-b404-a62c1e171da0-maas-api-openshift-ca" pod "d0746779-2ad5-4002-b404-a62c1e171da0" (UID: "d0746779-2ad5-4002-b404-a62c1e171da0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:25:09.472358 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:09.472329 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0746779-2ad5-4002-b404-a62c1e171da0-maas-api-xks-ca" pod "d0746779-2ad5-4002-b404-a62c1e171da0" (UID: "d0746779-2ad5-4002-b404-a62c1e171da0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:25:09.472572 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:09.472548 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0746779-2ad5-4002-b404-a62c1e171da0-kube-api-access-dxt5b" pod "d0746779-2ad5-4002-b404-a62c1e171da0" (UID: "d0746779-2ad5-4002-b404-a62c1e171da0"). InnerVolumeSpecName "kube-api-access-dxt5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:25:09.571749 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:09.571709 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0746779-2ad5-4002-b404-a62c1e171da0-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:25:09.571749 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:09.571745 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxt5b\" (UniqueName: \"kubernetes.io/projected/d0746779-2ad5-4002-b404-a62c1e171da0-kube-api-access-dxt5b\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:25:09.572092 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:09.571760 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0746779-2ad5-4002-b404-a62c1e171da0-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:25:10.134228 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:10.134201 5979 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 03 13:25:10.134403 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:10.134252 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9" podUID="26360457-2ef0-4c64-9a1c-4d62a783dc49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 03 13:25:10.219911 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:10.219875 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a821e75-8tt4c" Nov 03 13:25:10.220332 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:10.219960 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9a821e75-8tt4c" event={"ID":"d0746779-2ad5-4002-b404-a62c1e171da0","Type":"ContainerDied","Data":"919a89ad8128238d1a5ab9d1b11bbc93396c6b98b9553a993122d24e1dfc8025"} Nov 03 13:25:10.220332 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:10.220030 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="919a89ad8128238d1a5ab9d1b11bbc93396c6b98b9553a993122d24e1dfc8025" Nov 03 13:25:10.323417 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:10.323384 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-5bb463/distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw" Nov 03 13:25:10.336080 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:10.336055 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-5bb463/distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw" Nov 03 13:25:11.134047 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:11.134010 5979 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 03 13:25:11.134259 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:11.134063 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9" podUID="26360457-2ef0-4c64-9a1c-4d62a783dc49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 03 13:25:11.237169 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:11.237141 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5bb463/distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw" Nov 03 13:25:12.022617 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:12.022584 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2620536c-tnzcp"] Nov 03 13:25:12.023195 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:12.023176 5979 cpu_manager.go:414] "containerMap: removing container" podUID="d0746779-2ad5-4002-b404-a62c1e171da0" containerName="revoke-keys" Nov 03 13:25:12.023297 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:12.023197 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0746779-2ad5-4002-b404-a62c1e171da0" containerName="revoke-keys" Nov 03 13:25:12.023366 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:12.023353 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0746779-2ad5-4002-b404-a62c1e171da0" containerName="revoke-keys" Nov 03 13:25:12.027103 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:12.027083 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2620536c-tnzcp" Nov 03 13:25:12.076914 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:12.076875 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93aefb28-r8bjv"] Nov 03 13:25:12.088701 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:12.088567 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 13:25:12.088701 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:12.088600 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 13:25:12.104436 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:12.104395 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b95dcf2b-e814-4b75-a234-82b5b765e52a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2620536c-tnzcp\" (UID: \"b95dcf2b-e814-4b75-a234-82b5b765e52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2620536c-tnzcp" Nov 03 13:25:12.104632 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:12.104579 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b95dcf2b-e814-4b75-a234-82b5b765e52a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2620536c-tnzcp\" (UID: \"b95dcf2b-e814-4b75-a234-82b5b765e52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2620536c-tnzcp" Nov 03 13:25:12.104718 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:12.104680 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvcwx\" (UniqueName: \"kubernetes.io/projected/b95dcf2b-e814-4b75-a234-82b5b765e52a-kube-api-access-bvcwx\") pod \"maas-api-revoke-sub-2620536c-tnzcp\" (UID: \"b95dcf2b-e814-4b75-a234-82b5b765e52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2620536c-tnzcp" Nov 03 13:25:12.120158 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:12.120127 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:25:12.124114 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:12.124085 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93aefb28-r8bjv"] Nov 03 13:25:12.126884 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:12.126863 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2620536c-tnzcp"] Nov 03 13:25:12.129116 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:12.129093 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:25:12.133921 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:12.133896 5979 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 03 13:25:12.134032 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:12.133941 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9" podUID="26360457-2ef0-4c64-9a1c-4d62a783dc49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 03 13:25:12.205526 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:12.205464 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b95dcf2b-e814-4b75-a234-82b5b765e52a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2620536c-tnzcp\" (UID: \"b95dcf2b-e814-4b75-a234-82b5b765e52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2620536c-tnzcp" Nov 03 13:25:12.205702 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:12.205555 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b95dcf2b-e814-4b75-a234-82b5b765e52a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2620536c-tnzcp\" (UID: \"b95dcf2b-e814-4b75-a234-82b5b765e52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2620536c-tnzcp" Nov 03 13:25:12.205702 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:12.205598 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvcwx\" (UniqueName: \"kubernetes.io/projected/b95dcf2b-e814-4b75-a234-82b5b765e52a-kube-api-access-bvcwx\") pod \"maas-api-revoke-sub-2620536c-tnzcp\" (UID: \"b95dcf2b-e814-4b75-a234-82b5b765e52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2620536c-tnzcp" Nov 03 13:25:12.206124 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:12.206103 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b95dcf2b-e814-4b75-a234-82b5b765e52a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2620536c-tnzcp\" (UID: \"b95dcf2b-e814-4b75-a234-82b5b765e52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2620536c-tnzcp" Nov 03 13:25:12.208113 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:12.208069 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b95dcf2b-e814-4b75-a234-82b5b765e52a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2620536c-tnzcp\" (UID: \"b95dcf2b-e814-4b75-a234-82b5b765e52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2620536c-tnzcp" Nov 03 13:25:12.270104 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:12.270071 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvcwx\" (UniqueName: \"kubernetes.io/projected/b95dcf2b-e814-4b75-a234-82b5b765e52a-kube-api-access-bvcwx\") pod \"maas-api-revoke-sub-2620536c-tnzcp\" (UID: \"b95dcf2b-e814-4b75-a234-82b5b765e52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2620536c-tnzcp" Nov 03 13:25:12.341388 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:12.341318 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2620536c-tnzcp" Nov 03 13:25:12.742974 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:12.742941 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2620536c-tnzcp"] Nov 03 13:25:12.749495 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:25:12.749438 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb95dcf2b_e814_4b75_a234_82b5b765e52a.slice/crio-4bdcb32d04c268ab8eba3ae5151700c1217d0b69edc1e53ca2a21220a2203821 WatchSource:0}: Error finding container 4bdcb32d04c268ab8eba3ae5151700c1217d0b69edc1e53ca2a21220a2203821: Status 404 returned error can't find the container with id 4bdcb32d04c268ab8eba3ae5151700c1217d0b69edc1e53ca2a21220a2203821 Nov 03 13:25:13.134729 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:13.134688 5979 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 03 13:25:13.134869 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:13.134753 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9" podUID="26360457-2ef0-4c64-9a1c-4d62a783dc49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 03 13:25:13.241839 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:13.241802 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2620536c-tnzcp" event={"ID":"b95dcf2b-e814-4b75-a234-82b5b765e52a","Type":"ContainerStarted","Data":"7648ebd2f307d4fdf1953fa81f86a36878b3e7fe4f32570569e93b4cf9969aae"} Nov 03 13:25:13.242034 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:13.241867 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2620536c-tnzcp" event={"ID":"b95dcf2b-e814-4b75-a234-82b5b765e52a","Type":"ContainerStarted","Data":"4bdcb32d04c268ab8eba3ae5151700c1217d0b69edc1e53ca2a21220a2203821"} Nov 03 13:25:13.351074 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:13.350974 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2620536c-tnzcp" podStartSLOduration=2.350959323 podStartE2EDuration="2.350959323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:25:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:25:13.348660921 +0000 UTC m=+1195.359349176" watchObservedRunningTime="2026-11-03 13:25:13.350959323 +0000 UTC m=+1195.361647599" Nov 03 13:25:14.115528 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:14.115442 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f1800fb-b1b8-4113-a50d-01fa71e064e0" path="/var/lib/kubelet/pods/8f1800fb-b1b8-4113-a50d-01fa71e064e0/volumes" Nov 03 13:25:14.138162 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:14.138133 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9" Nov 03 13:25:14.138516 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:14.138496 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9" Nov 03 13:25:14.139264 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:14.139245 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9" Nov 03 13:25:14.249840 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:14.249798 5979 generic.go:356] "Generic (PLEG): container finished" podID="b95dcf2b-e814-4b75-a234-82b5b765e52a" containerID="7648ebd2f307d4fdf1953fa81f86a36878b3e7fe4f32570569e93b4cf9969aae" exitCode=0 Nov 03 13:25:14.250020 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:14.249842 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2620536c-tnzcp" event={"ID":"b95dcf2b-e814-4b75-a234-82b5b765e52a","Type":"ContainerDied","Data":"7648ebd2f307d4fdf1953fa81f86a36878b3e7fe4f32570569e93b4cf9969aae"} Nov 03 13:25:15.401371 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:15.401341 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2620536c-tnzcp" Nov 03 13:25:15.456598 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:15.456567 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b95dcf2b-e814-4b75-a234-82b5b765e52a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b95dcf2b-e814-4b75-a234-82b5b765e52a-maas-api-openshift-ca\") pod \"b95dcf2b-e814-4b75-a234-82b5b765e52a\" (UID: \"b95dcf2b-e814-4b75-a234-82b5b765e52a\") " Nov 03 13:25:15.456772 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:15.456685 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b95dcf2b-e814-4b75-a234-82b5b765e52a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b95dcf2b-e814-4b75-a234-82b5b765e52a-maas-api-xks-ca\") pod \"b95dcf2b-e814-4b75-a234-82b5b765e52a\" (UID: \"b95dcf2b-e814-4b75-a234-82b5b765e52a\") " Nov 03 13:25:15.456772 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:15.456739 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b95dcf2b-e814-4b75-a234-82b5b765e52a-kube-api-access-bvcwx\" (UniqueName: \"kubernetes.io/projected/b95dcf2b-e814-4b75-a234-82b5b765e52a-kube-api-access-bvcwx\") pod \"b95dcf2b-e814-4b75-a234-82b5b765e52a\" (UID: \"b95dcf2b-e814-4b75-a234-82b5b765e52a\") " Nov 03 13:25:15.456999 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:15.456968 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b95dcf2b-e814-4b75-a234-82b5b765e52a-maas-api-openshift-ca" pod "b95dcf2b-e814-4b75-a234-82b5b765e52a" (UID: "b95dcf2b-e814-4b75-a234-82b5b765e52a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:25:15.457323 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:15.457301 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b95dcf2b-e814-4b75-a234-82b5b765e52a-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:25:15.458600 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:15.458579 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b95dcf2b-e814-4b75-a234-82b5b765e52a-maas-api-xks-ca" pod "b95dcf2b-e814-4b75-a234-82b5b765e52a" (UID: "b95dcf2b-e814-4b75-a234-82b5b765e52a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:25:15.458738 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:15.458718 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b95dcf2b-e814-4b75-a234-82b5b765e52a-kube-api-access-bvcwx" pod "b95dcf2b-e814-4b75-a234-82b5b765e52a" (UID: "b95dcf2b-e814-4b75-a234-82b5b765e52a"). InnerVolumeSpecName "kube-api-access-bvcwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:25:15.558076 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:15.557995 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b95dcf2b-e814-4b75-a234-82b5b765e52a-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:25:15.558076 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:15.558026 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvcwx\" (UniqueName: \"kubernetes.io/projected/b95dcf2b-e814-4b75-a234-82b5b765e52a-kube-api-access-bvcwx\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:25:15.878210 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:15.878126 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-42c707/distinct-w0-42c707-kserve-64949d6cb8-kq4w9" Nov 03 13:25:15.890448 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:15.890422 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-42c707/distinct-w0-42c707-kserve-64949d6cb8-kq4w9" Nov 03 13:25:16.263803 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:16.263774 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2620536c-tnzcp" Nov 03 13:25:16.263995 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:16.263804 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2620536c-tnzcp" event={"ID":"b95dcf2b-e814-4b75-a234-82b5b765e52a","Type":"ContainerDied","Data":"4bdcb32d04c268ab8eba3ae5151700c1217d0b69edc1e53ca2a21220a2203821"} Nov 03 13:25:16.263995 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:16.263856 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4bdcb32d04c268ab8eba3ae5151700c1217d0b69edc1e53ca2a21220a2203821" Nov 03 13:25:16.275560 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:16.275536 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-42c707/distinct-w0-42c707-kserve-64949d6cb8-kq4w9" Nov 03 13:25:20.736496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:20.736150 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd"] Nov 03 13:25:20.740393 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:20.737064 5979 cpu_manager.go:414] "containerMap: removing container" podUID="b95dcf2b-e814-4b75-a234-82b5b765e52a" containerName="revoke-keys" Nov 03 13:25:20.740393 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:20.737088 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b95dcf2b-e814-4b75-a234-82b5b765e52a" containerName="revoke-keys" Nov 03 13:25:20.740393 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:20.737285 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="b95dcf2b-e814-4b75-a234-82b5b765e52a" containerName="revoke-keys" Nov 03 13:25:20.749495 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:20.746726 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd" Nov 03 13:25:20.831440 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:20.831402 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd"] Nov 03 13:25:20.869752 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:20.869709 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e80110d-950d-452f-9fdf-eecb56840a3c-istio-token\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd\" (UID: \"3e80110d-950d-452f-9fdf-eecb56840a3c\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd" Nov 03 13:25:20.870154 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:20.870107 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e80110d-950d-452f-9fdf-eecb56840a3c-credential-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd\" (UID: \"3e80110d-950d-452f-9fdf-eecb56840a3c\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd" Nov 03 13:25:20.870407 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:20.870386 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e80110d-950d-452f-9fdf-eecb56840a3c-istio-podinfo\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd\" (UID: \"3e80110d-950d-452f-9fdf-eecb56840a3c\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd" Nov 03 13:25:20.870680 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:20.870653 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e80110d-950d-452f-9fdf-eecb56840a3c-workload-certs\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd\" (UID: \"3e80110d-950d-452f-9fdf-eecb56840a3c\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd" Nov 03 13:25:20.870957 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:20.870919 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e80110d-950d-452f-9fdf-eecb56840a3c-workload-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd\" (UID: \"3e80110d-950d-452f-9fdf-eecb56840a3c\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd" Nov 03 13:25:20.871195 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:20.871176 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e80110d-950d-452f-9fdf-eecb56840a3c-istiod-ca-cert\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd\" (UID: \"3e80110d-950d-452f-9fdf-eecb56840a3c\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd" Nov 03 13:25:20.871399 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:20.871380 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nktj8\" (UniqueName: \"kubernetes.io/projected/3e80110d-950d-452f-9fdf-eecb56840a3c-kube-api-access-nktj8\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd\" (UID: \"3e80110d-950d-452f-9fdf-eecb56840a3c\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd" Nov 03 13:25:20.875704 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:20.875680 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e80110d-950d-452f-9fdf-eecb56840a3c-istio-data\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd\" (UID: \"3e80110d-950d-452f-9fdf-eecb56840a3c\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd" Nov 03 13:25:20.875950 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:20.875930 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e80110d-950d-452f-9fdf-eecb56840a3c-istio-envoy\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd\" (UID: \"3e80110d-950d-452f-9fdf-eecb56840a3c\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd" Nov 03 13:25:20.983504 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:20.978203 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e80110d-950d-452f-9fdf-eecb56840a3c-istiod-ca-cert\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd\" (UID: \"3e80110d-950d-452f-9fdf-eecb56840a3c\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd" Nov 03 13:25:20.983917 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:20.983763 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nktj8\" (UniqueName: \"kubernetes.io/projected/3e80110d-950d-452f-9fdf-eecb56840a3c-kube-api-access-nktj8\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd\" (UID: \"3e80110d-950d-452f-9fdf-eecb56840a3c\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd" Nov 03 13:25:20.988604 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:20.988520 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e80110d-950d-452f-9fdf-eecb56840a3c-istio-data\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd\" (UID: \"3e80110d-950d-452f-9fdf-eecb56840a3c\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd" Nov 03 13:25:20.988831 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:20.988813 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e80110d-950d-452f-9fdf-eecb56840a3c-istio-envoy\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd\" (UID: \"3e80110d-950d-452f-9fdf-eecb56840a3c\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd" Nov 03 13:25:20.989018 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:20.989004 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e80110d-950d-452f-9fdf-eecb56840a3c-istio-token\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd\" (UID: \"3e80110d-950d-452f-9fdf-eecb56840a3c\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd" Nov 03 13:25:20.989164 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:20.989151 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e80110d-950d-452f-9fdf-eecb56840a3c-credential-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd\" (UID: \"3e80110d-950d-452f-9fdf-eecb56840a3c\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd" Nov 03 13:25:20.989309 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:20.989292 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e80110d-950d-452f-9fdf-eecb56840a3c-istio-podinfo\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd\" (UID: \"3e80110d-950d-452f-9fdf-eecb56840a3c\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd" Nov 03 13:25:21.019550 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:20.996958 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e80110d-950d-452f-9fdf-eecb56840a3c-workload-certs\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd\" (UID: \"3e80110d-950d-452f-9fdf-eecb56840a3c\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd" Nov 03 13:25:21.019550 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:20.997086 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e80110d-950d-452f-9fdf-eecb56840a3c-workload-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd\" (UID: \"3e80110d-950d-452f-9fdf-eecb56840a3c\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd" Nov 03 13:25:21.019550 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:20.997563 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e80110d-950d-452f-9fdf-eecb56840a3c-workload-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd\" (UID: \"3e80110d-950d-452f-9fdf-eecb56840a3c\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd" Nov 03 13:25:21.019550 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:20.997828 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e80110d-950d-452f-9fdf-eecb56840a3c-workload-certs\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd\" (UID: \"3e80110d-950d-452f-9fdf-eecb56840a3c\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd" Nov 03 13:25:21.019550 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:20.998088 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e80110d-950d-452f-9fdf-eecb56840a3c-istio-data\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd\" (UID: \"3e80110d-950d-452f-9fdf-eecb56840a3c\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd" Nov 03 13:25:21.019550 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:20.998337 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e80110d-950d-452f-9fdf-eecb56840a3c-credential-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd\" (UID: \"3e80110d-950d-452f-9fdf-eecb56840a3c\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd" Nov 03 13:25:21.022650 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:21.022610 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e80110d-950d-452f-9fdf-eecb56840a3c-istiod-ca-cert\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd\" (UID: \"3e80110d-950d-452f-9fdf-eecb56840a3c\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd" Nov 03 13:25:21.101582 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:21.088571 5979 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:25:21.101582 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:21.088632 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b" podUID="7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:25:21.104694 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:21.104630 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nktj8\" (UniqueName: \"kubernetes.io/projected/3e80110d-950d-452f-9fdf-eecb56840a3c-kube-api-access-nktj8\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd\" (UID: \"3e80110d-950d-452f-9fdf-eecb56840a3c\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd" Nov 03 13:25:21.241073 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:21.240984 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e80110d-950d-452f-9fdf-eecb56840a3c-istio-envoy\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd\" (UID: \"3e80110d-950d-452f-9fdf-eecb56840a3c\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd" Nov 03 13:25:21.242227 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:21.242169 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e80110d-950d-452f-9fdf-eecb56840a3c-istio-token\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd\" (UID: \"3e80110d-950d-452f-9fdf-eecb56840a3c\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd" Nov 03 13:25:21.317495 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:21.316792 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e80110d-950d-452f-9fdf-eecb56840a3c-istio-podinfo\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd\" (UID: \"3e80110d-950d-452f-9fdf-eecb56840a3c\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd" Nov 03 13:25:21.392211 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:21.392181 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd" Nov 03 13:25:22.103503 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:22.103397 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd"] Nov 03 13:25:22.109783 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:25:22.109744 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e80110d_950d_452f_9fdf_eecb56840a3c.slice/crio-684b3ce026539ac2b84ba20efbf741b721766e8d2fa3a52c6b4997a56299ac9a WatchSource:0}: Error finding container 684b3ce026539ac2b84ba20efbf741b721766e8d2fa3a52c6b4997a56299ac9a: Status 404 returned error can't find the container with id 684b3ce026539ac2b84ba20efbf741b721766e8d2fa3a52c6b4997a56299ac9a Nov 03 13:25:22.112612 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:22.112574 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:25:22.112713 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:22.112660 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:25:22.112713 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:22.112704 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:25:22.402722 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:22.402680 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd" event={"ID":"3e80110d-950d-452f-9fdf-eecb56840a3c","Type":"ContainerStarted","Data":"bb0eeff0d2aa46c774377d182ba268af2f4372bbb98a29e51607aa1949913f6d"} Nov 03 13:25:22.402898 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:22.402745 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd" event={"ID":"3e80110d-950d-452f-9fdf-eecb56840a3c","Type":"ContainerStarted","Data":"684b3ce026539ac2b84ba20efbf741b721766e8d2fa3a52c6b4997a56299ac9a"} Nov 03 13:25:22.515566 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:22.515512 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd" podStartSLOduration=2.51549574 podStartE2EDuration="2.51549574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:25:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:25:22.514329024 +0000 UTC m=+1204.525017266" watchObservedRunningTime="2026-11-03 13:25:22.51549574 +0000 UTC m=+1204.526183984" Nov 03 13:25:22.794885 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:22.794812 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3dc9b19-p5jwq"] Nov 03 13:25:22.799168 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:22.799150 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3dc9b19-p5jwq" Nov 03 13:25:22.848930 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:22.848896 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:25:22.868538 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:22.849230 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 13:25:22.868753 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:22.864703 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 13:25:22.868846 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:22.864772 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:25:22.868929 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:22.866603 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ab6d0b5-79bc-45d5-8b77-75f5d5fd49cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3dc9b19-p5jwq\" (UID: \"2ab6d0b5-79bc-45d5-8b77-75f5d5fd49cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3dc9b19-p5jwq" Nov 03 13:25:22.869040 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:22.869025 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ab6d0b5-79bc-45d5-8b77-75f5d5fd49cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3dc9b19-p5jwq\" (UID: \"2ab6d0b5-79bc-45d5-8b77-75f5d5fd49cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3dc9b19-p5jwq" Nov 03 13:25:22.869256 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:22.869240 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn8t4\" (UniqueName: \"kubernetes.io/projected/2ab6d0b5-79bc-45d5-8b77-75f5d5fd49cd-kube-api-access-mn8t4\") pod \"maas-api-revoke-sub-d3dc9b19-p5jwq\" (UID: \"2ab6d0b5-79bc-45d5-8b77-75f5d5fd49cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3dc9b19-p5jwq" Nov 03 13:25:22.872135 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:22.872105 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3dc9b19-p5jwq"] Nov 03 13:25:22.970494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:22.970437 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ab6d0b5-79bc-45d5-8b77-75f5d5fd49cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3dc9b19-p5jwq\" (UID: \"2ab6d0b5-79bc-45d5-8b77-75f5d5fd49cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3dc9b19-p5jwq" Nov 03 13:25:22.970673 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:22.970504 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ab6d0b5-79bc-45d5-8b77-75f5d5fd49cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3dc9b19-p5jwq\" (UID: \"2ab6d0b5-79bc-45d5-8b77-75f5d5fd49cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3dc9b19-p5jwq" Nov 03 13:25:22.970673 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:22.970627 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn8t4\" (UniqueName: \"kubernetes.io/projected/2ab6d0b5-79bc-45d5-8b77-75f5d5fd49cd-kube-api-access-mn8t4\") pod \"maas-api-revoke-sub-d3dc9b19-p5jwq\" (UID: \"2ab6d0b5-79bc-45d5-8b77-75f5d5fd49cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3dc9b19-p5jwq" Nov 03 13:25:22.971673 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:22.971607 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ab6d0b5-79bc-45d5-8b77-75f5d5fd49cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3dc9b19-p5jwq\" (UID: \"2ab6d0b5-79bc-45d5-8b77-75f5d5fd49cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3dc9b19-p5jwq" Nov 03 13:25:22.973556 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:22.973530 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ab6d0b5-79bc-45d5-8b77-75f5d5fd49cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3dc9b19-p5jwq\" (UID: \"2ab6d0b5-79bc-45d5-8b77-75f5d5fd49cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3dc9b19-p5jwq" Nov 03 13:25:23.026940 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:23.026907 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn8t4\" (UniqueName: \"kubernetes.io/projected/2ab6d0b5-79bc-45d5-8b77-75f5d5fd49cd-kube-api-access-mn8t4\") pod \"maas-api-revoke-sub-d3dc9b19-p5jwq\" (UID: \"2ab6d0b5-79bc-45d5-8b77-75f5d5fd49cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3dc9b19-p5jwq" Nov 03 13:25:23.071184 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:23.071084 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f67e1b40-bdr95"] Nov 03 13:25:23.076119 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:23.076097 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67e1b40-bdr95" Nov 03 13:25:23.110036 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:23.109988 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f67e1b40-bdr95"] Nov 03 13:25:23.112036 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:23.112011 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3dc9b19-p5jwq" Nov 03 13:25:23.181413 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:23.181366 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv4wq\" (UniqueName: \"kubernetes.io/projected/f23a7bba-1d47-4cb9-853a-5adbe0e34430-kube-api-access-hv4wq\") pod \"maas-api-revoke-sub-f67e1b40-bdr95\" (UID: \"f23a7bba-1d47-4cb9-853a-5adbe0e34430\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67e1b40-bdr95" Nov 03 13:25:23.181629 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:23.181569 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f23a7bba-1d47-4cb9-853a-5adbe0e34430-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f67e1b40-bdr95\" (UID: \"f23a7bba-1d47-4cb9-853a-5adbe0e34430\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67e1b40-bdr95" Nov 03 13:25:23.181729 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:23.181687 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f23a7bba-1d47-4cb9-853a-5adbe0e34430-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f67e1b40-bdr95\" (UID: \"f23a7bba-1d47-4cb9-853a-5adbe0e34430\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67e1b40-bdr95" Nov 03 13:25:23.284494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:23.283806 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv4wq\" (UniqueName: \"kubernetes.io/projected/f23a7bba-1d47-4cb9-853a-5adbe0e34430-kube-api-access-hv4wq\") pod \"maas-api-revoke-sub-f67e1b40-bdr95\" (UID: \"f23a7bba-1d47-4cb9-853a-5adbe0e34430\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67e1b40-bdr95" Nov 03 13:25:23.284494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:23.283961 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f23a7bba-1d47-4cb9-853a-5adbe0e34430-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f67e1b40-bdr95\" (UID: \"f23a7bba-1d47-4cb9-853a-5adbe0e34430\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67e1b40-bdr95" Nov 03 13:25:23.284494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:23.284439 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f23a7bba-1d47-4cb9-853a-5adbe0e34430-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f67e1b40-bdr95\" (UID: \"f23a7bba-1d47-4cb9-853a-5adbe0e34430\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67e1b40-bdr95" Nov 03 13:25:23.285136 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:23.285092 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f23a7bba-1d47-4cb9-853a-5adbe0e34430-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f67e1b40-bdr95\" (UID: \"f23a7bba-1d47-4cb9-853a-5adbe0e34430\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67e1b40-bdr95" Nov 03 13:25:23.287082 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:23.287050 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f23a7bba-1d47-4cb9-853a-5adbe0e34430-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f67e1b40-bdr95\" (UID: \"f23a7bba-1d47-4cb9-853a-5adbe0e34430\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67e1b40-bdr95" Nov 03 13:25:23.338337 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:23.338264 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv4wq\" (UniqueName: \"kubernetes.io/projected/f23a7bba-1d47-4cb9-853a-5adbe0e34430-kube-api-access-hv4wq\") pod \"maas-api-revoke-sub-f67e1b40-bdr95\" (UID: \"f23a7bba-1d47-4cb9-853a-5adbe0e34430\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67e1b40-bdr95" Nov 03 13:25:23.389886 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:23.389856 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67e1b40-bdr95" Nov 03 13:25:23.394269 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:23.394126 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd" Nov 03 13:25:23.395633 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:23.395609 5979 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 13:25:23.395746 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:23.395659 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd" podUID="3e80110d-950d-452f-9fdf-eecb56840a3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 13:25:23.532848 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:23.532811 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3dc9b19-p5jwq"] Nov 03 13:25:23.537829 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:25:23.537794 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ab6d0b5_79bc_45d5_8b77_75f5d5fd49cd.slice/crio-695de040e76da6938642ffb14e969a53ccfcef13f8e1a022af0b62e7b2a6d956 WatchSource:0}: Error finding container 695de040e76da6938642ffb14e969a53ccfcef13f8e1a022af0b62e7b2a6d956: Status 404 returned error can't find the container with id 695de040e76da6938642ffb14e969a53ccfcef13f8e1a022af0b62e7b2a6d956 Nov 03 13:25:23.791791 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:23.791760 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f67e1b40-bdr95"] Nov 03 13:25:23.799060 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:25:23.799038 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf23a7bba_1d47_4cb9_853a_5adbe0e34430.slice/crio-85abfce0ab947b835e6b8c1e27f410efcd5468631644ddd1767c16837d52a26a WatchSource:0}: Error finding container 85abfce0ab947b835e6b8c1e27f410efcd5468631644ddd1767c16837d52a26a: Status 404 returned error can't find the container with id 85abfce0ab947b835e6b8c1e27f410efcd5468631644ddd1767c16837d52a26a Nov 03 13:25:24.395367 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:24.395333 5979 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 13:25:24.395791 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:24.395389 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd" podUID="3e80110d-950d-452f-9fdf-eecb56840a3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 13:25:24.417662 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:24.417624 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3dc9b19-p5jwq" event={"ID":"2ab6d0b5-79bc-45d5-8b77-75f5d5fd49cd","Type":"ContainerStarted","Data":"bae4a2d6284260a520d8bfa3ab90704ebfbf97859eeb5271321c8a0f6e85e2e8"} Nov 03 13:25:24.417824 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:24.417685 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3dc9b19-p5jwq" event={"ID":"2ab6d0b5-79bc-45d5-8b77-75f5d5fd49cd","Type":"ContainerStarted","Data":"695de040e76da6938642ffb14e969a53ccfcef13f8e1a022af0b62e7b2a6d956"} Nov 03 13:25:24.419385 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:24.419344 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67e1b40-bdr95" event={"ID":"f23a7bba-1d47-4cb9-853a-5adbe0e34430","Type":"ContainerStarted","Data":"0411fea64680cc5a963fe7383b8ff76160cca398c44c627598d8f77b76291621"} Nov 03 13:25:24.419551 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:24.419408 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67e1b40-bdr95" event={"ID":"f23a7bba-1d47-4cb9-853a-5adbe0e34430","Type":"ContainerStarted","Data":"85abfce0ab947b835e6b8c1e27f410efcd5468631644ddd1767c16837d52a26a"} Nov 03 13:25:24.570642 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:24.570573 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3dc9b19-p5jwq" podStartSLOduration=2.5705528600000003 podStartE2EDuration="2.57055286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:25:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:25:24.557900698 +0000 UTC m=+1206.568588941" watchObservedRunningTime="2026-11-03 13:25:24.57055286 +0000 UTC m=+1206.581241102" Nov 03 13:25:24.673502 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:24.672772 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67e1b40-bdr95" podStartSLOduration=1.672752509 podStartE2EDuration="1.672752509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:25:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:25:24.670404031 +0000 UTC m=+1206.681092274" watchObservedRunningTime="2026-11-03 13:25:24.672752509 +0000 UTC m=+1206.683440752" Nov 03 13:25:25.395149 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:25.395113 5979 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 13:25:25.395343 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:25.395171 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd" podUID="3e80110d-950d-452f-9fdf-eecb56840a3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 13:25:25.426666 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:25.426634 5979 generic.go:356] "Generic (PLEG): container finished" podID="2ab6d0b5-79bc-45d5-8b77-75f5d5fd49cd" containerID="bae4a2d6284260a520d8bfa3ab90704ebfbf97859eeb5271321c8a0f6e85e2e8" exitCode=0 Nov 03 13:25:25.427031 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:25.426719 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3dc9b19-p5jwq" event={"ID":"2ab6d0b5-79bc-45d5-8b77-75f5d5fd49cd","Type":"ContainerDied","Data":"bae4a2d6284260a520d8bfa3ab90704ebfbf97859eeb5271321c8a0f6e85e2e8"} Nov 03 13:25:25.428189 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:25.428168 5979 generic.go:356] "Generic (PLEG): container finished" podID="f23a7bba-1d47-4cb9-853a-5adbe0e34430" containerID="0411fea64680cc5a963fe7383b8ff76160cca398c44c627598d8f77b76291621" exitCode=0 Nov 03 13:25:25.428296 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:25.428257 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67e1b40-bdr95" event={"ID":"f23a7bba-1d47-4cb9-853a-5adbe0e34430","Type":"ContainerDied","Data":"0411fea64680cc5a963fe7383b8ff76160cca398c44c627598d8f77b76291621"} Nov 03 13:25:26.395053 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:26.395021 5979 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 13:25:26.395227 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:26.395072 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd" podUID="3e80110d-950d-452f-9fdf-eecb56840a3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 13:25:26.677171 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:26.677144 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3dc9b19-p5jwq" Nov 03 13:25:26.703454 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:26.703429 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67e1b40-bdr95" Nov 03 13:25:26.734288 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:26.734261 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ab6d0b5-79bc-45d5-8b77-75f5d5fd49cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ab6d0b5-79bc-45d5-8b77-75f5d5fd49cd-maas-api-openshift-ca\") pod \"2ab6d0b5-79bc-45d5-8b77-75f5d5fd49cd\" (UID: \"2ab6d0b5-79bc-45d5-8b77-75f5d5fd49cd\") " Nov 03 13:25:26.734411 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:26.734291 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f23a7bba-1d47-4cb9-853a-5adbe0e34430-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f23a7bba-1d47-4cb9-853a-5adbe0e34430-maas-api-openshift-ca\") pod \"f23a7bba-1d47-4cb9-853a-5adbe0e34430\" (UID: \"f23a7bba-1d47-4cb9-853a-5adbe0e34430\") " Nov 03 13:25:26.734411 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:26.734378 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f23a7bba-1d47-4cb9-853a-5adbe0e34430-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f23a7bba-1d47-4cb9-853a-5adbe0e34430-maas-api-xks-ca\") pod \"f23a7bba-1d47-4cb9-853a-5adbe0e34430\" (UID: \"f23a7bba-1d47-4cb9-853a-5adbe0e34430\") " Nov 03 13:25:26.734568 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:26.734547 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ab6d0b5-79bc-45d5-8b77-75f5d5fd49cd-kube-api-access-mn8t4\" (UniqueName: \"kubernetes.io/projected/2ab6d0b5-79bc-45d5-8b77-75f5d5fd49cd-kube-api-access-mn8t4\") pod \"2ab6d0b5-79bc-45d5-8b77-75f5d5fd49cd\" (UID: \"2ab6d0b5-79bc-45d5-8b77-75f5d5fd49cd\") " Nov 03 13:25:26.734636 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:26.734598 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f23a7bba-1d47-4cb9-853a-5adbe0e34430-kube-api-access-hv4wq\" (UniqueName: \"kubernetes.io/projected/f23a7bba-1d47-4cb9-853a-5adbe0e34430-kube-api-access-hv4wq\") pod \"f23a7bba-1d47-4cb9-853a-5adbe0e34430\" (UID: \"f23a7bba-1d47-4cb9-853a-5adbe0e34430\") " Nov 03 13:25:26.734697 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:26.734641 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ab6d0b5-79bc-45d5-8b77-75f5d5fd49cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ab6d0b5-79bc-45d5-8b77-75f5d5fd49cd-maas-api-xks-ca\") pod \"2ab6d0b5-79bc-45d5-8b77-75f5d5fd49cd\" (UID: \"2ab6d0b5-79bc-45d5-8b77-75f5d5fd49cd\") " Nov 03 13:25:26.734697 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:26.734676 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ab6d0b5-79bc-45d5-8b77-75f5d5fd49cd-maas-api-openshift-ca" pod "2ab6d0b5-79bc-45d5-8b77-75f5d5fd49cd" (UID: "2ab6d0b5-79bc-45d5-8b77-75f5d5fd49cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:25:26.734767 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:26.734704 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f23a7bba-1d47-4cb9-853a-5adbe0e34430-maas-api-openshift-ca" pod "f23a7bba-1d47-4cb9-853a-5adbe0e34430" (UID: "f23a7bba-1d47-4cb9-853a-5adbe0e34430"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:25:26.735109 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:26.735086 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ab6d0b5-79bc-45d5-8b77-75f5d5fd49cd-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:25:26.735221 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:26.735114 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f23a7bba-1d47-4cb9-853a-5adbe0e34430-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:25:26.736834 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:26.736807 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ab6d0b5-79bc-45d5-8b77-75f5d5fd49cd-maas-api-xks-ca" pod "2ab6d0b5-79bc-45d5-8b77-75f5d5fd49cd" (UID: "2ab6d0b5-79bc-45d5-8b77-75f5d5fd49cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:25:26.736936 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:26.736833 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ab6d0b5-79bc-45d5-8b77-75f5d5fd49cd-kube-api-access-mn8t4" pod "2ab6d0b5-79bc-45d5-8b77-75f5d5fd49cd" (UID: "2ab6d0b5-79bc-45d5-8b77-75f5d5fd49cd"). InnerVolumeSpecName "kube-api-access-mn8t4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:25:26.736936 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:26.736813 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f23a7bba-1d47-4cb9-853a-5adbe0e34430-kube-api-access-hv4wq" pod "f23a7bba-1d47-4cb9-853a-5adbe0e34430" (UID: "f23a7bba-1d47-4cb9-853a-5adbe0e34430"). InnerVolumeSpecName "kube-api-access-hv4wq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:25:26.736936 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:26.736920 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f23a7bba-1d47-4cb9-853a-5adbe0e34430-maas-api-xks-ca" pod "f23a7bba-1d47-4cb9-853a-5adbe0e34430" (UID: "f23a7bba-1d47-4cb9-853a-5adbe0e34430"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:25:26.835892 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:26.835850 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f23a7bba-1d47-4cb9-853a-5adbe0e34430-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:25:26.835892 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:26.835879 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mn8t4\" (UniqueName: \"kubernetes.io/projected/2ab6d0b5-79bc-45d5-8b77-75f5d5fd49cd-kube-api-access-mn8t4\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:25:26.835892 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:26.835898 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv4wq\" (UniqueName: \"kubernetes.io/projected/f23a7bba-1d47-4cb9-853a-5adbe0e34430-kube-api-access-hv4wq\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:25:26.836122 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:26.835907 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ab6d0b5-79bc-45d5-8b77-75f5d5fd49cd-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:25:27.395272 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:27.395232 5979 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 13:25:27.395432 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:27.395291 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd" podUID="3e80110d-950d-452f-9fdf-eecb56840a3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 13:25:27.441538 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:27.441509 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3dc9b19-p5jwq" Nov 03 13:25:27.441693 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:27.441503 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3dc9b19-p5jwq" event={"ID":"2ab6d0b5-79bc-45d5-8b77-75f5d5fd49cd","Type":"ContainerDied","Data":"695de040e76da6938642ffb14e969a53ccfcef13f8e1a022af0b62e7b2a6d956"} Nov 03 13:25:27.441693 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:27.441625 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="695de040e76da6938642ffb14e969a53ccfcef13f8e1a022af0b62e7b2a6d956" Nov 03 13:25:27.442864 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:27.442843 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67e1b40-bdr95" Nov 03 13:25:27.442993 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:27.442879 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f67e1b40-bdr95" event={"ID":"f23a7bba-1d47-4cb9-853a-5adbe0e34430","Type":"ContainerDied","Data":"85abfce0ab947b835e6b8c1e27f410efcd5468631644ddd1767c16837d52a26a"} Nov 03 13:25:27.442993 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:27.442922 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85abfce0ab947b835e6b8c1e27f410efcd5468631644ddd1767c16837d52a26a" Nov 03 13:25:28.394832 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:28.394799 5979 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Nov 03 13:25:28.395227 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:28.394849 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd" podUID="3e80110d-950d-452f-9fdf-eecb56840a3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Nov 03 13:25:29.399583 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:29.399551 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd" Nov 03 13:25:29.400004 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:29.399847 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd" Nov 03 13:25:29.400664 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:29.400646 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd" Nov 03 13:25:31.852906 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:31.852860 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:25:33.072494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:33.067025 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0922e14c-lhk8f"] Nov 03 13:25:33.131552 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:33.131515 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0922e14c-lhk8f"] Nov 03 13:25:34.061080 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:34.061040 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e9b2751-h8fbf"] Nov 03 13:25:34.170686 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:34.170652 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="463dd10d-31ba-4f68-af24-f68b13a04095" path="/var/lib/kubelet/pods/463dd10d-31ba-4f68-af24-f68b13a04095/volumes" Nov 03 13:25:34.178995 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:34.178966 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e9b2751-h8fbf"] Nov 03 13:25:34.300588 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:34.300547 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd" podUID="a4b4df54-0aac-419d-8bb2-3258267bc088" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 13:25:34.436639 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:34.436592 5979 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd" podUID="a4b4df54-0aac-419d-8bb2-3258267bc088" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 13:25:34.611713 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:34.611671 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3c5dbc/premium-w4-3c5dbc-kserve-577d8db86-mc72z" podUID="64d52ec5-fb72-4616-8d20-8b90043af828" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:25:34.674555 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:34.674518 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759"] Nov 03 13:25:34.675752 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:34.675726 5979 cpu_manager.go:414] "containerMap: removing container" podUID="f23a7bba-1d47-4cb9-853a-5adbe0e34430" containerName="revoke-keys" Nov 03 13:25:34.675906 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:34.675883 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f23a7bba-1d47-4cb9-853a-5adbe0e34430" containerName="revoke-keys" Nov 03 13:25:34.676052 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:34.676040 5979 cpu_manager.go:414] "containerMap: removing container" podUID="2ab6d0b5-79bc-45d5-8b77-75f5d5fd49cd" containerName="revoke-keys" Nov 03 13:25:34.676147 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:34.676133 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ab6d0b5-79bc-45d5-8b77-75f5d5fd49cd" containerName="revoke-keys" Nov 03 13:25:34.676382 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:34.676365 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="f23a7bba-1d47-4cb9-853a-5adbe0e34430" containerName="revoke-keys" Nov 03 13:25:34.676533 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:34.676519 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ab6d0b5-79bc-45d5-8b77-75f5d5fd49cd" containerName="revoke-keys" Nov 03 13:25:34.705404 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:34.705319 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759" Nov 03 13:25:34.706206 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:34.706172 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf"] Nov 03 13:25:34.780494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:34.772925 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5bb463/premium-w3-5bb463-kserve-5f6ff6795c-jxmsr" podUID="8ee26f55-5c0c-4bd8-ad9b-fb8c03641698" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 13:25:34.904501 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:34.867360 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-credential-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759\" (UID: \"dc5abfae-7745-4857-b1f1-ff7d91a65ddd\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759" Nov 03 13:25:34.904501 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:34.867421 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2shnh\" (UniqueName: \"kubernetes.io/projected/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-kube-api-access-2shnh\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759\" (UID: \"dc5abfae-7745-4857-b1f1-ff7d91a65ddd\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759" Nov 03 13:25:34.904501 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:34.867544 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-istio-podinfo\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759\" (UID: \"dc5abfae-7745-4857-b1f1-ff7d91a65ddd\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759" Nov 03 13:25:34.991544 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:34.989321 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-istio-data\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759\" (UID: \"dc5abfae-7745-4857-b1f1-ff7d91a65ddd\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759" Nov 03 13:25:34.991544 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:34.989409 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-istio-envoy\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759\" (UID: \"dc5abfae-7745-4857-b1f1-ff7d91a65ddd\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759" Nov 03 13:25:34.991544 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:34.989512 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-workload-certs\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759\" (UID: \"dc5abfae-7745-4857-b1f1-ff7d91a65ddd\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759" Nov 03 13:25:34.991544 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:34.989572 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-istio-token\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759\" (UID: \"dc5abfae-7745-4857-b1f1-ff7d91a65ddd\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759" Nov 03 13:25:34.991544 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:34.989762 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-istiod-ca-cert\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759\" (UID: \"dc5abfae-7745-4857-b1f1-ff7d91a65ddd\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759" Nov 03 13:25:34.991544 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:34.989812 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-workload-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759\" (UID: \"dc5abfae-7745-4857-b1f1-ff7d91a65ddd\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759" Nov 03 13:25:35.098193 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.098150 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-istio-podinfo\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759\" (UID: \"dc5abfae-7745-4857-b1f1-ff7d91a65ddd\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759" Nov 03 13:25:35.105499 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.105448 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-istio-data\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759\" (UID: \"dc5abfae-7745-4857-b1f1-ff7d91a65ddd\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759" Nov 03 13:25:35.105749 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.105730 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-istio-envoy\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759\" (UID: \"dc5abfae-7745-4857-b1f1-ff7d91a65ddd\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759" Nov 03 13:25:35.105933 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.105917 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-workload-certs\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759\" (UID: \"dc5abfae-7745-4857-b1f1-ff7d91a65ddd\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759" Nov 03 13:25:35.106073 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.106059 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-istio-token\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759\" (UID: \"dc5abfae-7745-4857-b1f1-ff7d91a65ddd\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759" Nov 03 13:25:35.106206 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.106191 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-istiod-ca-cert\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759\" (UID: \"dc5abfae-7745-4857-b1f1-ff7d91a65ddd\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759" Nov 03 13:25:35.106323 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.106310 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-workload-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759\" (UID: \"dc5abfae-7745-4857-b1f1-ff7d91a65ddd\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759" Nov 03 13:25:35.106530 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.106512 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-credential-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759\" (UID: \"dc5abfae-7745-4857-b1f1-ff7d91a65ddd\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759" Nov 03 13:25:35.106660 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.106644 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2shnh\" (UniqueName: \"kubernetes.io/projected/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-kube-api-access-2shnh\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759\" (UID: \"dc5abfae-7745-4857-b1f1-ff7d91a65ddd\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759" Nov 03 13:25:35.107788 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.107764 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-istio-data\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759\" (UID: \"dc5abfae-7745-4857-b1f1-ff7d91a65ddd\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759" Nov 03 13:25:35.110808 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.110755 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-istiod-ca-cert\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759\" (UID: \"dc5abfae-7745-4857-b1f1-ff7d91a65ddd\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759" Nov 03 13:25:35.111214 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.111194 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-workload-certs\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759\" (UID: \"dc5abfae-7745-4857-b1f1-ff7d91a65ddd\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759" Nov 03 13:25:35.112068 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.112021 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-credential-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759\" (UID: \"dc5abfae-7745-4857-b1f1-ff7d91a65ddd\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759" Nov 03 13:25:35.113293 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.113265 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-workload-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759\" (UID: \"dc5abfae-7745-4857-b1f1-ff7d91a65ddd\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759" Nov 03 13:25:35.118610 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.117860 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-istio-envoy\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759\" (UID: \"dc5abfae-7745-4857-b1f1-ff7d91a65ddd\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759" Nov 03 13:25:35.118610 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.118208 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-istio-podinfo\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759\" (UID: \"dc5abfae-7745-4857-b1f1-ff7d91a65ddd\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759" Nov 03 13:25:35.152499 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.152029 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759"] Nov 03 13:25:35.152499 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.152098 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf"] Nov 03 13:25:35.152499 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.152145 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk"] Nov 03 13:25:35.157067 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.155983 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf" Nov 03 13:25:35.253547 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.228844 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/620d1fb2-9a2b-4e36-819e-6997985fbc1b-workload-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf\" (UID: \"620d1fb2-9a2b-4e36-819e-6997985fbc1b\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf" Nov 03 13:25:35.253547 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.228916 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/620d1fb2-9a2b-4e36-819e-6997985fbc1b-istio-token\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf\" (UID: \"620d1fb2-9a2b-4e36-819e-6997985fbc1b\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf" Nov 03 13:25:35.253547 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.228955 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/620d1fb2-9a2b-4e36-819e-6997985fbc1b-credential-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf\" (UID: \"620d1fb2-9a2b-4e36-819e-6997985fbc1b\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf" Nov 03 13:25:35.253547 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.228973 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/620d1fb2-9a2b-4e36-819e-6997985fbc1b-istio-data\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf\" (UID: \"620d1fb2-9a2b-4e36-819e-6997985fbc1b\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf" Nov 03 13:25:35.253547 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.229069 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs9rz\" (UniqueName: \"kubernetes.io/projected/620d1fb2-9a2b-4e36-819e-6997985fbc1b-kube-api-access-cs9rz\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf\" (UID: \"620d1fb2-9a2b-4e36-819e-6997985fbc1b\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf" Nov 03 13:25:35.253547 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.229098 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/620d1fb2-9a2b-4e36-819e-6997985fbc1b-istio-envoy\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf\" (UID: \"620d1fb2-9a2b-4e36-819e-6997985fbc1b\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf" Nov 03 13:25:35.253547 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.229141 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/620d1fb2-9a2b-4e36-819e-6997985fbc1b-istiod-ca-cert\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf\" (UID: \"620d1fb2-9a2b-4e36-819e-6997985fbc1b\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf" Nov 03 13:25:35.253547 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.229194 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/620d1fb2-9a2b-4e36-819e-6997985fbc1b-workload-certs\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf\" (UID: \"620d1fb2-9a2b-4e36-819e-6997985fbc1b\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf" Nov 03 13:25:35.253547 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.229314 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/620d1fb2-9a2b-4e36-819e-6997985fbc1b-istio-podinfo\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf\" (UID: \"620d1fb2-9a2b-4e36-819e-6997985fbc1b\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf" Nov 03 13:25:35.267730 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.267693 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk"] Nov 03 13:25:35.268060 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.268038 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk" Nov 03 13:25:35.288102 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.288024 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-istio-token\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759\" (UID: \"dc5abfae-7745-4857-b1f1-ff7d91a65ddd\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759" Nov 03 13:25:35.302491 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.301913 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd" podUID="a4b4df54-0aac-419d-8bb2-3258267bc088" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:25:35.307350 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.302683 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd" Nov 03 13:25:35.310193 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.309131 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2shnh\" (UniqueName: \"kubernetes.io/projected/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-kube-api-access-2shnh\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759\" (UID: \"dc5abfae-7745-4857-b1f1-ff7d91a65ddd\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759" Nov 03 13:25:35.322764 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.322738 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd" Nov 03 13:25:35.330780 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.330746 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/620d1fb2-9a2b-4e36-819e-6997985fbc1b-workload-certs\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf\" (UID: \"620d1fb2-9a2b-4e36-819e-6997985fbc1b\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf" Nov 03 13:25:35.330887 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.330791 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-istio-podinfo\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk\" (UID: \"5211f1d9-4181-4ad0-b83a-4e4e7a556be8\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk" Nov 03 13:25:35.330887 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.330828 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-istio-token\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk\" (UID: \"5211f1d9-4181-4ad0-b83a-4e4e7a556be8\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk" Nov 03 13:25:35.331070 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.331048 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-credential-socket\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk\" (UID: \"5211f1d9-4181-4ad0-b83a-4e4e7a556be8\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk" Nov 03 13:25:35.331133 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.331109 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/620d1fb2-9a2b-4e36-819e-6997985fbc1b-workload-certs\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf\" (UID: \"620d1fb2-9a2b-4e36-819e-6997985fbc1b\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf" Nov 03 13:25:35.331193 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.331131 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-workload-certs\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk\" (UID: \"5211f1d9-4181-4ad0-b83a-4e4e7a556be8\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk" Nov 03 13:25:35.331247 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.331192 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/620d1fb2-9a2b-4e36-819e-6997985fbc1b-istio-podinfo\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf\" (UID: \"620d1fb2-9a2b-4e36-819e-6997985fbc1b\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf" Nov 03 13:25:35.331247 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.331230 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-workload-socket\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk\" (UID: \"5211f1d9-4181-4ad0-b83a-4e4e7a556be8\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk" Nov 03 13:25:35.331366 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.331344 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/620d1fb2-9a2b-4e36-819e-6997985fbc1b-workload-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf\" (UID: \"620d1fb2-9a2b-4e36-819e-6997985fbc1b\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf" Nov 03 13:25:35.331420 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.331384 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-istio-envoy\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk\" (UID: \"5211f1d9-4181-4ad0-b83a-4e4e7a556be8\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk" Nov 03 13:25:35.331485 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.331427 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/620d1fb2-9a2b-4e36-819e-6997985fbc1b-istio-token\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf\" (UID: \"620d1fb2-9a2b-4e36-819e-6997985fbc1b\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf" Nov 03 13:25:35.331547 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.331513 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/620d1fb2-9a2b-4e36-819e-6997985fbc1b-credential-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf\" (UID: \"620d1fb2-9a2b-4e36-819e-6997985fbc1b\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf" Nov 03 13:25:35.331603 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.331542 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/620d1fb2-9a2b-4e36-819e-6997985fbc1b-istio-data\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf\" (UID: \"620d1fb2-9a2b-4e36-819e-6997985fbc1b\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf" Nov 03 13:25:35.331748 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.331723 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-istiod-ca-cert\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk\" (UID: \"5211f1d9-4181-4ad0-b83a-4e4e7a556be8\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk" Nov 03 13:25:35.331822 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.331767 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-istio-data\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk\" (UID: \"5211f1d9-4181-4ad0-b83a-4e4e7a556be8\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk" Nov 03 13:25:35.331822 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.331797 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k8dt\" (UniqueName: \"kubernetes.io/projected/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-kube-api-access-2k8dt\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk\" (UID: \"5211f1d9-4181-4ad0-b83a-4e4e7a556be8\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk" Nov 03 13:25:35.331945 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.331860 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs9rz\" (UniqueName: \"kubernetes.io/projected/620d1fb2-9a2b-4e36-819e-6997985fbc1b-kube-api-access-cs9rz\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf\" (UID: \"620d1fb2-9a2b-4e36-819e-6997985fbc1b\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf" Nov 03 13:25:35.331945 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.331903 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/620d1fb2-9a2b-4e36-819e-6997985fbc1b-istio-envoy\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf\" (UID: \"620d1fb2-9a2b-4e36-819e-6997985fbc1b\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf" Nov 03 13:25:35.332053 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.331962 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/620d1fb2-9a2b-4e36-819e-6997985fbc1b-istiod-ca-cert\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf\" (UID: \"620d1fb2-9a2b-4e36-819e-6997985fbc1b\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf" Nov 03 13:25:35.332806 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.332582 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/620d1fb2-9a2b-4e36-819e-6997985fbc1b-credential-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf\" (UID: \"620d1fb2-9a2b-4e36-819e-6997985fbc1b\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf" Nov 03 13:25:35.333699 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.333266 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/620d1fb2-9a2b-4e36-819e-6997985fbc1b-istio-data\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf\" (UID: \"620d1fb2-9a2b-4e36-819e-6997985fbc1b\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf" Nov 03 13:25:35.333699 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.333445 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/620d1fb2-9a2b-4e36-819e-6997985fbc1b-istiod-ca-cert\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf\" (UID: \"620d1fb2-9a2b-4e36-819e-6997985fbc1b\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf" Nov 03 13:25:35.333699 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.333530 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/620d1fb2-9a2b-4e36-819e-6997985fbc1b-workload-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf\" (UID: \"620d1fb2-9a2b-4e36-819e-6997985fbc1b\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf" Nov 03 13:25:35.334526 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.334497 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/620d1fb2-9a2b-4e36-819e-6997985fbc1b-istio-envoy\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf\" (UID: \"620d1fb2-9a2b-4e36-819e-6997985fbc1b\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf" Nov 03 13:25:35.336022 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.336001 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/620d1fb2-9a2b-4e36-819e-6997985fbc1b-istio-podinfo\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf\" (UID: \"620d1fb2-9a2b-4e36-819e-6997985fbc1b\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf" Nov 03 13:25:35.385567 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.385532 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/620d1fb2-9a2b-4e36-819e-6997985fbc1b-istio-token\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf\" (UID: \"620d1fb2-9a2b-4e36-819e-6997985fbc1b\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf" Nov 03 13:25:35.385698 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.385574 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs9rz\" (UniqueName: \"kubernetes.io/projected/620d1fb2-9a2b-4e36-819e-6997985fbc1b-kube-api-access-cs9rz\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf\" (UID: \"620d1fb2-9a2b-4e36-819e-6997985fbc1b\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf" Nov 03 13:25:35.433411 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.433381 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-istio-podinfo\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk\" (UID: \"5211f1d9-4181-4ad0-b83a-4e4e7a556be8\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk" Nov 03 13:25:35.433622 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.433418 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-istio-token\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk\" (UID: \"5211f1d9-4181-4ad0-b83a-4e4e7a556be8\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk" Nov 03 13:25:35.433622 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.433553 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-credential-socket\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk\" (UID: \"5211f1d9-4181-4ad0-b83a-4e4e7a556be8\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk" Nov 03 13:25:35.433622 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.433604 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-workload-certs\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk\" (UID: \"5211f1d9-4181-4ad0-b83a-4e4e7a556be8\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk" Nov 03 13:25:35.433815 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.433661 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-workload-socket\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk\" (UID: \"5211f1d9-4181-4ad0-b83a-4e4e7a556be8\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk" Nov 03 13:25:35.433815 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.433704 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-istio-envoy\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk\" (UID: \"5211f1d9-4181-4ad0-b83a-4e4e7a556be8\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk" Nov 03 13:25:35.433815 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.433803 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-istiod-ca-cert\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk\" (UID: \"5211f1d9-4181-4ad0-b83a-4e4e7a556be8\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk" Nov 03 13:25:35.433980 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.433832 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-istio-data\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk\" (UID: \"5211f1d9-4181-4ad0-b83a-4e4e7a556be8\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk" Nov 03 13:25:35.433980 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.433857 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k8dt\" (UniqueName: \"kubernetes.io/projected/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-kube-api-access-2k8dt\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk\" (UID: \"5211f1d9-4181-4ad0-b83a-4e4e7a556be8\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk" Nov 03 13:25:35.433980 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.433948 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-credential-socket\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk\" (UID: \"5211f1d9-4181-4ad0-b83a-4e4e7a556be8\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk" Nov 03 13:25:35.434133 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.434034 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-workload-certs\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk\" (UID: \"5211f1d9-4181-4ad0-b83a-4e4e7a556be8\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk" Nov 03 13:25:35.434257 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.434229 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-istio-data\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk\" (UID: \"5211f1d9-4181-4ad0-b83a-4e4e7a556be8\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk" Nov 03 13:25:35.434391 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.434358 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-workload-socket\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk\" (UID: \"5211f1d9-4181-4ad0-b83a-4e4e7a556be8\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk" Nov 03 13:25:35.434544 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.434521 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-istiod-ca-cert\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk\" (UID: \"5211f1d9-4181-4ad0-b83a-4e4e7a556be8\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk" Nov 03 13:25:35.435905 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.435876 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-istio-podinfo\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk\" (UID: \"5211f1d9-4181-4ad0-b83a-4e4e7a556be8\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk" Nov 03 13:25:35.436313 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.436292 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-istio-envoy\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk\" (UID: \"5211f1d9-4181-4ad0-b83a-4e4e7a556be8\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk" Nov 03 13:25:35.482526 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.482497 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c33b5d69-fh6bc"] Nov 03 13:25:35.487516 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.487489 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-istio-token\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk\" (UID: \"5211f1d9-4181-4ad0-b83a-4e4e7a556be8\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk" Nov 03 13:25:35.487642 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.487624 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k8dt\" (UniqueName: \"kubernetes.io/projected/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-kube-api-access-2k8dt\") pod \"e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk\" (UID: \"5211f1d9-4181-4ad0-b83a-4e4e7a556be8\") " pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk" Nov 03 13:25:35.488011 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.487997 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c33b5d69-fh6bc" Nov 03 13:25:35.534284 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.534221 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759" Nov 03 13:25:35.534656 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.534635 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csdc8\" (UniqueName: \"kubernetes.io/projected/be076614-167d-49f2-850a-8649432fb5dc-kube-api-access-csdc8\") pod \"maas-api-revoke-sub-c33b5d69-fh6bc\" (UID: \"be076614-167d-49f2-850a-8649432fb5dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c33b5d69-fh6bc" Nov 03 13:25:35.534782 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.534761 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be076614-167d-49f2-850a-8649432fb5dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c33b5d69-fh6bc\" (UID: \"be076614-167d-49f2-850a-8649432fb5dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c33b5d69-fh6bc" Nov 03 13:25:35.534825 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.534805 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be076614-167d-49f2-850a-8649432fb5dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c33b5d69-fh6bc\" (UID: \"be076614-167d-49f2-850a-8649432fb5dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c33b5d69-fh6bc" Nov 03 13:25:35.538162 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.537698 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:25:35.538162 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.537761 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 13:25:35.538162 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.537895 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 13:25:35.538162 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.537978 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:25:35.541304 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.541278 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c33b5d69-fh6bc"] Nov 03 13:25:35.572881 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.572841 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf" Nov 03 13:25:35.611765 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.611484 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk" Nov 03 13:25:35.636672 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.636525 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be076614-167d-49f2-850a-8649432fb5dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c33b5d69-fh6bc\" (UID: \"be076614-167d-49f2-850a-8649432fb5dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c33b5d69-fh6bc" Nov 03 13:25:35.636672 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.636583 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be076614-167d-49f2-850a-8649432fb5dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c33b5d69-fh6bc\" (UID: \"be076614-167d-49f2-850a-8649432fb5dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c33b5d69-fh6bc" Nov 03 13:25:35.637381 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.636922 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csdc8\" (UniqueName: \"kubernetes.io/projected/be076614-167d-49f2-850a-8649432fb5dc-kube-api-access-csdc8\") pod \"maas-api-revoke-sub-c33b5d69-fh6bc\" (UID: \"be076614-167d-49f2-850a-8649432fb5dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c33b5d69-fh6bc" Nov 03 13:25:35.640174 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.640150 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be076614-167d-49f2-850a-8649432fb5dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c33b5d69-fh6bc\" (UID: \"be076614-167d-49f2-850a-8649432fb5dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c33b5d69-fh6bc" Nov 03 13:25:35.648683 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.648642 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be076614-167d-49f2-850a-8649432fb5dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c33b5d69-fh6bc\" (UID: \"be076614-167d-49f2-850a-8649432fb5dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c33b5d69-fh6bc" Nov 03 13:25:35.698513 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.698301 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csdc8\" (UniqueName: \"kubernetes.io/projected/be076614-167d-49f2-850a-8649432fb5dc-kube-api-access-csdc8\") pod \"maas-api-revoke-sub-c33b5d69-fh6bc\" (UID: \"be076614-167d-49f2-850a-8649432fb5dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c33b5d69-fh6bc" Nov 03 13:25:35.802142 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.802058 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c33b5d69-fh6bc" Nov 03 13:25:35.980417 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.980372 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759"] Nov 03 13:25:35.984271 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.984245 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf"] Nov 03 13:25:35.986652 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:25:35.986626 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc5abfae_7745_4857_b1f1_ff7d91a65ddd.slice/crio-f8d2769f98f5443810d1295886aa52ddcfc418b8cd93ccdbc0a12e4b5ad5956a WatchSource:0}: Error finding container f8d2769f98f5443810d1295886aa52ddcfc418b8cd93ccdbc0a12e4b5ad5956a: Status 404 returned error can't find the container with id f8d2769f98f5443810d1295886aa52ddcfc418b8cd93ccdbc0a12e4b5ad5956a Nov 03 13:25:35.990834 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:25:35.990803 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod620d1fb2_9a2b_4e36_819e_6997985fbc1b.slice/crio-3dd7e62c0a6bf29bb265c538a2dd0d47f4232abd5726e83e136f70bd32fc4c7c WatchSource:0}: Error finding container 3dd7e62c0a6bf29bb265c538a2dd0d47f4232abd5726e83e136f70bd32fc4c7c: Status 404 returned error can't find the container with id 3dd7e62c0a6bf29bb265c538a2dd0d47f4232abd5726e83e136f70bd32fc4c7c Nov 03 13:25:35.991407 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.991372 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:25:35.991499 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.991459 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:25:35.991547 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.991526 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:25:35.993694 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.993663 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:25:35.993795 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.993723 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:25:35.993795 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:35.993755 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:25:36.057949 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:36.055037 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk"] Nov 03 13:25:36.096511 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:36.077534 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:25:36.096511 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:36.077628 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:25:36.096511 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:36.077706 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:25:36.135334 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:36.135275 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="289c69a7-8c3a-4376-8634-fe80d013f1bf" path="/var/lib/kubelet/pods/289c69a7-8c3a-4376-8634-fe80d013f1bf/volumes" Nov 03 13:25:36.212095 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:36.212061 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c33b5d69-fh6bc"] Nov 03 13:25:36.620523 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:36.620429 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759" event={"ID":"dc5abfae-7745-4857-b1f1-ff7d91a65ddd","Type":"ContainerStarted","Data":"38ee84a1ef40cc675de03cf7b86ef2a79808c566217ca0644e0c478c3482ee45"} Nov 03 13:25:36.621081 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:36.621053 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759" event={"ID":"dc5abfae-7745-4857-b1f1-ff7d91a65ddd","Type":"ContainerStarted","Data":"f8d2769f98f5443810d1295886aa52ddcfc418b8cd93ccdbc0a12e4b5ad5956a"} Nov 03 13:25:36.640984 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:36.640946 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk" event={"ID":"5211f1d9-4181-4ad0-b83a-4e4e7a556be8","Type":"ContainerStarted","Data":"95d28afb5b88bab8ea99934962ff3b61b20bda95db7fa5baff3736a423a418b8"} Nov 03 13:25:36.642374 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:36.642156 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk" event={"ID":"5211f1d9-4181-4ad0-b83a-4e4e7a556be8","Type":"ContainerStarted","Data":"004028a21494b5acfd90ae1a7b19f0c014a7b1a1a0c3efffa8d681cc709b042e"} Nov 03 13:25:36.645135 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:36.645111 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c33b5d69-fh6bc" event={"ID":"be076614-167d-49f2-850a-8649432fb5dc","Type":"ContainerStarted","Data":"6104abbaa3038caca876a68135e5fa34805ea0cf620410ec5323a3a45381c893"} Nov 03 13:25:36.651019 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:36.650233 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf" event={"ID":"620d1fb2-9a2b-4e36-819e-6997985fbc1b","Type":"ContainerStarted","Data":"034a9e2816fda727fe7e62094aca88d7049e132615fc3cdd1ec916d33882dc92"} Nov 03 13:25:36.651019 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:36.650284 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf" event={"ID":"620d1fb2-9a2b-4e36-819e-6997985fbc1b","Type":"ContainerStarted","Data":"3dd7e62c0a6bf29bb265c538a2dd0d47f4232abd5726e83e136f70bd32fc4c7c"} Nov 03 13:25:36.942693 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:36.936892 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759" podStartSLOduration=2.936859341 podStartE2EDuration="2.936859341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:25:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:25:36.757864654 +0000 UTC m=+1218.768552897" watchObservedRunningTime="2026-11-03 13:25:36.936859341 +0000 UTC m=+1218.947547584" Nov 03 13:25:37.050981 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:37.050887 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf" podStartSLOduration=3.050863997 podStartE2EDuration="3.050863997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:25:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:25:36.934669865 +0000 UTC m=+1218.945358119" watchObservedRunningTime="2026-11-03 13:25:37.050863997 +0000 UTC m=+1219.061552241" Nov 03 13:25:37.199554 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:37.184079 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk" podStartSLOduration=3.184057024 podStartE2EDuration="3.184057024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:25:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:25:37.150995263 +0000 UTC m=+1219.161683505" watchObservedRunningTime="2026-11-03 13:25:37.184057024 +0000 UTC m=+1219.194745266" Nov 03 13:25:37.468048 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:25:37.467921 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:25:37.536843 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:37.536243 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759" Nov 03 13:25:37.536843 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:37.536373 5979 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 03 13:25:37.536843 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:37.536414 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759" podUID="dc5abfae-7745-4857-b1f1-ff7d91a65ddd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 03 13:25:37.576167 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:37.575743 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf" Nov 03 13:25:37.576167 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:37.575878 5979 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 13:25:37.576167 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:37.575929 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf" podUID="620d1fb2-9a2b-4e36-819e-6997985fbc1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 13:25:37.623871 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:37.616900 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk" Nov 03 13:25:37.626569 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:37.626063 5979 patch_prober.go:29] interesting pod/e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 13:25:37.626569 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:37.626118 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk" podUID="5211f1d9-4181-4ad0-b83a-4e4e7a556be8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 13:25:37.873079 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:37.871840 5979 generic.go:356] "Generic (PLEG): container finished" podID="be076614-167d-49f2-850a-8649432fb5dc" containerID="48695bbce91e8b1303288143ad4dc1868cdb7e734e566d92b8401433ffa013e3" exitCode=0 Nov 03 13:25:37.873079 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:37.872764 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c33b5d69-fh6bc" event={"ID":"be076614-167d-49f2-850a-8649432fb5dc","Type":"ContainerDied","Data":"48695bbce91e8b1303288143ad4dc1868cdb7e734e566d92b8401433ffa013e3"} Nov 03 13:25:38.535480 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:38.535422 5979 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 03 13:25:38.535639 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:38.535504 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759" podUID="dc5abfae-7745-4857-b1f1-ff7d91a65ddd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 03 13:25:38.580502 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:38.578915 5979 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 13:25:38.580502 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:38.578973 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf" podUID="620d1fb2-9a2b-4e36-819e-6997985fbc1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 13:25:38.617497 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:38.616524 5979 patch_prober.go:29] interesting pod/e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 13:25:38.617497 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:38.616588 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk" podUID="5211f1d9-4181-4ad0-b83a-4e4e7a556be8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 13:25:39.540936 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:39.540903 5979 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 03 13:25:39.541605 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:39.541569 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759" podUID="dc5abfae-7745-4857-b1f1-ff7d91a65ddd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 03 13:25:39.579097 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:39.575861 5979 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 13:25:39.579097 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:39.575927 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf" podUID="620d1fb2-9a2b-4e36-819e-6997985fbc1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 13:25:39.579097 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:39.576951 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c33b5d69-fh6bc" Nov 03 13:25:39.612939 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:39.612903 5979 patch_prober.go:29] interesting pod/e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 13:25:39.613226 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:39.613179 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk" podUID="5211f1d9-4181-4ad0-b83a-4e4e7a556be8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 13:25:39.719710 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:39.719623 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be076614-167d-49f2-850a-8649432fb5dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be076614-167d-49f2-850a-8649432fb5dc-maas-api-openshift-ca\") pod \"be076614-167d-49f2-850a-8649432fb5dc\" (UID: \"be076614-167d-49f2-850a-8649432fb5dc\") " Nov 03 13:25:39.720092 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:39.720066 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be076614-167d-49f2-850a-8649432fb5dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be076614-167d-49f2-850a-8649432fb5dc-maas-api-xks-ca\") pod \"be076614-167d-49f2-850a-8649432fb5dc\" (UID: \"be076614-167d-49f2-850a-8649432fb5dc\") " Nov 03 13:25:39.720231 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:39.720216 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be076614-167d-49f2-850a-8649432fb5dc-kube-api-access-csdc8\" (UniqueName: \"kubernetes.io/projected/be076614-167d-49f2-850a-8649432fb5dc-kube-api-access-csdc8\") pod \"be076614-167d-49f2-850a-8649432fb5dc\" (UID: \"be076614-167d-49f2-850a-8649432fb5dc\") " Nov 03 13:25:39.722678 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:39.722650 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be076614-167d-49f2-850a-8649432fb5dc-maas-api-openshift-ca" pod "be076614-167d-49f2-850a-8649432fb5dc" (UID: "be076614-167d-49f2-850a-8649432fb5dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:25:39.743228 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:39.743171 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be076614-167d-49f2-850a-8649432fb5dc-maas-api-xks-ca" pod "be076614-167d-49f2-850a-8649432fb5dc" (UID: "be076614-167d-49f2-850a-8649432fb5dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:25:39.762293 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:39.762240 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be076614-167d-49f2-850a-8649432fb5dc-kube-api-access-csdc8" pod "be076614-167d-49f2-850a-8649432fb5dc" (UID: "be076614-167d-49f2-850a-8649432fb5dc"). InnerVolumeSpecName "kube-api-access-csdc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:25:39.831019 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:39.830947 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be076614-167d-49f2-850a-8649432fb5dc-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:25:39.831019 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:39.830976 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csdc8\" (UniqueName: \"kubernetes.io/projected/be076614-167d-49f2-850a-8649432fb5dc-kube-api-access-csdc8\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:25:39.831019 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:39.830986 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be076614-167d-49f2-850a-8649432fb5dc-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:25:39.927532 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:39.927495 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c33b5d69-fh6bc" Nov 03 13:25:39.927715 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:39.927518 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c33b5d69-fh6bc" event={"ID":"be076614-167d-49f2-850a-8649432fb5dc","Type":"ContainerDied","Data":"6104abbaa3038caca876a68135e5fa34805ea0cf620410ec5323a3a45381c893"} Nov 03 13:25:39.927715 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:39.927583 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6104abbaa3038caca876a68135e5fa34805ea0cf620410ec5323a3a45381c893" Nov 03 13:25:40.534670 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:40.534639 5979 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 03 13:25:40.534816 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:40.534688 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759" podUID="dc5abfae-7745-4857-b1f1-ff7d91a65ddd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 03 13:25:40.577943 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:40.577901 5979 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 13:25:40.578590 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:40.578522 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf" podUID="620d1fb2-9a2b-4e36-819e-6997985fbc1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 13:25:40.615186 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:40.615060 5979 patch_prober.go:29] interesting pod/e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 13:25:40.615186 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:40.615123 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk" podUID="5211f1d9-4181-4ad0-b83a-4e4e7a556be8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 13:25:41.240072 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:41.240038 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7"] Nov 03 13:25:41.240646 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:41.240630 5979 cpu_manager.go:414] "containerMap: removing container" podUID="be076614-167d-49f2-850a-8649432fb5dc" containerName="revoke-keys" Nov 03 13:25:41.240716 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:41.240646 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be076614-167d-49f2-850a-8649432fb5dc" containerName="revoke-keys" Nov 03 13:25:41.240770 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:41.240761 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="be076614-167d-49f2-850a-8649432fb5dc" containerName="revoke-keys" Nov 03 13:25:41.265496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:41.258758 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7" Nov 03 13:25:41.348257 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:41.344604 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7"] Nov 03 13:25:41.437081 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:41.437037 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f4a018c-c28f-4a81-b727-e276ae0156d3-workload-certs\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7\" (UID: \"1f4a018c-c28f-4a81-b727-e276ae0156d3\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7" Nov 03 13:25:41.437296 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:41.437092 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f4a018c-c28f-4a81-b727-e276ae0156d3-workload-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7\" (UID: \"1f4a018c-c28f-4a81-b727-e276ae0156d3\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7" Nov 03 13:25:41.437296 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:41.437123 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g74z\" (UniqueName: \"kubernetes.io/projected/1f4a018c-c28f-4a81-b727-e276ae0156d3-kube-api-access-9g74z\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7\" (UID: \"1f4a018c-c28f-4a81-b727-e276ae0156d3\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7" Nov 03 13:25:41.437296 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:41.437154 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f4a018c-c28f-4a81-b727-e276ae0156d3-istio-envoy\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7\" (UID: \"1f4a018c-c28f-4a81-b727-e276ae0156d3\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7" Nov 03 13:25:41.437296 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:41.437218 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f4a018c-c28f-4a81-b727-e276ae0156d3-credential-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7\" (UID: \"1f4a018c-c28f-4a81-b727-e276ae0156d3\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7" Nov 03 13:25:41.437296 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:41.437263 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f4a018c-c28f-4a81-b727-e276ae0156d3-istio-podinfo\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7\" (UID: \"1f4a018c-c28f-4a81-b727-e276ae0156d3\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7" Nov 03 13:25:41.437296 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:41.437289 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f4a018c-c28f-4a81-b727-e276ae0156d3-istiod-ca-cert\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7\" (UID: \"1f4a018c-c28f-4a81-b727-e276ae0156d3\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7" Nov 03 13:25:41.437686 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:41.437356 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f4a018c-c28f-4a81-b727-e276ae0156d3-istio-data\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7\" (UID: \"1f4a018c-c28f-4a81-b727-e276ae0156d3\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7" Nov 03 13:25:41.437686 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:41.437494 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f4a018c-c28f-4a81-b727-e276ae0156d3-istio-token\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7\" (UID: \"1f4a018c-c28f-4a81-b727-e276ae0156d3\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7" Nov 03 13:25:41.535457 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:41.535384 5979 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 03 13:25:41.535457 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:41.535435 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759" podUID="dc5abfae-7745-4857-b1f1-ff7d91a65ddd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 03 13:25:41.539078 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:41.539047 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f4a018c-c28f-4a81-b727-e276ae0156d3-istio-token\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7\" (UID: \"1f4a018c-c28f-4a81-b727-e276ae0156d3\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7" Nov 03 13:25:41.539217 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:41.539136 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f4a018c-c28f-4a81-b727-e276ae0156d3-workload-certs\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7\" (UID: \"1f4a018c-c28f-4a81-b727-e276ae0156d3\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7" Nov 03 13:25:41.539217 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:41.539155 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f4a018c-c28f-4a81-b727-e276ae0156d3-workload-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7\" (UID: \"1f4a018c-c28f-4a81-b727-e276ae0156d3\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7" Nov 03 13:25:41.539217 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:41.539173 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g74z\" (UniqueName: \"kubernetes.io/projected/1f4a018c-c28f-4a81-b727-e276ae0156d3-kube-api-access-9g74z\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7\" (UID: \"1f4a018c-c28f-4a81-b727-e276ae0156d3\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7" Nov 03 13:25:41.539217 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:41.539193 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f4a018c-c28f-4a81-b727-e276ae0156d3-istio-envoy\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7\" (UID: \"1f4a018c-c28f-4a81-b727-e276ae0156d3\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7" Nov 03 13:25:41.539433 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:41.539232 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f4a018c-c28f-4a81-b727-e276ae0156d3-credential-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7\" (UID: \"1f4a018c-c28f-4a81-b727-e276ae0156d3\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7" Nov 03 13:25:41.539433 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:41.539256 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f4a018c-c28f-4a81-b727-e276ae0156d3-istio-podinfo\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7\" (UID: \"1f4a018c-c28f-4a81-b727-e276ae0156d3\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7" Nov 03 13:25:41.539433 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:41.539273 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f4a018c-c28f-4a81-b727-e276ae0156d3-istiod-ca-cert\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7\" (UID: \"1f4a018c-c28f-4a81-b727-e276ae0156d3\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7" Nov 03 13:25:41.539433 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:41.539315 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f4a018c-c28f-4a81-b727-e276ae0156d3-istio-data\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7\" (UID: \"1f4a018c-c28f-4a81-b727-e276ae0156d3\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7" Nov 03 13:25:41.539751 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:41.539731 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f4a018c-c28f-4a81-b727-e276ae0156d3-istio-data\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7\" (UID: \"1f4a018c-c28f-4a81-b727-e276ae0156d3\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7" Nov 03 13:25:41.539751 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:41.539745 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f4a018c-c28f-4a81-b727-e276ae0156d3-credential-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7\" (UID: \"1f4a018c-c28f-4a81-b727-e276ae0156d3\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7" Nov 03 13:25:41.539929 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:41.539905 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f4a018c-c28f-4a81-b727-e276ae0156d3-workload-certs\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7\" (UID: \"1f4a018c-c28f-4a81-b727-e276ae0156d3\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7" Nov 03 13:25:41.540262 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:41.540239 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f4a018c-c28f-4a81-b727-e276ae0156d3-istiod-ca-cert\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7\" (UID: \"1f4a018c-c28f-4a81-b727-e276ae0156d3\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7" Nov 03 13:25:41.540457 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:41.540440 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f4a018c-c28f-4a81-b727-e276ae0156d3-workload-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7\" (UID: \"1f4a018c-c28f-4a81-b727-e276ae0156d3\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7" Nov 03 13:25:41.542070 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:41.542050 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f4a018c-c28f-4a81-b727-e276ae0156d3-istio-podinfo\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7\" (UID: \"1f4a018c-c28f-4a81-b727-e276ae0156d3\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7" Nov 03 13:25:41.542600 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:41.542575 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f4a018c-c28f-4a81-b727-e276ae0156d3-istio-envoy\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7\" (UID: \"1f4a018c-c28f-4a81-b727-e276ae0156d3\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7" Nov 03 13:25:41.594511 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:41.575864 5979 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 13:25:41.594511 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:41.575938 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf" podUID="620d1fb2-9a2b-4e36-819e-6997985fbc1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 13:25:41.619510 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:41.616864 5979 patch_prober.go:29] interesting pod/e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 13:25:41.619510 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:41.616924 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk" podUID="5211f1d9-4181-4ad0-b83a-4e4e7a556be8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 13:25:41.626219 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:41.626186 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f4a018c-c28f-4a81-b727-e276ae0156d3-istio-token\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7\" (UID: \"1f4a018c-c28f-4a81-b727-e276ae0156d3\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7" Nov 03 13:25:41.627091 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:41.627065 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g74z\" (UniqueName: \"kubernetes.io/projected/1f4a018c-c28f-4a81-b727-e276ae0156d3-kube-api-access-9g74z\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7\" (UID: \"1f4a018c-c28f-4a81-b727-e276ae0156d3\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7" Nov 03 13:25:41.891504 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:41.889716 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7" Nov 03 13:25:42.540101 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:42.540020 5979 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 03 13:25:42.620496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:42.616625 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759" podUID="dc5abfae-7745-4857-b1f1-ff7d91a65ddd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 03 13:25:42.637495 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:42.630638 5979 patch_prober.go:29] interesting pod/e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 13:25:42.637495 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:42.630701 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk" podUID="5211f1d9-4181-4ad0-b83a-4e4e7a556be8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 13:25:42.637495 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:42.630912 5979 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 13:25:42.637495 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:42.630938 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf" podUID="620d1fb2-9a2b-4e36-819e-6997985fbc1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 13:25:42.917433 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:42.917383 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:25:42.917588 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:42.917535 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:25:42.917588 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:42.917572 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:25:42.919406 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:42.919213 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7"] Nov 03 13:25:42.995232 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:42.995134 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7" event={"ID":"1f4a018c-c28f-4a81-b727-e276ae0156d3","Type":"ContainerStarted","Data":"ebf5c2674be0f886fa7804b6e6024414c3251984d8938a5038fbefddcade86fa"} Nov 03 13:25:43.120158 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:43.120108 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8575f973-dm7n6"] Nov 03 13:25:43.131677 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:43.131232 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8575f973-dm7n6" Nov 03 13:25:43.194535 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:43.192210 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 13:25:43.194822 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:43.194710 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 13:25:43.194983 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:43.194968 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:25:43.195643 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:43.195614 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:25:43.198013 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:43.197982 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8575f973-dm7n6"] Nov 03 13:25:43.237949 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:43.235822 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lffln\" (UniqueName: \"kubernetes.io/projected/731dc017-0497-42c3-9a66-001519ede38b-kube-api-access-lffln\") pod \"maas-api-revoke-sub-8575f973-dm7n6\" (UID: \"731dc017-0497-42c3-9a66-001519ede38b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8575f973-dm7n6" Nov 03 13:25:43.237949 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:43.235980 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/731dc017-0497-42c3-9a66-001519ede38b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8575f973-dm7n6\" (UID: \"731dc017-0497-42c3-9a66-001519ede38b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8575f973-dm7n6" Nov 03 13:25:43.237949 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:43.236112 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/731dc017-0497-42c3-9a66-001519ede38b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8575f973-dm7n6\" (UID: \"731dc017-0497-42c3-9a66-001519ede38b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8575f973-dm7n6" Nov 03 13:25:43.337802 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:43.337762 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lffln\" (UniqueName: \"kubernetes.io/projected/731dc017-0497-42c3-9a66-001519ede38b-kube-api-access-lffln\") pod \"maas-api-revoke-sub-8575f973-dm7n6\" (UID: \"731dc017-0497-42c3-9a66-001519ede38b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8575f973-dm7n6" Nov 03 13:25:43.337973 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:43.337820 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/731dc017-0497-42c3-9a66-001519ede38b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8575f973-dm7n6\" (UID: \"731dc017-0497-42c3-9a66-001519ede38b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8575f973-dm7n6" Nov 03 13:25:43.337973 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:43.337904 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/731dc017-0497-42c3-9a66-001519ede38b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8575f973-dm7n6\" (UID: \"731dc017-0497-42c3-9a66-001519ede38b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8575f973-dm7n6" Nov 03 13:25:43.339634 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:43.338880 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/731dc017-0497-42c3-9a66-001519ede38b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8575f973-dm7n6\" (UID: \"731dc017-0497-42c3-9a66-001519ede38b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8575f973-dm7n6" Nov 03 13:25:43.344126 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:43.344100 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/731dc017-0497-42c3-9a66-001519ede38b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8575f973-dm7n6\" (UID: \"731dc017-0497-42c3-9a66-001519ede38b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8575f973-dm7n6" Nov 03 13:25:43.406254 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:43.406178 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lffln\" (UniqueName: \"kubernetes.io/projected/731dc017-0497-42c3-9a66-001519ede38b-kube-api-access-lffln\") pod \"maas-api-revoke-sub-8575f973-dm7n6\" (UID: \"731dc017-0497-42c3-9a66-001519ede38b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8575f973-dm7n6" Nov 03 13:25:43.477686 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:43.477651 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8575f973-dm7n6" Nov 03 13:25:43.535490 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:43.535427 5979 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Nov 03 13:25:43.536431 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:43.535730 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759" podUID="dc5abfae-7745-4857-b1f1-ff7d91a65ddd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Nov 03 13:25:43.578033 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:43.576930 5979 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Nov 03 13:25:43.578033 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:43.576989 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf" podUID="620d1fb2-9a2b-4e36-819e-6997985fbc1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Nov 03 13:25:43.636887 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:43.627538 5979 patch_prober.go:29] interesting pod/e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Nov 03 13:25:43.636887 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:43.627597 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk" podUID="5211f1d9-4181-4ad0-b83a-4e4e7a556be8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Nov 03 13:25:43.928634 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:43.928588 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8575f973-dm7n6"] Nov 03 13:25:44.009548 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:44.009514 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8575f973-dm7n6" event={"ID":"731dc017-0497-42c3-9a66-001519ede38b","Type":"ContainerStarted","Data":"04df778dfe49428647e06e512342cf7d814ef009f0dacba90f7d13b544ee1df3"} Nov 03 13:25:44.012157 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:44.012131 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7" event={"ID":"1f4a018c-c28f-4a81-b727-e276ae0156d3","Type":"ContainerStarted","Data":"9edcf518b5926b6444322e10b9d784c097113d24911329ae56638bd93d64f5e2"} Nov 03 13:25:44.141597 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:44.141536 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7" podStartSLOduration=3.141517273 podStartE2EDuration="3.141517273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:25:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:25:44.140627404 +0000 UTC m=+1226.151315641" watchObservedRunningTime="2026-11-03 13:25:44.141517273 +0000 UTC m=+1226.152205516" Nov 03 13:25:44.816668 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:44.816563 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf" Nov 03 13:25:44.817939 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:44.817908 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf" Nov 03 13:25:44.819584 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:44.819555 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf" Nov 03 13:25:44.895495 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:44.894559 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7" Nov 03 13:25:44.895495 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:44.894693 5979 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 13:25:44.895495 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:44.894738 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7" podUID="1f4a018c-c28f-4a81-b727-e276ae0156d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 13:25:45.021804 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:45.021771 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8575f973-dm7n6" event={"ID":"731dc017-0497-42c3-9a66-001519ede38b","Type":"ContainerStarted","Data":"5a553281075023fd355f3b59b4919913e05d26e102701d0eb606a96bbd3e1b5b"} Nov 03 13:25:45.195530 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:45.194069 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8575f973-dm7n6" podStartSLOduration=2.194046449 podStartE2EDuration="2.194046449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:25:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:25:45.193205261 +0000 UTC m=+1227.203893506" watchObservedRunningTime="2026-11-03 13:25:45.194046449 +0000 UTC m=+1227.204734692" Nov 03 13:25:45.223333 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:45.223303 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk" Nov 03 13:25:45.223728 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:45.223704 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk" Nov 03 13:25:45.224822 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:45.224803 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk" Nov 03 13:25:45.307097 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:45.307070 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759" Nov 03 13:25:45.307406 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:45.307379 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759" Nov 03 13:25:45.308618 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:45.308599 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759" Nov 03 13:25:45.893506 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:45.892900 5979 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 13:25:45.893506 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:45.892968 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7" podUID="1f4a018c-c28f-4a81-b727-e276ae0156d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 13:25:45.995492 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:45.992878 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5295ab29-6d58657867-lbm4z"] Nov 03 13:25:46.014807 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:46.014768 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5295ab29-6d58657867-lbm4z" Nov 03 13:25:46.031393 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:46.031367 5979 generic.go:356] "Generic (PLEG): container finished" podID="731dc017-0497-42c3-9a66-001519ede38b" containerID="5a553281075023fd355f3b59b4919913e05d26e102701d0eb606a96bbd3e1b5b" exitCode=0 Nov 03 13:25:46.031553 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:46.031451 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8575f973-dm7n6" event={"ID":"731dc017-0497-42c3-9a66-001519ede38b","Type":"ContainerDied","Data":"5a553281075023fd355f3b59b4919913e05d26e102701d0eb606a96bbd3e1b5b"} Nov 03 13:25:46.042413 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:46.042380 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5295ab29-6d58657867-lbm4z"] Nov 03 13:25:46.104491 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:46.073785 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-5295ab29\"" Nov 03 13:25:46.104491 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:46.074556 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-5295ab29-dockercfg-696sf\"" Nov 03 13:25:46.137211 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:46.136655 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhms2\" (UniqueName: \"kubernetes.io/projected/8ad148b9-3b17-46b3-8dca-0d36e3b43aa3-kube-api-access-xhms2\") pod \"payload-pre-processing-e2e-mt-5295ab29-6d58657867-lbm4z\" (UID: \"8ad148b9-3b17-46b3-8dca-0d36e3b43aa3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5295ab29-6d58657867-lbm4z" Nov 03 13:25:46.137211 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:46.136871 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ad148b9-3b17-46b3-8dca-0d36e3b43aa3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5295ab29-6d58657867-lbm4z\" (UID: \"8ad148b9-3b17-46b3-8dca-0d36e3b43aa3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5295ab29-6d58657867-lbm4z" Nov 03 13:25:46.311778 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:46.311680 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhms2\" (UniqueName: \"kubernetes.io/projected/8ad148b9-3b17-46b3-8dca-0d36e3b43aa3-kube-api-access-xhms2\") pod \"payload-pre-processing-e2e-mt-5295ab29-6d58657867-lbm4z\" (UID: \"8ad148b9-3b17-46b3-8dca-0d36e3b43aa3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5295ab29-6d58657867-lbm4z" Nov 03 13:25:46.312130 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:46.312107 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ad148b9-3b17-46b3-8dca-0d36e3b43aa3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5295ab29-6d58657867-lbm4z\" (UID: \"8ad148b9-3b17-46b3-8dca-0d36e3b43aa3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5295ab29-6d58657867-lbm4z" Nov 03 13:25:46.313327 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:46.313302 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ad148b9-3b17-46b3-8dca-0d36e3b43aa3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5295ab29-6d58657867-lbm4z\" (UID: \"8ad148b9-3b17-46b3-8dca-0d36e3b43aa3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5295ab29-6d58657867-lbm4z" Nov 03 13:25:46.314813 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:46.314794 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5295ab29-79c7656b49-qjvj4"] Nov 03 13:25:46.320056 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:46.320039 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5295ab29-79c7656b49-qjvj4" Nov 03 13:25:46.322442 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:46.322419 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5295ab29-79c7656b49-qjvj4"] Nov 03 13:25:46.372500 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:46.371638 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhms2\" (UniqueName: \"kubernetes.io/projected/8ad148b9-3b17-46b3-8dca-0d36e3b43aa3-kube-api-access-xhms2\") pod \"payload-pre-processing-e2e-mt-5295ab29-6d58657867-lbm4z\" (UID: \"8ad148b9-3b17-46b3-8dca-0d36e3b43aa3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5295ab29-6d58657867-lbm4z" Nov 03 13:25:46.413958 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:46.413921 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fab5fbe-10b6-4598-b135-30d3f1eae97b-plugins-config-volume\") pod \"payload-processing-e2e-mt-5295ab29-79c7656b49-qjvj4\" (UID: \"5fab5fbe-10b6-4598-b135-30d3f1eae97b\") " pod="openshift-ingress/payload-processing-e2e-mt-5295ab29-79c7656b49-qjvj4" Nov 03 13:25:46.414236 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:46.414209 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkhlj\" (UniqueName: \"kubernetes.io/projected/5fab5fbe-10b6-4598-b135-30d3f1eae97b-kube-api-access-wkhlj\") pod \"payload-processing-e2e-mt-5295ab29-79c7656b49-qjvj4\" (UID: \"5fab5fbe-10b6-4598-b135-30d3f1eae97b\") " pod="openshift-ingress/payload-processing-e2e-mt-5295ab29-79c7656b49-qjvj4" Nov 03 13:25:46.515643 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:46.515606 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkhlj\" (UniqueName: \"kubernetes.io/projected/5fab5fbe-10b6-4598-b135-30d3f1eae97b-kube-api-access-wkhlj\") pod \"payload-processing-e2e-mt-5295ab29-79c7656b49-qjvj4\" (UID: \"5fab5fbe-10b6-4598-b135-30d3f1eae97b\") " pod="openshift-ingress/payload-processing-e2e-mt-5295ab29-79c7656b49-qjvj4" Nov 03 13:25:46.515804 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:46.515772 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fab5fbe-10b6-4598-b135-30d3f1eae97b-plugins-config-volume\") pod \"payload-processing-e2e-mt-5295ab29-79c7656b49-qjvj4\" (UID: \"5fab5fbe-10b6-4598-b135-30d3f1eae97b\") " pod="openshift-ingress/payload-processing-e2e-mt-5295ab29-79c7656b49-qjvj4" Nov 03 13:25:46.516543 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:46.516518 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fab5fbe-10b6-4598-b135-30d3f1eae97b-plugins-config-volume\") pod \"payload-processing-e2e-mt-5295ab29-79c7656b49-qjvj4\" (UID: \"5fab5fbe-10b6-4598-b135-30d3f1eae97b\") " pod="openshift-ingress/payload-processing-e2e-mt-5295ab29-79c7656b49-qjvj4" Nov 03 13:25:46.570714 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:46.570625 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkhlj\" (UniqueName: \"kubernetes.io/projected/5fab5fbe-10b6-4598-b135-30d3f1eae97b-kube-api-access-wkhlj\") pod \"payload-processing-e2e-mt-5295ab29-79c7656b49-qjvj4\" (UID: \"5fab5fbe-10b6-4598-b135-30d3f1eae97b\") " pod="openshift-ingress/payload-processing-e2e-mt-5295ab29-79c7656b49-qjvj4" Nov 03 13:25:46.631106 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:46.631072 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5295ab29-6d58657867-lbm4z" Nov 03 13:25:46.643139 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:46.643108 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5295ab29-79c7656b49-qjvj4" Nov 03 13:25:46.891075 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:46.891049 5979 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 13:25:46.891247 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:46.891102 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7" podUID="1f4a018c-c28f-4a81-b727-e276ae0156d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 13:25:47.077985 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:47.077960 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5295ab29-6d58657867-lbm4z"] Nov 03 13:25:47.083094 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:25:47.083064 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ad148b9_3b17_46b3_8dca_0d36e3b43aa3.slice/crio-f446401f808816475302f8602bb87fe2643c3d735acb93e8fae41a0a67f15b79 WatchSource:0}: Error finding container f446401f808816475302f8602bb87fe2643c3d735acb93e8fae41a0a67f15b79: Status 404 returned error can't find the container with id f446401f808816475302f8602bb87fe2643c3d735acb93e8fae41a0a67f15b79 Nov 03 13:25:47.092731 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:47.092462 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5295ab29-79c7656b49-qjvj4"] Nov 03 13:25:47.101075 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:25:47.101012 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fab5fbe_10b6_4598_b135_30d3f1eae97b.slice/crio-02ead4a953d38c568c22220bdd086f56a82efad2e45fc7c77db4c2dc97f9a012 WatchSource:0}: Error finding container 02ead4a953d38c568c22220bdd086f56a82efad2e45fc7c77db4c2dc97f9a012: Status 404 returned error can't find the container with id 02ead4a953d38c568c22220bdd086f56a82efad2e45fc7c77db4c2dc97f9a012 Nov 03 13:25:47.283252 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:47.283227 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8575f973-dm7n6" Nov 03 13:25:47.427929 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:47.427839 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/731dc017-0497-42c3-9a66-001519ede38b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/731dc017-0497-42c3-9a66-001519ede38b-maas-api-xks-ca\") pod \"731dc017-0497-42c3-9a66-001519ede38b\" (UID: \"731dc017-0497-42c3-9a66-001519ede38b\") " Nov 03 13:25:47.427929 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:47.427896 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/731dc017-0497-42c3-9a66-001519ede38b-kube-api-access-lffln\" (UniqueName: \"kubernetes.io/projected/731dc017-0497-42c3-9a66-001519ede38b-kube-api-access-lffln\") pod \"731dc017-0497-42c3-9a66-001519ede38b\" (UID: \"731dc017-0497-42c3-9a66-001519ede38b\") " Nov 03 13:25:47.428125 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:47.427978 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/731dc017-0497-42c3-9a66-001519ede38b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/731dc017-0497-42c3-9a66-001519ede38b-maas-api-openshift-ca\") pod \"731dc017-0497-42c3-9a66-001519ede38b\" (UID: \"731dc017-0497-42c3-9a66-001519ede38b\") " Nov 03 13:25:47.428409 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:47.428379 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/731dc017-0497-42c3-9a66-001519ede38b-maas-api-openshift-ca" pod "731dc017-0497-42c3-9a66-001519ede38b" (UID: "731dc017-0497-42c3-9a66-001519ede38b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:25:47.428569 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:47.428520 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/731dc017-0497-42c3-9a66-001519ede38b-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:25:47.429919 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:47.429884 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/731dc017-0497-42c3-9a66-001519ede38b-maas-api-xks-ca" pod "731dc017-0497-42c3-9a66-001519ede38b" (UID: "731dc017-0497-42c3-9a66-001519ede38b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:25:47.430055 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:47.430035 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/731dc017-0497-42c3-9a66-001519ede38b-kube-api-access-lffln" pod "731dc017-0497-42c3-9a66-001519ede38b" (UID: "731dc017-0497-42c3-9a66-001519ede38b"). InnerVolumeSpecName "kube-api-access-lffln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:25:47.530103 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:47.530060 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/731dc017-0497-42c3-9a66-001519ede38b-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:25:47.530103 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:47.530097 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lffln\" (UniqueName: \"kubernetes.io/projected/731dc017-0497-42c3-9a66-001519ede38b-kube-api-access-lffln\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:25:47.891303 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:47.891273 5979 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 13:25:47.891466 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:47.891330 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7" podUID="1f4a018c-c28f-4a81-b727-e276ae0156d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 13:25:48.082235 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:48.082193 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5295ab29-6d58657867-lbm4z" event={"ID":"8ad148b9-3b17-46b3-8dca-0d36e3b43aa3","Type":"ContainerStarted","Data":"d3f939f4aba2f3ae4b37c5810b085e214b64dabebb63d8a5f7406ded7f68c2a6"} Nov 03 13:25:48.082755 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:48.082265 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-5295ab29-6d58657867-lbm4z" Nov 03 13:25:48.082755 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:48.082290 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5295ab29-6d58657867-lbm4z" event={"ID":"8ad148b9-3b17-46b3-8dca-0d36e3b43aa3","Type":"ContainerStarted","Data":"f446401f808816475302f8602bb87fe2643c3d735acb93e8fae41a0a67f15b79"} Nov 03 13:25:48.083591 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:48.083570 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8575f973-dm7n6" Nov 03 13:25:48.083712 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:48.083600 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8575f973-dm7n6" event={"ID":"731dc017-0497-42c3-9a66-001519ede38b","Type":"ContainerDied","Data":"04df778dfe49428647e06e512342cf7d814ef009f0dacba90f7d13b544ee1df3"} Nov 03 13:25:48.083712 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:48.083641 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04df778dfe49428647e06e512342cf7d814ef009f0dacba90f7d13b544ee1df3" Nov 03 13:25:48.086686 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:48.086664 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5295ab29-79c7656b49-qjvj4" event={"ID":"5fab5fbe-10b6-4598-b135-30d3f1eae97b","Type":"ContainerStarted","Data":"760ef3889538ff1ef10caa083f4ae2951a334c99b47a20d27f3a53fb329fcd36"} Nov 03 13:25:48.086787 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:48.086713 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5295ab29-79c7656b49-qjvj4" event={"ID":"5fab5fbe-10b6-4598-b135-30d3f1eae97b","Type":"ContainerStarted","Data":"02ead4a953d38c568c22220bdd086f56a82efad2e45fc7c77db4c2dc97f9a012"} Nov 03 13:25:48.086787 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:48.086745 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-5295ab29-79c7656b49-qjvj4" Nov 03 13:25:48.192321 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:48.192217 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-5295ab29-6d58657867-lbm4z" podStartSLOduration=3.192203483 podStartE2EDuration="3.192203483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:25:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:25:48.188896395 +0000 UTC m=+1230.199584637" watchObservedRunningTime="2026-11-03 13:25:48.192203483 +0000 UTC m=+1230.202891743" Nov 03 13:25:48.405494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:48.404684 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-5295ab29-79c7656b49-qjvj4" podStartSLOduration=2.404664399 podStartE2EDuration="2.404664399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:25:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:25:48.399007432 +0000 UTC m=+1230.409695681" watchObservedRunningTime="2026-11-03 13:25:48.404664399 +0000 UTC m=+1230.415352644" Nov 03 13:25:48.893934 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:48.893890 5979 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 13:25:48.894251 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:48.894220 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7" podUID="1f4a018c-c28f-4a81-b727-e276ae0156d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 13:25:49.047492 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:49.047379 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb46c11f-jq7sn"] Nov 03 13:25:49.053966 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:49.053931 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb46c11f-jq7sn"] Nov 03 13:25:49.891651 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:49.891605 5979 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 13:25:49.892264 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:49.892229 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7" podUID="1f4a018c-c28f-4a81-b727-e276ae0156d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 13:25:50.517497 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:50.512359 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8efac707-fd2a-473e-afce-7e3aa9caeaa9" path="/var/lib/kubelet/pods/8efac707-fd2a-473e-afce-7e3aa9caeaa9/volumes" Nov 03 13:25:50.892915 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:50.892879 5979 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Nov 03 13:25:50.893537 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:50.893500 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7" podUID="1f4a018c-c28f-4a81-b727-e276ae0156d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Nov 03 13:25:51.291700 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:51.291584 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5bb463/distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw" podUID="f7c171f9-4569-431e-a891-5cae3328ce47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:25:51.292405 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:51.291961 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42c707/distinct-w0-42c707-kserve-64949d6cb8-kq4w9" podUID="430d8f95-204c-4243-b771-a03e30303e77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded" Nov 03 13:25:51.335637 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:51.335584 5979 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5bb463/distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw" podUID="f7c171f9-4569-431e-a891-5cae3328ce47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:25:51.671356 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:51.671311 5979 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5bb463/premium-w3-5bb463-kserve-5f6ff6795c-jxmsr" podUID="8ee26f55-5c0c-4bd8-ad9b-fb8c03641698" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": context deadline exceeded" Nov 03 13:25:51.741658 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:51.741600 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5bb463/premium-w3-5bb463-kserve-5f6ff6795c-jxmsr" podUID="8ee26f55-5c0c-4bd8-ad9b-fb8c03641698" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:25:52.303597 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:52.303546 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42c707/distinct-w0-42c707-kserve-64949d6cb8-kq4w9" podUID="430d8f95-204c-4243-b771-a03e30303e77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:25:52.320623 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:52.320574 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5bb463/distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw" podUID="f7c171f9-4569-431e-a891-5cae3328ce47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded" Nov 03 13:25:52.322030 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:52.321998 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-42c707/distinct-w0-42c707-kserve-64949d6cb8-kq4w9" Nov 03 13:25:52.322165 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:52.322066 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5bb463/distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw" Nov 03 13:25:52.343823 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:52.339758 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd" podUID="a4b4df54-0aac-419d-8bb2-3258267bc088" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:25:52.619818 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:52.605829 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3c5dbc/premium-w4-3c5dbc-kserve-577d8db86-mc72z" podUID="64d52ec5-fb72-4616-8d20-8b90043af828" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:25:52.798622 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:52.798572 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5bb463/premium-w3-5bb463-kserve-5f6ff6795c-jxmsr" podUID="8ee26f55-5c0c-4bd8-ad9b-fb8c03641698" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:25:52.821496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:52.799275 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5bb463/premium-w3-5bb463-kserve-5f6ff6795c-jxmsr" Nov 03 13:25:53.007363 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:52.979568 5979 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:25:53.007363 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:52.979670 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7" podUID="1f4a018c-c28f-4a81-b727-e276ae0156d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:25:53.321719 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:53.321617 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5bb463/distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw" podUID="f7c171f9-4569-431e-a891-5cae3328ce47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:25:53.322646 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:53.322611 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42c707/distinct-w0-42c707-kserve-64949d6cb8-kq4w9" podUID="430d8f95-204c-4243-b771-a03e30303e77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:25:53.340867 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:53.340826 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd" podUID="a4b4df54-0aac-419d-8bb2-3258267bc088" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:25:53.341488 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:53.341448 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd" Nov 03 13:25:53.454502 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:53.454195 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7" Nov 03 13:25:53.458504 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:53.455389 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7" Nov 03 13:25:53.462731 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:53.462707 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7" Nov 03 13:25:53.515301 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:53.507366 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5bb463/distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw" Nov 03 13:25:53.604683 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:53.604590 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5bb463/premium-w3-5bb463-kserve-5f6ff6795c-jxmsr" Nov 03 13:25:53.713015 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:53.618933 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3c5dbc/premium-w4-3c5dbc-kserve-577d8db86-mc72z" podUID="64d52ec5-fb72-4616-8d20-8b90043af828" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Nov 03 13:25:53.713015 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:53.619505 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3c5dbc/premium-w4-3c5dbc-kserve-577d8db86-mc72z" Nov 03 13:25:53.733711 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:53.733683 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd" Nov 03 13:25:53.745959 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:53.745926 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3c5dbc/premium-w4-3c5dbc-kserve-577d8db86-mc72z" Nov 03 13:25:53.796568 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:53.796540 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-42c707/distinct-w0-42c707-kserve-64949d6cb8-kq4w9" Nov 03 13:25:54.267704 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:54.267662 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d567b768-j8mp7"] Nov 03 13:25:54.269199 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:54.269175 5979 cpu_manager.go:414] "containerMap: removing container" podUID="731dc017-0497-42c3-9a66-001519ede38b" containerName="revoke-keys" Nov 03 13:25:54.269253 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:54.269203 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="731dc017-0497-42c3-9a66-001519ede38b" containerName="revoke-keys" Nov 03 13:25:54.269621 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:54.269603 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="731dc017-0497-42c3-9a66-001519ede38b" containerName="revoke-keys" Nov 03 13:25:54.276068 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:54.276044 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d567b768-j8mp7" Nov 03 13:25:54.282942 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:54.282916 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ef15fe1-f088-46d7-adce-da164a1b0b77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d567b768-j8mp7\" (UID: \"9ef15fe1-f088-46d7-adce-da164a1b0b77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d567b768-j8mp7" Nov 03 13:25:54.283135 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:54.283118 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8jv2\" (UniqueName: \"kubernetes.io/projected/9ef15fe1-f088-46d7-adce-da164a1b0b77-kube-api-access-l8jv2\") pod \"maas-api-revoke-sub-d567b768-j8mp7\" (UID: \"9ef15fe1-f088-46d7-adce-da164a1b0b77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d567b768-j8mp7" Nov 03 13:25:54.283196 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:54.283163 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ef15fe1-f088-46d7-adce-da164a1b0b77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d567b768-j8mp7\" (UID: \"9ef15fe1-f088-46d7-adce-da164a1b0b77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d567b768-j8mp7" Nov 03 13:25:54.312025 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:54.311996 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d567b768-j8mp7"] Nov 03 13:25:54.325547 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:54.325263 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:25:54.325547 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:54.325312 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:25:54.325547 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:54.325354 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 13:25:54.325547 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:54.325422 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 13:25:54.384305 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:54.384276 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8jv2\" (UniqueName: \"kubernetes.io/projected/9ef15fe1-f088-46d7-adce-da164a1b0b77-kube-api-access-l8jv2\") pod \"maas-api-revoke-sub-d567b768-j8mp7\" (UID: \"9ef15fe1-f088-46d7-adce-da164a1b0b77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d567b768-j8mp7" Nov 03 13:25:54.384544 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:54.384322 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ef15fe1-f088-46d7-adce-da164a1b0b77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d567b768-j8mp7\" (UID: \"9ef15fe1-f088-46d7-adce-da164a1b0b77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d567b768-j8mp7" Nov 03 13:25:54.384544 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:54.384375 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ef15fe1-f088-46d7-adce-da164a1b0b77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d567b768-j8mp7\" (UID: \"9ef15fe1-f088-46d7-adce-da164a1b0b77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d567b768-j8mp7" Nov 03 13:25:54.385223 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:54.385199 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ef15fe1-f088-46d7-adce-da164a1b0b77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d567b768-j8mp7\" (UID: \"9ef15fe1-f088-46d7-adce-da164a1b0b77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d567b768-j8mp7" Nov 03 13:25:54.386664 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:54.386640 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ef15fe1-f088-46d7-adce-da164a1b0b77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d567b768-j8mp7\" (UID: \"9ef15fe1-f088-46d7-adce-da164a1b0b77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d567b768-j8mp7" Nov 03 13:25:54.438507 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:54.438467 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8jv2\" (UniqueName: \"kubernetes.io/projected/9ef15fe1-f088-46d7-adce-da164a1b0b77-kube-api-access-l8jv2\") pod \"maas-api-revoke-sub-d567b768-j8mp7\" (UID: \"9ef15fe1-f088-46d7-adce-da164a1b0b77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d567b768-j8mp7" Nov 03 13:25:54.596131 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:54.595843 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d567b768-j8mp7" Nov 03 13:25:55.043879 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:55.043791 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d567b768-j8mp7"] Nov 03 13:25:55.051020 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:25:55.050991 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ef15fe1_f088_46d7_adce_da164a1b0b77.slice/crio-01c824028fd7ed3bca829c8d622f215ee847e84d0b591d8e0e5c184bc37dca00 WatchSource:0}: Error finding container 01c824028fd7ed3bca829c8d622f215ee847e84d0b591d8e0e5c184bc37dca00: Status 404 returned error can't find the container with id 01c824028fd7ed3bca829c8d622f215ee847e84d0b591d8e0e5c184bc37dca00 Nov 03 13:25:55.055030 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:55.055003 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe460bed-7q2pg"] Nov 03 13:25:55.089981 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:55.089951 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe460bed-7q2pg"] Nov 03 13:25:55.408746 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:55.408711 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d567b768-j8mp7" event={"ID":"9ef15fe1-f088-46d7-adce-da164a1b0b77","Type":"ContainerStarted","Data":"0aaad4592654f5b7575b9552852d8d6b7fe78830210dca27f0d9ca04247e1c02"} Nov 03 13:25:55.409098 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:55.408770 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d567b768-j8mp7" event={"ID":"9ef15fe1-f088-46d7-adce-da164a1b0b77","Type":"ContainerStarted","Data":"01c824028fd7ed3bca829c8d622f215ee847e84d0b591d8e0e5c184bc37dca00"} Nov 03 13:25:55.518256 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:55.518207 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d567b768-j8mp7" podStartSLOduration=1.518193573 podStartE2EDuration="1.518193573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:25:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:25:55.516192752 +0000 UTC m=+1237.526880997" watchObservedRunningTime="2026-11-03 13:25:55.518193573 +0000 UTC m=+1237.528881814" Nov 03 13:25:56.115605 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:56.115572 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="057489b7-ac12-4d95-9bc1-bf100a735dd2" path="/var/lib/kubelet/pods/057489b7-ac12-4d95-9bc1-bf100a735dd2/volumes" Nov 03 13:25:56.416709 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:56.416673 5979 generic.go:356] "Generic (PLEG): container finished" podID="9ef15fe1-f088-46d7-adce-da164a1b0b77" containerID="0aaad4592654f5b7575b9552852d8d6b7fe78830210dca27f0d9ca04247e1c02" exitCode=0 Nov 03 13:25:56.417220 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:56.416716 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d567b768-j8mp7" event={"ID":"9ef15fe1-f088-46d7-adce-da164a1b0b77","Type":"ContainerDied","Data":"0aaad4592654f5b7575b9552852d8d6b7fe78830210dca27f0d9ca04247e1c02"} Nov 03 13:25:57.576052 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:57.576026 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d567b768-j8mp7" Nov 03 13:25:57.630296 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:57.630265 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ef15fe1-f088-46d7-adce-da164a1b0b77-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ef15fe1-f088-46d7-adce-da164a1b0b77-maas-api-openshift-ca\") pod \"9ef15fe1-f088-46d7-adce-da164a1b0b77\" (UID: \"9ef15fe1-f088-46d7-adce-da164a1b0b77\") " Nov 03 13:25:57.630461 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:57.630353 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ef15fe1-f088-46d7-adce-da164a1b0b77-kube-api-access-l8jv2\" (UniqueName: \"kubernetes.io/projected/9ef15fe1-f088-46d7-adce-da164a1b0b77-kube-api-access-l8jv2\") pod \"9ef15fe1-f088-46d7-adce-da164a1b0b77\" (UID: \"9ef15fe1-f088-46d7-adce-da164a1b0b77\") " Nov 03 13:25:57.630529 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:57.630465 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ef15fe1-f088-46d7-adce-da164a1b0b77-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ef15fe1-f088-46d7-adce-da164a1b0b77-maas-api-xks-ca\") pod \"9ef15fe1-f088-46d7-adce-da164a1b0b77\" (UID: \"9ef15fe1-f088-46d7-adce-da164a1b0b77\") " Nov 03 13:25:57.630658 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:57.630636 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ef15fe1-f088-46d7-adce-da164a1b0b77-maas-api-openshift-ca" pod "9ef15fe1-f088-46d7-adce-da164a1b0b77" (UID: "9ef15fe1-f088-46d7-adce-da164a1b0b77"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:25:57.630978 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:57.630957 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ef15fe1-f088-46d7-adce-da164a1b0b77-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:25:57.632407 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:57.632380 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ef15fe1-f088-46d7-adce-da164a1b0b77-maas-api-xks-ca" pod "9ef15fe1-f088-46d7-adce-da164a1b0b77" (UID: "9ef15fe1-f088-46d7-adce-da164a1b0b77"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:25:57.632555 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:57.632408 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ef15fe1-f088-46d7-adce-da164a1b0b77-kube-api-access-l8jv2" pod "9ef15fe1-f088-46d7-adce-da164a1b0b77" (UID: "9ef15fe1-f088-46d7-adce-da164a1b0b77"). InnerVolumeSpecName "kube-api-access-l8jv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:25:57.732386 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:57.732303 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8jv2\" (UniqueName: \"kubernetes.io/projected/9ef15fe1-f088-46d7-adce-da164a1b0b77-kube-api-access-l8jv2\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:25:57.732386 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:57.732333 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ef15fe1-f088-46d7-adce-da164a1b0b77-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:25:58.431718 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:58.431689 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d567b768-j8mp7" Nov 03 13:25:58.431869 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:58.431720 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d567b768-j8mp7" event={"ID":"9ef15fe1-f088-46d7-adce-da164a1b0b77","Type":"ContainerDied","Data":"01c824028fd7ed3bca829c8d622f215ee847e84d0b591d8e0e5c184bc37dca00"} Nov 03 13:25:58.431869 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:58.431794 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01c824028fd7ed3bca829c8d622f215ee847e84d0b591d8e0e5c184bc37dca00" Nov 03 13:25:59.096686 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:59.096656 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-5295ab29-6d58657867-lbm4z" Nov 03 13:25:59.097073 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:25:59.096737 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-5295ab29-79c7656b49-qjvj4" Nov 03 13:26:05.043296 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:05.043263 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07596b90-87n7l"] Nov 03 13:26:05.089646 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:05.089613 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07596b90-87n7l"] Nov 03 13:26:06.047965 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:06.047930 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0494db61-rrv4c"] Nov 03 13:26:06.094338 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:06.094303 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0494db61-rrv4c"] Nov 03 13:26:06.116112 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:06.116077 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bc1e136-292c-425b-b481-3836bb343d3b" path="/var/lib/kubelet/pods/6bc1e136-292c-425b-b481-3836bb343d3b/volumes" Nov 03 13:26:06.116645 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:06.116623 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b68be428-2283-4a4f-a55e-b9bb113d8d3b" path="/var/lib/kubelet/pods/b68be428-2283-4a4f-a55e-b9bb113d8d3b/volumes" Nov 03 13:26:07.235002 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:26:07.234968 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:26:11.490520 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:11.490485 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb"] Nov 03 13:26:11.491148 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:11.491133 5979 cpu_manager.go:414] "containerMap: removing container" podUID="9ef15fe1-f088-46d7-adce-da164a1b0b77" containerName="revoke-keys" Nov 03 13:26:11.491189 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:11.491150 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ef15fe1-f088-46d7-adce-da164a1b0b77" containerName="revoke-keys" Nov 03 13:26:11.491242 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:11.491233 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ef15fe1-f088-46d7-adce-da164a1b0b77" containerName="revoke-keys" Nov 03 13:26:11.497873 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:11.497845 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb" Nov 03 13:26:11.623913 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:11.623862 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-workload-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb\" (UID: \"cfd3eded-c3de-4d6b-9a1e-c284549c6c18\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb" Nov 03 13:26:11.623913 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:11.623910 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-workload-certs\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb\" (UID: \"cfd3eded-c3de-4d6b-9a1e-c284549c6c18\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb" Nov 03 13:26:11.624151 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:11.624035 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-istio-podinfo\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb\" (UID: \"cfd3eded-c3de-4d6b-9a1e-c284549c6c18\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb" Nov 03 13:26:11.624151 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:11.624065 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-istio-token\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb\" (UID: \"cfd3eded-c3de-4d6b-9a1e-c284549c6c18\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb" Nov 03 13:26:11.624151 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:11.624082 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-istiod-ca-cert\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb\" (UID: \"cfd3eded-c3de-4d6b-9a1e-c284549c6c18\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb" Nov 03 13:26:11.624297 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:11.624209 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-credential-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb\" (UID: \"cfd3eded-c3de-4d6b-9a1e-c284549c6c18\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb" Nov 03 13:26:11.624297 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:11.624267 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-istio-data\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb\" (UID: \"cfd3eded-c3de-4d6b-9a1e-c284549c6c18\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb" Nov 03 13:26:11.624297 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:11.624285 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7vql\" (UniqueName: \"kubernetes.io/projected/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-kube-api-access-d7vql\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb\" (UID: \"cfd3eded-c3de-4d6b-9a1e-c284549c6c18\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb" Nov 03 13:26:11.624442 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:11.624384 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-istio-envoy\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb\" (UID: \"cfd3eded-c3de-4d6b-9a1e-c284549c6c18\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb" Nov 03 13:26:11.693099 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:11.693058 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb"] Nov 03 13:26:11.726104 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:11.726067 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-workload-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb\" (UID: \"cfd3eded-c3de-4d6b-9a1e-c284549c6c18\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb" Nov 03 13:26:11.726262 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:11.726118 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-workload-certs\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb\" (UID: \"cfd3eded-c3de-4d6b-9a1e-c284549c6c18\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb" Nov 03 13:26:11.726262 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:11.726228 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-istio-podinfo\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb\" (UID: \"cfd3eded-c3de-4d6b-9a1e-c284549c6c18\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb" Nov 03 13:26:11.726262 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:11.726253 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-istio-token\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb\" (UID: \"cfd3eded-c3de-4d6b-9a1e-c284549c6c18\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb" Nov 03 13:26:11.726443 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:11.726280 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-istiod-ca-cert\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb\" (UID: \"cfd3eded-c3de-4d6b-9a1e-c284549c6c18\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb" Nov 03 13:26:11.726443 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:11.726334 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-credential-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb\" (UID: \"cfd3eded-c3de-4d6b-9a1e-c284549c6c18\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb" Nov 03 13:26:11.726443 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:11.726406 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-istio-data\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb\" (UID: \"cfd3eded-c3de-4d6b-9a1e-c284549c6c18\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb" Nov 03 13:26:11.726443 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:11.726432 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7vql\" (UniqueName: \"kubernetes.io/projected/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-kube-api-access-d7vql\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb\" (UID: \"cfd3eded-c3de-4d6b-9a1e-c284549c6c18\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb" Nov 03 13:26:11.726673 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:11.726519 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-workload-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb\" (UID: \"cfd3eded-c3de-4d6b-9a1e-c284549c6c18\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb" Nov 03 13:26:11.726673 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:11.726551 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-istio-envoy\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb\" (UID: \"cfd3eded-c3de-4d6b-9a1e-c284549c6c18\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb" Nov 03 13:26:11.726673 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:11.726615 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-workload-certs\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb\" (UID: \"cfd3eded-c3de-4d6b-9a1e-c284549c6c18\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb" Nov 03 13:26:11.726846 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:11.726810 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-istio-data\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb\" (UID: \"cfd3eded-c3de-4d6b-9a1e-c284549c6c18\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb" Nov 03 13:26:11.727281 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:11.727252 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-istiod-ca-cert\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb\" (UID: \"cfd3eded-c3de-4d6b-9a1e-c284549c6c18\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb" Nov 03 13:26:11.727396 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:11.727259 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-credential-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb\" (UID: \"cfd3eded-c3de-4d6b-9a1e-c284549c6c18\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb" Nov 03 13:26:11.728841 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:11.728812 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-istio-podinfo\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb\" (UID: \"cfd3eded-c3de-4d6b-9a1e-c284549c6c18\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb" Nov 03 13:26:11.729055 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:11.729038 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-istio-envoy\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb\" (UID: \"cfd3eded-c3de-4d6b-9a1e-c284549c6c18\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb" Nov 03 13:26:11.782818 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:11.782746 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-istio-token\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb\" (UID: \"cfd3eded-c3de-4d6b-9a1e-c284549c6c18\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb" Nov 03 13:26:11.783696 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:11.783674 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7vql\" (UniqueName: \"kubernetes.io/projected/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-kube-api-access-d7vql\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb\" (UID: \"cfd3eded-c3de-4d6b-9a1e-c284549c6c18\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb" Nov 03 13:26:11.813705 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:11.813668 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb" Nov 03 13:26:12.152166 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:12.152133 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4"] Nov 03 13:26:12.152606 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:12.152352 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4" podUID="01e6f154-38b8-47bf-8f9b-436fa4cf1436" containerName="istio-proxy" containerID="cri-o://c5e4a2dd70e2ae178f24143eff509a5ba3b66e4d4a02725b02841013f1b4e6e2" gracePeriod=30 Nov 03 13:26:12.301018 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:26:12.300984 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfd3eded_c3de_4d6b_9a1e_c284549c6c18.slice/crio-9012e4c0b0cbbc9534d9feb16bf34319f9cbcfb4ba9fa0f5c34a1f554378336d WatchSource:0}: Error finding container 9012e4c0b0cbbc9534d9feb16bf34319f9cbcfb4ba9fa0f5c34a1f554378336d: Status 404 returned error can't find the container with id 9012e4c0b0cbbc9534d9feb16bf34319f9cbcfb4ba9fa0f5c34a1f554378336d Nov 03 13:26:12.303555 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:12.303525 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:26:12.303638 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:12.303590 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:26:12.303638 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:12.303617 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:26:12.318199 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:12.318171 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb"] Nov 03 13:26:12.403583 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:12.403515 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:26:12.533709 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:12.533666 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb" event={"ID":"cfd3eded-c3de-4d6b-9a1e-c284549c6c18","Type":"ContainerStarted","Data":"405a543b62ebb99ccd78f604fd577bd100d66d8b417afb35d33618fca28d407a"} Nov 03 13:26:12.534171 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:12.533742 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb" event={"ID":"cfd3eded-c3de-4d6b-9a1e-c284549c6c18","Type":"ContainerStarted","Data":"9012e4c0b0cbbc9534d9feb16bf34319f9cbcfb4ba9fa0f5c34a1f554378336d"} Nov 03 13:26:12.647536 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:12.647487 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb" podStartSLOduration=1.647458772 podStartE2EDuration="1.647458772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:26:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:26:12.643114184 +0000 UTC m=+1254.653802428" watchObservedRunningTime="2026-11-03 13:26:12.647458772 +0000 UTC m=+1254.658147013" Nov 03 13:26:12.825552 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:12.819694 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb" Nov 03 13:26:13.817143 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:13.817108 5979 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 03 13:26:13.817755 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:13.817722 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb" podUID="cfd3eded-c3de-4d6b-9a1e-c284549c6c18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 03 13:26:14.328508 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:14.326562 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5bb463/distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw" podUID="f7c171f9-4569-431e-a891-5cae3328ce47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:26:14.397211 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:14.397160 5979 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd" podUID="a4b4df54-0aac-419d-8bb2-3258267bc088" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:26:14.610682 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:14.610576 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5bb463/premium-w3-5bb463-kserve-5f6ff6795c-jxmsr" podUID="8ee26f55-5c0c-4bd8-ad9b-fb8c03641698" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:26:14.757711 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:14.757663 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3c5dbc/premium-w4-3c5dbc-kserve-577d8db86-mc72z" podUID="64d52ec5-fb72-4616-8d20-8b90043af828" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:26:14.783972 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:14.773150 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd" podUID="a4b4df54-0aac-419d-8bb2-3258267bc088" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:26:14.823497 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:14.803220 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42c707/distinct-w0-42c707-kserve-64949d6cb8-kq4w9" podUID="430d8f95-204c-4243-b771-a03e30303e77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:26:14.825436 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:14.825401 5979 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 03 13:26:14.825686 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:14.825648 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb" podUID="cfd3eded-c3de-4d6b-9a1e-c284549c6c18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 03 13:26:14.925746 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:14.925588 5979 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:26:14.925746 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:14.925671 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh" podUID="821c73f7-cf2f-414c-8aea-1aff84e017af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:26:15.328753 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:15.328644 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5bb463/distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw" podUID="f7c171f9-4569-431e-a891-5cae3328ce47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": context deadline exceeded" Nov 03 13:26:15.330099 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:15.330077 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5bb463/distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw" Nov 03 13:26:15.665524 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:15.659557 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5bb463/premium-w3-5bb463-kserve-5f6ff6795c-jxmsr" podUID="8ee26f55-5c0c-4bd8-ad9b-fb8c03641698" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Nov 03 13:26:15.665524 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:15.660359 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5bb463/premium-w3-5bb463-kserve-5f6ff6795c-jxmsr" Nov 03 13:26:15.797822 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:15.788544 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd" podUID="a4b4df54-0aac-419d-8bb2-3258267bc088" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:26:15.806228 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:15.806197 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd" Nov 03 13:26:15.852834 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:15.843608 5979 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 03 13:26:15.853427 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:15.853392 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb" podUID="cfd3eded-c3de-4d6b-9a1e-c284549c6c18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 03 13:26:15.853613 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:15.844321 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-42c707/distinct-w0-42c707-kserve-64949d6cb8-kq4w9" podUID="430d8f95-204c-4243-b771-a03e30303e77" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:26:15.854806 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:15.854785 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-42c707/distinct-w0-42c707-kserve-64949d6cb8-kq4w9" Nov 03 13:26:15.893564 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:15.893526 5979 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:26:15.893860 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:15.893827 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf" podUID="620d1fb2-9a2b-4e36-819e-6997985fbc1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:26:16.417223 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:16.417163 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5bb463/distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw" podUID="f7c171f9-4569-431e-a891-5cae3328ce47" containerName="main" probeResult="failure" output="Get \"https://10.134.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 13:26:16.496738 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:16.496703 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60a9f79b-2kx52"] Nov 03 13:26:16.520996 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:16.520964 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60a9f79b-2kx52"] Nov 03 13:26:16.525497 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:16.521296 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a9f79b-2kx52" Nov 03 13:26:16.534027 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:16.533288 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5bb463/distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw" Nov 03 13:26:16.536201 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:16.536177 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd" Nov 03 13:26:16.536500 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:16.536390 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-42c707/distinct-w0-42c707-kserve-64949d6cb8-kq4w9" Nov 03 13:26:16.543819 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:16.543798 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5bb463/premium-w3-5bb463-kserve-5f6ff6795c-jxmsr" Nov 03 13:26:16.573878 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:16.573028 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a8f32fb-a894-49b1-8da8-3cccbca598cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60a9f79b-2kx52\" (UID: \"1a8f32fb-a894-49b1-8da8-3cccbca598cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a9f79b-2kx52" Nov 03 13:26:16.573878 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:16.573168 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wzgs\" (UniqueName: \"kubernetes.io/projected/1a8f32fb-a894-49b1-8da8-3cccbca598cc-kube-api-access-5wzgs\") pod \"maas-api-revoke-sub-60a9f79b-2kx52\" (UID: \"1a8f32fb-a894-49b1-8da8-3cccbca598cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a9f79b-2kx52" Nov 03 13:26:16.584737 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:16.582201 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:26:16.584737 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:16.582383 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 13:26:16.584737 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:16.582669 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 13:26:16.584737 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:16.582203 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:26:16.588204 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:16.587575 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a8f32fb-a894-49b1-8da8-3cccbca598cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60a9f79b-2kx52\" (UID: \"1a8f32fb-a894-49b1-8da8-3cccbca598cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a9f79b-2kx52" Nov 03 13:26:16.688943 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:16.688851 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a8f32fb-a894-49b1-8da8-3cccbca598cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60a9f79b-2kx52\" (UID: \"1a8f32fb-a894-49b1-8da8-3cccbca598cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a9f79b-2kx52" Nov 03 13:26:16.688943 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:16.688935 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wzgs\" (UniqueName: \"kubernetes.io/projected/1a8f32fb-a894-49b1-8da8-3cccbca598cc-kube-api-access-5wzgs\") pod \"maas-api-revoke-sub-60a9f79b-2kx52\" (UID: \"1a8f32fb-a894-49b1-8da8-3cccbca598cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a9f79b-2kx52" Nov 03 13:26:16.689173 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:16.689016 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a8f32fb-a894-49b1-8da8-3cccbca598cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60a9f79b-2kx52\" (UID: \"1a8f32fb-a894-49b1-8da8-3cccbca598cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a9f79b-2kx52" Nov 03 13:26:16.689492 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:16.689451 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a8f32fb-a894-49b1-8da8-3cccbca598cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60a9f79b-2kx52\" (UID: \"1a8f32fb-a894-49b1-8da8-3cccbca598cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a9f79b-2kx52" Nov 03 13:26:16.691305 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:16.691281 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a8f32fb-a894-49b1-8da8-3cccbca598cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60a9f79b-2kx52\" (UID: \"1a8f32fb-a894-49b1-8da8-3cccbca598cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a9f79b-2kx52" Nov 03 13:26:16.742421 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:16.742385 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wzgs\" (UniqueName: \"kubernetes.io/projected/1a8f32fb-a894-49b1-8da8-3cccbca598cc-kube-api-access-5wzgs\") pod \"maas-api-revoke-sub-60a9f79b-2kx52\" (UID: \"1a8f32fb-a894-49b1-8da8-3cccbca598cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a9f79b-2kx52" Nov 03 13:26:16.815114 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:16.815083 5979 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 03 13:26:16.815277 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:16.815137 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb" podUID="cfd3eded-c3de-4d6b-9a1e-c284549c6c18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 03 13:26:16.898264 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:16.898235 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a9f79b-2kx52" Nov 03 13:26:17.302803 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:17.302718 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60a9f79b-2kx52"] Nov 03 13:26:17.308418 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:26:17.308388 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a8f32fb_a894_49b1_8da8_3cccbca598cc.slice/crio-c91707f196f1d0a8d2e8e4d6e21b6cae27c313ffadb61fd20a5ee61d1f9f72d5 WatchSource:0}: Error finding container c91707f196f1d0a8d2e8e4d6e21b6cae27c313ffadb61fd20a5ee61d1f9f72d5: Status 404 returned error can't find the container with id c91707f196f1d0a8d2e8e4d6e21b6cae27c313ffadb61fd20a5ee61d1f9f72d5 Nov 03 13:26:17.814388 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:17.814356 5979 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 03 13:26:17.814552 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:17.814405 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb" podUID="cfd3eded-c3de-4d6b-9a1e-c284549c6c18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 03 13:26:17.872110 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:17.872076 5979 generic.go:356] "Generic (PLEG): container finished" podID="01e6f154-38b8-47bf-8f9b-436fa4cf1436" containerID="c5e4a2dd70e2ae178f24143eff509a5ba3b66e4d4a02725b02841013f1b4e6e2" exitCode=0 Nov 03 13:26:17.872300 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:17.872148 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4" event={"ID":"01e6f154-38b8-47bf-8f9b-436fa4cf1436","Type":"ContainerDied","Data":"c5e4a2dd70e2ae178f24143eff509a5ba3b66e4d4a02725b02841013f1b4e6e2"} Nov 03 13:26:17.872300 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:17.872237 5979 scope.go:122] "RemoveContainer" containerID="9a1ed4e0b255d0d777095b119f9eebff6f036c31428b8681be748dbb940a5d1a" Nov 03 13:26:17.875408 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:17.875382 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a9f79b-2kx52" event={"ID":"1a8f32fb-a894-49b1-8da8-3cccbca598cc","Type":"ContainerStarted","Data":"7b26f60f94453ea1e97ed5ae4e3343e50989071b14d629c22ab73c7b529e6254"} Nov 03 13:26:17.875539 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:17.875434 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a9f79b-2kx52" event={"ID":"1a8f32fb-a894-49b1-8da8-3cccbca598cc","Type":"ContainerStarted","Data":"c91707f196f1d0a8d2e8e4d6e21b6cae27c313ffadb61fd20a5ee61d1f9f72d5"} Nov 03 13:26:17.909501 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:17.909463 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4" Nov 03 13:26:18.009549 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:18.009045 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a9f79b-2kx52" podStartSLOduration=2.00902179 podStartE2EDuration="2.00902179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:26:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:26:17.982895859 +0000 UTC m=+1259.993584102" watchObservedRunningTime="2026-11-03 13:26:18.00902179 +0000 UTC m=+1260.019710032" Nov 03 13:26:18.137236 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:18.137200 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01e6f154-38b8-47bf-8f9b-436fa4cf1436-istio-token\" (UniqueName: \"kubernetes.io/projected/01e6f154-38b8-47bf-8f9b-436fa4cf1436-istio-token\") pod \"01e6f154-38b8-47bf-8f9b-436fa4cf1436\" (UID: \"01e6f154-38b8-47bf-8f9b-436fa4cf1436\") " Nov 03 13:26:18.137412 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:18.137269 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01e6f154-38b8-47bf-8f9b-436fa4cf1436-kube-api-access-c98bt\" (UniqueName: \"kubernetes.io/projected/01e6f154-38b8-47bf-8f9b-436fa4cf1436-kube-api-access-c98bt\") pod \"01e6f154-38b8-47bf-8f9b-436fa4cf1436\" (UID: \"01e6f154-38b8-47bf-8f9b-436fa4cf1436\") " Nov 03 13:26:18.137412 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:18.137382 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/01e6f154-38b8-47bf-8f9b-436fa4cf1436-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01e6f154-38b8-47bf-8f9b-436fa4cf1436-istio-podinfo\") pod \"01e6f154-38b8-47bf-8f9b-436fa4cf1436\" (UID: \"01e6f154-38b8-47bf-8f9b-436fa4cf1436\") " Nov 03 13:26:18.137575 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:18.137417 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01e6f154-38b8-47bf-8f9b-436fa4cf1436-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01e6f154-38b8-47bf-8f9b-436fa4cf1436-credential-socket\") pod \"01e6f154-38b8-47bf-8f9b-436fa4cf1436\" (UID: \"01e6f154-38b8-47bf-8f9b-436fa4cf1436\") " Nov 03 13:26:18.137656 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:18.137600 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01e6f154-38b8-47bf-8f9b-436fa4cf1436-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01e6f154-38b8-47bf-8f9b-436fa4cf1436-istio-data\") pod \"01e6f154-38b8-47bf-8f9b-436fa4cf1436\" (UID: \"01e6f154-38b8-47bf-8f9b-436fa4cf1436\") " Nov 03 13:26:18.137740 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:18.137712 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01e6f154-38b8-47bf-8f9b-436fa4cf1436-credential-socket" pod "01e6f154-38b8-47bf-8f9b-436fa4cf1436" (UID: "01e6f154-38b8-47bf-8f9b-436fa4cf1436"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:26:18.137875 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:18.137852 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01e6f154-38b8-47bf-8f9b-436fa4cf1436-istio-data" pod "01e6f154-38b8-47bf-8f9b-436fa4cf1436" (UID: "01e6f154-38b8-47bf-8f9b-436fa4cf1436"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:26:18.138290 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:18.138265 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01e6f154-38b8-47bf-8f9b-436fa4cf1436-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01e6f154-38b8-47bf-8f9b-436fa4cf1436-workload-socket\") pod \"01e6f154-38b8-47bf-8f9b-436fa4cf1436\" (UID: \"01e6f154-38b8-47bf-8f9b-436fa4cf1436\") " Nov 03 13:26:18.138391 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:18.138334 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01e6f154-38b8-47bf-8f9b-436fa4cf1436-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01e6f154-38b8-47bf-8f9b-436fa4cf1436-istio-envoy\") pod \"01e6f154-38b8-47bf-8f9b-436fa4cf1436\" (UID: \"01e6f154-38b8-47bf-8f9b-436fa4cf1436\") " Nov 03 13:26:18.138457 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:18.138427 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01e6f154-38b8-47bf-8f9b-436fa4cf1436-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01e6f154-38b8-47bf-8f9b-436fa4cf1436-workload-certs\") pod \"01e6f154-38b8-47bf-8f9b-436fa4cf1436\" (UID: \"01e6f154-38b8-47bf-8f9b-436fa4cf1436\") " Nov 03 13:26:18.138538 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:18.138456 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01e6f154-38b8-47bf-8f9b-436fa4cf1436-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01e6f154-38b8-47bf-8f9b-436fa4cf1436-istiod-ca-cert\") pod \"01e6f154-38b8-47bf-8f9b-436fa4cf1436\" (UID: \"01e6f154-38b8-47bf-8f9b-436fa4cf1436\") " Nov 03 13:26:18.139043 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:18.139018 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01e6f154-38b8-47bf-8f9b-436fa4cf1436-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:26:18.139119 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:18.139046 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01e6f154-38b8-47bf-8f9b-436fa4cf1436-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:26:18.139423 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:18.139402 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01e6f154-38b8-47bf-8f9b-436fa4cf1436-istiod-ca-cert" pod "01e6f154-38b8-47bf-8f9b-436fa4cf1436" (UID: "01e6f154-38b8-47bf-8f9b-436fa4cf1436"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:26:18.139735 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:18.139707 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01e6f154-38b8-47bf-8f9b-436fa4cf1436-istio-podinfo" pod "01e6f154-38b8-47bf-8f9b-436fa4cf1436" (UID: "01e6f154-38b8-47bf-8f9b-436fa4cf1436"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:26:18.139832 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:18.139748 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01e6f154-38b8-47bf-8f9b-436fa4cf1436-istio-token" pod "01e6f154-38b8-47bf-8f9b-436fa4cf1436" (UID: "01e6f154-38b8-47bf-8f9b-436fa4cf1436"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:26:18.139832 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:18.139818 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01e6f154-38b8-47bf-8f9b-436fa4cf1436-kube-api-access-c98bt" pod "01e6f154-38b8-47bf-8f9b-436fa4cf1436" (UID: "01e6f154-38b8-47bf-8f9b-436fa4cf1436"). InnerVolumeSpecName "kube-api-access-c98bt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:26:18.139922 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:18.139835 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01e6f154-38b8-47bf-8f9b-436fa4cf1436-workload-socket" pod "01e6f154-38b8-47bf-8f9b-436fa4cf1436" (UID: "01e6f154-38b8-47bf-8f9b-436fa4cf1436"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:26:18.139922 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:18.139890 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01e6f154-38b8-47bf-8f9b-436fa4cf1436-workload-certs" pod "01e6f154-38b8-47bf-8f9b-436fa4cf1436" (UID: "01e6f154-38b8-47bf-8f9b-436fa4cf1436"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:26:18.141315 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:18.141291 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01e6f154-38b8-47bf-8f9b-436fa4cf1436-istio-envoy" pod "01e6f154-38b8-47bf-8f9b-436fa4cf1436" (UID: "01e6f154-38b8-47bf-8f9b-436fa4cf1436"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:26:18.240392 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:18.240356 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01e6f154-38b8-47bf-8f9b-436fa4cf1436-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:26:18.240392 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:18.240386 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01e6f154-38b8-47bf-8f9b-436fa4cf1436-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:26:18.240634 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:18.240401 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01e6f154-38b8-47bf-8f9b-436fa4cf1436-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:26:18.240634 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:18.240417 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c98bt\" (UniqueName: \"kubernetes.io/projected/01e6f154-38b8-47bf-8f9b-436fa4cf1436-kube-api-access-c98bt\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:26:18.240634 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:18.240430 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01e6f154-38b8-47bf-8f9b-436fa4cf1436-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:26:18.240634 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:18.240441 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01e6f154-38b8-47bf-8f9b-436fa4cf1436-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:26:18.240634 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:18.240453 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01e6f154-38b8-47bf-8f9b-436fa4cf1436-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:26:18.418361 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:18.418331 5979 scope.go:122] "RemoveContainer" containerID="13f26f6159be9f1afdba03b9ca1a53290a3e754a9d2567e51451c4b605a62760" Nov 03 13:26:18.436063 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:18.436037 5979 scope.go:122] "RemoveContainer" containerID="51b3ad4623534410f3f945a318e3b202ef2570630e7fdd9d055cc26806ce0299" Nov 03 13:26:18.469189 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:18.469160 5979 scope.go:122] "RemoveContainer" containerID="c5e4a2dd70e2ae178f24143eff509a5ba3b66e4d4a02725b02841013f1b4e6e2" Nov 03 13:26:18.815064 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:18.815027 5979 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 03 13:26:18.815232 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:18.815082 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb" podUID="cfd3eded-c3de-4d6b-9a1e-c284549c6c18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 03 13:26:18.882220 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:18.882180 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4" event={"ID":"01e6f154-38b8-47bf-8f9b-436fa4cf1436","Type":"ContainerDied","Data":"00a5afe5ccb26049911a2ba4eabcb755268092def35fa421f9a867b7c159129b"} Nov 03 13:26:18.884026 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:18.884001 5979 generic.go:356] "Generic (PLEG): container finished" podID="1a8f32fb-a894-49b1-8da8-3cccbca598cc" containerID="7b26f60f94453ea1e97ed5ae4e3343e50989071b14d629c22ab73c7b529e6254" exitCode=0 Nov 03 13:26:18.884156 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:18.884085 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a9f79b-2kx52" event={"ID":"1a8f32fb-a894-49b1-8da8-3cccbca598cc","Type":"ContainerDied","Data":"7b26f60f94453ea1e97ed5ae4e3343e50989071b14d629c22ab73c7b529e6254"} Nov 03 13:26:18.884156 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:18.884115 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4" Nov 03 13:26:19.223441 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:19.223407 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4"] Nov 03 13:26:19.227771 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:19.227745 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7wwf4"] Nov 03 13:26:19.815093 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:19.815055 5979 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 03 13:26:19.815288 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:19.815120 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb" podUID="cfd3eded-c3de-4d6b-9a1e-c284549c6c18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 03 13:26:20.119229 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:20.119197 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01e6f154-38b8-47bf-8f9b-436fa4cf1436" path="/var/lib/kubelet/pods/01e6f154-38b8-47bf-8f9b-436fa4cf1436/volumes" Nov 03 13:26:20.205549 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:20.205523 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a9f79b-2kx52" Nov 03 13:26:20.297904 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:20.297874 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a8f32fb-a894-49b1-8da8-3cccbca598cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a8f32fb-a894-49b1-8da8-3cccbca598cc-maas-api-openshift-ca\") pod \"1a8f32fb-a894-49b1-8da8-3cccbca598cc\" (UID: \"1a8f32fb-a894-49b1-8da8-3cccbca598cc\") " Nov 03 13:26:20.298345 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:20.297968 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a8f32fb-a894-49b1-8da8-3cccbca598cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a8f32fb-a894-49b1-8da8-3cccbca598cc-maas-api-xks-ca\") pod \"1a8f32fb-a894-49b1-8da8-3cccbca598cc\" (UID: \"1a8f32fb-a894-49b1-8da8-3cccbca598cc\") " Nov 03 13:26:20.298345 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:20.298082 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a8f32fb-a894-49b1-8da8-3cccbca598cc-kube-api-access-5wzgs\" (UniqueName: \"kubernetes.io/projected/1a8f32fb-a894-49b1-8da8-3cccbca598cc-kube-api-access-5wzgs\") pod \"1a8f32fb-a894-49b1-8da8-3cccbca598cc\" (UID: \"1a8f32fb-a894-49b1-8da8-3cccbca598cc\") " Nov 03 13:26:20.298345 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:20.298230 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a8f32fb-a894-49b1-8da8-3cccbca598cc-maas-api-openshift-ca" pod "1a8f32fb-a894-49b1-8da8-3cccbca598cc" (UID: "1a8f32fb-a894-49b1-8da8-3cccbca598cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:26:20.298646 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:20.298629 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a8f32fb-a894-49b1-8da8-3cccbca598cc-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:26:20.300172 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:20.300148 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a8f32fb-a894-49b1-8da8-3cccbca598cc-maas-api-xks-ca" pod "1a8f32fb-a894-49b1-8da8-3cccbca598cc" (UID: "1a8f32fb-a894-49b1-8da8-3cccbca598cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:26:20.300299 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:20.300273 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a8f32fb-a894-49b1-8da8-3cccbca598cc-kube-api-access-5wzgs" pod "1a8f32fb-a894-49b1-8da8-3cccbca598cc" (UID: "1a8f32fb-a894-49b1-8da8-3cccbca598cc"). InnerVolumeSpecName "kube-api-access-5wzgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:26:20.399754 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:20.399720 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wzgs\" (UniqueName: \"kubernetes.io/projected/1a8f32fb-a894-49b1-8da8-3cccbca598cc-kube-api-access-5wzgs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:26:20.399754 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:20.399753 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a8f32fb-a894-49b1-8da8-3cccbca598cc-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:26:20.814597 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:20.814512 5979 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Nov 03 13:26:20.814741 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:20.814585 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb" podUID="cfd3eded-c3de-4d6b-9a1e-c284549c6c18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Nov 03 13:26:20.933667 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:20.933637 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a9f79b-2kx52" Nov 03 13:26:20.933841 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:20.933672 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60a9f79b-2kx52" event={"ID":"1a8f32fb-a894-49b1-8da8-3cccbca598cc","Type":"ContainerDied","Data":"c91707f196f1d0a8d2e8e4d6e21b6cae27c313ffadb61fd20a5ee61d1f9f72d5"} Nov 03 13:26:20.933841 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:20.933755 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c91707f196f1d0a8d2e8e4d6e21b6cae27c313ffadb61fd20a5ee61d1f9f72d5" Nov 03 13:26:21.940309 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:21.940279 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb" Nov 03 13:26:21.941203 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:21.941168 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb" Nov 03 13:26:21.942131 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:21.942110 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb" Nov 03 13:26:37.256243 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:26:37.256201 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:26:38.496496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:38.495451 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e11b3dc0-flp98"] Nov 03 13:26:38.496496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:38.496435 5979 cpu_manager.go:414] "containerMap: removing container" podUID="01e6f154-38b8-47bf-8f9b-436fa4cf1436" containerName="istio-proxy" Nov 03 13:26:38.496496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:38.496456 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01e6f154-38b8-47bf-8f9b-436fa4cf1436" containerName="istio-proxy" Nov 03 13:26:38.496496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:38.496499 5979 cpu_manager.go:414] "containerMap: removing container" podUID="1a8f32fb-a894-49b1-8da8-3cccbca598cc" containerName="revoke-keys" Nov 03 13:26:38.496496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:38.496507 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a8f32fb-a894-49b1-8da8-3cccbca598cc" containerName="revoke-keys" Nov 03 13:26:38.497234 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:38.496545 5979 cpu_manager.go:414] "containerMap: removing container" podUID="01e6f154-38b8-47bf-8f9b-436fa4cf1436" containerName="istio-proxy" Nov 03 13:26:38.497234 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:38.496555 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01e6f154-38b8-47bf-8f9b-436fa4cf1436" containerName="istio-proxy" Nov 03 13:26:38.497234 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:38.496720 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a8f32fb-a894-49b1-8da8-3cccbca598cc" containerName="revoke-keys" Nov 03 13:26:38.497234 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:38.496739 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="01e6f154-38b8-47bf-8f9b-436fa4cf1436" containerName="istio-proxy" Nov 03 13:26:38.522768 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:38.522736 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11b3dc0-flp98" Nov 03 13:26:38.546531 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:38.541156 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e11b3dc0-flp98"] Nov 03 13:26:38.611743 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:38.607913 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:26:38.611743 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:38.608090 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:26:38.611743 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:38.609564 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccdc18d4-45e3-497e-9f79-454392831e04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e11b3dc0-flp98\" (UID: \"ccdc18d4-45e3-497e-9f79-454392831e04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11b3dc0-flp98" Nov 03 13:26:38.611743 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:38.609613 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52v4j\" (UniqueName: \"kubernetes.io/projected/ccdc18d4-45e3-497e-9f79-454392831e04-kube-api-access-52v4j\") pod \"maas-api-revoke-sub-e11b3dc0-flp98\" (UID: \"ccdc18d4-45e3-497e-9f79-454392831e04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11b3dc0-flp98" Nov 03 13:26:38.611743 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:38.609647 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccdc18d4-45e3-497e-9f79-454392831e04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e11b3dc0-flp98\" (UID: \"ccdc18d4-45e3-497e-9f79-454392831e04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11b3dc0-flp98" Nov 03 13:26:38.654652 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:38.654624 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 13:26:38.654868 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:38.654811 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 13:26:38.710559 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:38.710530 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccdc18d4-45e3-497e-9f79-454392831e04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e11b3dc0-flp98\" (UID: \"ccdc18d4-45e3-497e-9f79-454392831e04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11b3dc0-flp98" Nov 03 13:26:38.710756 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:38.710590 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52v4j\" (UniqueName: \"kubernetes.io/projected/ccdc18d4-45e3-497e-9f79-454392831e04-kube-api-access-52v4j\") pod \"maas-api-revoke-sub-e11b3dc0-flp98\" (UID: \"ccdc18d4-45e3-497e-9f79-454392831e04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11b3dc0-flp98" Nov 03 13:26:38.710756 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:38.710645 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccdc18d4-45e3-497e-9f79-454392831e04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e11b3dc0-flp98\" (UID: \"ccdc18d4-45e3-497e-9f79-454392831e04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11b3dc0-flp98" Nov 03 13:26:38.711365 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:38.711336 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccdc18d4-45e3-497e-9f79-454392831e04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e11b3dc0-flp98\" (UID: \"ccdc18d4-45e3-497e-9f79-454392831e04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11b3dc0-flp98" Nov 03 13:26:38.713453 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:38.713426 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccdc18d4-45e3-497e-9f79-454392831e04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e11b3dc0-flp98\" (UID: \"ccdc18d4-45e3-497e-9f79-454392831e04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11b3dc0-flp98" Nov 03 13:26:38.764961 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:38.764891 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52v4j\" (UniqueName: \"kubernetes.io/projected/ccdc18d4-45e3-497e-9f79-454392831e04-kube-api-access-52v4j\") pod \"maas-api-revoke-sub-e11b3dc0-flp98\" (UID: \"ccdc18d4-45e3-497e-9f79-454392831e04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11b3dc0-flp98" Nov 03 13:26:38.947272 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:38.947235 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11b3dc0-flp98" Nov 03 13:26:39.322858 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:39.322825 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e11b3dc0-flp98"] Nov 03 13:26:39.338090 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:26:39.338057 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccdc18d4_45e3_497e_9f79_454392831e04.slice/crio-0eeb0c02d3594996f8b06f0c5af95626a64c2953d5acece9a08f6ce0f0447396 WatchSource:0}: Error finding container 0eeb0c02d3594996f8b06f0c5af95626a64c2953d5acece9a08f6ce0f0447396: Status 404 returned error can't find the container with id 0eeb0c02d3594996f8b06f0c5af95626a64c2953d5acece9a08f6ce0f0447396 Nov 03 13:26:40.089763 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:40.089719 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11b3dc0-flp98" event={"ID":"ccdc18d4-45e3-497e-9f79-454392831e04","Type":"ContainerStarted","Data":"c2b839e72b5321a77bd94d966ff62660f84ef96187e9e109091b253d0dcf4beb"} Nov 03 13:26:40.089763 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:40.089788 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11b3dc0-flp98" event={"ID":"ccdc18d4-45e3-497e-9f79-454392831e04","Type":"ContainerStarted","Data":"0eeb0c02d3594996f8b06f0c5af95626a64c2953d5acece9a08f6ce0f0447396"} Nov 03 13:26:40.199287 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:40.199225 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11b3dc0-flp98" podStartSLOduration=2.199208289 podStartE2EDuration="2.199208289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:26:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:26:40.197978577 +0000 UTC m=+1282.208666816" watchObservedRunningTime="2026-11-03 13:26:40.199208289 +0000 UTC m=+1282.209896531" Nov 03 13:26:41.125376 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:41.125330 5979 generic.go:356] "Generic (PLEG): container finished" podID="ccdc18d4-45e3-497e-9f79-454392831e04" containerID="c2b839e72b5321a77bd94d966ff62660f84ef96187e9e109091b253d0dcf4beb" exitCode=0 Nov 03 13:26:41.125941 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:41.125405 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11b3dc0-flp98" event={"ID":"ccdc18d4-45e3-497e-9f79-454392831e04","Type":"ContainerDied","Data":"c2b839e72b5321a77bd94d966ff62660f84ef96187e9e109091b253d0dcf4beb"} Nov 03 13:26:41.850496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:41.847568 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-mksft container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:26:41.850496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:41.847632 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mksft" podUID="42cb885d-1072-4057-ab7d-71fe9f166f7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:26:42.500494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:42.491451 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11b3dc0-flp98" Nov 03 13:26:42.583366 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:42.583325 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccdc18d4-45e3-497e-9f79-454392831e04-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccdc18d4-45e3-497e-9f79-454392831e04-maas-api-openshift-ca\") pod \"ccdc18d4-45e3-497e-9f79-454392831e04\" (UID: \"ccdc18d4-45e3-497e-9f79-454392831e04\") " Nov 03 13:26:42.583628 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:42.583391 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccdc18d4-45e3-497e-9f79-454392831e04-kube-api-access-52v4j\" (UniqueName: \"kubernetes.io/projected/ccdc18d4-45e3-497e-9f79-454392831e04-kube-api-access-52v4j\") pod \"ccdc18d4-45e3-497e-9f79-454392831e04\" (UID: \"ccdc18d4-45e3-497e-9f79-454392831e04\") " Nov 03 13:26:42.583628 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:42.583550 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccdc18d4-45e3-497e-9f79-454392831e04-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccdc18d4-45e3-497e-9f79-454392831e04-maas-api-xks-ca\") pod \"ccdc18d4-45e3-497e-9f79-454392831e04\" (UID: \"ccdc18d4-45e3-497e-9f79-454392831e04\") " Nov 03 13:26:42.600517 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:42.597320 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccdc18d4-45e3-497e-9f79-454392831e04-kube-api-access-52v4j" pod "ccdc18d4-45e3-497e-9f79-454392831e04" (UID: "ccdc18d4-45e3-497e-9f79-454392831e04"). InnerVolumeSpecName "kube-api-access-52v4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:26:42.636965 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:42.604762 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccdc18d4-45e3-497e-9f79-454392831e04-maas-api-openshift-ca" pod "ccdc18d4-45e3-497e-9f79-454392831e04" (UID: "ccdc18d4-45e3-497e-9f79-454392831e04"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:26:42.652985 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:42.652713 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccdc18d4-45e3-497e-9f79-454392831e04-maas-api-xks-ca" pod "ccdc18d4-45e3-497e-9f79-454392831e04" (UID: "ccdc18d4-45e3-497e-9f79-454392831e04"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:26:42.685849 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:42.685817 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccdc18d4-45e3-497e-9f79-454392831e04-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:26:42.685849 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:42.685847 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52v4j\" (UniqueName: \"kubernetes.io/projected/ccdc18d4-45e3-497e-9f79-454392831e04-kube-api-access-52v4j\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:26:42.686091 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:42.685859 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccdc18d4-45e3-497e-9f79-454392831e04-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:26:43.139137 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:43.139107 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11b3dc0-flp98" event={"ID":"ccdc18d4-45e3-497e-9f79-454392831e04","Type":"ContainerDied","Data":"0eeb0c02d3594996f8b06f0c5af95626a64c2953d5acece9a08f6ce0f0447396"} Nov 03 13:26:43.139280 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:43.139145 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e11b3dc0-flp98" Nov 03 13:26:43.139280 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:43.139151 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0eeb0c02d3594996f8b06f0c5af95626a64c2953d5acece9a08f6ce0f0447396" Nov 03 13:26:51.788797 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:51.788765 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8"] Nov 03 13:26:51.789360 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:51.789346 5979 cpu_manager.go:414] "containerMap: removing container" podUID="ccdc18d4-45e3-497e-9f79-454392831e04" containerName="revoke-keys" Nov 03 13:26:51.789407 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:51.789361 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccdc18d4-45e3-497e-9f79-454392831e04" containerName="revoke-keys" Nov 03 13:26:51.789496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:51.789461 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="01e6f154-38b8-47bf-8f9b-436fa4cf1436" containerName="istio-proxy" Nov 03 13:26:51.789496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:51.789491 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccdc18d4-45e3-497e-9f79-454392831e04" containerName="revoke-keys" Nov 03 13:26:51.793544 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:51.793526 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8" Nov 03 13:26:51.842663 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:51.842629 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-30074e-openshift-default-dockercfg-j6dvj\"" Nov 03 13:26:51.894038 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:51.894001 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8"] Nov 03 13:26:51.910265 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:51.910236 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-istio-podinfo\") pod \"e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8\" (UID: \"53fab3c0-d97b-4c5d-ad31-798afd7c5b6c\") " pod="openshift-ingress/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8" Nov 03 13:26:51.910412 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:51.910290 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-workload-socket\") pod \"e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8\" (UID: \"53fab3c0-d97b-4c5d-ad31-798afd7c5b6c\") " pod="openshift-ingress/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8" Nov 03 13:26:51.910412 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:51.910369 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-istio-envoy\") pod \"e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8\" (UID: \"53fab3c0-d97b-4c5d-ad31-798afd7c5b6c\") " pod="openshift-ingress/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8" Nov 03 13:26:51.910526 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:51.910457 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-istio-token\") pod \"e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8\" (UID: \"53fab3c0-d97b-4c5d-ad31-798afd7c5b6c\") " pod="openshift-ingress/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8" Nov 03 13:26:51.910563 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:51.910520 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-credential-socket\") pod \"e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8\" (UID: \"53fab3c0-d97b-4c5d-ad31-798afd7c5b6c\") " pod="openshift-ingress/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8" Nov 03 13:26:51.910639 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:51.910621 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-istio-data\") pod \"e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8\" (UID: \"53fab3c0-d97b-4c5d-ad31-798afd7c5b6c\") " pod="openshift-ingress/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8" Nov 03 13:26:51.910678 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:51.910665 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-workload-certs\") pod \"e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8\" (UID: \"53fab3c0-d97b-4c5d-ad31-798afd7c5b6c\") " pod="openshift-ingress/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8" Nov 03 13:26:51.910711 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:51.910694 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-istiod-ca-cert\") pod \"e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8\" (UID: \"53fab3c0-d97b-4c5d-ad31-798afd7c5b6c\") " pod="openshift-ingress/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8" Nov 03 13:26:51.910812 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:51.910793 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxl67\" (UniqueName: \"kubernetes.io/projected/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-kube-api-access-wxl67\") pod \"e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8\" (UID: \"53fab3c0-d97b-4c5d-ad31-798afd7c5b6c\") " pod="openshift-ingress/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8" Nov 03 13:26:52.015511 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:52.014898 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxl67\" (UniqueName: \"kubernetes.io/projected/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-kube-api-access-wxl67\") pod \"e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8\" (UID: \"53fab3c0-d97b-4c5d-ad31-798afd7c5b6c\") " pod="openshift-ingress/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8" Nov 03 13:26:52.021091 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:52.017044 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-istio-podinfo\") pod \"e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8\" (UID: \"53fab3c0-d97b-4c5d-ad31-798afd7c5b6c\") " pod="openshift-ingress/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8" Nov 03 13:26:52.021091 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:52.017142 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-workload-socket\") pod \"e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8\" (UID: \"53fab3c0-d97b-4c5d-ad31-798afd7c5b6c\") " pod="openshift-ingress/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8" Nov 03 13:26:52.021091 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:52.017214 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-istio-envoy\") pod \"e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8\" (UID: \"53fab3c0-d97b-4c5d-ad31-798afd7c5b6c\") " pod="openshift-ingress/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8" Nov 03 13:26:52.021091 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:52.017298 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-istio-token\") pod \"e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8\" (UID: \"53fab3c0-d97b-4c5d-ad31-798afd7c5b6c\") " pod="openshift-ingress/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8" Nov 03 13:26:52.021091 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:52.017371 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-credential-socket\") pod \"e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8\" (UID: \"53fab3c0-d97b-4c5d-ad31-798afd7c5b6c\") " pod="openshift-ingress/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8" Nov 03 13:26:52.021091 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:52.017663 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-istio-data\") pod \"e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8\" (UID: \"53fab3c0-d97b-4c5d-ad31-798afd7c5b6c\") " pod="openshift-ingress/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8" Nov 03 13:26:52.021091 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:52.017761 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-workload-certs\") pod \"e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8\" (UID: \"53fab3c0-d97b-4c5d-ad31-798afd7c5b6c\") " pod="openshift-ingress/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8" Nov 03 13:26:52.021091 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:52.017822 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-istiod-ca-cert\") pod \"e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8\" (UID: \"53fab3c0-d97b-4c5d-ad31-798afd7c5b6c\") " pod="openshift-ingress/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8" Nov 03 13:26:52.021091 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:52.019494 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-istiod-ca-cert\") pod \"e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8\" (UID: \"53fab3c0-d97b-4c5d-ad31-798afd7c5b6c\") " pod="openshift-ingress/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8" Nov 03 13:26:52.021091 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:52.020434 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-workload-socket\") pod \"e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8\" (UID: \"53fab3c0-d97b-4c5d-ad31-798afd7c5b6c\") " pod="openshift-ingress/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8" Nov 03 13:26:52.034831 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:52.021831 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-workload-certs\") pod \"e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8\" (UID: \"53fab3c0-d97b-4c5d-ad31-798afd7c5b6c\") " pod="openshift-ingress/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8" Nov 03 13:26:52.034831 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:52.022164 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-credential-socket\") pod \"e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8\" (UID: \"53fab3c0-d97b-4c5d-ad31-798afd7c5b6c\") " pod="openshift-ingress/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8" Nov 03 13:26:52.034831 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:52.022455 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-istio-data\") pod \"e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8\" (UID: \"53fab3c0-d97b-4c5d-ad31-798afd7c5b6c\") " pod="openshift-ingress/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8" Nov 03 13:26:52.036086 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:52.036049 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-istio-podinfo\") pod \"e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8\" (UID: \"53fab3c0-d97b-4c5d-ad31-798afd7c5b6c\") " pod="openshift-ingress/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8" Nov 03 13:26:52.037754 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:52.037732 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-istio-envoy\") pod \"e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8\" (UID: \"53fab3c0-d97b-4c5d-ad31-798afd7c5b6c\") " pod="openshift-ingress/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8" Nov 03 13:26:52.071187 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:52.071116 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxl67\" (UniqueName: \"kubernetes.io/projected/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-kube-api-access-wxl67\") pod \"e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8\" (UID: \"53fab3c0-d97b-4c5d-ad31-798afd7c5b6c\") " pod="openshift-ingress/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8" Nov 03 13:26:52.076847 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:52.076821 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-istio-token\") pod \"e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8\" (UID: \"53fab3c0-d97b-4c5d-ad31-798afd7c5b6c\") " pod="openshift-ingress/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8" Nov 03 13:26:52.106984 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:52.106715 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8" Nov 03 13:26:52.601097 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:52.601060 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8"] Nov 03 13:26:52.607696 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:26:52.607669 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53fab3c0_d97b_4c5d_ad31_798afd7c5b6c.slice/crio-c6baab293447a30adf71f78758975b18d45ff29684cbc73f8a62a200e600f7c4 WatchSource:0}: Error finding container c6baab293447a30adf71f78758975b18d45ff29684cbc73f8a62a200e600f7c4: Status 404 returned error can't find the container with id c6baab293447a30adf71f78758975b18d45ff29684cbc73f8a62a200e600f7c4 Nov 03 13:26:52.610028 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:52.609994 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:26:52.610108 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:52.610062 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:26:52.610108 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:52.610090 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:26:53.208383 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:53.208345 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8" event={"ID":"53fab3c0-d97b-4c5d-ad31-798afd7c5b6c","Type":"ContainerStarted","Data":"ff7a2dcb437a5a04242e4663480438260cbdca082dd0eb14c0da32c2e7fee8f7"} Nov 03 13:26:53.208904 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:53.208406 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8" event={"ID":"53fab3c0-d97b-4c5d-ad31-798afd7c5b6c","Type":"ContainerStarted","Data":"c6baab293447a30adf71f78758975b18d45ff29684cbc73f8a62a200e600f7c4"} Nov 03 13:26:53.321205 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:53.321150 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8" podStartSLOduration=2.321136554 podStartE2EDuration="2.321136554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:26:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:26:53.318584881 +0000 UTC m=+1295.329273123" watchObservedRunningTime="2026-11-03 13:26:53.321136554 +0000 UTC m=+1295.331824795" Nov 03 13:26:53.612576 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:53.612496 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:26:54.108392 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:54.108357 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8" Nov 03 13:26:54.116769 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:54.116743 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8" Nov 03 13:26:54.214842 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:54.214815 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8" Nov 03 13:26:54.215717 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:26:54.215694 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8" Nov 03 13:27:02.176769 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:02.176734 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-30074e-5dd5666c67-gkb7j"] Nov 03 13:27:02.182382 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:02.182358 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-30074e-5dd5666c67-gkb7j" Nov 03 13:27:02.269508 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:02.264807 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-30074e-dockercfg-x7zdk\"" Nov 03 13:27:02.278456 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:02.278420 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-30074e-5dd5666c67-gkb7j"] Nov 03 13:27:02.326499 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:02.322317 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-30074e\"" Nov 03 13:27:02.328265 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:02.328229 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p68b6\" (UniqueName: \"kubernetes.io/projected/f59b1527-f24d-47bb-bbd2-3de97661eeee-kube-api-access-p68b6\") pod \"payload-pre-processing-e2e-api-a-30074e-5dd5666c67-gkb7j\" (UID: \"f59b1527-f24d-47bb-bbd2-3de97661eeee\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-30074e-5dd5666c67-gkb7j" Nov 03 13:27:02.328485 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:02.328448 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f59b1527-f24d-47bb-bbd2-3de97661eeee-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-30074e-5dd5666c67-gkb7j\" (UID: \"f59b1527-f24d-47bb-bbd2-3de97661eeee\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-30074e-5dd5666c67-gkb7j" Nov 03 13:27:02.339238 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:02.339186 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-30074e-7986f65ffc-82gzw"] Nov 03 13:27:02.348536 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:02.346361 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-30074e-7986f65ffc-82gzw" Nov 03 13:27:02.454136 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:02.454048 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-30074e-7986f65ffc-82gzw"] Nov 03 13:27:02.458499 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:02.457303 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5hrt\" (UniqueName: \"kubernetes.io/projected/ef6ffd56-1244-4430-9740-436407eb0ebe-kube-api-access-m5hrt\") pod \"payload-processing-e2e-api-a-30074e-7986f65ffc-82gzw\" (UID: \"ef6ffd56-1244-4430-9740-436407eb0ebe\") " pod="openshift-ingress/payload-processing-e2e-api-a-30074e-7986f65ffc-82gzw" Nov 03 13:27:02.458499 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:02.457381 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef6ffd56-1244-4430-9740-436407eb0ebe-plugins-config-volume\") pod \"payload-processing-e2e-api-a-30074e-7986f65ffc-82gzw\" (UID: \"ef6ffd56-1244-4430-9740-436407eb0ebe\") " pod="openshift-ingress/payload-processing-e2e-api-a-30074e-7986f65ffc-82gzw" Nov 03 13:27:02.458499 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:02.457522 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p68b6\" (UniqueName: \"kubernetes.io/projected/f59b1527-f24d-47bb-bbd2-3de97661eeee-kube-api-access-p68b6\") pod \"payload-pre-processing-e2e-api-a-30074e-5dd5666c67-gkb7j\" (UID: \"f59b1527-f24d-47bb-bbd2-3de97661eeee\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-30074e-5dd5666c67-gkb7j" Nov 03 13:27:02.458499 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:02.457553 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f59b1527-f24d-47bb-bbd2-3de97661eeee-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-30074e-5dd5666c67-gkb7j\" (UID: \"f59b1527-f24d-47bb-bbd2-3de97661eeee\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-30074e-5dd5666c67-gkb7j" Nov 03 13:27:02.458795 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:02.458584 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f59b1527-f24d-47bb-bbd2-3de97661eeee-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-30074e-5dd5666c67-gkb7j\" (UID: \"f59b1527-f24d-47bb-bbd2-3de97661eeee\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-30074e-5dd5666c67-gkb7j" Nov 03 13:27:02.513633 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:02.513599 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p68b6\" (UniqueName: \"kubernetes.io/projected/f59b1527-f24d-47bb-bbd2-3de97661eeee-kube-api-access-p68b6\") pod \"payload-pre-processing-e2e-api-a-30074e-5dd5666c67-gkb7j\" (UID: \"f59b1527-f24d-47bb-bbd2-3de97661eeee\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-30074e-5dd5666c67-gkb7j" Nov 03 13:27:02.558369 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:02.558338 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5hrt\" (UniqueName: \"kubernetes.io/projected/ef6ffd56-1244-4430-9740-436407eb0ebe-kube-api-access-m5hrt\") pod \"payload-processing-e2e-api-a-30074e-7986f65ffc-82gzw\" (UID: \"ef6ffd56-1244-4430-9740-436407eb0ebe\") " pod="openshift-ingress/payload-processing-e2e-api-a-30074e-7986f65ffc-82gzw" Nov 03 13:27:02.558563 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:02.558388 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef6ffd56-1244-4430-9740-436407eb0ebe-plugins-config-volume\") pod \"payload-processing-e2e-api-a-30074e-7986f65ffc-82gzw\" (UID: \"ef6ffd56-1244-4430-9740-436407eb0ebe\") " pod="openshift-ingress/payload-processing-e2e-api-a-30074e-7986f65ffc-82gzw" Nov 03 13:27:02.559046 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:02.559025 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef6ffd56-1244-4430-9740-436407eb0ebe-plugins-config-volume\") pod \"payload-processing-e2e-api-a-30074e-7986f65ffc-82gzw\" (UID: \"ef6ffd56-1244-4430-9740-436407eb0ebe\") " pod="openshift-ingress/payload-processing-e2e-api-a-30074e-7986f65ffc-82gzw" Nov 03 13:27:02.612614 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:02.612586 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5hrt\" (UniqueName: \"kubernetes.io/projected/ef6ffd56-1244-4430-9740-436407eb0ebe-kube-api-access-m5hrt\") pod \"payload-processing-e2e-api-a-30074e-7986f65ffc-82gzw\" (UID: \"ef6ffd56-1244-4430-9740-436407eb0ebe\") " pod="openshift-ingress/payload-processing-e2e-api-a-30074e-7986f65ffc-82gzw" Nov 03 13:27:02.770621 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:02.770541 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-30074e-7986f65ffc-82gzw" Nov 03 13:27:02.797699 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:02.797664 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-30074e-5dd5666c67-gkb7j" Nov 03 13:27:03.152140 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:03.152109 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-30074e-7986f65ffc-82gzw"] Nov 03 13:27:03.157692 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:27:03.157665 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef6ffd56_1244_4430_9740_436407eb0ebe.slice/crio-82bbb7d740d1280fd9e2905f897109d5fcc8bd6b3e348c6f5c9dd87fcb1f6fe2 WatchSource:0}: Error finding container 82bbb7d740d1280fd9e2905f897109d5fcc8bd6b3e348c6f5c9dd87fcb1f6fe2: Status 404 returned error can't find the container with id 82bbb7d740d1280fd9e2905f897109d5fcc8bd6b3e348c6f5c9dd87fcb1f6fe2 Nov 03 13:27:03.203303 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:03.203273 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-30074e-5dd5666c67-gkb7j"] Nov 03 13:27:03.209774 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:27:03.209749 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf59b1527_f24d_47bb_bbd2_3de97661eeee.slice/crio-b0235964855c9593a0ab5f2ad5a56d5587916905753c2b97bcb6d91096c3a4fb WatchSource:0}: Error finding container b0235964855c9593a0ab5f2ad5a56d5587916905753c2b97bcb6d91096c3a4fb: Status 404 returned error can't find the container with id b0235964855c9593a0ab5f2ad5a56d5587916905753c2b97bcb6d91096c3a4fb Nov 03 13:27:03.330545 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:03.330507 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-30074e-7986f65ffc-82gzw" event={"ID":"ef6ffd56-1244-4430-9740-436407eb0ebe","Type":"ContainerStarted","Data":"42309357fb4917487bee8cb84fd479bdc1b9af23e1de9bbb36d3fafc9159ffb6"} Nov 03 13:27:03.330747 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:03.330574 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-30074e-7986f65ffc-82gzw" event={"ID":"ef6ffd56-1244-4430-9740-436407eb0ebe","Type":"ContainerStarted","Data":"82bbb7d740d1280fd9e2905f897109d5fcc8bd6b3e348c6f5c9dd87fcb1f6fe2"} Nov 03 13:27:03.330747 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:03.330702 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-30074e-7986f65ffc-82gzw" Nov 03 13:27:03.332089 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:03.332065 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-30074e-5dd5666c67-gkb7j" event={"ID":"f59b1527-f24d-47bb-bbd2-3de97661eeee","Type":"ContainerStarted","Data":"7e50e4e5c0ffc192c0511b98f2edba69f71ed1588d20e2c55cb51ea21b896a79"} Nov 03 13:27:03.332179 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:03.332102 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-30074e-5dd5666c67-gkb7j" event={"ID":"f59b1527-f24d-47bb-bbd2-3de97661eeee","Type":"ContainerStarted","Data":"b0235964855c9593a0ab5f2ad5a56d5587916905753c2b97bcb6d91096c3a4fb"} Nov 03 13:27:03.332222 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:03.332207 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-30074e-5dd5666c67-gkb7j" Nov 03 13:27:03.442845 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:03.442747 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-30074e-7986f65ffc-82gzw" podStartSLOduration=1.442733078 podStartE2EDuration="1.442733078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:27:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:27:03.439556519 +0000 UTC m=+1305.450244761" watchObservedRunningTime="2026-11-03 13:27:03.442733078 +0000 UTC m=+1305.453421320" Nov 03 13:27:03.553869 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:03.553812 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-30074e-5dd5666c67-gkb7j" podStartSLOduration=1.5537945720000002 podStartE2EDuration="1.553794572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:27:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:27:03.553328603 +0000 UTC m=+1305.564016845" watchObservedRunningTime="2026-11-03 13:27:03.553794572 +0000 UTC m=+1305.564482815" Nov 03 13:27:07.234135 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:27:07.234091 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:27:11.957548 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:11.957511 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7"] Nov 03 13:27:11.962245 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:11.962222 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7" Nov 03 13:27:12.034972 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:12.034930 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-istiod-ca-cert\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7\" (UID: \"42d9f732-f0cc-4e09-8ef5-cce6fc157dad\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7" Nov 03 13:27:12.035149 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:12.034997 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqkrg\" (UniqueName: \"kubernetes.io/projected/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-kube-api-access-jqkrg\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7\" (UID: \"42d9f732-f0cc-4e09-8ef5-cce6fc157dad\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7" Nov 03 13:27:12.035193 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:12.035148 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-credential-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7\" (UID: \"42d9f732-f0cc-4e09-8ef5-cce6fc157dad\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7" Nov 03 13:27:12.035243 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:12.035186 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-istio-podinfo\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7\" (UID: \"42d9f732-f0cc-4e09-8ef5-cce6fc157dad\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7" Nov 03 13:27:12.035243 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:12.035214 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-workload-certs\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7\" (UID: \"42d9f732-f0cc-4e09-8ef5-cce6fc157dad\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7" Nov 03 13:27:12.035322 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:12.035259 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-istio-envoy\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7\" (UID: \"42d9f732-f0cc-4e09-8ef5-cce6fc157dad\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7" Nov 03 13:27:12.035362 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:12.035335 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-istio-token\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7\" (UID: \"42d9f732-f0cc-4e09-8ef5-cce6fc157dad\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7" Nov 03 13:27:12.035416 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:12.035370 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-workload-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7\" (UID: \"42d9f732-f0cc-4e09-8ef5-cce6fc157dad\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7" Nov 03 13:27:12.035506 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:12.035450 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-istio-data\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7\" (UID: \"42d9f732-f0cc-4e09-8ef5-cce6fc157dad\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7" Nov 03 13:27:12.137011 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:12.136963 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-istio-data\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7\" (UID: \"42d9f732-f0cc-4e09-8ef5-cce6fc157dad\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7" Nov 03 13:27:12.137198 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:12.137101 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-istiod-ca-cert\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7\" (UID: \"42d9f732-f0cc-4e09-8ef5-cce6fc157dad\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7" Nov 03 13:27:12.137198 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:12.137158 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqkrg\" (UniqueName: \"kubernetes.io/projected/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-kube-api-access-jqkrg\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7\" (UID: \"42d9f732-f0cc-4e09-8ef5-cce6fc157dad\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7" Nov 03 13:27:12.137325 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:12.137308 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-credential-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7\" (UID: \"42d9f732-f0cc-4e09-8ef5-cce6fc157dad\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7" Nov 03 13:27:12.137365 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:12.137342 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-istio-podinfo\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7\" (UID: \"42d9f732-f0cc-4e09-8ef5-cce6fc157dad\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7" Nov 03 13:27:12.137423 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:12.137378 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-workload-certs\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7\" (UID: \"42d9f732-f0cc-4e09-8ef5-cce6fc157dad\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7" Nov 03 13:27:12.137423 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:12.137401 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-istio-envoy\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7\" (UID: \"42d9f732-f0cc-4e09-8ef5-cce6fc157dad\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7" Nov 03 13:27:12.137584 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:12.137417 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-istio-data\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7\" (UID: \"42d9f732-f0cc-4e09-8ef5-cce6fc157dad\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7" Nov 03 13:27:12.137584 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:12.137548 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-istio-token\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7\" (UID: \"42d9f732-f0cc-4e09-8ef5-cce6fc157dad\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7" Nov 03 13:27:12.137584 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:12.137572 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-workload-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7\" (UID: \"42d9f732-f0cc-4e09-8ef5-cce6fc157dad\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7" Nov 03 13:27:12.137745 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:12.137713 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-credential-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7\" (UID: \"42d9f732-f0cc-4e09-8ef5-cce6fc157dad\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7" Nov 03 13:27:12.137850 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:12.137826 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-workload-certs\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7\" (UID: \"42d9f732-f0cc-4e09-8ef5-cce6fc157dad\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7" Nov 03 13:27:12.137905 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:12.137851 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-istiod-ca-cert\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7\" (UID: \"42d9f732-f0cc-4e09-8ef5-cce6fc157dad\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7" Nov 03 13:27:12.137950 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:12.137905 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-workload-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7\" (UID: \"42d9f732-f0cc-4e09-8ef5-cce6fc157dad\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7" Nov 03 13:27:12.139892 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:12.139863 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-istio-envoy\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7\" (UID: \"42d9f732-f0cc-4e09-8ef5-cce6fc157dad\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7" Nov 03 13:27:12.140134 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:12.140101 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-istio-podinfo\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7\" (UID: \"42d9f732-f0cc-4e09-8ef5-cce6fc157dad\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7" Nov 03 13:27:12.149826 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:12.149803 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7"] Nov 03 13:27:12.191852 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:12.191820 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-istio-token\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7\" (UID: \"42d9f732-f0cc-4e09-8ef5-cce6fc157dad\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7" Nov 03 13:27:12.192030 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:12.191947 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqkrg\" (UniqueName: \"kubernetes.io/projected/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-kube-api-access-jqkrg\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7\" (UID: \"42d9f732-f0cc-4e09-8ef5-cce6fc157dad\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7" Nov 03 13:27:12.276761 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:12.276679 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7" Nov 03 13:27:12.716496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:12.715126 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7"] Nov 03 13:27:12.756746 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:27:12.756716 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42d9f732_f0cc_4e09_8ef5_cce6fc157dad.slice/crio-ce8594714d02491d87a85bdef53bb653f3b1e2fe0a4a90d457da4797a744c0f0 WatchSource:0}: Error finding container ce8594714d02491d87a85bdef53bb653f3b1e2fe0a4a90d457da4797a744c0f0: Status 404 returned error can't find the container with id ce8594714d02491d87a85bdef53bb653f3b1e2fe0a4a90d457da4797a744c0f0 Nov 03 13:27:12.759945 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:12.759911 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:27:12.760034 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:12.759979 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:27:12.760034 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:12.760005 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:27:13.462497 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:13.443484 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7" event={"ID":"42d9f732-f0cc-4e09-8ef5-cce6fc157dad","Type":"ContainerStarted","Data":"ab52e4bdbd49931b70d29485333cc0e955891ea8c6700ec49000ecb8f3208308"} Nov 03 13:27:13.462497 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:13.443557 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7" event={"ID":"42d9f732-f0cc-4e09-8ef5-cce6fc157dad","Type":"ContainerStarted","Data":"ce8594714d02491d87a85bdef53bb653f3b1e2fe0a4a90d457da4797a744c0f0"} Nov 03 13:27:13.605213 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:13.605161 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7" podStartSLOduration=2.605146128 podStartE2EDuration="2.605146128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:27:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:27:13.603751867 +0000 UTC m=+1315.614440109" watchObservedRunningTime="2026-11-03 13:27:13.605146128 +0000 UTC m=+1315.615834370" Nov 03 13:27:14.277015 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:14.276971 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7" Nov 03 13:27:14.278865 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:14.278839 5979 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 03 13:27:14.278988 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:14.278885 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7" podUID="42d9f732-f0cc-4e09-8ef5-cce6fc157dad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 03 13:27:14.342951 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:14.342918 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-30074e-5dd5666c67-gkb7j" Nov 03 13:27:14.343138 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:14.342992 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-30074e-7986f65ffc-82gzw" Nov 03 13:27:15.277461 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:15.277428 5979 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 03 13:27:15.277870 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:15.277502 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7" podUID="42d9f732-f0cc-4e09-8ef5-cce6fc157dad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 03 13:27:16.277553 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:16.277516 5979 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 03 13:27:16.277992 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:16.277571 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7" podUID="42d9f732-f0cc-4e09-8ef5-cce6fc157dad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 03 13:27:17.277053 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:17.277009 5979 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 03 13:27:17.277267 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:17.277069 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7" podUID="42d9f732-f0cc-4e09-8ef5-cce6fc157dad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 03 13:27:18.278036 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:18.277999 5979 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 03 13:27:18.278549 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:18.278108 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7" podUID="42d9f732-f0cc-4e09-8ef5-cce6fc157dad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 03 13:27:18.506516 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:18.506483 5979 scope.go:122] "RemoveContainer" containerID="62b02f1874dd3e55cb02585b3552ec012e3c8335c43db15a46ef8e906ae8f6c9" Nov 03 13:27:18.516868 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:18.516847 5979 scope.go:122] "RemoveContainer" containerID="1ee0fdd8d860a17473e0eb3a92f970068cf649a5cbe3655f76ccc52af381ef72" Nov 03 13:27:18.526384 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:18.526364 5979 scope.go:122] "RemoveContainer" containerID="f03a37fa604ee9e3e87066f2faed6236739b03d6a166a2dd066f1ec55a002be4" Nov 03 13:27:18.535320 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:18.535292 5979 scope.go:122] "RemoveContainer" containerID="e20760b43fbcedf017ffa7a649d2f9f2c72b9871568347b376fe05a638ede239" Nov 03 13:27:18.572008 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:18.571982 5979 scope.go:122] "RemoveContainer" containerID="e9d0324e2eba7d820e1d4ea1c6fee494e9b6928346b6c0d26aa9c1f9c44b7c0e" Nov 03 13:27:18.583040 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:18.583022 5979 scope.go:122] "RemoveContainer" containerID="f84d8766f31a15e389a9087abc4a13116ed63f6e70dfe3efa8af8ca4b0a0da30" Nov 03 13:27:19.277448 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:19.277403 5979 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Nov 03 13:27:19.277692 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:19.277453 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7" podUID="42d9f732-f0cc-4e09-8ef5-cce6fc157dad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Nov 03 13:27:20.293906 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:20.293870 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7" Nov 03 13:27:20.295591 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:20.295570 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7" Nov 03 13:27:20.296003 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:20.295982 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7" Nov 03 13:27:21.038271 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:21.038236 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:27:37.235220 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:27:37.235182 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:27:50.209798 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:50.209764 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:27:53.699582 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:53.699550 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb"] Nov 03 13:27:53.704223 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:53.704204 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb" Nov 03 13:27:53.708749 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:53.708727 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp8xn\" (UniqueName: \"kubernetes.io/projected/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-kube-api-access-qp8xn\") pod \"e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb\" (UID: \"e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5\") " pod="openshift-ingress/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb" Nov 03 13:27:53.708854 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:53.708759 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-istio-token\") pod \"e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb\" (UID: \"e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5\") " pod="openshift-ingress/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb" Nov 03 13:27:53.708854 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:53.708781 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-istiod-ca-cert\") pod \"e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb\" (UID: \"e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5\") " pod="openshift-ingress/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb" Nov 03 13:27:53.708951 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:53.708873 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-credential-socket\") pod \"e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb\" (UID: \"e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5\") " pod="openshift-ingress/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb" Nov 03 13:27:53.708951 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:53.708915 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-workload-socket\") pod \"e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb\" (UID: \"e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5\") " pod="openshift-ingress/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb" Nov 03 13:27:53.709057 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:53.708965 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-istio-podinfo\") pod \"e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb\" (UID: \"e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5\") " pod="openshift-ingress/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb" Nov 03 13:27:53.709057 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:53.708992 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-istio-data\") pod \"e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb\" (UID: \"e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5\") " pod="openshift-ingress/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb" Nov 03 13:27:53.709172 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:53.709097 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-workload-certs\") pod \"e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb\" (UID: \"e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5\") " pod="openshift-ingress/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb" Nov 03 13:27:53.709172 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:53.709138 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-istio-envoy\") pod \"e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb\" (UID: \"e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5\") " pod="openshift-ingress/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb" Nov 03 13:27:53.800324 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:53.800272 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb"] Nov 03 13:27:53.810093 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:53.810064 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-istio-podinfo\") pod \"e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb\" (UID: \"e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5\") " pod="openshift-ingress/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb" Nov 03 13:27:53.810250 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:53.810114 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-istio-data\") pod \"e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb\" (UID: \"e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5\") " pod="openshift-ingress/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb" Nov 03 13:27:53.810250 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:53.810199 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-workload-certs\") pod \"e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb\" (UID: \"e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5\") " pod="openshift-ingress/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb" Nov 03 13:27:53.810364 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:53.810246 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-istio-envoy\") pod \"e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb\" (UID: \"e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5\") " pod="openshift-ingress/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb" Nov 03 13:27:53.810364 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:53.810354 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp8xn\" (UniqueName: \"kubernetes.io/projected/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-kube-api-access-qp8xn\") pod \"e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb\" (UID: \"e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5\") " pod="openshift-ingress/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb" Nov 03 13:27:53.810551 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:53.810395 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-istio-token\") pod \"e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb\" (UID: \"e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5\") " pod="openshift-ingress/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb" Nov 03 13:27:53.810551 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:53.810431 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-istiod-ca-cert\") pod \"e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb\" (UID: \"e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5\") " pod="openshift-ingress/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb" Nov 03 13:27:53.810551 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:53.810505 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-credential-socket\") pod \"e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb\" (UID: \"e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5\") " pod="openshift-ingress/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb" Nov 03 13:27:53.810710 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:53.810661 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-istio-data\") pod \"e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb\" (UID: \"e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5\") " pod="openshift-ingress/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb" Nov 03 13:27:53.810710 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:53.810681 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-workload-certs\") pod \"e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb\" (UID: \"e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5\") " pod="openshift-ingress/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb" Nov 03 13:27:53.810828 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:53.810793 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-workload-socket\") pod \"e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb\" (UID: \"e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5\") " pod="openshift-ingress/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb" Nov 03 13:27:53.811228 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:53.811203 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-workload-socket\") pod \"e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb\" (UID: \"e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5\") " pod="openshift-ingress/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb" Nov 03 13:27:53.811389 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:53.811229 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-credential-socket\") pod \"e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb\" (UID: \"e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5\") " pod="openshift-ingress/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb" Nov 03 13:27:53.811563 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:53.811502 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-istiod-ca-cert\") pod \"e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb\" (UID: \"e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5\") " pod="openshift-ingress/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb" Nov 03 13:27:53.813232 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:53.813208 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-istio-podinfo\") pod \"e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb\" (UID: \"e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5\") " pod="openshift-ingress/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb" Nov 03 13:27:53.815757 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:53.815735 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-istio-envoy\") pod \"e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb\" (UID: \"e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5\") " pod="openshift-ingress/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb" Nov 03 13:27:53.825804 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:53.825779 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-1fdd09-openshift-default-dockercfg-995sk\"" Nov 03 13:27:53.910711 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:53.904355 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-istio-token\") pod \"e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb\" (UID: \"e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5\") " pod="openshift-ingress/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb" Nov 03 13:27:53.923514 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:53.923162 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp8xn\" (UniqueName: \"kubernetes.io/projected/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-kube-api-access-qp8xn\") pod \"e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb\" (UID: \"e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5\") " pod="openshift-ingress/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb" Nov 03 13:27:54.025114 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:54.024787 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb" Nov 03 13:27:54.578263 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:54.578228 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb"] Nov 03 13:27:54.584572 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:27:54.584543 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0ad1760_927c_4eaf_9ae9_0a2e85f4c0d5.slice/crio-799d7f0dbfe8a95fce53c58400d1be47766e2931f875a6c1fb1366c1658fcaa5 WatchSource:0}: Error finding container 799d7f0dbfe8a95fce53c58400d1be47766e2931f875a6c1fb1366c1658fcaa5: Status 404 returned error can't find the container with id 799d7f0dbfe8a95fce53c58400d1be47766e2931f875a6c1fb1366c1658fcaa5 Nov 03 13:27:54.587056 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:54.587021 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:27:54.587182 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:54.587092 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:27:54.587182 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:54.587138 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:27:54.895153 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:54.895112 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb" event={"ID":"e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5","Type":"ContainerStarted","Data":"c015efe631d4b0927cfa8378487110d1a80d7128c49f1c5bd103219ffa4cdb3c"} Nov 03 13:27:54.895588 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:54.895178 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb" event={"ID":"e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5","Type":"ContainerStarted","Data":"799d7f0dbfe8a95fce53c58400d1be47766e2931f875a6c1fb1366c1658fcaa5"} Nov 03 13:27:55.024432 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:55.021342 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb" podStartSLOduration=2.021002111 podStartE2EDuration="2.021002111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:27:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:27:55.016539817 +0000 UTC m=+1357.027228059" watchObservedRunningTime="2026-11-03 13:27:55.021002111 +0000 UTC m=+1357.031690366" Nov 03 13:27:55.025955 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:55.025929 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb" Nov 03 13:27:55.027110 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:55.027084 5979 patch_prober.go:29] interesting pod/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 03 13:27:55.027237 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:55.027137 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb" podUID="e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 03 13:27:56.030840 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:56.030811 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb" Nov 03 13:27:56.909519 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:56.909485 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb" Nov 03 13:27:56.910419 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:56.910399 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb" Nov 03 13:27:57.752166 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:57.752133 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cfe580d-p9kpx"] Nov 03 13:27:57.757413 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:57.757390 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cfe580d-p9kpx" Nov 03 13:27:57.809156 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:57.809120 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d24c04d4-388d-4eea-8aa0-2d14c245b017-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cfe580d-p9kpx\" (UID: \"d24c04d4-388d-4eea-8aa0-2d14c245b017\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cfe580d-p9kpx" Nov 03 13:27:57.809340 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:57.809178 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d24c04d4-388d-4eea-8aa0-2d14c245b017-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cfe580d-p9kpx\" (UID: \"d24c04d4-388d-4eea-8aa0-2d14c245b017\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cfe580d-p9kpx" Nov 03 13:27:57.809340 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:57.809205 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb48m\" (UniqueName: \"kubernetes.io/projected/d24c04d4-388d-4eea-8aa0-2d14c245b017-kube-api-access-hb48m\") pod \"maas-api-revoke-sub-5cfe580d-p9kpx\" (UID: \"d24c04d4-388d-4eea-8aa0-2d14c245b017\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cfe580d-p9kpx" Nov 03 13:27:57.810408 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:57.810380 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cfe580d-p9kpx"] Nov 03 13:27:57.817514 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:57.815627 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 13:27:57.817514 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:57.815673 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 13:27:57.819267 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:57.819229 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:27:57.850838 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:57.850786 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:27:57.911098 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:57.911053 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d24c04d4-388d-4eea-8aa0-2d14c245b017-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cfe580d-p9kpx\" (UID: \"d24c04d4-388d-4eea-8aa0-2d14c245b017\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cfe580d-p9kpx" Nov 03 13:27:57.911293 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:57.911135 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d24c04d4-388d-4eea-8aa0-2d14c245b017-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cfe580d-p9kpx\" (UID: \"d24c04d4-388d-4eea-8aa0-2d14c245b017\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cfe580d-p9kpx" Nov 03 13:27:57.911293 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:57.911185 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb48m\" (UniqueName: \"kubernetes.io/projected/d24c04d4-388d-4eea-8aa0-2d14c245b017-kube-api-access-hb48m\") pod \"maas-api-revoke-sub-5cfe580d-p9kpx\" (UID: \"d24c04d4-388d-4eea-8aa0-2d14c245b017\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cfe580d-p9kpx" Nov 03 13:27:57.912003 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:57.911966 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d24c04d4-388d-4eea-8aa0-2d14c245b017-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cfe580d-p9kpx\" (UID: \"d24c04d4-388d-4eea-8aa0-2d14c245b017\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cfe580d-p9kpx" Nov 03 13:27:57.913668 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:57.913624 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d24c04d4-388d-4eea-8aa0-2d14c245b017-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cfe580d-p9kpx\" (UID: \"d24c04d4-388d-4eea-8aa0-2d14c245b017\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cfe580d-p9kpx" Nov 03 13:27:57.966564 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:57.966529 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb48m\" (UniqueName: \"kubernetes.io/projected/d24c04d4-388d-4eea-8aa0-2d14c245b017-kube-api-access-hb48m\") pod \"maas-api-revoke-sub-5cfe580d-p9kpx\" (UID: \"d24c04d4-388d-4eea-8aa0-2d14c245b017\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cfe580d-p9kpx" Nov 03 13:27:58.073698 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:58.073617 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cfe580d-p9kpx" Nov 03 13:27:58.480067 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:58.480033 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cfe580d-p9kpx"] Nov 03 13:27:58.498926 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:27:58.486088 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd24c04d4_388d_4eea_8aa0_2d14c245b017.slice/crio-8fd2c0c344ed3c0adc685a0a33b5610ab72bb276ae98cf04d45e19311f56a581 WatchSource:0}: Error finding container 8fd2c0c344ed3c0adc685a0a33b5610ab72bb276ae98cf04d45e19311f56a581: Status 404 returned error can't find the container with id 8fd2c0c344ed3c0adc685a0a33b5610ab72bb276ae98cf04d45e19311f56a581 Nov 03 13:27:58.931118 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:58.931079 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cfe580d-p9kpx" event={"ID":"d24c04d4-388d-4eea-8aa0-2d14c245b017","Type":"ContainerStarted","Data":"5b8a3322f250b20945fe1dd93cd7f5b55b40a5b49a618c3d562dea5fd89e9f5b"} Nov 03 13:27:58.931520 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:58.931167 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cfe580d-p9kpx" event={"ID":"d24c04d4-388d-4eea-8aa0-2d14c245b017","Type":"ContainerStarted","Data":"8fd2c0c344ed3c0adc685a0a33b5610ab72bb276ae98cf04d45e19311f56a581"} Nov 03 13:27:59.039688 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:59.039621 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cfe580d-p9kpx" podStartSLOduration=2.039604733 podStartE2EDuration="2.039604733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:27:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:27:59.039360779 +0000 UTC m=+1361.050049064" watchObservedRunningTime="2026-11-03 13:27:59.039604733 +0000 UTC m=+1361.050292976" Nov 03 13:27:59.943025 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:59.942987 5979 generic.go:356] "Generic (PLEG): container finished" podID="d24c04d4-388d-4eea-8aa0-2d14c245b017" containerID="5b8a3322f250b20945fe1dd93cd7f5b55b40a5b49a618c3d562dea5fd89e9f5b" exitCode=0 Nov 03 13:27:59.943422 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:27:59.943058 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cfe580d-p9kpx" event={"ID":"d24c04d4-388d-4eea-8aa0-2d14c245b017","Type":"ContainerDied","Data":"5b8a3322f250b20945fe1dd93cd7f5b55b40a5b49a618c3d562dea5fd89e9f5b"} Nov 03 13:28:00.951385 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:00.951353 5979 generic.go:356] "Generic (PLEG): container finished" podID="53fab3c0-d97b-4c5d-ad31-798afd7c5b6c" containerID="ff7a2dcb437a5a04242e4663480438260cbdca082dd0eb14c0da32c2e7fee8f7" exitCode=0 Nov 03 13:28:00.951915 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:00.951428 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8" event={"ID":"53fab3c0-d97b-4c5d-ad31-798afd7c5b6c","Type":"ContainerDied","Data":"ff7a2dcb437a5a04242e4663480438260cbdca082dd0eb14c0da32c2e7fee8f7"} Nov 03 13:28:00.952269 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:00.952078 5979 scope.go:122] "RemoveContainer" containerID="ff7a2dcb437a5a04242e4663480438260cbdca082dd0eb14c0da32c2e7fee8f7" Nov 03 13:28:00.953692 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:00.953664 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:28:00.953775 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:00.953730 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:28:00.953775 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:00.953765 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:28:01.124728 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:01.124698 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cfe580d-p9kpx" Nov 03 13:28:01.161975 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:01.161937 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d24c04d4-388d-4eea-8aa0-2d14c245b017-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d24c04d4-388d-4eea-8aa0-2d14c245b017-maas-api-openshift-ca\") pod \"d24c04d4-388d-4eea-8aa0-2d14c245b017\" (UID: \"d24c04d4-388d-4eea-8aa0-2d14c245b017\") " Nov 03 13:28:01.162209 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:01.162188 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d24c04d4-388d-4eea-8aa0-2d14c245b017-kube-api-access-hb48m\" (UniqueName: \"kubernetes.io/projected/d24c04d4-388d-4eea-8aa0-2d14c245b017-kube-api-access-hb48m\") pod \"d24c04d4-388d-4eea-8aa0-2d14c245b017\" (UID: \"d24c04d4-388d-4eea-8aa0-2d14c245b017\") " Nov 03 13:28:01.162358 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:01.162338 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d24c04d4-388d-4eea-8aa0-2d14c245b017-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d24c04d4-388d-4eea-8aa0-2d14c245b017-maas-api-xks-ca\") pod \"d24c04d4-388d-4eea-8aa0-2d14c245b017\" (UID: \"d24c04d4-388d-4eea-8aa0-2d14c245b017\") " Nov 03 13:28:01.162429 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:01.162347 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d24c04d4-388d-4eea-8aa0-2d14c245b017-maas-api-openshift-ca" pod "d24c04d4-388d-4eea-8aa0-2d14c245b017" (UID: "d24c04d4-388d-4eea-8aa0-2d14c245b017"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:28:01.163704 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:01.163672 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d24c04d4-388d-4eea-8aa0-2d14c245b017-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:28:01.166638 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:01.166615 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d24c04d4-388d-4eea-8aa0-2d14c245b017-maas-api-xks-ca" pod "d24c04d4-388d-4eea-8aa0-2d14c245b017" (UID: "d24c04d4-388d-4eea-8aa0-2d14c245b017"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:28:01.166784 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:01.166675 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d24c04d4-388d-4eea-8aa0-2d14c245b017-kube-api-access-hb48m" pod "d24c04d4-388d-4eea-8aa0-2d14c245b017" (UID: "d24c04d4-388d-4eea-8aa0-2d14c245b017"). InnerVolumeSpecName "kube-api-access-hb48m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:28:01.266290 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:01.266194 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb48m\" (UniqueName: \"kubernetes.io/projected/d24c04d4-388d-4eea-8aa0-2d14c245b017-kube-api-access-hb48m\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:28:01.266290 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:01.266235 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d24c04d4-388d-4eea-8aa0-2d14c245b017-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:28:01.841494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:01.807208 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1fdd09-5f897ffc9f-m85wq"] Nov 03 13:28:01.841494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:01.820569 5979 cpu_manager.go:414] "containerMap: removing container" podUID="d24c04d4-388d-4eea-8aa0-2d14c245b017" containerName="revoke-keys" Nov 03 13:28:01.841494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:01.820598 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d24c04d4-388d-4eea-8aa0-2d14c245b017" containerName="revoke-keys" Nov 03 13:28:01.841494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:01.820796 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="d24c04d4-388d-4eea-8aa0-2d14c245b017" containerName="revoke-keys" Nov 03 13:28:01.847824 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:01.847644 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1fdd09-5f897ffc9f-m85wq"] Nov 03 13:28:01.847824 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:01.847660 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fdd09-5f897ffc9f-m85wq" Nov 03 13:28:01.916497 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:01.910938 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-1fdd09\"" Nov 03 13:28:01.947504 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:01.944279 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-1fdd09-dockercfg-nhs5d\"" Nov 03 13:28:02.053606 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:02.017709 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hbcd\" (UniqueName: \"kubernetes.io/projected/d8de58c5-0f63-43b8-91f1-ce1830f4b438-kube-api-access-8hbcd\") pod \"payload-pre-processing-e2e-api-a-1fdd09-5f897ffc9f-m85wq\" (UID: \"d8de58c5-0f63-43b8-91f1-ce1830f4b438\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fdd09-5f897ffc9f-m85wq" Nov 03 13:28:02.053606 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:02.017792 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8de58c5-0f63-43b8-91f1-ce1830f4b438-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-1fdd09-5f897ffc9f-m85wq\" (UID: \"d8de58c5-0f63-43b8-91f1-ce1830f4b438\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fdd09-5f897ffc9f-m85wq" Nov 03 13:28:02.053606 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:02.040530 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1fdd09-68b57fc9d7-jpz5x"] Nov 03 13:28:02.143556 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:02.125363 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hbcd\" (UniqueName: \"kubernetes.io/projected/d8de58c5-0f63-43b8-91f1-ce1830f4b438-kube-api-access-8hbcd\") pod \"payload-pre-processing-e2e-api-a-1fdd09-5f897ffc9f-m85wq\" (UID: \"d8de58c5-0f63-43b8-91f1-ce1830f4b438\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fdd09-5f897ffc9f-m85wq" Nov 03 13:28:02.143556 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:02.125432 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8de58c5-0f63-43b8-91f1-ce1830f4b438-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-1fdd09-5f897ffc9f-m85wq\" (UID: \"d8de58c5-0f63-43b8-91f1-ce1830f4b438\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fdd09-5f897ffc9f-m85wq" Nov 03 13:28:02.143556 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:02.126502 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8de58c5-0f63-43b8-91f1-ce1830f4b438-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-1fdd09-5f897ffc9f-m85wq\" (UID: \"d8de58c5-0f63-43b8-91f1-ce1830f4b438\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fdd09-5f897ffc9f-m85wq" Nov 03 13:28:02.160980 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:02.153029 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1fdd09-68b57fc9d7-jpz5x" Nov 03 13:28:02.204678 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:02.204648 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1fdd09-68b57fc9d7-jpz5x"] Nov 03 13:28:02.204871 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:02.204712 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8" event={"ID":"53fab3c0-d97b-4c5d-ad31-798afd7c5b6c","Type":"ContainerStarted","Data":"cce14b17abb038ab6260fc4aff6cffa4ae4ffe694ede6d492e2b8fa06d62b478"} Nov 03 13:28:02.207819 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:02.207793 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hbcd\" (UniqueName: \"kubernetes.io/projected/d8de58c5-0f63-43b8-91f1-ce1830f4b438-kube-api-access-8hbcd\") pod \"payload-pre-processing-e2e-api-a-1fdd09-5f897ffc9f-m85wq\" (UID: \"d8de58c5-0f63-43b8-91f1-ce1830f4b438\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fdd09-5f897ffc9f-m85wq" Nov 03 13:28:02.209306 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:02.209282 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cfe580d-p9kpx" event={"ID":"d24c04d4-388d-4eea-8aa0-2d14c245b017","Type":"ContainerDied","Data":"8fd2c0c344ed3c0adc685a0a33b5610ab72bb276ae98cf04d45e19311f56a581"} Nov 03 13:28:02.209505 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:02.209454 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8fd2c0c344ed3c0adc685a0a33b5610ab72bb276ae98cf04d45e19311f56a581" Nov 03 13:28:02.209621 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:02.209300 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cfe580d-p9kpx" Nov 03 13:28:02.226573 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:02.226546 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a895b353-a4b2-48bc-86ae-236473d418ca-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1fdd09-68b57fc9d7-jpz5x\" (UID: \"a895b353-a4b2-48bc-86ae-236473d418ca\") " pod="openshift-ingress/payload-processing-e2e-api-a-1fdd09-68b57fc9d7-jpz5x" Nov 03 13:28:02.226697 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:02.226654 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whj6p\" (UniqueName: \"kubernetes.io/projected/a895b353-a4b2-48bc-86ae-236473d418ca-kube-api-access-whj6p\") pod \"payload-processing-e2e-api-a-1fdd09-68b57fc9d7-jpz5x\" (UID: \"a895b353-a4b2-48bc-86ae-236473d418ca\") " pod="openshift-ingress/payload-processing-e2e-api-a-1fdd09-68b57fc9d7-jpz5x" Nov 03 13:28:02.328728 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:02.328686 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a895b353-a4b2-48bc-86ae-236473d418ca-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1fdd09-68b57fc9d7-jpz5x\" (UID: \"a895b353-a4b2-48bc-86ae-236473d418ca\") " pod="openshift-ingress/payload-processing-e2e-api-a-1fdd09-68b57fc9d7-jpz5x" Nov 03 13:28:02.328917 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:02.328771 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whj6p\" (UniqueName: \"kubernetes.io/projected/a895b353-a4b2-48bc-86ae-236473d418ca-kube-api-access-whj6p\") pod \"payload-processing-e2e-api-a-1fdd09-68b57fc9d7-jpz5x\" (UID: \"a895b353-a4b2-48bc-86ae-236473d418ca\") " pod="openshift-ingress/payload-processing-e2e-api-a-1fdd09-68b57fc9d7-jpz5x" Nov 03 13:28:02.329451 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:02.329430 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a895b353-a4b2-48bc-86ae-236473d418ca-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1fdd09-68b57fc9d7-jpz5x\" (UID: \"a895b353-a4b2-48bc-86ae-236473d418ca\") " pod="openshift-ingress/payload-processing-e2e-api-a-1fdd09-68b57fc9d7-jpz5x" Nov 03 13:28:02.386696 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:02.386659 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whj6p\" (UniqueName: \"kubernetes.io/projected/a895b353-a4b2-48bc-86ae-236473d418ca-kube-api-access-whj6p\") pod \"payload-processing-e2e-api-a-1fdd09-68b57fc9d7-jpz5x\" (UID: \"a895b353-a4b2-48bc-86ae-236473d418ca\") " pod="openshift-ingress/payload-processing-e2e-api-a-1fdd09-68b57fc9d7-jpz5x" Nov 03 13:28:02.467546 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:02.466721 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fdd09-5f897ffc9f-m85wq" Nov 03 13:28:02.525397 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:02.525329 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1fdd09-68b57fc9d7-jpz5x" Nov 03 13:28:02.871993 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:02.868545 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1fdd09-5f897ffc9f-m85wq"] Nov 03 13:28:02.974132 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:02.974103 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1fdd09-68b57fc9d7-jpz5x"] Nov 03 13:28:02.974297 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:28:02.974271 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda895b353_a4b2_48bc_86ae_236473d418ca.slice/crio-4feccea93c607687863fbba223d7dcb2a6f33684a8d83c0a1c892b2380de2697 WatchSource:0}: Error finding container 4feccea93c607687863fbba223d7dcb2a6f33684a8d83c0a1c892b2380de2697: Status 404 returned error can't find the container with id 4feccea93c607687863fbba223d7dcb2a6f33684a8d83c0a1c892b2380de2697 Nov 03 13:28:03.107998 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:03.107956 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8" Nov 03 13:28:03.108604 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:03.108578 5979 patch_prober.go:29] interesting pod/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 03 13:28:03.108716 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:03.108633 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8" podUID="53fab3c0-d97b-4c5d-ad31-798afd7c5b6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 03 13:28:03.217544 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:03.217501 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1fdd09-68b57fc9d7-jpz5x" event={"ID":"a895b353-a4b2-48bc-86ae-236473d418ca","Type":"ContainerStarted","Data":"3d4caaf6e852c139668ccbf822c09fb5bfd97a9e93fe6d6d05fa3dd96f2464c8"} Nov 03 13:28:03.217753 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:03.217721 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-1fdd09-68b57fc9d7-jpz5x" Nov 03 13:28:03.217753 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:03.217742 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1fdd09-68b57fc9d7-jpz5x" event={"ID":"a895b353-a4b2-48bc-86ae-236473d418ca","Type":"ContainerStarted","Data":"4feccea93c607687863fbba223d7dcb2a6f33684a8d83c0a1c892b2380de2697"} Nov 03 13:28:03.219108 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:03.219081 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fdd09-5f897ffc9f-m85wq" event={"ID":"d8de58c5-0f63-43b8-91f1-ce1830f4b438","Type":"ContainerStarted","Data":"523e80ec9b9dcb8cf1afaf9612105a0efbcf77392a685db2db1af2c212311590"} Nov 03 13:28:03.219239 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:03.219119 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fdd09-5f897ffc9f-m85wq" event={"ID":"d8de58c5-0f63-43b8-91f1-ce1830f4b438","Type":"ContainerStarted","Data":"d22e1e863b059543d86b130a7713eaa451cb7bd661e2b26cdf789cc0c2d4f28e"} Nov 03 13:28:03.219300 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:03.219275 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fdd09-5f897ffc9f-m85wq" Nov 03 13:28:03.341662 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:03.341608 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-1fdd09-68b57fc9d7-jpz5x" podStartSLOduration=2.341595334 podStartE2EDuration="2.341595334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:28:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:28:03.33820479 +0000 UTC m=+1365.348893033" watchObservedRunningTime="2026-11-03 13:28:03.341595334 +0000 UTC m=+1365.352283576" Nov 03 13:28:03.466495 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:03.462827 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fdd09-5f897ffc9f-m85wq" podStartSLOduration=2.462806423 podStartE2EDuration="2.462806423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:28:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:28:03.459865886 +0000 UTC m=+1365.470554129" watchObservedRunningTime="2026-11-03 13:28:03.462806423 +0000 UTC m=+1365.473494667" Nov 03 13:28:04.108259 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:04.108222 5979 patch_prober.go:29] interesting pod/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 03 13:28:04.108706 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:04.108286 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8" podUID="53fab3c0-d97b-4c5d-ad31-798afd7c5b6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 03 13:28:05.108038 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:05.108001 5979 patch_prober.go:29] interesting pod/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 03 13:28:05.108229 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:05.108064 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8" podUID="53fab3c0-d97b-4c5d-ad31-798afd7c5b6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 03 13:28:06.108272 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:06.108234 5979 patch_prober.go:29] interesting pod/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 03 13:28:06.108715 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:06.108297 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8" podUID="53fab3c0-d97b-4c5d-ad31-798afd7c5b6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 03 13:28:07.107709 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:07.107671 5979 patch_prober.go:29] interesting pod/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Nov 03 13:28:07.107920 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:07.107728 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8" podUID="53fab3c0-d97b-4c5d-ad31-798afd7c5b6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Nov 03 13:28:07.235073 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:28:07.235036 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:28:08.117529 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:08.117499 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8" Nov 03 13:28:08.117989 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:08.117967 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8" Nov 03 13:28:08.118814 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:08.118792 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8" Nov 03 13:28:14.234929 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:14.234887 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-1fdd09-68b57fc9d7-jpz5x" Nov 03 13:28:14.241073 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:14.241044 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fdd09-5f897ffc9f-m85wq" Nov 03 13:28:14.741790 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:14.741758 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37ae8021-5bwfl"] Nov 03 13:28:14.745791 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:14.745769 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ae8021-5bwfl" Nov 03 13:28:14.790044 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:14.790010 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65772\" (UniqueName: \"kubernetes.io/projected/adbe9b58-f764-4b66-bf64-fcc44f186a4d-kube-api-access-65772\") pod \"maas-api-revoke-sub-37ae8021-5bwfl\" (UID: \"adbe9b58-f764-4b66-bf64-fcc44f186a4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ae8021-5bwfl" Nov 03 13:28:14.790203 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:14.790076 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adbe9b58-f764-4b66-bf64-fcc44f186a4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37ae8021-5bwfl\" (UID: \"adbe9b58-f764-4b66-bf64-fcc44f186a4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ae8021-5bwfl" Nov 03 13:28:14.790305 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:14.790280 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adbe9b58-f764-4b66-bf64-fcc44f186a4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37ae8021-5bwfl\" (UID: \"adbe9b58-f764-4b66-bf64-fcc44f186a4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ae8021-5bwfl" Nov 03 13:28:14.803390 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:14.803240 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:28:14.803390 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:14.803262 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 13:28:14.807866 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:14.807837 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37ae8021-5bwfl"] Nov 03 13:28:14.848509 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:14.848459 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:28:14.848689 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:14.848512 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 13:28:14.891356 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:14.891314 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65772\" (UniqueName: \"kubernetes.io/projected/adbe9b58-f764-4b66-bf64-fcc44f186a4d-kube-api-access-65772\") pod \"maas-api-revoke-sub-37ae8021-5bwfl\" (UID: \"adbe9b58-f764-4b66-bf64-fcc44f186a4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ae8021-5bwfl" Nov 03 13:28:14.891579 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:14.891401 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adbe9b58-f764-4b66-bf64-fcc44f186a4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37ae8021-5bwfl\" (UID: \"adbe9b58-f764-4b66-bf64-fcc44f186a4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ae8021-5bwfl" Nov 03 13:28:14.891579 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:14.891545 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adbe9b58-f764-4b66-bf64-fcc44f186a4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37ae8021-5bwfl\" (UID: \"adbe9b58-f764-4b66-bf64-fcc44f186a4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ae8021-5bwfl" Nov 03 13:28:14.892228 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:14.892207 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adbe9b58-f764-4b66-bf64-fcc44f186a4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37ae8021-5bwfl\" (UID: \"adbe9b58-f764-4b66-bf64-fcc44f186a4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ae8021-5bwfl" Nov 03 13:28:14.893901 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:14.893878 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adbe9b58-f764-4b66-bf64-fcc44f186a4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37ae8021-5bwfl\" (UID: \"adbe9b58-f764-4b66-bf64-fcc44f186a4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ae8021-5bwfl" Nov 03 13:28:14.947290 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:14.947257 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65772\" (UniqueName: \"kubernetes.io/projected/adbe9b58-f764-4b66-bf64-fcc44f186a4d-kube-api-access-65772\") pod \"maas-api-revoke-sub-37ae8021-5bwfl\" (UID: \"adbe9b58-f764-4b66-bf64-fcc44f186a4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ae8021-5bwfl" Nov 03 13:28:15.060673 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:15.060584 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ae8021-5bwfl" Nov 03 13:28:15.447838 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:15.447804 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37ae8021-5bwfl"] Nov 03 13:28:15.453080 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:28:15.453051 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadbe9b58_f764_4b66_bf64_fcc44f186a4d.slice/crio-59d4b62dd5c2a31d684a5604b607bc99647315761b124be35e37b25369e29d43 WatchSource:0}: Error finding container 59d4b62dd5c2a31d684a5604b607bc99647315761b124be35e37b25369e29d43: Status 404 returned error can't find the container with id 59d4b62dd5c2a31d684a5604b607bc99647315761b124be35e37b25369e29d43 Nov 03 13:28:16.333350 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:16.333311 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ae8021-5bwfl" event={"ID":"adbe9b58-f764-4b66-bf64-fcc44f186a4d","Type":"ContainerStarted","Data":"2eeb7233bdc9b02b3b4d2f968f604665b63e4fa70af980c86f7a2d78a7d02e18"} Nov 03 13:28:16.333516 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:16.333373 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ae8021-5bwfl" event={"ID":"adbe9b58-f764-4b66-bf64-fcc44f186a4d","Type":"ContainerStarted","Data":"59d4b62dd5c2a31d684a5604b607bc99647315761b124be35e37b25369e29d43"} Nov 03 13:28:16.448090 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:16.448036 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ae8021-5bwfl" podStartSLOduration=2.448021545 podStartE2EDuration="2.448021545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:28:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:28:16.445575197 +0000 UTC m=+1378.456263454" watchObservedRunningTime="2026-11-03 13:28:16.448021545 +0000 UTC m=+1378.458709786" Nov 03 13:28:17.342278 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:17.342246 5979 generic.go:356] "Generic (PLEG): container finished" podID="adbe9b58-f764-4b66-bf64-fcc44f186a4d" containerID="2eeb7233bdc9b02b3b4d2f968f604665b63e4fa70af980c86f7a2d78a7d02e18" exitCode=0 Nov 03 13:28:17.342479 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:17.342286 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ae8021-5bwfl" event={"ID":"adbe9b58-f764-4b66-bf64-fcc44f186a4d","Type":"ContainerDied","Data":"2eeb7233bdc9b02b3b4d2f968f604665b63e4fa70af980c86f7a2d78a7d02e18"} Nov 03 13:28:18.499396 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:18.499371 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ae8021-5bwfl" Nov 03 13:28:18.540119 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:18.540090 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adbe9b58-f764-4b66-bf64-fcc44f186a4d-kube-api-access-65772\" (UniqueName: \"kubernetes.io/projected/adbe9b58-f764-4b66-bf64-fcc44f186a4d-kube-api-access-65772\") pod \"adbe9b58-f764-4b66-bf64-fcc44f186a4d\" (UID: \"adbe9b58-f764-4b66-bf64-fcc44f186a4d\") " Nov 03 13:28:18.540260 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:18.540179 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adbe9b58-f764-4b66-bf64-fcc44f186a4d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adbe9b58-f764-4b66-bf64-fcc44f186a4d-maas-api-openshift-ca\") pod \"adbe9b58-f764-4b66-bf64-fcc44f186a4d\" (UID: \"adbe9b58-f764-4b66-bf64-fcc44f186a4d\") " Nov 03 13:28:18.540260 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:18.540212 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/adbe9b58-f764-4b66-bf64-fcc44f186a4d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adbe9b58-f764-4b66-bf64-fcc44f186a4d-maas-api-xks-ca\") pod \"adbe9b58-f764-4b66-bf64-fcc44f186a4d\" (UID: \"adbe9b58-f764-4b66-bf64-fcc44f186a4d\") " Nov 03 13:28:18.540596 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:18.540565 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adbe9b58-f764-4b66-bf64-fcc44f186a4d-maas-api-openshift-ca" pod "adbe9b58-f764-4b66-bf64-fcc44f186a4d" (UID: "adbe9b58-f764-4b66-bf64-fcc44f186a4d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:28:18.540854 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:18.540837 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adbe9b58-f764-4b66-bf64-fcc44f186a4d-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:28:18.542313 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:18.542280 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adbe9b58-f764-4b66-bf64-fcc44f186a4d-maas-api-xks-ca" pod "adbe9b58-f764-4b66-bf64-fcc44f186a4d" (UID: "adbe9b58-f764-4b66-bf64-fcc44f186a4d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:28:18.542427 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:18.542376 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adbe9b58-f764-4b66-bf64-fcc44f186a4d-kube-api-access-65772" pod "adbe9b58-f764-4b66-bf64-fcc44f186a4d" (UID: "adbe9b58-f764-4b66-bf64-fcc44f186a4d"). InnerVolumeSpecName "kube-api-access-65772". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:28:18.642204 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:18.642171 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65772\" (UniqueName: \"kubernetes.io/projected/adbe9b58-f764-4b66-bf64-fcc44f186a4d-kube-api-access-65772\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:28:18.642373 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:18.642208 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adbe9b58-f764-4b66-bf64-fcc44f186a4d-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:28:19.358381 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:19.358354 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ae8021-5bwfl" Nov 03 13:28:19.358617 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:19.358382 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ae8021-5bwfl" event={"ID":"adbe9b58-f764-4b66-bf64-fcc44f186a4d","Type":"ContainerDied","Data":"59d4b62dd5c2a31d684a5604b607bc99647315761b124be35e37b25369e29d43"} Nov 03 13:28:19.358617 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:19.358427 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59d4b62dd5c2a31d684a5604b607bc99647315761b124be35e37b25369e29d43" Nov 03 13:28:25.169498 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:25.169433 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:28:29.560935 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:29.560902 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bb55afd-ch2rj"] Nov 03 13:28:29.561737 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:29.561717 5979 cpu_manager.go:414] "containerMap: removing container" podUID="adbe9b58-f764-4b66-bf64-fcc44f186a4d" containerName="revoke-keys" Nov 03 13:28:29.561737 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:29.561739 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adbe9b58-f764-4b66-bf64-fcc44f186a4d" containerName="revoke-keys" Nov 03 13:28:29.561882 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:29.561870 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="adbe9b58-f764-4b66-bf64-fcc44f186a4d" containerName="revoke-keys" Nov 03 13:28:29.565080 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:29.565058 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb55afd-ch2rj" Nov 03 13:28:29.615831 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:29.615797 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf"] Nov 03 13:28:29.619550 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:29.619530 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf" Nov 03 13:28:29.634164 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:29.634139 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 13:28:29.668691 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:29.668656 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:28:29.668991 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:29.668954 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:28:29.669241 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:29.669226 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 13:28:29.707556 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:29.688924 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqn22\" (UniqueName: \"kubernetes.io/projected/23cdc8aa-735c-49b4-8952-3750dddfab79-kube-api-access-cqn22\") pod \"maas-api-revoke-sub-1bb55afd-ch2rj\" (UID: \"23cdc8aa-735c-49b4-8952-3750dddfab79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb55afd-ch2rj" Nov 03 13:28:29.708937 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:29.708900 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23cdc8aa-735c-49b4-8952-3750dddfab79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1bb55afd-ch2rj\" (UID: \"23cdc8aa-735c-49b4-8952-3750dddfab79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb55afd-ch2rj" Nov 03 13:28:29.709231 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:29.709211 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23cdc8aa-735c-49b4-8952-3750dddfab79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1bb55afd-ch2rj\" (UID: \"23cdc8aa-735c-49b4-8952-3750dddfab79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb55afd-ch2rj" Nov 03 13:28:29.716730 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:29.716706 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bb55afd-ch2rj"] Nov 03 13:28:29.724316 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:29.724292 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf"] Nov 03 13:28:29.840544 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:29.826953 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-abe0e0-openshift-default-dockercfg-s4qcw\"" Nov 03 13:28:29.908528 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:29.894861 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqn22\" (UniqueName: \"kubernetes.io/projected/23cdc8aa-735c-49b4-8952-3750dddfab79-kube-api-access-cqn22\") pod \"maas-api-revoke-sub-1bb55afd-ch2rj\" (UID: \"23cdc8aa-735c-49b4-8952-3750dddfab79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb55afd-ch2rj" Nov 03 13:28:29.908528 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:29.905996 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h4bp\" (UniqueName: \"kubernetes.io/projected/fa17ed42-8832-4fea-b962-9029c6e6dbb9-kube-api-access-8h4bp\") pod \"e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf\" (UID: \"fa17ed42-8832-4fea-b962-9029c6e6dbb9\") " pod="openshift-ingress/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf" Nov 03 13:28:29.908528 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:29.906098 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa17ed42-8832-4fea-b962-9029c6e6dbb9-credential-socket\") pod \"e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf\" (UID: \"fa17ed42-8832-4fea-b962-9029c6e6dbb9\") " pod="openshift-ingress/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf" Nov 03 13:28:29.908528 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:29.906147 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa17ed42-8832-4fea-b962-9029c6e6dbb9-istio-token\") pod \"e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf\" (UID: \"fa17ed42-8832-4fea-b962-9029c6e6dbb9\") " pod="openshift-ingress/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf" Nov 03 13:28:29.908528 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:29.906241 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa17ed42-8832-4fea-b962-9029c6e6dbb9-istio-data\") pod \"e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf\" (UID: \"fa17ed42-8832-4fea-b962-9029c6e6dbb9\") " pod="openshift-ingress/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf" Nov 03 13:28:29.908528 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:29.906285 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23cdc8aa-735c-49b4-8952-3750dddfab79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1bb55afd-ch2rj\" (UID: \"23cdc8aa-735c-49b4-8952-3750dddfab79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb55afd-ch2rj" Nov 03 13:28:29.908528 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:29.906312 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa17ed42-8832-4fea-b962-9029c6e6dbb9-istiod-ca-cert\") pod \"e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf\" (UID: \"fa17ed42-8832-4fea-b962-9029c6e6dbb9\") " pod="openshift-ingress/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf" Nov 03 13:28:29.908528 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:29.906357 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23cdc8aa-735c-49b4-8952-3750dddfab79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1bb55afd-ch2rj\" (UID: \"23cdc8aa-735c-49b4-8952-3750dddfab79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb55afd-ch2rj" Nov 03 13:28:29.908528 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:29.906381 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa17ed42-8832-4fea-b962-9029c6e6dbb9-istio-envoy\") pod \"e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf\" (UID: \"fa17ed42-8832-4fea-b962-9029c6e6dbb9\") " pod="openshift-ingress/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf" Nov 03 13:28:29.908528 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:29.906487 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa17ed42-8832-4fea-b962-9029c6e6dbb9-istio-podinfo\") pod \"e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf\" (UID: \"fa17ed42-8832-4fea-b962-9029c6e6dbb9\") " pod="openshift-ingress/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf" Nov 03 13:28:29.908528 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:29.906518 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa17ed42-8832-4fea-b962-9029c6e6dbb9-workload-certs\") pod \"e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf\" (UID: \"fa17ed42-8832-4fea-b962-9029c6e6dbb9\") " pod="openshift-ingress/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf" Nov 03 13:28:29.908528 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:29.906575 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa17ed42-8832-4fea-b962-9029c6e6dbb9-workload-socket\") pod \"e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf\" (UID: \"fa17ed42-8832-4fea-b962-9029c6e6dbb9\") " pod="openshift-ingress/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf" Nov 03 13:28:29.908528 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:29.907519 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23cdc8aa-735c-49b4-8952-3750dddfab79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1bb55afd-ch2rj\" (UID: \"23cdc8aa-735c-49b4-8952-3750dddfab79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb55afd-ch2rj" Nov 03 13:28:29.999494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:29.983576 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqn22\" (UniqueName: \"kubernetes.io/projected/23cdc8aa-735c-49b4-8952-3750dddfab79-kube-api-access-cqn22\") pod \"maas-api-revoke-sub-1bb55afd-ch2rj\" (UID: \"23cdc8aa-735c-49b4-8952-3750dddfab79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb55afd-ch2rj" Nov 03 13:28:30.017494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:30.016160 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa17ed42-8832-4fea-b962-9029c6e6dbb9-istio-podinfo\") pod \"e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf\" (UID: \"fa17ed42-8832-4fea-b962-9029c6e6dbb9\") " pod="openshift-ingress/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf" Nov 03 13:28:30.017494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:30.016217 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa17ed42-8832-4fea-b962-9029c6e6dbb9-workload-certs\") pod \"e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf\" (UID: \"fa17ed42-8832-4fea-b962-9029c6e6dbb9\") " pod="openshift-ingress/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf" Nov 03 13:28:30.017494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:30.016267 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa17ed42-8832-4fea-b962-9029c6e6dbb9-workload-socket\") pod \"e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf\" (UID: \"fa17ed42-8832-4fea-b962-9029c6e6dbb9\") " pod="openshift-ingress/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf" Nov 03 13:28:30.017494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:30.016439 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h4bp\" (UniqueName: \"kubernetes.io/projected/fa17ed42-8832-4fea-b962-9029c6e6dbb9-kube-api-access-8h4bp\") pod \"e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf\" (UID: \"fa17ed42-8832-4fea-b962-9029c6e6dbb9\") " pod="openshift-ingress/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf" Nov 03 13:28:30.017494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:30.016532 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa17ed42-8832-4fea-b962-9029c6e6dbb9-credential-socket\") pod \"e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf\" (UID: \"fa17ed42-8832-4fea-b962-9029c6e6dbb9\") " pod="openshift-ingress/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf" Nov 03 13:28:30.017494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:30.016577 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa17ed42-8832-4fea-b962-9029c6e6dbb9-istio-token\") pod \"e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf\" (UID: \"fa17ed42-8832-4fea-b962-9029c6e6dbb9\") " pod="openshift-ingress/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf" Nov 03 13:28:30.017494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:30.016660 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa17ed42-8832-4fea-b962-9029c6e6dbb9-istio-data\") pod \"e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf\" (UID: \"fa17ed42-8832-4fea-b962-9029c6e6dbb9\") " pod="openshift-ingress/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf" Nov 03 13:28:30.017494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:30.016696 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa17ed42-8832-4fea-b962-9029c6e6dbb9-istiod-ca-cert\") pod \"e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf\" (UID: \"fa17ed42-8832-4fea-b962-9029c6e6dbb9\") " pod="openshift-ingress/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf" Nov 03 13:28:30.017494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:30.016739 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa17ed42-8832-4fea-b962-9029c6e6dbb9-istio-envoy\") pod \"e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf\" (UID: \"fa17ed42-8832-4fea-b962-9029c6e6dbb9\") " pod="openshift-ingress/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf" Nov 03 13:28:30.021551 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:30.018515 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa17ed42-8832-4fea-b962-9029c6e6dbb9-credential-socket\") pod \"e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf\" (UID: \"fa17ed42-8832-4fea-b962-9029c6e6dbb9\") " pod="openshift-ingress/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf" Nov 03 13:28:30.021551 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:30.018791 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa17ed42-8832-4fea-b962-9029c6e6dbb9-workload-certs\") pod \"e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf\" (UID: \"fa17ed42-8832-4fea-b962-9029c6e6dbb9\") " pod="openshift-ingress/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf" Nov 03 13:28:30.021551 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:30.019811 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa17ed42-8832-4fea-b962-9029c6e6dbb9-istio-data\") pod \"e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf\" (UID: \"fa17ed42-8832-4fea-b962-9029c6e6dbb9\") " pod="openshift-ingress/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf" Nov 03 13:28:30.021551 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:30.020705 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa17ed42-8832-4fea-b962-9029c6e6dbb9-istiod-ca-cert\") pod \"e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf\" (UID: \"fa17ed42-8832-4fea-b962-9029c6e6dbb9\") " pod="openshift-ingress/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf" Nov 03 13:28:30.021551 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:30.019988 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa17ed42-8832-4fea-b962-9029c6e6dbb9-workload-socket\") pod \"e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf\" (UID: \"fa17ed42-8832-4fea-b962-9029c6e6dbb9\") " pod="openshift-ingress/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf" Nov 03 13:28:30.022074 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:30.022047 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23cdc8aa-735c-49b4-8952-3750dddfab79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1bb55afd-ch2rj\" (UID: \"23cdc8aa-735c-49b4-8952-3750dddfab79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb55afd-ch2rj" Nov 03 13:28:30.042621 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:30.035031 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa17ed42-8832-4fea-b962-9029c6e6dbb9-istio-envoy\") pod \"e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf\" (UID: \"fa17ed42-8832-4fea-b962-9029c6e6dbb9\") " pod="openshift-ingress/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf" Nov 03 13:28:30.042621 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:30.035949 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa17ed42-8832-4fea-b962-9029c6e6dbb9-istio-podinfo\") pod \"e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf\" (UID: \"fa17ed42-8832-4fea-b962-9029c6e6dbb9\") " pod="openshift-ingress/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf" Nov 03 13:28:30.101205 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:30.101071 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa17ed42-8832-4fea-b962-9029c6e6dbb9-istio-token\") pod \"e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf\" (UID: \"fa17ed42-8832-4fea-b962-9029c6e6dbb9\") " pod="openshift-ingress/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf" Nov 03 13:28:30.101382 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:30.101274 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h4bp\" (UniqueName: \"kubernetes.io/projected/fa17ed42-8832-4fea-b962-9029c6e6dbb9-kube-api-access-8h4bp\") pod \"e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf\" (UID: \"fa17ed42-8832-4fea-b962-9029c6e6dbb9\") " pod="openshift-ingress/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf" Nov 03 13:28:30.194586 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:30.194547 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb55afd-ch2rj" Nov 03 13:28:30.236593 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:30.236559 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf" Nov 03 13:28:30.348523 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:30.348491 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-84b78b6f97-fwtt2"] Nov 03 13:28:30.353728 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:30.353667 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-84b78b6f97-fwtt2" Nov 03 13:28:30.406435 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:30.406391 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-84b78b6f97-fwtt2"] Nov 03 13:28:30.444745 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:30.444714 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c-plugins-config-volume\") pod \"payload-processing-external-model-84b78b6f97-fwtt2\" (UID: \"15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c\") " pod="models-as-a-service/payload-processing-external-model-84b78b6f97-fwtt2" Nov 03 13:28:30.444924 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:30.444860 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c-provider-credentials\") pod \"payload-processing-external-model-84b78b6f97-fwtt2\" (UID: \"15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c\") " pod="models-as-a-service/payload-processing-external-model-84b78b6f97-fwtt2" Nov 03 13:28:30.444924 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:30.444920 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c-routing-overlay-volume\") pod \"payload-processing-external-model-84b78b6f97-fwtt2\" (UID: \"15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c\") " pod="models-as-a-service/payload-processing-external-model-84b78b6f97-fwtt2" Nov 03 13:28:30.445064 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:30.445044 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79ftj\" (UniqueName: \"kubernetes.io/projected/15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c-kube-api-access-79ftj\") pod \"payload-processing-external-model-84b78b6f97-fwtt2\" (UID: \"15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c\") " pod="models-as-a-service/payload-processing-external-model-84b78b6f97-fwtt2" Nov 03 13:28:30.546713 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:30.546674 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c-provider-credentials\") pod \"payload-processing-external-model-84b78b6f97-fwtt2\" (UID: \"15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c\") " pod="models-as-a-service/payload-processing-external-model-84b78b6f97-fwtt2" Nov 03 13:28:30.546891 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:30.546748 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c-routing-overlay-volume\") pod \"payload-processing-external-model-84b78b6f97-fwtt2\" (UID: \"15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c\") " pod="models-as-a-service/payload-processing-external-model-84b78b6f97-fwtt2" Nov 03 13:28:30.546944 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:30.546895 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79ftj\" (UniqueName: \"kubernetes.io/projected/15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c-kube-api-access-79ftj\") pod \"payload-processing-external-model-84b78b6f97-fwtt2\" (UID: \"15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c\") " pod="models-as-a-service/payload-processing-external-model-84b78b6f97-fwtt2" Nov 03 13:28:30.547012 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:30.546997 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c-plugins-config-volume\") pod \"payload-processing-external-model-84b78b6f97-fwtt2\" (UID: \"15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c\") " pod="models-as-a-service/payload-processing-external-model-84b78b6f97-fwtt2" Nov 03 13:28:30.547637 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:30.547599 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c-routing-overlay-volume\") pod \"payload-processing-external-model-84b78b6f97-fwtt2\" (UID: \"15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c\") " pod="models-as-a-service/payload-processing-external-model-84b78b6f97-fwtt2" Nov 03 13:28:30.547845 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:30.547819 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c-plugins-config-volume\") pod \"payload-processing-external-model-84b78b6f97-fwtt2\" (UID: \"15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c\") " pod="models-as-a-service/payload-processing-external-model-84b78b6f97-fwtt2" Nov 03 13:28:30.549279 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:30.549257 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c-provider-credentials\") pod \"payload-processing-external-model-84b78b6f97-fwtt2\" (UID: \"15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c\") " pod="models-as-a-service/payload-processing-external-model-84b78b6f97-fwtt2" Nov 03 13:28:30.583591 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:30.583547 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bb55afd-ch2rj"] Nov 03 13:28:30.589764 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:28:30.589733 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23cdc8aa_735c_49b4_8952_3750dddfab79.slice/crio-06e361253610a0509a8a8e58afe9c3c32a7f46c4ad2c0fa9e42d936ca586fa09 WatchSource:0}: Error finding container 06e361253610a0509a8a8e58afe9c3c32a7f46c4ad2c0fa9e42d936ca586fa09: Status 404 returned error can't find the container with id 06e361253610a0509a8a8e58afe9c3c32a7f46c4ad2c0fa9e42d936ca586fa09 Nov 03 13:28:30.603657 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:30.603631 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79ftj\" (UniqueName: \"kubernetes.io/projected/15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c-kube-api-access-79ftj\") pod \"payload-processing-external-model-84b78b6f97-fwtt2\" (UID: \"15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c\") " pod="models-as-a-service/payload-processing-external-model-84b78b6f97-fwtt2" Nov 03 13:28:30.668581 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:30.668549 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-84b78b6f97-fwtt2" Nov 03 13:28:30.686265 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:30.686239 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf"] Nov 03 13:28:30.691707 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:28:30.691677 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa17ed42_8832_4fea_b962_9029c6e6dbb9.slice/crio-9220fa135b50f1d093d7152926f14cc29b384849f8b32e593595b58bc0cefc01 WatchSource:0}: Error finding container 9220fa135b50f1d093d7152926f14cc29b384849f8b32e593595b58bc0cefc01: Status 404 returned error can't find the container with id 9220fa135b50f1d093d7152926f14cc29b384849f8b32e593595b58bc0cefc01 Nov 03 13:28:30.694776 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:30.694750 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:28:30.694863 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:30.694810 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:28:30.694863 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:30.694840 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:28:31.072496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:31.068382 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-84b78b6f97-fwtt2"] Nov 03 13:28:31.154771 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:28:31.154748 5979 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-7d775f439052: secret "agc-cred-7d775f439052" not found Nov 03 13:28:31.155075 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:28:31.154777 5979 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-de9101746298: secret "agc-cred-de9101746298" not found Nov 03 13:28:31.155075 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:28:31.154791 5979 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7c48b6b6bf-2prt4: [secret "agc-cred-7d775f439052" not found, secret "agc-cred-de9101746298" not found] Nov 03 13:28:31.155075 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:28:31.154848 5979 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4-provider-credentials podName:b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4 nodeName:}" failed. No retries permitted until 2026-11-03 13:28:31.654828653 +0000 UTC m=+1393.665516874 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4-provider-credentials") pod "payload-processing-external-model-7c48b6b6bf-2prt4" (UID: "b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4") : [secret "agc-cred-7d775f439052" not found, secret "agc-cred-de9101746298" not found] Nov 03 13:28:31.519374 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:31.519342 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-84b78b6f97-fwtt2" event={"ID":"15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c","Type":"ContainerStarted","Data":"c519e202e55eef0db45a72c00639edb64bc16a1ae6146ae57b3d6ae2ef5cfef1"} Nov 03 13:28:31.519625 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:31.519395 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-84b78b6f97-fwtt2" event={"ID":"15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c","Type":"ContainerStarted","Data":"a27cb70a8aff9c3c6036703e22bd1c0cd755790e892282585db947596537b685"} Nov 03 13:28:31.519625 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:31.519555 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-84b78b6f97-fwtt2" Nov 03 13:28:31.521162 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:31.521132 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf" event={"ID":"fa17ed42-8832-4fea-b962-9029c6e6dbb9","Type":"ContainerStarted","Data":"0638dfc254a429df1d30b3e1bd1175552ef5d69f8b77c529acf1f3fa2970ab8b"} Nov 03 13:28:31.521301 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:31.521173 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf" event={"ID":"fa17ed42-8832-4fea-b962-9029c6e6dbb9","Type":"ContainerStarted","Data":"9220fa135b50f1d093d7152926f14cc29b384849f8b32e593595b58bc0cefc01"} Nov 03 13:28:31.522665 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:31.522647 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb55afd-ch2rj" event={"ID":"23cdc8aa-735c-49b4-8952-3750dddfab79","Type":"ContainerStarted","Data":"5351b77dbd0e72fa7aab841085efd2d04e7f1fcf75b2ed45a5ff570f7bfd302b"} Nov 03 13:28:31.522749 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:31.522679 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb55afd-ch2rj" event={"ID":"23cdc8aa-735c-49b4-8952-3750dddfab79","Type":"ContainerStarted","Data":"06e361253610a0509a8a8e58afe9c3c32a7f46c4ad2c0fa9e42d936ca586fa09"} Nov 03 13:28:31.633353 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:31.633301 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-84b78b6f97-fwtt2" podStartSLOduration=1.633287425 podStartE2EDuration="1.633287425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:28:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:28:31.631723024 +0000 UTC m=+1393.642411286" watchObservedRunningTime="2026-11-03 13:28:31.633287425 +0000 UTC m=+1393.643975667" Nov 03 13:28:31.661882 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:28:31.661851 5979 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-7d775f439052: secret "agc-cred-7d775f439052" not found Nov 03 13:28:31.661882 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:28:31.661882 5979 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-de9101746298: secret "agc-cred-de9101746298" not found Nov 03 13:28:31.662041 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:28:31.661894 5979 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7c48b6b6bf-2prt4: [secret "agc-cred-7d775f439052" not found, secret "agc-cred-de9101746298" not found] Nov 03 13:28:31.662041 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:28:31.661974 5979 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4-provider-credentials podName:b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4 nodeName:}" failed. No retries permitted until 2026-11-03 13:28:32.661957257 +0000 UTC m=+1394.672645479 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4-provider-credentials") pod "payload-processing-external-model-7c48b6b6bf-2prt4" (UID: "b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4") : [secret "agc-cred-7d775f439052" not found, secret "agc-cred-de9101746298" not found] Nov 03 13:28:31.759482 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:31.759419 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf" podStartSLOduration=2.759405993 podStartE2EDuration="2.759405993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:28:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:28:31.758028779 +0000 UTC m=+1393.768717021" watchObservedRunningTime="2026-11-03 13:28:31.759405993 +0000 UTC m=+1393.770094234" Nov 03 13:28:31.871392 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:31.871298 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb55afd-ch2rj" podStartSLOduration=2.8712872689999998 podStartE2EDuration="2.871287269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:28:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:28:31.870164515 +0000 UTC m=+1393.880852774" watchObservedRunningTime="2026-11-03 13:28:31.871287269 +0000 UTC m=+1393.881975558" Nov 03 13:28:32.237649 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:32.237617 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf" Nov 03 13:28:32.242938 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:32.242911 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf" Nov 03 13:28:32.532621 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:32.532535 5979 generic.go:356] "Generic (PLEG): container finished" podID="23cdc8aa-735c-49b4-8952-3750dddfab79" containerID="5351b77dbd0e72fa7aab841085efd2d04e7f1fcf75b2ed45a5ff570f7bfd302b" exitCode=0 Nov 03 13:28:32.532621 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:32.532594 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb55afd-ch2rj" event={"ID":"23cdc8aa-735c-49b4-8952-3750dddfab79","Type":"ContainerDied","Data":"5351b77dbd0e72fa7aab841085efd2d04e7f1fcf75b2ed45a5ff570f7bfd302b"} Nov 03 13:28:32.532939 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:32.532922 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf" Nov 03 13:28:32.533751 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:32.533730 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf" Nov 03 13:28:32.675806 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:28:32.675768 5979 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-7d775f439052: secret "agc-cred-7d775f439052" not found Nov 03 13:28:32.675806 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:28:32.675796 5979 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-de9101746298: secret "agc-cred-de9101746298" not found Nov 03 13:28:32.675806 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:28:32.675805 5979 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7c48b6b6bf-2prt4: [secret "agc-cred-7d775f439052" not found, secret "agc-cred-de9101746298" not found] Nov 03 13:28:32.676401 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:28:32.675861 5979 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4-provider-credentials podName:b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4 nodeName:}" failed. No retries permitted until 2026-11-03 13:28:34.675848482 +0000 UTC m=+1396.686536702 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4-provider-credentials") pod "payload-processing-external-model-7c48b6b6bf-2prt4" (UID: "b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4") : [secret "agc-cred-7d775f439052" not found, secret "agc-cred-de9101746298" not found] Nov 03 13:28:33.701461 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:33.701434 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb55afd-ch2rj" Nov 03 13:28:33.794004 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:33.793917 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23cdc8aa-735c-49b4-8952-3750dddfab79-kube-api-access-cqn22\" (UniqueName: \"kubernetes.io/projected/23cdc8aa-735c-49b4-8952-3750dddfab79-kube-api-access-cqn22\") pod \"23cdc8aa-735c-49b4-8952-3750dddfab79\" (UID: \"23cdc8aa-735c-49b4-8952-3750dddfab79\") " Nov 03 13:28:33.794175 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:33.794056 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23cdc8aa-735c-49b4-8952-3750dddfab79-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23cdc8aa-735c-49b4-8952-3750dddfab79-maas-api-openshift-ca\") pod \"23cdc8aa-735c-49b4-8952-3750dddfab79\" (UID: \"23cdc8aa-735c-49b4-8952-3750dddfab79\") " Nov 03 13:28:33.794175 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:33.794139 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23cdc8aa-735c-49b4-8952-3750dddfab79-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23cdc8aa-735c-49b4-8952-3750dddfab79-maas-api-xks-ca\") pod \"23cdc8aa-735c-49b4-8952-3750dddfab79\" (UID: \"23cdc8aa-735c-49b4-8952-3750dddfab79\") " Nov 03 13:28:33.794394 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:33.794366 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23cdc8aa-735c-49b4-8952-3750dddfab79-maas-api-openshift-ca" pod "23cdc8aa-735c-49b4-8952-3750dddfab79" (UID: "23cdc8aa-735c-49b4-8952-3750dddfab79"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:28:33.794801 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:33.794784 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23cdc8aa-735c-49b4-8952-3750dddfab79-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:28:33.796083 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:33.796062 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23cdc8aa-735c-49b4-8952-3750dddfab79-maas-api-xks-ca" pod "23cdc8aa-735c-49b4-8952-3750dddfab79" (UID: "23cdc8aa-735c-49b4-8952-3750dddfab79"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:28:33.796155 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:33.796094 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23cdc8aa-735c-49b4-8952-3750dddfab79-kube-api-access-cqn22" pod "23cdc8aa-735c-49b4-8952-3750dddfab79" (UID: "23cdc8aa-735c-49b4-8952-3750dddfab79"). InnerVolumeSpecName "kube-api-access-cqn22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:28:33.896753 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:33.896717 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23cdc8aa-735c-49b4-8952-3750dddfab79-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:28:33.896753 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:33.896747 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqn22\" (UniqueName: \"kubernetes.io/projected/23cdc8aa-735c-49b4-8952-3750dddfab79-kube-api-access-cqn22\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:28:34.551040 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:34.551003 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb55afd-ch2rj" event={"ID":"23cdc8aa-735c-49b4-8952-3750dddfab79","Type":"ContainerDied","Data":"06e361253610a0509a8a8e58afe9c3c32a7f46c4ad2c0fa9e42d936ca586fa09"} Nov 03 13:28:34.551251 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:34.551061 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06e361253610a0509a8a8e58afe9c3c32a7f46c4ad2c0fa9e42d936ca586fa09" Nov 03 13:28:34.551251 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:34.551023 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1bb55afd-ch2rj" Nov 03 13:28:34.707573 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:28:34.707537 5979 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-7d775f439052: secret "agc-cred-7d775f439052" not found Nov 03 13:28:34.707573 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:28:34.707562 5979 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-de9101746298: secret "agc-cred-de9101746298" not found Nov 03 13:28:34.707573 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:28:34.707573 5979 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7c48b6b6bf-2prt4: [secret "agc-cred-7d775f439052" not found, secret "agc-cred-de9101746298" not found] Nov 03 13:28:34.708028 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:28:34.707635 5979 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4-provider-credentials podName:b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4 nodeName:}" failed. No retries permitted until 2026-11-03 13:28:38.707620143 +0000 UTC m=+1400.718308363 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4-provider-credentials") pod "payload-processing-external-model-7c48b6b6bf-2prt4" (UID: "b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4") : [secret "agc-cred-7d775f439052" not found, secret "agc-cred-de9101746298" not found] Nov 03 13:28:37.236222 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:28:37.236187 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:28:38.604493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:38.597143 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-abe0e0-766d597f55-t5mk2"] Nov 03 13:28:38.604493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:38.598183 5979 cpu_manager.go:414] "containerMap: removing container" podUID="23cdc8aa-735c-49b4-8952-3750dddfab79" containerName="revoke-keys" Nov 03 13:28:38.604493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:38.598204 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23cdc8aa-735c-49b4-8952-3750dddfab79" containerName="revoke-keys" Nov 03 13:28:38.604493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:38.598413 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="23cdc8aa-735c-49b4-8952-3750dddfab79" containerName="revoke-keys" Nov 03 13:28:38.613724 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:38.613698 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-abe0e0-766d597f55-t5mk2" Nov 03 13:28:38.633889 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:38.633026 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-abe0e0-766d597f55-t5mk2"] Nov 03 13:28:38.681859 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:38.681829 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-abe0e0\"" Nov 03 13:28:38.682257 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:38.682240 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-abe0e0-dockercfg-ht992\"" Nov 03 13:28:38.754887 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:38.754843 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wprxj\" (UniqueName: \"kubernetes.io/projected/f266d718-6f0c-4972-b21b-db7533013291-kube-api-access-wprxj\") pod \"payload-pre-processing-e2e-api-b-abe0e0-766d597f55-t5mk2\" (UID: \"f266d718-6f0c-4972-b21b-db7533013291\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-abe0e0-766d597f55-t5mk2" Nov 03 13:28:38.798526 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:38.791923 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f266d718-6f0c-4972-b21b-db7533013291-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-abe0e0-766d597f55-t5mk2\" (UID: \"f266d718-6f0c-4972-b21b-db7533013291\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-abe0e0-766d597f55-t5mk2" Nov 03 13:28:38.798526 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:28:38.792210 5979 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-7d775f439052: secret "agc-cred-7d775f439052" not found Nov 03 13:28:38.798526 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:28:38.792226 5979 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-de9101746298: secret "agc-cred-de9101746298" not found Nov 03 13:28:38.798526 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:28:38.792238 5979 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7c48b6b6bf-2prt4: [secret "agc-cred-7d775f439052" not found, secret "agc-cred-de9101746298" not found] Nov 03 13:28:38.798526 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:28:38.792306 5979 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4-provider-credentials podName:b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4 nodeName:}" failed. No retries permitted until 2026-11-03 13:28:46.792286011 +0000 UTC m=+1408.802974235 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4-provider-credentials") pod "payload-processing-external-model-7c48b6b6bf-2prt4" (UID: "b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4") : [secret "agc-cred-7d775f439052" not found, secret "agc-cred-de9101746298" not found] Nov 03 13:28:38.894534 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:38.894105 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f266d718-6f0c-4972-b21b-db7533013291-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-abe0e0-766d597f55-t5mk2\" (UID: \"f266d718-6f0c-4972-b21b-db7533013291\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-abe0e0-766d597f55-t5mk2" Nov 03 13:28:38.894534 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:38.894227 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wprxj\" (UniqueName: \"kubernetes.io/projected/f266d718-6f0c-4972-b21b-db7533013291-kube-api-access-wprxj\") pod \"payload-pre-processing-e2e-api-b-abe0e0-766d597f55-t5mk2\" (UID: \"f266d718-6f0c-4972-b21b-db7533013291\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-abe0e0-766d597f55-t5mk2" Nov 03 13:28:38.897525 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:38.895322 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f266d718-6f0c-4972-b21b-db7533013291-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-abe0e0-766d597f55-t5mk2\" (UID: \"f266d718-6f0c-4972-b21b-db7533013291\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-abe0e0-766d597f55-t5mk2" Nov 03 13:28:38.905034 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:38.904998 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-abe0e0-6d6c498cf9-lktgw"] Nov 03 13:28:38.911422 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:38.911272 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-abe0e0-6d6c498cf9-lktgw" Nov 03 13:28:38.915022 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:38.914999 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-abe0e0-6d6c498cf9-lktgw"] Nov 03 13:28:38.948983 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:38.948953 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wprxj\" (UniqueName: \"kubernetes.io/projected/f266d718-6f0c-4972-b21b-db7533013291-kube-api-access-wprxj\") pod \"payload-pre-processing-e2e-api-b-abe0e0-766d597f55-t5mk2\" (UID: \"f266d718-6f0c-4972-b21b-db7533013291\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-abe0e0-766d597f55-t5mk2" Nov 03 13:28:38.951702 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:38.951678 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-abe0e0-766d597f55-t5mk2" Nov 03 13:28:39.097924 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:39.097892 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b8fe7cb-3ae1-474f-a573-cea5cae96854-plugins-config-volume\") pod \"payload-processing-e2e-api-b-abe0e0-6d6c498cf9-lktgw\" (UID: \"1b8fe7cb-3ae1-474f-a573-cea5cae96854\") " pod="openshift-ingress/payload-processing-e2e-api-b-abe0e0-6d6c498cf9-lktgw" Nov 03 13:28:39.098097 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:39.098052 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t54jr\" (UniqueName: \"kubernetes.io/projected/1b8fe7cb-3ae1-474f-a573-cea5cae96854-kube-api-access-t54jr\") pod \"payload-processing-e2e-api-b-abe0e0-6d6c498cf9-lktgw\" (UID: \"1b8fe7cb-3ae1-474f-a573-cea5cae96854\") " pod="openshift-ingress/payload-processing-e2e-api-b-abe0e0-6d6c498cf9-lktgw" Nov 03 13:28:39.200126 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:39.200037 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t54jr\" (UniqueName: \"kubernetes.io/projected/1b8fe7cb-3ae1-474f-a573-cea5cae96854-kube-api-access-t54jr\") pod \"payload-processing-e2e-api-b-abe0e0-6d6c498cf9-lktgw\" (UID: \"1b8fe7cb-3ae1-474f-a573-cea5cae96854\") " pod="openshift-ingress/payload-processing-e2e-api-b-abe0e0-6d6c498cf9-lktgw" Nov 03 13:28:39.200588 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:39.200331 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b8fe7cb-3ae1-474f-a573-cea5cae96854-plugins-config-volume\") pod \"payload-processing-e2e-api-b-abe0e0-6d6c498cf9-lktgw\" (UID: \"1b8fe7cb-3ae1-474f-a573-cea5cae96854\") " pod="openshift-ingress/payload-processing-e2e-api-b-abe0e0-6d6c498cf9-lktgw" Nov 03 13:28:39.201116 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:39.201097 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b8fe7cb-3ae1-474f-a573-cea5cae96854-plugins-config-volume\") pod \"payload-processing-e2e-api-b-abe0e0-6d6c498cf9-lktgw\" (UID: \"1b8fe7cb-3ae1-474f-a573-cea5cae96854\") " pod="openshift-ingress/payload-processing-e2e-api-b-abe0e0-6d6c498cf9-lktgw" Nov 03 13:28:39.254332 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:39.254294 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t54jr\" (UniqueName: \"kubernetes.io/projected/1b8fe7cb-3ae1-474f-a573-cea5cae96854-kube-api-access-t54jr\") pod \"payload-processing-e2e-api-b-abe0e0-6d6c498cf9-lktgw\" (UID: \"1b8fe7cb-3ae1-474f-a573-cea5cae96854\") " pod="openshift-ingress/payload-processing-e2e-api-b-abe0e0-6d6c498cf9-lktgw" Nov 03 13:28:39.328735 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:39.328698 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-abe0e0-766d597f55-t5mk2"] Nov 03 13:28:39.334181 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:28:39.334141 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf266d718_6f0c_4972_b21b_db7533013291.slice/crio-19e504de467b9753f70a2bb2e10d02d1c23cc06ab4539d6e7c0344fd637a74d7 WatchSource:0}: Error finding container 19e504de467b9753f70a2bb2e10d02d1c23cc06ab4539d6e7c0344fd637a74d7: Status 404 returned error can't find the container with id 19e504de467b9753f70a2bb2e10d02d1c23cc06ab4539d6e7c0344fd637a74d7 Nov 03 13:28:39.531913 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:39.531809 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-abe0e0-6d6c498cf9-lktgw" Nov 03 13:28:39.621632 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:39.621596 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-abe0e0-766d597f55-t5mk2" event={"ID":"f266d718-6f0c-4972-b21b-db7533013291","Type":"ContainerStarted","Data":"094bb5adb816224db5f9bc89202c0038d5aea0443b79f5b769f7f09aa071158d"} Nov 03 13:28:39.622112 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:39.621660 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-abe0e0-766d597f55-t5mk2" event={"ID":"f266d718-6f0c-4972-b21b-db7533013291","Type":"ContainerStarted","Data":"19e504de467b9753f70a2bb2e10d02d1c23cc06ab4539d6e7c0344fd637a74d7"} Nov 03 13:28:39.622112 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:39.621693 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-abe0e0-766d597f55-t5mk2" Nov 03 13:28:39.730600 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:39.730527 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-abe0e0-766d597f55-t5mk2" podStartSLOduration=1.7305103590000002 podStartE2EDuration="1.730510359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:28:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:28:39.72986966 +0000 UTC m=+1401.740557885" watchObservedRunningTime="2026-11-03 13:28:39.730510359 +0000 UTC m=+1401.741198610" Nov 03 13:28:39.933136 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:39.932116 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-abe0e0-6d6c498cf9-lktgw"] Nov 03 13:28:40.631935 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:40.631900 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-abe0e0-6d6c498cf9-lktgw" event={"ID":"1b8fe7cb-3ae1-474f-a573-cea5cae96854","Type":"ContainerStarted","Data":"9d66874e1f30507d5ad763160d3ff3c1d16c6dbdc3fcf770836add1f044fece8"} Nov 03 13:28:40.632447 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:40.631954 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-abe0e0-6d6c498cf9-lktgw" event={"ID":"1b8fe7cb-3ae1-474f-a573-cea5cae96854","Type":"ContainerStarted","Data":"14e8b1f8b584d1551ddfc0a1ea2dc15c504bd7a8c99b556a88be2a1bff2b7612"} Nov 03 13:28:40.632447 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:40.632153 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-abe0e0-6d6c498cf9-lktgw" Nov 03 13:28:40.743206 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:40.743151 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-abe0e0-6d6c498cf9-lktgw" podStartSLOduration=2.7431356 podStartE2EDuration="2.7431356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:28:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:28:40.740150316 +0000 UTC m=+1402.750838558" watchObservedRunningTime="2026-11-03 13:28:40.7431356 +0000 UTC m=+1402.753823841" Nov 03 13:28:41.057259 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:41.057222 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fcf7766-ptb4c"] Nov 03 13:28:41.061041 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:41.061015 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fcf7766-ptb4c"] Nov 03 13:28:41.871749 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:41.871715 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-84b78b6f97-fwtt2"] Nov 03 13:28:41.874565 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:41.874534 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-84b78b6f97-fwtt2" podUID="15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:28:41.878904 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:41.878880 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7c48b6b6bf-2prt4"] Nov 03 13:28:42.182857 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:42.182820 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a32d8b0-e28d-4f9a-9bad-461cd71473fa" path="/var/lib/kubelet/pods/7a32d8b0-e28d-4f9a-9bad-461cd71473fa/volumes" Nov 03 13:28:46.872240 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:46.872201 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-84b78b6f97-fwtt2" podUID="15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c" containerName="payload-processing" containerID="cri-o://c519e202e55eef0db45a72c00639edb64bc16a1ae6146ae57b3d6ae2ef5cfef1" gracePeriod=25 Nov 03 13:28:46.879551 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:46.879504 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7c48b6b6bf-2prt4" podUID="b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4" containerName="payload-processing" containerID="cri-o://c84d1214c12d75e9e3aca0b2d37d5c2251a60225a386c6af6d862d411907bbef" gracePeriod=25 Nov 03 13:28:46.888103 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:28:46.888079 5979 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-6d0760cb6b9f: secret "agc-cred-6d0760cb6b9f" not found Nov 03 13:28:46.888103 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:28:46.888101 5979 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-7d775f439052: secret "agc-cred-7d775f439052" not found Nov 03 13:28:46.888103 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:28:46.888108 5979 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-de9101746298: secret "agc-cred-de9101746298" not found Nov 03 13:28:46.888265 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:28:46.888117 5979 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7c48b6b6bf-2prt4: [secret "agc-cred-6d0760cb6b9f" not found, secret "agc-cred-7d775f439052" not found, secret "agc-cred-de9101746298" not found] Nov 03 13:28:46.888265 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:28:46.888182 5979 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4-provider-credentials podName:b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4 nodeName:}" failed. No retries permitted until 2026-11-03 13:29:02.8881664 +0000 UTC m=+1424.898854620 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4-provider-credentials") pod "payload-processing-external-model-7c48b6b6bf-2prt4" (UID: "b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4") : [secret "agc-cred-6d0760cb6b9f" not found, secret "agc-cred-7d775f439052" not found, secret "agc-cred-de9101746298" not found] Nov 03 13:28:47.165405 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:47.165375 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-84b78b6f97-fwtt2" Nov 03 13:28:47.191290 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:47.191266 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7c48b6b6bf-2prt4" Nov 03 13:28:47.293164 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:47.293136 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4-routing-overlay-volume\") pod \"b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4\" (UID: \"b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4\") " Nov 03 13:28:47.293324 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:47.293249 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c-plugins-config-volume\") pod \"15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c\" (UID: \"15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c\") " Nov 03 13:28:47.293406 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:47.293356 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c-routing-overlay-volume\") pod \"15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c\" (UID: \"15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c\") " Nov 03 13:28:47.293760 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:47.293408 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4-kube-api-access-xl6gd\" (UniqueName: \"kubernetes.io/projected/b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4-kube-api-access-xl6gd\") pod \"b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4\" (UID: \"b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4\") " Nov 03 13:28:47.293760 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:47.293463 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c-provider-credentials\") pod \"15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c\" (UID: \"15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c\") " Nov 03 13:28:47.293760 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:47.293574 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c-kube-api-access-79ftj\" (UniqueName: \"kubernetes.io/projected/15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c-kube-api-access-79ftj\") pod \"15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c\" (UID: \"15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c\") " Nov 03 13:28:47.293760 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:47.293615 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4-routing-overlay-volume" pod "b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4" (UID: "b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:28:47.293760 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:47.293632 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4-provider-credentials\") pod \"b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4\" (UID: \"b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4\") " Nov 03 13:28:47.293760 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:47.293660 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4-plugins-config-volume\") pod \"b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4\" (UID: \"b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4\") " Nov 03 13:28:47.294436 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:47.294408 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4-plugins-config-volume" pod "b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4" (UID: "b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:28:47.294577 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:47.294442 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c-routing-overlay-volume" pod "15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c" (UID: "15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:28:47.294720 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:47.294696 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c-plugins-config-volume" pod "15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c" (UID: "15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:28:47.294855 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:47.294831 5979 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c-routing-overlay-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:28:47.294928 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:47.294885 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:28:47.294928 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:47.294903 5979 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4-routing-overlay-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:28:47.296783 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:47.296758 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4-kube-api-access-xl6gd" pod "b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4" (UID: "b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4"). InnerVolumeSpecName "kube-api-access-xl6gd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:28:47.297173 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:47.297154 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c-kube-api-access-79ftj" pod "15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c" (UID: "15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c"). InnerVolumeSpecName "kube-api-access-79ftj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:28:47.297406 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:47.297384 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c-provider-credentials" pod "15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c" (UID: "15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:28:47.299702 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:47.299667 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4-provider-credentials" pod "b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4" (UID: "b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:28:47.396402 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:47.396306 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:28:47.396402 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:47.396338 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl6gd\" (UniqueName: \"kubernetes.io/projected/b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4-kube-api-access-xl6gd\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:28:47.396402 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:47.396353 5979 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c-provider-credentials\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:28:47.396402 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:47.396366 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79ftj\" (UniqueName: \"kubernetes.io/projected/15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c-kube-api-access-79ftj\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:28:47.396402 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:47.396378 5979 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4-provider-credentials\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:28:47.832549 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:47.832437 5979 generic.go:356] "Generic (PLEG): container finished" podID="15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c" containerID="c519e202e55eef0db45a72c00639edb64bc16a1ae6146ae57b3d6ae2ef5cfef1" exitCode=0 Nov 03 13:28:47.832549 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:47.832503 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-84b78b6f97-fwtt2" Nov 03 13:28:47.832549 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:47.832517 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-84b78b6f97-fwtt2" event={"ID":"15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c","Type":"ContainerDied","Data":"c519e202e55eef0db45a72c00639edb64bc16a1ae6146ae57b3d6ae2ef5cfef1"} Nov 03 13:28:47.832839 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:47.832578 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-84b78b6f97-fwtt2" event={"ID":"15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c","Type":"ContainerDied","Data":"a27cb70a8aff9c3c6036703e22bd1c0cd755790e892282585db947596537b685"} Nov 03 13:28:47.832839 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:47.832602 5979 scope.go:122] "RemoveContainer" containerID="c519e202e55eef0db45a72c00639edb64bc16a1ae6146ae57b3d6ae2ef5cfef1" Nov 03 13:28:47.834017 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:47.833994 5979 generic.go:356] "Generic (PLEG): container finished" podID="b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4" containerID="c84d1214c12d75e9e3aca0b2d37d5c2251a60225a386c6af6d862d411907bbef" exitCode=0 Nov 03 13:28:47.834154 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:47.834035 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7c48b6b6bf-2prt4" Nov 03 13:28:47.834154 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:47.834049 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7c48b6b6bf-2prt4" event={"ID":"b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4","Type":"ContainerDied","Data":"c84d1214c12d75e9e3aca0b2d37d5c2251a60225a386c6af6d862d411907bbef"} Nov 03 13:28:47.834154 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:47.834084 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7c48b6b6bf-2prt4" event={"ID":"b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4","Type":"ContainerDied","Data":"ddc09c507e0a3bca87a56bafe3bf6aa9006520aeae96a5f1db4f501a74f60f1d"} Nov 03 13:28:47.844414 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:47.844396 5979 scope.go:122] "RemoveContainer" containerID="c519e202e55eef0db45a72c00639edb64bc16a1ae6146ae57b3d6ae2ef5cfef1" Nov 03 13:28:47.844811 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:28:47.844781 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c519e202e55eef0db45a72c00639edb64bc16a1ae6146ae57b3d6ae2ef5cfef1\": container with ID starting with c519e202e55eef0db45a72c00639edb64bc16a1ae6146ae57b3d6ae2ef5cfef1 not found: ID does not exist" containerID="c519e202e55eef0db45a72c00639edb64bc16a1ae6146ae57b3d6ae2ef5cfef1" Nov 03 13:28:47.844920 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:47.844820 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c519e202e55eef0db45a72c00639edb64bc16a1ae6146ae57b3d6ae2ef5cfef1"} err="failed to get container status \"c519e202e55eef0db45a72c00639edb64bc16a1ae6146ae57b3d6ae2ef5cfef1\": rpc error: code = NotFound desc = could not find container \"c519e202e55eef0db45a72c00639edb64bc16a1ae6146ae57b3d6ae2ef5cfef1\": container with ID starting with c519e202e55eef0db45a72c00639edb64bc16a1ae6146ae57b3d6ae2ef5cfef1 not found: ID does not exist" Nov 03 13:28:47.844920 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:47.844844 5979 scope.go:122] "RemoveContainer" containerID="c84d1214c12d75e9e3aca0b2d37d5c2251a60225a386c6af6d862d411907bbef" Nov 03 13:28:47.854344 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:47.854322 5979 scope.go:122] "RemoveContainer" containerID="c84d1214c12d75e9e3aca0b2d37d5c2251a60225a386c6af6d862d411907bbef" Nov 03 13:28:47.854641 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:28:47.854620 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c84d1214c12d75e9e3aca0b2d37d5c2251a60225a386c6af6d862d411907bbef\": container with ID starting with c84d1214c12d75e9e3aca0b2d37d5c2251a60225a386c6af6d862d411907bbef not found: ID does not exist" containerID="c84d1214c12d75e9e3aca0b2d37d5c2251a60225a386c6af6d862d411907bbef" Nov 03 13:28:47.854729 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:47.854649 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c84d1214c12d75e9e3aca0b2d37d5c2251a60225a386c6af6d862d411907bbef"} err="failed to get container status \"c84d1214c12d75e9e3aca0b2d37d5c2251a60225a386c6af6d862d411907bbef\": rpc error: code = NotFound desc = could not find container \"c84d1214c12d75e9e3aca0b2d37d5c2251a60225a386c6af6d862d411907bbef\": container with ID starting with c84d1214c12d75e9e3aca0b2d37d5c2251a60225a386c6af6d862d411907bbef not found: ID does not exist" Nov 03 13:28:47.990205 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:47.990160 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7c48b6b6bf-2prt4"] Nov 03 13:28:47.997408 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:47.997375 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7c48b6b6bf-2prt4"] Nov 03 13:28:48.114544 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:48.110977 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-84b78b6f97-fwtt2"] Nov 03 13:28:48.174660 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:48.174625 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4" path="/var/lib/kubelet/pods/b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4/volumes" Nov 03 13:28:48.175201 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:48.175177 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-84b78b6f97-fwtt2"] Nov 03 13:28:50.116310 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:50.116278 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c" path="/var/lib/kubelet/pods/15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c/volumes" Nov 03 13:28:50.635586 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:50.635558 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-abe0e0-766d597f55-t5mk2" Nov 03 13:28:51.786532 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:51.782509 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-abe0e0-6d6c498cf9-lktgw" Nov 03 13:28:57.055550 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:57.055513 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21564fc7-b4kkc"] Nov 03 13:28:57.102370 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:57.102338 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21564fc7-b4kkc"] Nov 03 13:28:58.116442 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:28:58.116411 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56b539c3-1509-49a0-9a52-009dc1c0f3f7" path="/var/lib/kubelet/pods/56b539c3-1509-49a0-9a52-009dc1c0f3f7/volumes" Nov 03 13:29:05.024373 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:05.024341 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5295ab29-zqqnv"] Nov 03 13:29:05.025101 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:05.025083 5979 cpu_manager.go:414] "containerMap: removing container" podUID="15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c" containerName="payload-processing" Nov 03 13:29:05.025163 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:05.025102 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c" containerName="payload-processing" Nov 03 13:29:05.025163 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:05.025122 5979 cpu_manager.go:414] "containerMap: removing container" podUID="b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4" containerName="payload-processing" Nov 03 13:29:05.025163 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:05.025128 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4" containerName="payload-processing" Nov 03 13:29:05.025261 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:05.025240 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="15c412e3-e9c3-4a62-bd38-47c2c0f4fa2c" containerName="payload-processing" Nov 03 13:29:05.025261 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:05.025251 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3d318e3-0c74-4f44-ba58-6cd1bd0f39f4" containerName="payload-processing" Nov 03 13:29:05.030557 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:05.030539 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5295ab29-zqqnv" Nov 03 13:29:05.084359 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:05.084313 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5295ab29-zqqnv"] Nov 03 13:29:05.122742 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:05.122708 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn7px\" (UniqueName: \"kubernetes.io/projected/416c0968-c727-4bbc-996a-a63c72331f51-kube-api-access-xn7px\") pod \"maas-api-revoke-keys-e2e-mt-5295ab29-zqqnv\" (UID: \"416c0968-c727-4bbc-996a-a63c72331f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5295ab29-zqqnv" Nov 03 13:29:05.122913 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:05.122782 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/416c0968-c727-4bbc-996a-a63c72331f51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-5295ab29-zqqnv\" (UID: \"416c0968-c727-4bbc-996a-a63c72331f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5295ab29-zqqnv" Nov 03 13:29:05.122978 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:05.122952 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/416c0968-c727-4bbc-996a-a63c72331f51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-5295ab29-zqqnv\" (UID: \"416c0968-c727-4bbc-996a-a63c72331f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5295ab29-zqqnv" Nov 03 13:29:05.125960 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:05.125933 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:29:05.125960 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:05.125952 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 13:29:05.126167 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:05.126073 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 13:29:05.126226 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:05.126164 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:29:05.224593 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:05.224554 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn7px\" (UniqueName: \"kubernetes.io/projected/416c0968-c727-4bbc-996a-a63c72331f51-kube-api-access-xn7px\") pod \"maas-api-revoke-keys-e2e-mt-5295ab29-zqqnv\" (UID: \"416c0968-c727-4bbc-996a-a63c72331f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5295ab29-zqqnv" Nov 03 13:29:05.224807 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:05.224635 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/416c0968-c727-4bbc-996a-a63c72331f51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-5295ab29-zqqnv\" (UID: \"416c0968-c727-4bbc-996a-a63c72331f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5295ab29-zqqnv" Nov 03 13:29:05.224807 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:05.224688 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/416c0968-c727-4bbc-996a-a63c72331f51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-5295ab29-zqqnv\" (UID: \"416c0968-c727-4bbc-996a-a63c72331f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5295ab29-zqqnv" Nov 03 13:29:05.225283 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:05.225260 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/416c0968-c727-4bbc-996a-a63c72331f51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-5295ab29-zqqnv\" (UID: \"416c0968-c727-4bbc-996a-a63c72331f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5295ab29-zqqnv" Nov 03 13:29:05.227086 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:05.227061 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/416c0968-c727-4bbc-996a-a63c72331f51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-5295ab29-zqqnv\" (UID: \"416c0968-c727-4bbc-996a-a63c72331f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5295ab29-zqqnv" Nov 03 13:29:05.286075 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:05.285999 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn7px\" (UniqueName: \"kubernetes.io/projected/416c0968-c727-4bbc-996a-a63c72331f51-kube-api-access-xn7px\") pod \"maas-api-revoke-keys-e2e-mt-5295ab29-zqqnv\" (UID: \"416c0968-c727-4bbc-996a-a63c72331f51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5295ab29-zqqnv" Nov 03 13:29:05.343928 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:05.343890 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5295ab29-zqqnv" Nov 03 13:29:05.727448 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:05.727411 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5295ab29-zqqnv"] Nov 03 13:29:05.735300 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:29:05.735274 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod416c0968_c727_4bbc_996a_a63c72331f51.slice/crio-c48a364364fe3353d408febb8679fd61d335a7c9d8356e353d57adf204dd055d WatchSource:0}: Error finding container c48a364364fe3353d408febb8679fd61d335a7c9d8356e353d57adf204dd055d: Status 404 returned error can't find the container with id c48a364364fe3353d408febb8679fd61d335a7c9d8356e353d57adf204dd055d Nov 03 13:29:06.011958 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:06.011868 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5295ab29-zqqnv" event={"ID":"416c0968-c727-4bbc-996a-a63c72331f51","Type":"ContainerStarted","Data":"c48a364364fe3353d408febb8679fd61d335a7c9d8356e353d57adf204dd055d"} Nov 03 13:29:07.021572 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:07.021482 5979 generic.go:356] "Generic (PLEG): container finished" podID="416c0968-c727-4bbc-996a-a63c72331f51" containerID="5471d8e4b233b9a9f7afcb7c04815b86f3d5d1b42ddc7c71b9074c90502cbca7" exitCode=0 Nov 03 13:29:07.021957 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:07.021567 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5295ab29-zqqnv" event={"ID":"416c0968-c727-4bbc-996a-a63c72331f51","Type":"ContainerDied","Data":"5471d8e4b233b9a9f7afcb7c04815b86f3d5d1b42ddc7c71b9074c90502cbca7"} Nov 03 13:29:07.235060 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:29:07.235015 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:29:08.184079 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:08.184054 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5295ab29-zqqnv" Nov 03 13:29:08.268169 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:08.268138 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/416c0968-c727-4bbc-996a-a63c72331f51-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/416c0968-c727-4bbc-996a-a63c72331f51-maas-api-xks-ca\") pod \"416c0968-c727-4bbc-996a-a63c72331f51\" (UID: \"416c0968-c727-4bbc-996a-a63c72331f51\") " Nov 03 13:29:08.268363 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:08.268201 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/416c0968-c727-4bbc-996a-a63c72331f51-kube-api-access-xn7px\" (UniqueName: \"kubernetes.io/projected/416c0968-c727-4bbc-996a-a63c72331f51-kube-api-access-xn7px\") pod \"416c0968-c727-4bbc-996a-a63c72331f51\" (UID: \"416c0968-c727-4bbc-996a-a63c72331f51\") " Nov 03 13:29:08.268363 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:08.268296 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/416c0968-c727-4bbc-996a-a63c72331f51-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/416c0968-c727-4bbc-996a-a63c72331f51-maas-api-openshift-ca\") pod \"416c0968-c727-4bbc-996a-a63c72331f51\" (UID: \"416c0968-c727-4bbc-996a-a63c72331f51\") " Nov 03 13:29:08.268726 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:08.268701 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/416c0968-c727-4bbc-996a-a63c72331f51-maas-api-openshift-ca" pod "416c0968-c727-4bbc-996a-a63c72331f51" (UID: "416c0968-c727-4bbc-996a-a63c72331f51"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:29:08.268839 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:08.268825 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/416c0968-c727-4bbc-996a-a63c72331f51-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:29:08.270154 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:08.270127 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/416c0968-c727-4bbc-996a-a63c72331f51-maas-api-xks-ca" pod "416c0968-c727-4bbc-996a-a63c72331f51" (UID: "416c0968-c727-4bbc-996a-a63c72331f51"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:29:08.270367 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:08.270352 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/416c0968-c727-4bbc-996a-a63c72331f51-kube-api-access-xn7px" pod "416c0968-c727-4bbc-996a-a63c72331f51" (UID: "416c0968-c727-4bbc-996a-a63c72331f51"). InnerVolumeSpecName "kube-api-access-xn7px". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:29:08.370743 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:08.370707 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/416c0968-c727-4bbc-996a-a63c72331f51-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:29:08.370743 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:08.370736 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn7px\" (UniqueName: \"kubernetes.io/projected/416c0968-c727-4bbc-996a-a63c72331f51-kube-api-access-xn7px\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:29:09.045700 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:09.045668 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5295ab29-zqqnv" Nov 03 13:29:09.045855 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:09.045707 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5295ab29-zqqnv" event={"ID":"416c0968-c727-4bbc-996a-a63c72331f51","Type":"ContainerDied","Data":"c48a364364fe3353d408febb8679fd61d335a7c9d8356e353d57adf204dd055d"} Nov 03 13:29:09.045855 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:09.045795 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c48a364364fe3353d408febb8679fd61d335a7c9d8356e353d57adf204dd055d" Nov 03 13:29:13.045512 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:13.045462 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a0494c9-pvzls"] Nov 03 13:29:13.143498 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:13.138902 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a0494c9-pvzls"] Nov 03 13:29:13.338924 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:13.338839 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:29:14.046576 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:14.046540 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8841c477-xzd8t"] Nov 03 13:29:14.093018 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:14.092978 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8841c477-xzd8t"] Nov 03 13:29:14.115554 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:14.115520 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0604ce99-cf1e-4a2c-a1f4-b6363a0fa865" path="/var/lib/kubelet/pods/0604ce99-cf1e-4a2c-a1f4-b6363a0fa865/volumes" Nov 03 13:29:14.116012 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:14.115996 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="353ec922-1cbb-465a-aefa-60803be0959e" path="/var/lib/kubelet/pods/353ec922-1cbb-465a-aefa-60803be0959e/volumes" Nov 03 13:29:16.567415 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:16.567372 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5295ab29-79c7656b49-qjvj4"] Nov 03 13:29:16.673336 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:16.673274 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5295ab29-6d58657867-lbm4z"] Nov 03 13:29:21.568760 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:21.568716 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-5295ab29-79c7656b49-qjvj4" podUID="5fab5fbe-10b6-4598-b135-30d3f1eae97b" containerName="payload-processing" containerID="cri-o://760ef3889538ff1ef10caa083f4ae2951a334c99b47a20d27f3a53fb329fcd36" gracePeriod=25 Nov 03 13:29:21.674586 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:21.674539 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-5295ab29-6d58657867-lbm4z" podUID="8ad148b9-3b17-46b3-8dca-0d36e3b43aa3" containerName="payload-pre-processing" containerID="cri-o://d3f939f4aba2f3ae4b37c5810b085e214b64dabebb63d8a5f7406ded7f68c2a6" gracePeriod=25 Nov 03 13:29:24.447334 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:24.447290 5979 generic.go:356] "Generic (PLEG): container finished" podID="e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5" containerID="c015efe631d4b0927cfa8378487110d1a80d7128c49f1c5bd103219ffa4cdb3c" exitCode=0 Nov 03 13:29:24.447914 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:24.447367 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb" event={"ID":"e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5","Type":"ContainerDied","Data":"c015efe631d4b0927cfa8378487110d1a80d7128c49f1c5bd103219ffa4cdb3c"} Nov 03 13:29:24.447993 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:24.447942 5979 scope.go:122] "RemoveContainer" containerID="c015efe631d4b0927cfa8378487110d1a80d7128c49f1c5bd103219ffa4cdb3c" Nov 03 13:29:24.449615 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:24.449576 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:29:24.449745 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:24.449658 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:29:24.449745 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:24.449708 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:29:25.078500 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:25.077549 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e321db9b-gthpp"] Nov 03 13:29:25.101216 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:25.101185 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e321db9b-gthpp"] Nov 03 13:29:25.477437 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:25.477397 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb" event={"ID":"e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5","Type":"ContainerStarted","Data":"d50528b2a85504ba67be25c95423ac67172534ed201e3edef2d05eba08509966"} Nov 03 13:29:26.025522 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:26.025487 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb" Nov 03 13:29:26.026418 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:26.026390 5979 patch_prober.go:29] interesting pod/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 03 13:29:26.026562 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:26.026430 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb" podUID="e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 03 13:29:26.125082 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:26.125044 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f32e29d-157b-4baf-a2bf-01fe1102fac5" path="/var/lib/kubelet/pods/4f32e29d-157b-4baf-a2bf-01fe1102fac5/volumes" Nov 03 13:29:26.910291 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:26.910254 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb" Nov 03 13:29:27.029381 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:27.027044 5979 patch_prober.go:29] interesting pod/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 03 13:29:27.029381 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:27.027122 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb" podUID="e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 03 13:29:28.031495 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:28.025886 5979 patch_prober.go:29] interesting pod/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 03 13:29:28.031495 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:28.025950 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb" podUID="e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 03 13:29:29.033543 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:29.032505 5979 patch_prober.go:29] interesting pod/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 03 13:29:29.033543 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:29.032574 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb" podUID="e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 03 13:29:29.096120 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:29.096088 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-5295ab29-79c7656b49-qjvj4" podUID="5fab5fbe-10b6-4598-b135-30d3f1eae97b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:29:29.096340 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:29.096157 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-5295ab29-6d58657867-lbm4z" podUID="8ad148b9-3b17-46b3-8dca-0d36e3b43aa3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:29:30.025930 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:30.025892 5979 patch_prober.go:29] interesting pod/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Nov 03 13:29:30.026139 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:30.025950 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb" podUID="e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Nov 03 13:29:31.030164 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:31.030131 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb" Nov 03 13:29:31.031236 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:31.031213 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb" Nov 03 13:29:37.234805 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:29:37.234763 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:29:39.104522 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:39.101038 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-5295ab29-79c7656b49-qjvj4" podUID="5fab5fbe-10b6-4598-b135-30d3f1eae97b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:29:39.104522 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:39.101323 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-5295ab29-6d58657867-lbm4z" podUID="8ad148b9-3b17-46b3-8dca-0d36e3b43aa3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:29:40.369029 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:40.368991 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw"] Nov 03 13:29:40.375787 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:40.375746 5979 cpu_manager.go:414] "containerMap: removing container" podUID="416c0968-c727-4bbc-996a-a63c72331f51" containerName="revoke-keys" Nov 03 13:29:40.376040 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:40.376021 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="416c0968-c727-4bbc-996a-a63c72331f51" containerName="revoke-keys" Nov 03 13:29:40.376438 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:40.376422 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="416c0968-c727-4bbc-996a-a63c72331f51" containerName="revoke-keys" Nov 03 13:29:40.387847 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:40.387812 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw" Nov 03 13:29:40.442156 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:40.442112 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-461b2d-openshift-default-dockercfg-8wtbs\"" Nov 03 13:29:40.506141 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:40.506103 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw"] Nov 03 13:29:40.510844 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:40.510814 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-istio-podinfo\") pod \"e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw\" (UID: \"e5478d2d-1afc-4d18-9e91-e1965f3f51c0\") " pod="openshift-ingress/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw" Nov 03 13:29:40.511166 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:40.511142 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-workload-certs\") pod \"e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw\" (UID: \"e5478d2d-1afc-4d18-9e91-e1965f3f51c0\") " pod="openshift-ingress/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw" Nov 03 13:29:40.511487 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:40.511455 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-istio-data\") pod \"e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw\" (UID: \"e5478d2d-1afc-4d18-9e91-e1965f3f51c0\") " pod="openshift-ingress/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw" Nov 03 13:29:40.511720 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:40.511702 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-istio-envoy\") pod \"e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw\" (UID: \"e5478d2d-1afc-4d18-9e91-e1965f3f51c0\") " pod="openshift-ingress/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw" Nov 03 13:29:40.511906 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:40.511877 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zd4d\" (UniqueName: \"kubernetes.io/projected/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-kube-api-access-4zd4d\") pod \"e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw\" (UID: \"e5478d2d-1afc-4d18-9e91-e1965f3f51c0\") " pod="openshift-ingress/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw" Nov 03 13:29:40.512143 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:40.512123 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-istiod-ca-cert\") pod \"e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw\" (UID: \"e5478d2d-1afc-4d18-9e91-e1965f3f51c0\") " pod="openshift-ingress/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw" Nov 03 13:29:40.512329 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:40.512310 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-workload-socket\") pod \"e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw\" (UID: \"e5478d2d-1afc-4d18-9e91-e1965f3f51c0\") " pod="openshift-ingress/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw" Nov 03 13:29:40.512526 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:40.512506 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-istio-token\") pod \"e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw\" (UID: \"e5478d2d-1afc-4d18-9e91-e1965f3f51c0\") " pod="openshift-ingress/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw" Nov 03 13:29:40.512710 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:40.512694 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-credential-socket\") pod \"e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw\" (UID: \"e5478d2d-1afc-4d18-9e91-e1965f3f51c0\") " pod="openshift-ingress/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw" Nov 03 13:29:40.619543 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:40.615404 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-workload-certs\") pod \"e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw\" (UID: \"e5478d2d-1afc-4d18-9e91-e1965f3f51c0\") " pod="openshift-ingress/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw" Nov 03 13:29:40.619543 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:40.615524 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-istio-data\") pod \"e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw\" (UID: \"e5478d2d-1afc-4d18-9e91-e1965f3f51c0\") " pod="openshift-ingress/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw" Nov 03 13:29:40.619543 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:40.615609 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-istio-envoy\") pod \"e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw\" (UID: \"e5478d2d-1afc-4d18-9e91-e1965f3f51c0\") " pod="openshift-ingress/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw" Nov 03 13:29:40.619543 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:40.615649 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zd4d\" (UniqueName: \"kubernetes.io/projected/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-kube-api-access-4zd4d\") pod \"e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw\" (UID: \"e5478d2d-1afc-4d18-9e91-e1965f3f51c0\") " pod="openshift-ingress/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw" Nov 03 13:29:40.619543 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:40.615753 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-istiod-ca-cert\") pod \"e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw\" (UID: \"e5478d2d-1afc-4d18-9e91-e1965f3f51c0\") " pod="openshift-ingress/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw" Nov 03 13:29:40.619543 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:40.615789 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-workload-socket\") pod \"e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw\" (UID: \"e5478d2d-1afc-4d18-9e91-e1965f3f51c0\") " pod="openshift-ingress/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw" Nov 03 13:29:40.619543 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:40.615813 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-istio-token\") pod \"e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw\" (UID: \"e5478d2d-1afc-4d18-9e91-e1965f3f51c0\") " pod="openshift-ingress/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw" Nov 03 13:29:40.619543 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:40.615855 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-credential-socket\") pod \"e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw\" (UID: \"e5478d2d-1afc-4d18-9e91-e1965f3f51c0\") " pod="openshift-ingress/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw" Nov 03 13:29:40.619543 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:40.616005 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-istio-podinfo\") pod \"e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw\" (UID: \"e5478d2d-1afc-4d18-9e91-e1965f3f51c0\") " pod="openshift-ingress/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw" Nov 03 13:29:40.632130 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:40.632092 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-workload-certs\") pod \"e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw\" (UID: \"e5478d2d-1afc-4d18-9e91-e1965f3f51c0\") " pod="openshift-ingress/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw" Nov 03 13:29:40.665496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:40.633611 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-istiod-ca-cert\") pod \"e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw\" (UID: \"e5478d2d-1afc-4d18-9e91-e1965f3f51c0\") " pod="openshift-ingress/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw" Nov 03 13:29:40.680652 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:40.664004 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-workload-socket\") pod \"e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw\" (UID: \"e5478d2d-1afc-4d18-9e91-e1965f3f51c0\") " pod="openshift-ingress/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw" Nov 03 13:29:40.680652 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:40.664249 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-credential-socket\") pod \"e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw\" (UID: \"e5478d2d-1afc-4d18-9e91-e1965f3f51c0\") " pod="openshift-ingress/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw" Nov 03 13:29:40.680652 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:40.674845 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-istio-data\") pod \"e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw\" (UID: \"e5478d2d-1afc-4d18-9e91-e1965f3f51c0\") " pod="openshift-ingress/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw" Nov 03 13:29:40.706888 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:40.706801 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-istio-envoy\") pod \"e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw\" (UID: \"e5478d2d-1afc-4d18-9e91-e1965f3f51c0\") " pod="openshift-ingress/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw" Nov 03 13:29:40.734496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:40.730031 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-istio-podinfo\") pod \"e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw\" (UID: \"e5478d2d-1afc-4d18-9e91-e1965f3f51c0\") " pod="openshift-ingress/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw" Nov 03 13:29:40.749507 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:40.744548 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-istio-token\") pod \"e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw\" (UID: \"e5478d2d-1afc-4d18-9e91-e1965f3f51c0\") " pod="openshift-ingress/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw" Nov 03 13:29:40.749507 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:40.748155 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zd4d\" (UniqueName: \"kubernetes.io/projected/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-kube-api-access-4zd4d\") pod \"e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw\" (UID: \"e5478d2d-1afc-4d18-9e91-e1965f3f51c0\") " pod="openshift-ingress/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw" Nov 03 13:29:40.753075 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:40.752813 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw" Nov 03 13:29:41.373656 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:41.373620 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw"] Nov 03 13:29:41.374100 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:29:41.374069 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5478d2d_1afc_4d18_9e91_e1965f3f51c0.slice/crio-cb53c77d2317236788eff372f383a4c359a5639fd85431e03de9c0e6fef19619 WatchSource:0}: Error finding container cb53c77d2317236788eff372f383a4c359a5639fd85431e03de9c0e6fef19619: Status 404 returned error can't find the container with id cb53c77d2317236788eff372f383a4c359a5639fd85431e03de9c0e6fef19619 Nov 03 13:29:41.380607 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:41.380561 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:29:41.380717 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:41.380638 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:29:41.380717 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:41.380666 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:29:41.717009 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:41.716637 5979 generic.go:356] "Generic (PLEG): container finished" podID="5fab5fbe-10b6-4598-b135-30d3f1eae97b" containerID="760ef3889538ff1ef10caa083f4ae2951a334c99b47a20d27f3a53fb329fcd36" exitCode=0 Nov 03 13:29:41.717009 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:41.716730 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5295ab29-79c7656b49-qjvj4" event={"ID":"5fab5fbe-10b6-4598-b135-30d3f1eae97b","Type":"ContainerDied","Data":"760ef3889538ff1ef10caa083f4ae2951a334c99b47a20d27f3a53fb329fcd36"} Nov 03 13:29:41.719053 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:41.719027 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw" event={"ID":"e5478d2d-1afc-4d18-9e91-e1965f3f51c0","Type":"ContainerStarted","Data":"4c37c527438e7778be040ed075fd4a776f6721feb812f6539f6939850020aba2"} Nov 03 13:29:41.719148 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:41.719072 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw" event={"ID":"e5478d2d-1afc-4d18-9e91-e1965f3f51c0","Type":"ContainerStarted","Data":"cb53c77d2317236788eff372f383a4c359a5639fd85431e03de9c0e6fef19619"} Nov 03 13:29:41.770983 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:41.757602 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw" Nov 03 13:29:41.934441 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:41.934411 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5295ab29-79c7656b49-qjvj4" Nov 03 13:29:42.049667 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.045613 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fab5fbe-10b6-4598-b135-30d3f1eae97b-kube-api-access-wkhlj\" (UniqueName: \"kubernetes.io/projected/5fab5fbe-10b6-4598-b135-30d3f1eae97b-kube-api-access-wkhlj\") pod \"5fab5fbe-10b6-4598-b135-30d3f1eae97b\" (UID: \"5fab5fbe-10b6-4598-b135-30d3f1eae97b\") " Nov 03 13:29:42.104498 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.059048 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fab5fbe-10b6-4598-b135-30d3f1eae97b-kube-api-access-wkhlj" pod "5fab5fbe-10b6-4598-b135-30d3f1eae97b" (UID: "5fab5fbe-10b6-4598-b135-30d3f1eae97b"). InnerVolumeSpecName "kube-api-access-wkhlj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:29:42.104498 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.083989 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw" podStartSLOduration=2.083968657 podStartE2EDuration="2.083968657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:29:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:29:41.865924768 +0000 UTC m=+1463.876613011" watchObservedRunningTime="2026-11-03 13:29:42.083968657 +0000 UTC m=+1464.094656900" Nov 03 13:29:42.104498 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.093906 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fab5fbe-10b6-4598-b135-30d3f1eae97b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fab5fbe-10b6-4598-b135-30d3f1eae97b-plugins-config-volume\") pod \"5fab5fbe-10b6-4598-b135-30d3f1eae97b\" (UID: \"5fab5fbe-10b6-4598-b135-30d3f1eae97b\") " Nov 03 13:29:42.121497 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.121076 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fab5fbe-10b6-4598-b135-30d3f1eae97b-plugins-config-volume" pod "5fab5fbe-10b6-4598-b135-30d3f1eae97b" (UID: "5fab5fbe-10b6-4598-b135-30d3f1eae97b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:29:42.167509 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.166140 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkhlj\" (UniqueName: \"kubernetes.io/projected/5fab5fbe-10b6-4598-b135-30d3f1eae97b-kube-api-access-wkhlj\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:29:42.167509 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.166176 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5fab5fbe-10b6-4598-b135-30d3f1eae97b-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:29:42.195512 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.194247 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1866534f-pcvnf"] Nov 03 13:29:42.195512 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.194297 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1866534f-pcvnf"] Nov 03 13:29:42.207279 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.207256 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5295ab29-6d58657867-lbm4z" Nov 03 13:29:42.268636 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.268598 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ad148b9-3b17-46b3-8dca-0d36e3b43aa3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ad148b9-3b17-46b3-8dca-0d36e3b43aa3-plugins-config-volume\") pod \"8ad148b9-3b17-46b3-8dca-0d36e3b43aa3\" (UID: \"8ad148b9-3b17-46b3-8dca-0d36e3b43aa3\") " Nov 03 13:29:42.268849 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.268691 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ad148b9-3b17-46b3-8dca-0d36e3b43aa3-kube-api-access-xhms2\" (UniqueName: \"kubernetes.io/projected/8ad148b9-3b17-46b3-8dca-0d36e3b43aa3-kube-api-access-xhms2\") pod \"8ad148b9-3b17-46b3-8dca-0d36e3b43aa3\" (UID: \"8ad148b9-3b17-46b3-8dca-0d36e3b43aa3\") " Nov 03 13:29:42.269580 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.269544 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ad148b9-3b17-46b3-8dca-0d36e3b43aa3-plugins-config-volume" pod "8ad148b9-3b17-46b3-8dca-0d36e3b43aa3" (UID: "8ad148b9-3b17-46b3-8dca-0d36e3b43aa3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:29:42.271113 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.271087 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ad148b9-3b17-46b3-8dca-0d36e3b43aa3-kube-api-access-xhms2" pod "8ad148b9-3b17-46b3-8dca-0d36e3b43aa3" (UID: "8ad148b9-3b17-46b3-8dca-0d36e3b43aa3"). InnerVolumeSpecName "kube-api-access-xhms2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:29:42.370671 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.370548 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ad148b9-3b17-46b3-8dca-0d36e3b43aa3-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:29:42.370671 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.370581 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhms2\" (UniqueName: \"kubernetes.io/projected/8ad148b9-3b17-46b3-8dca-0d36e3b43aa3-kube-api-access-xhms2\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:29:42.728018 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.727989 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5295ab29-79c7656b49-qjvj4" Nov 03 13:29:42.728516 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.728038 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5295ab29-79c7656b49-qjvj4" event={"ID":"5fab5fbe-10b6-4598-b135-30d3f1eae97b","Type":"ContainerDied","Data":"02ead4a953d38c568c22220bdd086f56a82efad2e45fc7c77db4c2dc97f9a012"} Nov 03 13:29:42.728516 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.728110 5979 scope.go:122] "RemoveContainer" containerID="760ef3889538ff1ef10caa083f4ae2951a334c99b47a20d27f3a53fb329fcd36" Nov 03 13:29:42.729927 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.729900 5979 generic.go:356] "Generic (PLEG): container finished" podID="8ad148b9-3b17-46b3-8dca-0d36e3b43aa3" containerID="d3f939f4aba2f3ae4b37c5810b085e214b64dabebb63d8a5f7406ded7f68c2a6" exitCode=0 Nov 03 13:29:42.730057 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.729942 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5295ab29-6d58657867-lbm4z" Nov 03 13:29:42.730057 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.729983 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5295ab29-6d58657867-lbm4z" event={"ID":"8ad148b9-3b17-46b3-8dca-0d36e3b43aa3","Type":"ContainerDied","Data":"d3f939f4aba2f3ae4b37c5810b085e214b64dabebb63d8a5f7406ded7f68c2a6"} Nov 03 13:29:42.730057 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.730029 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5295ab29-6d58657867-lbm4z" event={"ID":"8ad148b9-3b17-46b3-8dca-0d36e3b43aa3","Type":"ContainerDied","Data":"f446401f808816475302f8602bb87fe2643c3d735acb93e8fae41a0a67f15b79"} Nov 03 13:29:42.739116 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.739099 5979 scope.go:122] "RemoveContainer" containerID="d3f939f4aba2f3ae4b37c5810b085e214b64dabebb63d8a5f7406ded7f68c2a6" Nov 03 13:29:42.773710 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.773685 5979 scope.go:122] "RemoveContainer" containerID="d3f939f4aba2f3ae4b37c5810b085e214b64dabebb63d8a5f7406ded7f68c2a6" Nov 03 13:29:42.774120 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:29:42.774089 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3f939f4aba2f3ae4b37c5810b085e214b64dabebb63d8a5f7406ded7f68c2a6\": container with ID starting with d3f939f4aba2f3ae4b37c5810b085e214b64dabebb63d8a5f7406ded7f68c2a6 not found: ID does not exist" containerID="d3f939f4aba2f3ae4b37c5810b085e214b64dabebb63d8a5f7406ded7f68c2a6" Nov 03 13:29:42.774204 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.774120 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3f939f4aba2f3ae4b37c5810b085e214b64dabebb63d8a5f7406ded7f68c2a6"} err="failed to get container status \"d3f939f4aba2f3ae4b37c5810b085e214b64dabebb63d8a5f7406ded7f68c2a6\": rpc error: code = NotFound desc = could not find container \"d3f939f4aba2f3ae4b37c5810b085e214b64dabebb63d8a5f7406ded7f68c2a6\": container with ID starting with d3f939f4aba2f3ae4b37c5810b085e214b64dabebb63d8a5f7406ded7f68c2a6 not found: ID does not exist" Nov 03 13:29:42.778351 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.778331 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw" Nov 03 13:29:42.807915 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.807890 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8"] Nov 03 13:29:42.808582 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.808565 5979 cpu_manager.go:414] "containerMap: removing container" podUID="8ad148b9-3b17-46b3-8dca-0d36e3b43aa3" containerName="payload-pre-processing" Nov 03 13:29:42.808582 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.808581 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ad148b9-3b17-46b3-8dca-0d36e3b43aa3" containerName="payload-pre-processing" Nov 03 13:29:42.808706 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.808634 5979 cpu_manager.go:414] "containerMap: removing container" podUID="5fab5fbe-10b6-4598-b135-30d3f1eae97b" containerName="payload-processing" Nov 03 13:29:42.808706 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.808639 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fab5fbe-10b6-4598-b135-30d3f1eae97b" containerName="payload-processing" Nov 03 13:29:42.808804 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.808793 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fab5fbe-10b6-4598-b135-30d3f1eae97b" containerName="payload-processing" Nov 03 13:29:42.808851 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.808809 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ad148b9-3b17-46b3-8dca-0d36e3b43aa3" containerName="payload-pre-processing" Nov 03 13:29:42.813835 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.813817 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8" Nov 03 13:29:42.865496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.864839 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-cdb532-openshift-default-dockercfg-q2fdz\"" Nov 03 13:29:42.877365 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.877333 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/337bd1a1-184a-4942-a921-b6a7a0df9130-istio-data\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8\" (UID: \"337bd1a1-184a-4942-a921-b6a7a0df9130\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8" Nov 03 13:29:42.877569 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.877423 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/337bd1a1-184a-4942-a921-b6a7a0df9130-istio-envoy\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8\" (UID: \"337bd1a1-184a-4942-a921-b6a7a0df9130\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8" Nov 03 13:29:42.877569 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.877532 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/337bd1a1-184a-4942-a921-b6a7a0df9130-istiod-ca-cert\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8\" (UID: \"337bd1a1-184a-4942-a921-b6a7a0df9130\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8" Nov 03 13:29:42.877668 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.877598 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/337bd1a1-184a-4942-a921-b6a7a0df9130-workload-certs\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8\" (UID: \"337bd1a1-184a-4942-a921-b6a7a0df9130\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8" Nov 03 13:29:42.877668 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.877638 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdsmh\" (UniqueName: \"kubernetes.io/projected/337bd1a1-184a-4942-a921-b6a7a0df9130-kube-api-access-rdsmh\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8\" (UID: \"337bd1a1-184a-4942-a921-b6a7a0df9130\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8" Nov 03 13:29:42.877738 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.877722 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/337bd1a1-184a-4942-a921-b6a7a0df9130-workload-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8\" (UID: \"337bd1a1-184a-4942-a921-b6a7a0df9130\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8" Nov 03 13:29:42.877775 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.877740 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/337bd1a1-184a-4942-a921-b6a7a0df9130-credential-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8\" (UID: \"337bd1a1-184a-4942-a921-b6a7a0df9130\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8" Nov 03 13:29:42.877809 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.877770 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/337bd1a1-184a-4942-a921-b6a7a0df9130-istio-podinfo\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8\" (UID: \"337bd1a1-184a-4942-a921-b6a7a0df9130\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8" Nov 03 13:29:42.877881 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.877862 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/337bd1a1-184a-4942-a921-b6a7a0df9130-istio-token\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8\" (UID: \"337bd1a1-184a-4942-a921-b6a7a0df9130\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8" Nov 03 13:29:42.885390 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.885355 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5295ab29-6d58657867-lbm4z"] Nov 03 13:29:42.889829 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.889804 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5295ab29-6d58657867-lbm4z"] Nov 03 13:29:42.915121 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.915096 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8"] Nov 03 13:29:42.979167 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.979069 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/337bd1a1-184a-4942-a921-b6a7a0df9130-istiod-ca-cert\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8\" (UID: \"337bd1a1-184a-4942-a921-b6a7a0df9130\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8" Nov 03 13:29:42.979167 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.979165 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/337bd1a1-184a-4942-a921-b6a7a0df9130-workload-certs\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8\" (UID: \"337bd1a1-184a-4942-a921-b6a7a0df9130\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8" Nov 03 13:29:42.979404 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.979213 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdsmh\" (UniqueName: \"kubernetes.io/projected/337bd1a1-184a-4942-a921-b6a7a0df9130-kube-api-access-rdsmh\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8\" (UID: \"337bd1a1-184a-4942-a921-b6a7a0df9130\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8" Nov 03 13:29:42.979404 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.979298 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/337bd1a1-184a-4942-a921-b6a7a0df9130-workload-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8\" (UID: \"337bd1a1-184a-4942-a921-b6a7a0df9130\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8" Nov 03 13:29:42.979404 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.979324 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/337bd1a1-184a-4942-a921-b6a7a0df9130-credential-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8\" (UID: \"337bd1a1-184a-4942-a921-b6a7a0df9130\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8" Nov 03 13:29:42.979404 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.979367 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/337bd1a1-184a-4942-a921-b6a7a0df9130-istio-podinfo\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8\" (UID: \"337bd1a1-184a-4942-a921-b6a7a0df9130\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8" Nov 03 13:29:42.979404 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.979396 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/337bd1a1-184a-4942-a921-b6a7a0df9130-istio-token\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8\" (UID: \"337bd1a1-184a-4942-a921-b6a7a0df9130\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8" Nov 03 13:29:42.979744 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.979512 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/337bd1a1-184a-4942-a921-b6a7a0df9130-istio-data\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8\" (UID: \"337bd1a1-184a-4942-a921-b6a7a0df9130\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8" Nov 03 13:29:42.979744 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.979557 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/337bd1a1-184a-4942-a921-b6a7a0df9130-istio-envoy\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8\" (UID: \"337bd1a1-184a-4942-a921-b6a7a0df9130\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8" Nov 03 13:29:42.979744 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.979720 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/337bd1a1-184a-4942-a921-b6a7a0df9130-workload-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8\" (UID: \"337bd1a1-184a-4942-a921-b6a7a0df9130\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8" Nov 03 13:29:42.979928 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.979778 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/337bd1a1-184a-4942-a921-b6a7a0df9130-workload-certs\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8\" (UID: \"337bd1a1-184a-4942-a921-b6a7a0df9130\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8" Nov 03 13:29:42.980027 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.980005 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/337bd1a1-184a-4942-a921-b6a7a0df9130-credential-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8\" (UID: \"337bd1a1-184a-4942-a921-b6a7a0df9130\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8" Nov 03 13:29:42.980027 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.980016 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/337bd1a1-184a-4942-a921-b6a7a0df9130-istiod-ca-cert\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8\" (UID: \"337bd1a1-184a-4942-a921-b6a7a0df9130\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8" Nov 03 13:29:42.980228 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.980205 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/337bd1a1-184a-4942-a921-b6a7a0df9130-istio-data\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8\" (UID: \"337bd1a1-184a-4942-a921-b6a7a0df9130\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8" Nov 03 13:29:42.982550 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.982517 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/337bd1a1-184a-4942-a921-b6a7a0df9130-istio-envoy\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8\" (UID: \"337bd1a1-184a-4942-a921-b6a7a0df9130\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8" Nov 03 13:29:42.982708 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:42.982678 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/337bd1a1-184a-4942-a921-b6a7a0df9130-istio-podinfo\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8\" (UID: \"337bd1a1-184a-4942-a921-b6a7a0df9130\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8" Nov 03 13:29:43.006350 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:43.006322 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5295ab29-79c7656b49-qjvj4"] Nov 03 13:29:43.014833 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:43.014810 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5295ab29-79c7656b49-qjvj4"] Nov 03 13:29:43.039941 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:43.039908 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/337bd1a1-184a-4942-a921-b6a7a0df9130-istio-token\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8\" (UID: \"337bd1a1-184a-4942-a921-b6a7a0df9130\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8" Nov 03 13:29:43.040832 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:43.040815 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdsmh\" (UniqueName: \"kubernetes.io/projected/337bd1a1-184a-4942-a921-b6a7a0df9130-kube-api-access-rdsmh\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8\" (UID: \"337bd1a1-184a-4942-a921-b6a7a0df9130\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8" Nov 03 13:29:43.127089 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:43.127055 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8" Nov 03 13:29:43.590801 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:43.590760 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8"] Nov 03 13:29:43.595394 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:29:43.595361 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod337bd1a1_184a_4942_a921_b6a7a0df9130.slice/crio-61bfd55a0268e6fc6447332a108cb7cfdf309c936d03f11ad1975de114a63509 WatchSource:0}: Error finding container 61bfd55a0268e6fc6447332a108cb7cfdf309c936d03f11ad1975de114a63509: Status 404 returned error can't find the container with id 61bfd55a0268e6fc6447332a108cb7cfdf309c936d03f11ad1975de114a63509 Nov 03 13:29:43.598002 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:43.597960 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:29:43.598096 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:43.598028 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:29:43.598096 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:43.598058 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:29:43.742329 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:43.741213 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8" event={"ID":"337bd1a1-184a-4942-a921-b6a7a0df9130","Type":"ContainerStarted","Data":"d110700aec531b08e67013b3016382c18cc02599adfba48b658ce7f088e7a1ce"} Nov 03 13:29:43.742329 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:43.741272 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8" event={"ID":"337bd1a1-184a-4942-a921-b6a7a0df9130","Type":"ContainerStarted","Data":"61bfd55a0268e6fc6447332a108cb7cfdf309c936d03f11ad1975de114a63509"} Nov 03 13:29:43.757512 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:43.753049 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw" Nov 03 13:29:43.758059 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:43.758008 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw" Nov 03 13:29:43.862554 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:43.861600 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8" podStartSLOduration=1.8615805399999998 podStartE2EDuration="1.86158054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:29:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:29:43.858984699 +0000 UTC m=+1465.869672957" watchObservedRunningTime="2026-11-03 13:29:43.86158054 +0000 UTC m=+1465.872268803" Nov 03 13:29:44.117706 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:44.117621 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fab5fbe-10b6-4598-b135-30d3f1eae97b" path="/var/lib/kubelet/pods/5fab5fbe-10b6-4598-b135-30d3f1eae97b/volumes" Nov 03 13:29:44.118078 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:44.118062 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ad148b9-3b17-46b3-8dca-0d36e3b43aa3" path="/var/lib/kubelet/pods/8ad148b9-3b17-46b3-8dca-0d36e3b43aa3/volumes" Nov 03 13:29:44.118486 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:44.118454 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6ea1f5b-6ba3-4053-a5e5-28554fefccbb" path="/var/lib/kubelet/pods/d6ea1f5b-6ba3-4053-a5e5-28554fefccbb/volumes" Nov 03 13:29:44.127856 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:44.127828 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8" Nov 03 13:29:44.129418 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:44.129390 5979 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 03 13:29:44.129569 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:44.129436 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8" podUID="337bd1a1-184a-4942-a921-b6a7a0df9130" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 03 13:29:45.127673 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:45.127630 5979 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 03 13:29:45.128129 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:45.127691 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8" podUID="337bd1a1-184a-4942-a921-b6a7a0df9130" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 03 13:29:46.127428 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:46.127339 5979 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 03 13:29:46.127428 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:46.127391 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8" podUID="337bd1a1-184a-4942-a921-b6a7a0df9130" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 03 13:29:46.475137 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:46.475100 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7"] Nov 03 13:29:46.482499 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:46.482440 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7" Nov 03 13:29:46.537507 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:46.536566 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-istiod-ca-cert\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7\" (UID: \"dd4c2ad4-8ff1-40ba-91e5-5e4556213933\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7" Nov 03 13:29:46.537507 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:46.536615 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg29v\" (UniqueName: \"kubernetes.io/projected/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-kube-api-access-kg29v\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7\" (UID: \"dd4c2ad4-8ff1-40ba-91e5-5e4556213933\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7" Nov 03 13:29:46.537507 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:46.536668 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-istio-podinfo\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7\" (UID: \"dd4c2ad4-8ff1-40ba-91e5-5e4556213933\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7" Nov 03 13:29:46.537507 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:46.536722 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-istio-data\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7\" (UID: \"dd4c2ad4-8ff1-40ba-91e5-5e4556213933\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7" Nov 03 13:29:46.537507 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:46.536841 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-istio-token\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7\" (UID: \"dd4c2ad4-8ff1-40ba-91e5-5e4556213933\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7" Nov 03 13:29:46.537507 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:46.536878 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-credential-socket\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7\" (UID: \"dd4c2ad4-8ff1-40ba-91e5-5e4556213933\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7" Nov 03 13:29:46.537507 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:46.536980 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-istio-envoy\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7\" (UID: \"dd4c2ad4-8ff1-40ba-91e5-5e4556213933\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7" Nov 03 13:29:46.537507 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:46.537116 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-workload-socket\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7\" (UID: \"dd4c2ad4-8ff1-40ba-91e5-5e4556213933\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7" Nov 03 13:29:46.537507 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:46.537165 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-workload-certs\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7\" (UID: \"dd4c2ad4-8ff1-40ba-91e5-5e4556213933\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7" Nov 03 13:29:46.540450 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:46.539145 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-28d87c-openshift-default-dockercfg-lcrzm\"" Nov 03 13:29:46.679545 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:46.677005 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-istio-envoy\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7\" (UID: \"dd4c2ad4-8ff1-40ba-91e5-5e4556213933\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7" Nov 03 13:29:46.679545 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:46.677161 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-workload-socket\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7\" (UID: \"dd4c2ad4-8ff1-40ba-91e5-5e4556213933\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7" Nov 03 13:29:46.679545 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:46.677198 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-workload-certs\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7\" (UID: \"dd4c2ad4-8ff1-40ba-91e5-5e4556213933\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7" Nov 03 13:29:46.679545 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:46.677236 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-istiod-ca-cert\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7\" (UID: \"dd4c2ad4-8ff1-40ba-91e5-5e4556213933\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7" Nov 03 13:29:46.679545 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:46.677261 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg29v\" (UniqueName: \"kubernetes.io/projected/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-kube-api-access-kg29v\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7\" (UID: \"dd4c2ad4-8ff1-40ba-91e5-5e4556213933\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7" Nov 03 13:29:46.679545 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:46.677299 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-istio-podinfo\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7\" (UID: \"dd4c2ad4-8ff1-40ba-91e5-5e4556213933\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7" Nov 03 13:29:46.679545 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:46.677350 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-istio-data\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7\" (UID: \"dd4c2ad4-8ff1-40ba-91e5-5e4556213933\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7" Nov 03 13:29:46.679545 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:46.677486 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-istio-token\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7\" (UID: \"dd4c2ad4-8ff1-40ba-91e5-5e4556213933\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7" Nov 03 13:29:46.679545 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:46.677538 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-credential-socket\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7\" (UID: \"dd4c2ad4-8ff1-40ba-91e5-5e4556213933\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7" Nov 03 13:29:46.679545 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:46.678003 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-credential-socket\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7\" (UID: \"dd4c2ad4-8ff1-40ba-91e5-5e4556213933\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7" Nov 03 13:29:46.726539 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:46.724998 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-workload-socket\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7\" (UID: \"dd4c2ad4-8ff1-40ba-91e5-5e4556213933\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7" Nov 03 13:29:46.726539 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:46.725255 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-workload-certs\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7\" (UID: \"dd4c2ad4-8ff1-40ba-91e5-5e4556213933\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7" Nov 03 13:29:46.726539 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:46.725867 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-istiod-ca-cert\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7\" (UID: \"dd4c2ad4-8ff1-40ba-91e5-5e4556213933\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7" Nov 03 13:29:46.726539 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:46.726125 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-istio-data\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7\" (UID: \"dd4c2ad4-8ff1-40ba-91e5-5e4556213933\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7" Nov 03 13:29:46.787496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:46.780419 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-istio-envoy\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7\" (UID: \"dd4c2ad4-8ff1-40ba-91e5-5e4556213933\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7" Nov 03 13:29:46.984300 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:46.983127 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-istio-podinfo\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7\" (UID: \"dd4c2ad4-8ff1-40ba-91e5-5e4556213933\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7" Nov 03 13:29:47.131494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:47.131059 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7"] Nov 03 13:29:47.133563 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:47.133528 5979 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 03 13:29:47.133686 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:47.133601 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8" podUID="337bd1a1-184a-4942-a921-b6a7a0df9130" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 03 13:29:47.136638 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:47.136611 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-istio-token\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7\" (UID: \"dd4c2ad4-8ff1-40ba-91e5-5e4556213933\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7" Nov 03 13:29:47.138276 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:47.138242 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg29v\" (UniqueName: \"kubernetes.io/projected/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-kube-api-access-kg29v\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7\" (UID: \"dd4c2ad4-8ff1-40ba-91e5-5e4556213933\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7" Nov 03 13:29:47.421044 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:47.421003 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7" Nov 03 13:29:47.913421 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:47.913379 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7"] Nov 03 13:29:47.920421 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:29:47.920383 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd4c2ad4_8ff1_40ba_91e5_5e4556213933.slice/crio-e106c4d7857b264c76ea934db802a9a1c370bc731e7884ae6fbb2dd46dae97e6 WatchSource:0}: Error finding container e106c4d7857b264c76ea934db802a9a1c370bc731e7884ae6fbb2dd46dae97e6: Status 404 returned error can't find the container with id e106c4d7857b264c76ea934db802a9a1c370bc731e7884ae6fbb2dd46dae97e6 Nov 03 13:29:47.925555 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:47.925507 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:29:47.925648 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:47.925614 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:29:47.925692 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:47.925664 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:29:48.127927 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:48.127879 5979 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 03 13:29:48.128103 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:48.127955 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8" podUID="337bd1a1-184a-4942-a921-b6a7a0df9130" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 03 13:29:48.184627 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:48.184544 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7" event={"ID":"dd4c2ad4-8ff1-40ba-91e5-5e4556213933","Type":"ContainerStarted","Data":"7da7678ef1c981b9e5f62d2acaf668e2134b7db5116739fde82b81dffa45b13d"} Nov 03 13:29:48.184627 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:48.184614 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7" event={"ID":"dd4c2ad4-8ff1-40ba-91e5-5e4556213933","Type":"ContainerStarted","Data":"e106c4d7857b264c76ea934db802a9a1c370bc731e7884ae6fbb2dd46dae97e6"} Nov 03 13:29:48.370498 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:48.370163 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7" podStartSLOduration=2.370141382 podStartE2EDuration="2.370141382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:29:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:29:48.305033641 +0000 UTC m=+1470.315721896" watchObservedRunningTime="2026-11-03 13:29:48.370141382 +0000 UTC m=+1470.380829625" Nov 03 13:29:48.459546 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:48.458703 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7" Nov 03 13:29:48.522905 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:48.522863 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-461b2d-7d984ffbf9-262hb"] Nov 03 13:29:48.529623 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:48.528819 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-461b2d-7d984ffbf9-262hb" Nov 03 13:29:48.584832 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:48.580284 5979 patch_prober.go:29] interesting pod/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:29:48.584832 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:48.580343 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf" podUID="fa17ed42-8832-4fea-b962-9029c6e6dbb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:29:48.584832 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:48.581327 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-461b2d-dockercfg-8mb9m\"" Nov 03 13:29:48.584832 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:48.581398 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-461b2d\"" Nov 03 13:29:48.592493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:48.588880 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-461b2d-7d984ffbf9-262hb"] Nov 03 13:29:48.685528 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:48.669842 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28v2f\" (UniqueName: \"kubernetes.io/projected/be6c35e9-cf38-4a4b-a393-932c02403030-kube-api-access-28v2f\") pod \"payload-pre-processing-e2e-api-a-461b2d-7d984ffbf9-262hb\" (UID: \"be6c35e9-cf38-4a4b-a393-932c02403030\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-461b2d-7d984ffbf9-262hb" Nov 03 13:29:48.685528 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:48.670055 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be6c35e9-cf38-4a4b-a393-932c02403030-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-461b2d-7d984ffbf9-262hb\" (UID: \"be6c35e9-cf38-4a4b-a393-932c02403030\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-461b2d-7d984ffbf9-262hb" Nov 03 13:29:48.869543 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:48.867450 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28v2f\" (UniqueName: \"kubernetes.io/projected/be6c35e9-cf38-4a4b-a393-932c02403030-kube-api-access-28v2f\") pod \"payload-pre-processing-e2e-api-a-461b2d-7d984ffbf9-262hb\" (UID: \"be6c35e9-cf38-4a4b-a393-932c02403030\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-461b2d-7d984ffbf9-262hb" Nov 03 13:29:48.869543 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:48.867716 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be6c35e9-cf38-4a4b-a393-932c02403030-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-461b2d-7d984ffbf9-262hb\" (UID: \"be6c35e9-cf38-4a4b-a393-932c02403030\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-461b2d-7d984ffbf9-262hb" Nov 03 13:29:48.869543 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:48.868909 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be6c35e9-cf38-4a4b-a393-932c02403030-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-461b2d-7d984ffbf9-262hb\" (UID: \"be6c35e9-cf38-4a4b-a393-932c02403030\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-461b2d-7d984ffbf9-262hb" Nov 03 13:29:49.008791 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:49.008348 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28v2f\" (UniqueName: \"kubernetes.io/projected/be6c35e9-cf38-4a4b-a393-932c02403030-kube-api-access-28v2f\") pod \"payload-pre-processing-e2e-api-a-461b2d-7d984ffbf9-262hb\" (UID: \"be6c35e9-cf38-4a4b-a393-932c02403030\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-461b2d-7d984ffbf9-262hb" Nov 03 13:29:49.008791 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:49.008436 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-461b2d-6466bf6b8f-48fhn"] Nov 03 13:29:49.021496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:49.016446 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-461b2d-6466bf6b8f-48fhn" Nov 03 13:29:49.021496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:49.017347 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-461b2d-6466bf6b8f-48fhn"] Nov 03 13:29:49.128416 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:49.128337 5979 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 03 13:29:49.128570 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:49.128415 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8" podUID="337bd1a1-184a-4942-a921-b6a7a0df9130" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 03 13:29:49.168502 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:49.163731 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-461b2d-7d984ffbf9-262hb" Nov 03 13:29:49.174804 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:49.174767 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v27mc\" (UniqueName: \"kubernetes.io/projected/bb292b8a-a577-42de-a66b-646fdf0e22b8-kube-api-access-v27mc\") pod \"payload-processing-e2e-api-a-461b2d-6466bf6b8f-48fhn\" (UID: \"bb292b8a-a577-42de-a66b-646fdf0e22b8\") " pod="openshift-ingress/payload-processing-e2e-api-a-461b2d-6466bf6b8f-48fhn" Nov 03 13:29:49.174983 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:49.174963 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb292b8a-a577-42de-a66b-646fdf0e22b8-plugins-config-volume\") pod \"payload-processing-e2e-api-a-461b2d-6466bf6b8f-48fhn\" (UID: \"bb292b8a-a577-42de-a66b-646fdf0e22b8\") " pod="openshift-ingress/payload-processing-e2e-api-a-461b2d-6466bf6b8f-48fhn" Nov 03 13:29:49.280511 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:49.277846 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v27mc\" (UniqueName: \"kubernetes.io/projected/bb292b8a-a577-42de-a66b-646fdf0e22b8-kube-api-access-v27mc\") pod \"payload-processing-e2e-api-a-461b2d-6466bf6b8f-48fhn\" (UID: \"bb292b8a-a577-42de-a66b-646fdf0e22b8\") " pod="openshift-ingress/payload-processing-e2e-api-a-461b2d-6466bf6b8f-48fhn" Nov 03 13:29:49.280511 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:49.278986 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb292b8a-a577-42de-a66b-646fdf0e22b8-plugins-config-volume\") pod \"payload-processing-e2e-api-a-461b2d-6466bf6b8f-48fhn\" (UID: \"bb292b8a-a577-42de-a66b-646fdf0e22b8\") " pod="openshift-ingress/payload-processing-e2e-api-a-461b2d-6466bf6b8f-48fhn" Nov 03 13:29:49.280511 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:49.279983 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb292b8a-a577-42de-a66b-646fdf0e22b8-plugins-config-volume\") pod \"payload-processing-e2e-api-a-461b2d-6466bf6b8f-48fhn\" (UID: \"bb292b8a-a577-42de-a66b-646fdf0e22b8\") " pod="openshift-ingress/payload-processing-e2e-api-a-461b2d-6466bf6b8f-48fhn" Nov 03 13:29:49.331795 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:49.331745 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v27mc\" (UniqueName: \"kubernetes.io/projected/bb292b8a-a577-42de-a66b-646fdf0e22b8-kube-api-access-v27mc\") pod \"payload-processing-e2e-api-a-461b2d-6466bf6b8f-48fhn\" (UID: \"bb292b8a-a577-42de-a66b-646fdf0e22b8\") " pod="openshift-ingress/payload-processing-e2e-api-a-461b2d-6466bf6b8f-48fhn" Nov 03 13:29:49.367494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:49.366789 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-461b2d-6466bf6b8f-48fhn" Nov 03 13:29:49.435111 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:49.433790 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7" Nov 03 13:29:49.587849 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:49.587816 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-461b2d-7d984ffbf9-262hb"] Nov 03 13:29:49.635252 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:29:49.633896 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe6c35e9_cf38_4a4b_a393_932c02403030.slice/crio-1bd1f72c1dad8be1937839385c8fe7b4104c0d9ca08346e12117a8c8a39fb1b6 WatchSource:0}: Error finding container 1bd1f72c1dad8be1937839385c8fe7b4104c0d9ca08346e12117a8c8a39fb1b6: Status 404 returned error can't find the container with id 1bd1f72c1dad8be1937839385c8fe7b4104c0d9ca08346e12117a8c8a39fb1b6 Nov 03 13:29:49.774104 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:49.773894 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-461b2d-6466bf6b8f-48fhn"] Nov 03 13:29:49.774104 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:29:49.774039 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb292b8a_a577_42de_a66b_646fdf0e22b8.slice/crio-e9596a430131586bab7417487ecc65a889fa54d196ba38bfe5b84648c7a25437 WatchSource:0}: Error finding container e9596a430131586bab7417487ecc65a889fa54d196ba38bfe5b84648c7a25437: Status 404 returned error can't find the container with id e9596a430131586bab7417487ecc65a889fa54d196ba38bfe5b84648c7a25437 Nov 03 13:29:50.127960 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:50.127879 5979 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 03 13:29:50.127960 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:50.127937 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8" podUID="337bd1a1-184a-4942-a921-b6a7a0df9130" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 03 13:29:50.217142 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:50.217100 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-461b2d-7d984ffbf9-262hb" event={"ID":"be6c35e9-cf38-4a4b-a393-932c02403030","Type":"ContainerStarted","Data":"172910e9b60ba468df1baf2ff58e9b868aa048154002fe3dab07e6a1279a9584"} Nov 03 13:29:50.217389 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:50.217370 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-461b2d-7d984ffbf9-262hb" Nov 03 13:29:50.217520 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:50.217405 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-461b2d-7d984ffbf9-262hb" event={"ID":"be6c35e9-cf38-4a4b-a393-932c02403030","Type":"ContainerStarted","Data":"1bd1f72c1dad8be1937839385c8fe7b4104c0d9ca08346e12117a8c8a39fb1b6"} Nov 03 13:29:50.219098 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:50.219060 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-461b2d-6466bf6b8f-48fhn" event={"ID":"bb292b8a-a577-42de-a66b-646fdf0e22b8","Type":"ContainerStarted","Data":"015ca4a909b0339e1d848ea381ea24f38a1ec1f377cd2e719fdf9a303b27c8b1"} Nov 03 13:29:50.219207 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:50.219138 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-461b2d-6466bf6b8f-48fhn" event={"ID":"bb292b8a-a577-42de-a66b-646fdf0e22b8","Type":"ContainerStarted","Data":"e9596a430131586bab7417487ecc65a889fa54d196ba38bfe5b84648c7a25437"} Nov 03 13:29:50.219346 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:50.219320 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-461b2d-6466bf6b8f-48fhn" Nov 03 13:29:50.219606 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:50.219586 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7" Nov 03 13:29:50.220447 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:50.220428 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7" Nov 03 13:29:50.333619 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:50.333545 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-461b2d-7d984ffbf9-262hb" podStartSLOduration=2.33353219 podStartE2EDuration="2.33353219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:29:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:29:50.331338825 +0000 UTC m=+1472.342027068" watchObservedRunningTime="2026-11-03 13:29:50.33353219 +0000 UTC m=+1472.344220431" Nov 03 13:29:50.438256 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:50.438204 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-461b2d-6466bf6b8f-48fhn" podStartSLOduration=2.4381907099999998 podStartE2EDuration="2.43819071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:29:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:29:50.43690503 +0000 UTC m=+1472.447593271" watchObservedRunningTime="2026-11-03 13:29:50.43819071 +0000 UTC m=+1472.448878951" Nov 03 13:29:51.128402 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:51.128357 5979 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Nov 03 13:29:51.128848 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:51.128427 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8" podUID="337bd1a1-184a-4942-a921-b6a7a0df9130" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Nov 03 13:29:52.132061 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:52.132033 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8" Nov 03 13:29:52.132523 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:52.132376 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8" Nov 03 13:29:52.133244 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:52.133227 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8" Nov 03 13:29:52.240138 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:52.240056 5979 generic.go:356] "Generic (PLEG): container finished" podID="fa17ed42-8832-4fea-b962-9029c6e6dbb9" containerID="0638dfc254a429df1d30b3e1bd1175552ef5d69f8b77c529acf1f3fa2970ab8b" exitCode=0 Nov 03 13:29:52.240304 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:52.240138 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf" event={"ID":"fa17ed42-8832-4fea-b962-9029c6e6dbb9","Type":"ContainerDied","Data":"0638dfc254a429df1d30b3e1bd1175552ef5d69f8b77c529acf1f3fa2970ab8b"} Nov 03 13:29:52.240907 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:52.240882 5979 scope.go:122] "RemoveContainer" containerID="0638dfc254a429df1d30b3e1bd1175552ef5d69f8b77c529acf1f3fa2970ab8b" Nov 03 13:29:52.242480 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:52.242435 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:29:52.242584 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:52.242555 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:29:52.242625 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:52.242605 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:29:53.264495 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:53.255549 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf" event={"ID":"fa17ed42-8832-4fea-b962-9029c6e6dbb9","Type":"ContainerStarted","Data":"453ebeeb01d5af571ca02703d1b423cda52203f86f2355ad7249ba0d400804f1"} Nov 03 13:29:54.237404 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:54.237365 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf" Nov 03 13:29:54.237891 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:54.237870 5979 patch_prober.go:29] interesting pod/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 03 13:29:54.237942 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:54.237917 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf" podUID="fa17ed42-8832-4fea-b962-9029c6e6dbb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 03 13:29:55.066495 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:55.063949 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-28d87c-6c79c6f6dc-pvhkm"] Nov 03 13:29:55.120834 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:55.120803 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-28d87c-6c79c6f6dc-pvhkm" Nov 03 13:29:55.130669 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:55.130632 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-28d87c-6c79c6f6dc-pvhkm"] Nov 03 13:29:55.310494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:55.310084 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-28d87c-dockercfg-zhfpj\"" Nov 03 13:29:55.310494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:55.310194 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-28d87c\"" Nov 03 13:29:55.349582 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:55.340665 5979 patch_prober.go:29] interesting pod/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 03 13:29:55.349582 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:55.340726 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf" podUID="fa17ed42-8832-4fea-b962-9029c6e6dbb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 03 13:29:55.354496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:55.354125 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-28d87c-f8886f745-ggq7c"] Nov 03 13:29:55.435493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:55.415498 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-28d87c-f8886f745-ggq7c" Nov 03 13:29:55.448961 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:55.448924 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/332311c4-5b9c-4cb2-b48d-ec4308a75c41-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-28d87c-6c79c6f6dc-pvhkm\" (UID: \"332311c4-5b9c-4cb2-b48d-ec4308a75c41\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-28d87c-6c79c6f6dc-pvhkm" Nov 03 13:29:55.449317 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:55.449294 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d52v\" (UniqueName: \"kubernetes.io/projected/332311c4-5b9c-4cb2-b48d-ec4308a75c41-kube-api-access-5d52v\") pod \"payload-pre-processing-e2e-api-b-28d87c-6c79c6f6dc-pvhkm\" (UID: \"332311c4-5b9c-4cb2-b48d-ec4308a75c41\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-28d87c-6c79c6f6dc-pvhkm" Nov 03 13:29:55.493998 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:55.481637 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-28d87c-f8886f745-ggq7c"] Nov 03 13:29:55.552497 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:55.551486 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/332311c4-5b9c-4cb2-b48d-ec4308a75c41-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-28d87c-6c79c6f6dc-pvhkm\" (UID: \"332311c4-5b9c-4cb2-b48d-ec4308a75c41\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-28d87c-6c79c6f6dc-pvhkm" Nov 03 13:29:55.552497 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:55.551586 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d52v\" (UniqueName: \"kubernetes.io/projected/332311c4-5b9c-4cb2-b48d-ec4308a75c41-kube-api-access-5d52v\") pod \"payload-pre-processing-e2e-api-b-28d87c-6c79c6f6dc-pvhkm\" (UID: \"332311c4-5b9c-4cb2-b48d-ec4308a75c41\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-28d87c-6c79c6f6dc-pvhkm" Nov 03 13:29:55.552497 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:55.551685 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15e26ebd-f572-4f06-b7cc-55f415f492ad-plugins-config-volume\") pod \"payload-processing-e2e-api-b-28d87c-f8886f745-ggq7c\" (UID: \"15e26ebd-f572-4f06-b7cc-55f415f492ad\") " pod="openshift-ingress/payload-processing-e2e-api-b-28d87c-f8886f745-ggq7c" Nov 03 13:29:55.552497 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:55.551777 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57pj8\" (UniqueName: \"kubernetes.io/projected/15e26ebd-f572-4f06-b7cc-55f415f492ad-kube-api-access-57pj8\") pod \"payload-processing-e2e-api-b-28d87c-f8886f745-ggq7c\" (UID: \"15e26ebd-f572-4f06-b7cc-55f415f492ad\") " pod="openshift-ingress/payload-processing-e2e-api-b-28d87c-f8886f745-ggq7c" Nov 03 13:29:55.561522 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:55.553329 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/332311c4-5b9c-4cb2-b48d-ec4308a75c41-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-28d87c-6c79c6f6dc-pvhkm\" (UID: \"332311c4-5b9c-4cb2-b48d-ec4308a75c41\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-28d87c-6c79c6f6dc-pvhkm" Nov 03 13:29:55.606902 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:55.606830 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d52v\" (UniqueName: \"kubernetes.io/projected/332311c4-5b9c-4cb2-b48d-ec4308a75c41-kube-api-access-5d52v\") pod \"payload-pre-processing-e2e-api-b-28d87c-6c79c6f6dc-pvhkm\" (UID: \"332311c4-5b9c-4cb2-b48d-ec4308a75c41\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-28d87c-6c79c6f6dc-pvhkm" Nov 03 13:29:55.654551 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:55.654098 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57pj8\" (UniqueName: \"kubernetes.io/projected/15e26ebd-f572-4f06-b7cc-55f415f492ad-kube-api-access-57pj8\") pod \"payload-processing-e2e-api-b-28d87c-f8886f745-ggq7c\" (UID: \"15e26ebd-f572-4f06-b7cc-55f415f492ad\") " pod="openshift-ingress/payload-processing-e2e-api-b-28d87c-f8886f745-ggq7c" Nov 03 13:29:55.655122 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:55.655100 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-28d87c-6c79c6f6dc-pvhkm" Nov 03 13:29:55.655872 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:55.655848 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15e26ebd-f572-4f06-b7cc-55f415f492ad-plugins-config-volume\") pod \"payload-processing-e2e-api-b-28d87c-f8886f745-ggq7c\" (UID: \"15e26ebd-f572-4f06-b7cc-55f415f492ad\") " pod="openshift-ingress/payload-processing-e2e-api-b-28d87c-f8886f745-ggq7c" Nov 03 13:29:55.657077 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:55.656877 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15e26ebd-f572-4f06-b7cc-55f415f492ad-plugins-config-volume\") pod \"payload-processing-e2e-api-b-28d87c-f8886f745-ggq7c\" (UID: \"15e26ebd-f572-4f06-b7cc-55f415f492ad\") " pod="openshift-ingress/payload-processing-e2e-api-b-28d87c-f8886f745-ggq7c" Nov 03 13:29:55.711133 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:55.711102 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57pj8\" (UniqueName: \"kubernetes.io/projected/15e26ebd-f572-4f06-b7cc-55f415f492ad-kube-api-access-57pj8\") pod \"payload-processing-e2e-api-b-28d87c-f8886f745-ggq7c\" (UID: \"15e26ebd-f572-4f06-b7cc-55f415f492ad\") " pod="openshift-ingress/payload-processing-e2e-api-b-28d87c-f8886f745-ggq7c" Nov 03 13:29:55.806668 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:55.806638 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-28d87c-f8886f745-ggq7c" Nov 03 13:29:56.043492 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:56.043445 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-28d87c-6c79c6f6dc-pvhkm"] Nov 03 13:29:56.073012 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:29:56.072969 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod332311c4_5b9c_4cb2_b48d_ec4308a75c41.slice/crio-dd85e3d1c537712bb1fd9534c79508f37949a6dab5ec09a9c9e3c844177c21a9 WatchSource:0}: Error finding container dd85e3d1c537712bb1fd9534c79508f37949a6dab5ec09a9c9e3c844177c21a9: Status 404 returned error can't find the container with id dd85e3d1c537712bb1fd9534c79508f37949a6dab5ec09a9c9e3c844177c21a9 Nov 03 13:29:56.207247 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:56.207205 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-28d87c-f8886f745-ggq7c"] Nov 03 13:29:56.212453 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:29:56.212424 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15e26ebd_f572_4f06_b7cc_55f415f492ad.slice/crio-2774e048c543c9815de9f29c5eb21ca6e4d6c3807d64c2252d12a51c665d2f8d WatchSource:0}: Error finding container 2774e048c543c9815de9f29c5eb21ca6e4d6c3807d64c2252d12a51c665d2f8d: Status 404 returned error can't find the container with id 2774e048c543c9815de9f29c5eb21ca6e4d6c3807d64c2252d12a51c665d2f8d Nov 03 13:29:56.237952 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:56.237926 5979 patch_prober.go:29] interesting pod/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 03 13:29:56.238080 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:56.237973 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf" podUID="fa17ed42-8832-4fea-b962-9029c6e6dbb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 03 13:29:56.514619 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:56.514521 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-28d87c-f8886f745-ggq7c" event={"ID":"15e26ebd-f572-4f06-b7cc-55f415f492ad","Type":"ContainerStarted","Data":"8587e7ed08e92ba11caddb03e2794062c76520f060966cf5454777b593245dd4"} Nov 03 13:29:56.514619 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:56.514586 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-28d87c-f8886f745-ggq7c" Nov 03 13:29:56.514619 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:56.514614 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-28d87c-f8886f745-ggq7c" event={"ID":"15e26ebd-f572-4f06-b7cc-55f415f492ad","Type":"ContainerStarted","Data":"2774e048c543c9815de9f29c5eb21ca6e4d6c3807d64c2252d12a51c665d2f8d"} Nov 03 13:29:56.516341 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:56.516309 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-28d87c-6c79c6f6dc-pvhkm" event={"ID":"332311c4-5b9c-4cb2-b48d-ec4308a75c41","Type":"ContainerStarted","Data":"8e673f1f77c22e6ed530bf7fe1c9d8edf7c5bb1bfb164e56fd695ec6b5df5865"} Nov 03 13:29:56.516498 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:56.516367 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-28d87c-6c79c6f6dc-pvhkm" event={"ID":"332311c4-5b9c-4cb2-b48d-ec4308a75c41","Type":"ContainerStarted","Data":"dd85e3d1c537712bb1fd9534c79508f37949a6dab5ec09a9c9e3c844177c21a9"} Nov 03 13:29:56.516498 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:56.516401 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-28d87c-6c79c6f6dc-pvhkm" Nov 03 13:29:56.628407 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:56.628353 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-28d87c-f8886f745-ggq7c" podStartSLOduration=1.628339636 podStartE2EDuration="1.628339636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:29:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:29:56.626009095 +0000 UTC m=+1478.636697337" watchObservedRunningTime="2026-11-03 13:29:56.628339636 +0000 UTC m=+1478.639027917" Nov 03 13:29:56.737115 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:56.737057 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-28d87c-6c79c6f6dc-pvhkm" podStartSLOduration=2.737042988 podStartE2EDuration="2.737042988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:29:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:29:56.733278611 +0000 UTC m=+1478.743966852" watchObservedRunningTime="2026-11-03 13:29:56.737042988 +0000 UTC m=+1478.747731230" Nov 03 13:29:57.237808 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:57.237769 5979 patch_prober.go:29] interesting pod/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 03 13:29:57.238224 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:57.237826 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf" podUID="fa17ed42-8832-4fea-b962-9029c6e6dbb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 03 13:29:58.082951 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:58.082912 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cea9b59f-qkx4q"] Nov 03 13:29:58.129254 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:58.129220 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cea9b59f-qkx4q"] Nov 03 13:29:58.168498 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:29:58.160739 5979 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-sub-cea9b59f-qkx4q\" is forbidden: User \"system:node:ip-10-0-130-198.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-130-198.us-west-2.compute.internal' and this object" podUID="881d9604-a7b9-4fdc-83e9-e66011e961f9" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cea9b59f-qkx4q" Nov 03 13:29:58.237172 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:58.237139 5979 patch_prober.go:29] interesting pod/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Nov 03 13:29:58.237356 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:58.237192 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf" podUID="fa17ed42-8832-4fea-b962-9029c6e6dbb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Nov 03 13:29:58.427596 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:58.427557 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a111273-pdcvs"] Nov 03 13:29:58.432418 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:58.432394 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a111273-pdcvs" Nov 03 13:29:58.483122 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:58.483086 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 13:29:58.483310 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:58.483196 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 13:29:58.486451 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:58.486421 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a111273-pdcvs"] Nov 03 13:29:58.521438 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:58.521406 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01e5557b-1b62-4d56-843e-1685de2da7f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a111273-pdcvs\" (UID: \"01e5557b-1b62-4d56-843e-1685de2da7f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a111273-pdcvs" Nov 03 13:29:58.521666 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:58.521464 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d4hg\" (UniqueName: \"kubernetes.io/projected/01e5557b-1b62-4d56-843e-1685de2da7f1-kube-api-access-9d4hg\") pod \"maas-api-revoke-sub-4a111273-pdcvs\" (UID: \"01e5557b-1b62-4d56-843e-1685de2da7f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a111273-pdcvs" Nov 03 13:29:58.521666 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:58.521608 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01e5557b-1b62-4d56-843e-1685de2da7f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a111273-pdcvs\" (UID: \"01e5557b-1b62-4d56-843e-1685de2da7f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a111273-pdcvs" Nov 03 13:29:58.527416 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:58.526964 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:29:58.527416 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:58.527095 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:29:58.622911 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:58.622868 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01e5557b-1b62-4d56-843e-1685de2da7f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a111273-pdcvs\" (UID: \"01e5557b-1b62-4d56-843e-1685de2da7f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a111273-pdcvs" Nov 03 13:29:58.623156 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:58.622960 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d4hg\" (UniqueName: \"kubernetes.io/projected/01e5557b-1b62-4d56-843e-1685de2da7f1-kube-api-access-9d4hg\") pod \"maas-api-revoke-sub-4a111273-pdcvs\" (UID: \"01e5557b-1b62-4d56-843e-1685de2da7f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a111273-pdcvs" Nov 03 13:29:58.623156 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:58.623013 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01e5557b-1b62-4d56-843e-1685de2da7f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a111273-pdcvs\" (UID: \"01e5557b-1b62-4d56-843e-1685de2da7f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a111273-pdcvs" Nov 03 13:29:58.623881 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:58.623854 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01e5557b-1b62-4d56-843e-1685de2da7f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a111273-pdcvs\" (UID: \"01e5557b-1b62-4d56-843e-1685de2da7f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a111273-pdcvs" Nov 03 13:29:58.625628 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:58.625600 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01e5557b-1b62-4d56-843e-1685de2da7f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a111273-pdcvs\" (UID: \"01e5557b-1b62-4d56-843e-1685de2da7f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a111273-pdcvs" Nov 03 13:29:58.677463 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:58.677426 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d4hg\" (UniqueName: \"kubernetes.io/projected/01e5557b-1b62-4d56-843e-1685de2da7f1-kube-api-access-9d4hg\") pod \"maas-api-revoke-sub-4a111273-pdcvs\" (UID: \"01e5557b-1b62-4d56-843e-1685de2da7f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a111273-pdcvs" Nov 03 13:29:58.771998 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:58.771898 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a111273-pdcvs" Nov 03 13:29:59.056841 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:59.056742 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17e79185-qsj8x"] Nov 03 13:29:59.060198 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:59.060167 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17e79185-qsj8x"] Nov 03 13:29:59.163640 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:59.163594 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a111273-pdcvs"] Nov 03 13:29:59.169166 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:29:59.169136 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01e5557b_1b62_4d56_843e_1685de2da7f1.slice/crio-ffcd3efdf7181589086794ff370928a352e35e6c4ce1f45d7f36ee9257f413df WatchSource:0}: Error finding container ffcd3efdf7181589086794ff370928a352e35e6c4ce1f45d7f36ee9257f413df: Status 404 returned error can't find the container with id ffcd3efdf7181589086794ff370928a352e35e6c4ce1f45d7f36ee9257f413df Nov 03 13:29:59.241436 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:59.241402 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf" Nov 03 13:29:59.241725 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:59.241709 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf" Nov 03 13:29:59.242435 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:59.242413 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf" Nov 03 13:29:59.551875 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:29:59.551842 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a111273-pdcvs" event={"ID":"01e5557b-1b62-4d56-843e-1685de2da7f1","Type":"ContainerStarted","Data":"ffcd3efdf7181589086794ff370928a352e35e6c4ce1f45d7f36ee9257f413df"} Nov 03 13:30:00.118659 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.118624 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e02c865-a839-42dd-b387-c5444bfe0a22" path="/var/lib/kubelet/pods/6e02c865-a839-42dd-b387-c5444bfe0a22/volumes" Nov 03 13:30:00.119205 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.119189 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="881d9604-a7b9-4fdc-83e9-e66011e961f9" path="/var/lib/kubelet/pods/881d9604-a7b9-4fdc-83e9-e66011e961f9/volumes" Nov 03 13:30:00.161273 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.161250 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895210-b26x7"] Nov 03 13:30:00.165139 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.165118 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895210-b26x7" Nov 03 13:30:00.165229 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.165207 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5295ab29-29895210-tt82x"] Nov 03 13:30:00.168669 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.168648 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5295ab29-29895210-tt82x" Nov 03 13:30:00.170709 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.170690 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895210-8rs5b"] Nov 03 13:30:00.174992 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.174972 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895210-bml7d"] Nov 03 13:30:00.175101 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.175080 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895210-8rs5b" Nov 03 13:30:00.178514 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.178465 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895210-bml7d" Nov 03 13:30:00.207920 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.207882 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-abe0e0-29895210-fb7wr"] Nov 03 13:30:00.211704 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.211677 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-abe0e0-29895210-fb7wr" Nov 03 13:30:00.212881 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.212857 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895210-hlf2l"] Nov 03 13:30:00.216672 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.216644 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895210-hlf2l" Nov 03 13:30:00.254805 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.254775 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e268eef-9d3a-4459-a456-381085fdfa63-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5bb463-29895210-8rs5b\" (UID: \"8e268eef-9d3a-4459-a456-381085fdfa63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895210-8rs5b" Nov 03 13:30:00.254947 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.254862 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dndql\" (UniqueName: \"kubernetes.io/projected/54703132-f756-4072-bb59-170c003ac16b-kube-api-access-dndql\") pod \"maas-api-key-cleanup-e2e-mt-5295ab29-29895210-tt82x\" (UID: \"54703132-f756-4072-bb59-170c003ac16b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5295ab29-29895210-tt82x" Nov 03 13:30:00.254947 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.254905 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvkmf\" (UniqueName: \"kubernetes.io/projected/2666652f-479a-44fe-8f0c-17bc354aaaec-kube-api-access-mvkmf\") pod \"maas-api-key-cleanup-e2e-worker-w1-34b13c-29895210-b26x7\" (UID: \"2666652f-479a-44fe-8f0c-17bc354aaaec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895210-b26x7" Nov 03 13:30:00.254947 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.254937 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ab44829-1b15-408c-b6fd-647ba93cac2e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-abe0e0-29895210-fb7wr\" (UID: \"9ab44829-1b15-408c-b6fd-647ba93cac2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-abe0e0-29895210-fb7wr" Nov 03 13:30:00.255076 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.255042 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c1ef57f-1651-4415-a8b1-1f1dff93e5d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-cdb532-29895210-bml7d\" (UID: \"0c1ef57f-1651-4415-a8b1-1f1dff93e5d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895210-bml7d" Nov 03 13:30:00.255152 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.255140 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgk2h\" (UniqueName: \"kubernetes.io/projected/088feb81-52ce-4afc-8d07-aa9d2a094442-kube-api-access-fgk2h\") pod \"maas-api-key-cleanup-29895210-hlf2l\" (UID: \"088feb81-52ce-4afc-8d07-aa9d2a094442\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895210-hlf2l" Nov 03 13:30:00.255201 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.255189 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2666652f-479a-44fe-8f0c-17bc354aaaec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-34b13c-29895210-b26x7\" (UID: \"2666652f-479a-44fe-8f0c-17bc354aaaec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895210-b26x7" Nov 03 13:30:00.255253 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.255231 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54703132-f756-4072-bb59-170c003ac16b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-5295ab29-29895210-tt82x\" (UID: \"54703132-f756-4072-bb59-170c003ac16b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5295ab29-29895210-tt82x" Nov 03 13:30:00.255339 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.255320 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbfdw\" (UniqueName: \"kubernetes.io/projected/9ab44829-1b15-408c-b6fd-647ba93cac2e-kube-api-access-bbfdw\") pod \"maas-api-key-cleanup-e2e-api-b-abe0e0-29895210-fb7wr\" (UID: \"9ab44829-1b15-408c-b6fd-647ba93cac2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-abe0e0-29895210-fb7wr" Nov 03 13:30:00.255396 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.255373 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/088feb81-52ce-4afc-8d07-aa9d2a094442-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895210-hlf2l\" (UID: \"088feb81-52ce-4afc-8d07-aa9d2a094442\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895210-hlf2l" Nov 03 13:30:00.255524 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.255509 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw8jg\" (UniqueName: \"kubernetes.io/projected/8e268eef-9d3a-4459-a456-381085fdfa63-kube-api-access-nw8jg\") pod \"maas-api-key-cleanup-e2e-worker-w3-5bb463-29895210-8rs5b\" (UID: \"8e268eef-9d3a-4459-a456-381085fdfa63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895210-8rs5b" Nov 03 13:30:00.255600 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.255538 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxvz5\" (UniqueName: \"kubernetes.io/projected/0c1ef57f-1651-4415-a8b1-1f1dff93e5d4-kube-api-access-fxvz5\") pod \"maas-api-key-cleanup-e2e-worker-w2-cdb532-29895210-bml7d\" (UID: \"0c1ef57f-1651-4415-a8b1-1f1dff93e5d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895210-bml7d" Nov 03 13:30:00.262550 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.262513 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895210-b26x7"] Nov 03 13:30:00.266852 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.266828 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895210-8rs5b"] Nov 03 13:30:00.273713 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.272272 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5295ab29-29895210-tt82x"] Nov 03 13:30:00.321131 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.320201 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-abe0e0-29895210-fb7wr"] Nov 03 13:30:00.327096 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.327063 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895210-bml7d"] Nov 03 13:30:00.357719 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.357684 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw8jg\" (UniqueName: \"kubernetes.io/projected/8e268eef-9d3a-4459-a456-381085fdfa63-kube-api-access-nw8jg\") pod \"maas-api-key-cleanup-e2e-worker-w3-5bb463-29895210-8rs5b\" (UID: \"8e268eef-9d3a-4459-a456-381085fdfa63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895210-8rs5b" Nov 03 13:30:00.357901 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.357732 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxvz5\" (UniqueName: \"kubernetes.io/projected/0c1ef57f-1651-4415-a8b1-1f1dff93e5d4-kube-api-access-fxvz5\") pod \"maas-api-key-cleanup-e2e-worker-w2-cdb532-29895210-bml7d\" (UID: \"0c1ef57f-1651-4415-a8b1-1f1dff93e5d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895210-bml7d" Nov 03 13:30:00.357901 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.357787 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e268eef-9d3a-4459-a456-381085fdfa63-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5bb463-29895210-8rs5b\" (UID: \"8e268eef-9d3a-4459-a456-381085fdfa63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895210-8rs5b" Nov 03 13:30:00.357901 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.357864 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dndql\" (UniqueName: \"kubernetes.io/projected/54703132-f756-4072-bb59-170c003ac16b-kube-api-access-dndql\") pod \"maas-api-key-cleanup-e2e-mt-5295ab29-29895210-tt82x\" (UID: \"54703132-f756-4072-bb59-170c003ac16b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5295ab29-29895210-tt82x" Nov 03 13:30:00.358087 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.357908 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvkmf\" (UniqueName: \"kubernetes.io/projected/2666652f-479a-44fe-8f0c-17bc354aaaec-kube-api-access-mvkmf\") pod \"maas-api-key-cleanup-e2e-worker-w1-34b13c-29895210-b26x7\" (UID: \"2666652f-479a-44fe-8f0c-17bc354aaaec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895210-b26x7" Nov 03 13:30:00.358087 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.357939 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ab44829-1b15-408c-b6fd-647ba93cac2e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-abe0e0-29895210-fb7wr\" (UID: \"9ab44829-1b15-408c-b6fd-647ba93cac2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-abe0e0-29895210-fb7wr" Nov 03 13:30:00.358087 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.358006 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c1ef57f-1651-4415-a8b1-1f1dff93e5d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-cdb532-29895210-bml7d\" (UID: \"0c1ef57f-1651-4415-a8b1-1f1dff93e5d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895210-bml7d" Nov 03 13:30:00.358247 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.358114 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgk2h\" (UniqueName: \"kubernetes.io/projected/088feb81-52ce-4afc-8d07-aa9d2a094442-kube-api-access-fgk2h\") pod \"maas-api-key-cleanup-29895210-hlf2l\" (UID: \"088feb81-52ce-4afc-8d07-aa9d2a094442\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895210-hlf2l" Nov 03 13:30:00.359281 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.359251 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ab44829-1b15-408c-b6fd-647ba93cac2e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-abe0e0-29895210-fb7wr\" (UID: \"9ab44829-1b15-408c-b6fd-647ba93cac2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-abe0e0-29895210-fb7wr" Nov 03 13:30:00.359437 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.359335 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e268eef-9d3a-4459-a456-381085fdfa63-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5bb463-29895210-8rs5b\" (UID: \"8e268eef-9d3a-4459-a456-381085fdfa63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895210-8rs5b" Nov 03 13:30:00.359437 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.359386 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2666652f-479a-44fe-8f0c-17bc354aaaec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-34b13c-29895210-b26x7\" (UID: \"2666652f-479a-44fe-8f0c-17bc354aaaec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895210-b26x7" Nov 03 13:30:00.359437 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.359421 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54703132-f756-4072-bb59-170c003ac16b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-5295ab29-29895210-tt82x\" (UID: \"54703132-f756-4072-bb59-170c003ac16b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5295ab29-29895210-tt82x" Nov 03 13:30:00.359688 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.359526 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbfdw\" (UniqueName: \"kubernetes.io/projected/9ab44829-1b15-408c-b6fd-647ba93cac2e-kube-api-access-bbfdw\") pod \"maas-api-key-cleanup-e2e-api-b-abe0e0-29895210-fb7wr\" (UID: \"9ab44829-1b15-408c-b6fd-647ba93cac2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-abe0e0-29895210-fb7wr" Nov 03 13:30:00.359688 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.359616 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/088feb81-52ce-4afc-8d07-aa9d2a094442-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895210-hlf2l\" (UID: \"088feb81-52ce-4afc-8d07-aa9d2a094442\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895210-hlf2l" Nov 03 13:30:00.360149 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.360126 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54703132-f756-4072-bb59-170c003ac16b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-5295ab29-29895210-tt82x\" (UID: \"54703132-f756-4072-bb59-170c003ac16b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5295ab29-29895210-tt82x" Nov 03 13:30:00.360439 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.360413 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/088feb81-52ce-4afc-8d07-aa9d2a094442-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895210-hlf2l\" (UID: \"088feb81-52ce-4afc-8d07-aa9d2a094442\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895210-hlf2l" Nov 03 13:30:00.360811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.360790 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2666652f-479a-44fe-8f0c-17bc354aaaec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-34b13c-29895210-b26x7\" (UID: \"2666652f-479a-44fe-8f0c-17bc354aaaec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895210-b26x7" Nov 03 13:30:00.361272 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.361250 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c1ef57f-1651-4415-a8b1-1f1dff93e5d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-cdb532-29895210-bml7d\" (UID: \"0c1ef57f-1651-4415-a8b1-1f1dff93e5d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895210-bml7d" Nov 03 13:30:00.362295 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.362263 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895210-hlf2l"] Nov 03 13:30:00.418160 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.417939 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw8jg\" (UniqueName: \"kubernetes.io/projected/8e268eef-9d3a-4459-a456-381085fdfa63-kube-api-access-nw8jg\") pod \"maas-api-key-cleanup-e2e-worker-w3-5bb463-29895210-8rs5b\" (UID: \"8e268eef-9d3a-4459-a456-381085fdfa63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895210-8rs5b" Nov 03 13:30:00.421458 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.421070 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgk2h\" (UniqueName: \"kubernetes.io/projected/088feb81-52ce-4afc-8d07-aa9d2a094442-kube-api-access-fgk2h\") pod \"maas-api-key-cleanup-29895210-hlf2l\" (UID: \"088feb81-52ce-4afc-8d07-aa9d2a094442\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895210-hlf2l" Nov 03 13:30:00.421458 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.421413 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dndql\" (UniqueName: \"kubernetes.io/projected/54703132-f756-4072-bb59-170c003ac16b-kube-api-access-dndql\") pod \"maas-api-key-cleanup-e2e-mt-5295ab29-29895210-tt82x\" (UID: \"54703132-f756-4072-bb59-170c003ac16b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5295ab29-29895210-tt82x" Nov 03 13:30:00.423137 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.423108 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxvz5\" (UniqueName: \"kubernetes.io/projected/0c1ef57f-1651-4415-a8b1-1f1dff93e5d4-kube-api-access-fxvz5\") pod \"maas-api-key-cleanup-e2e-worker-w2-cdb532-29895210-bml7d\" (UID: \"0c1ef57f-1651-4415-a8b1-1f1dff93e5d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895210-bml7d" Nov 03 13:30:00.429619 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.429590 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvkmf\" (UniqueName: \"kubernetes.io/projected/2666652f-479a-44fe-8f0c-17bc354aaaec-kube-api-access-mvkmf\") pod \"maas-api-key-cleanup-e2e-worker-w1-34b13c-29895210-b26x7\" (UID: \"2666652f-479a-44fe-8f0c-17bc354aaaec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895210-b26x7" Nov 03 13:30:00.430365 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.430345 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbfdw\" (UniqueName: \"kubernetes.io/projected/9ab44829-1b15-408c-b6fd-647ba93cac2e-kube-api-access-bbfdw\") pod \"maas-api-key-cleanup-e2e-api-b-abe0e0-29895210-fb7wr\" (UID: \"9ab44829-1b15-408c-b6fd-647ba93cac2e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-abe0e0-29895210-fb7wr" Nov 03 13:30:00.549991 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.549960 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895210-b26x7" Nov 03 13:30:00.556793 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.556763 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5295ab29-29895210-tt82x" Nov 03 13:30:00.560648 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.560621 5979 generic.go:356] "Generic (PLEG): container finished" podID="01e5557b-1b62-4d56-843e-1685de2da7f1" containerID="acf9669ed1f269729e033615a7ab21135c6edf455ab6cabf9044b1604440175f" exitCode=0 Nov 03 13:30:00.560760 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.560710 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a111273-pdcvs" event={"ID":"01e5557b-1b62-4d56-843e-1685de2da7f1","Type":"ContainerDied","Data":"acf9669ed1f269729e033615a7ab21135c6edf455ab6cabf9044b1604440175f"} Nov 03 13:30:00.568442 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.568421 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895210-8rs5b" Nov 03 13:30:00.580084 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.580047 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895210-bml7d" Nov 03 13:30:00.600436 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.597245 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-abe0e0-29895210-fb7wr" Nov 03 13:30:00.602908 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.602293 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895210-hlf2l" Nov 03 13:30:00.949829 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.949801 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895210-b26x7"] Nov 03 13:30:00.953940 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:30:00.953907 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2666652f_479a_44fe_8f0c_17bc354aaaec.slice/crio-4eb5f6fe3a61f86a3b52952281ff558df2f2ce2256acab76335639f6a8db5214 WatchSource:0}: Error finding container 4eb5f6fe3a61f86a3b52952281ff558df2f2ce2256acab76335639f6a8db5214: Status 404 returned error can't find the container with id 4eb5f6fe3a61f86a3b52952281ff558df2f2ce2256acab76335639f6a8db5214 Nov 03 13:30:00.998402 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:00.998378 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5295ab29-29895210-tt82x"] Nov 03 13:30:01.001489 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:30:01.001452 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54703132_f756_4072_bb59_170c003ac16b.slice/crio-4562538bb8b605242579e84756212e0f6959497e0592cd03696522bed73cd633 WatchSource:0}: Error finding container 4562538bb8b605242579e84756212e0f6959497e0592cd03696522bed73cd633: Status 404 returned error can't find the container with id 4562538bb8b605242579e84756212e0f6959497e0592cd03696522bed73cd633 Nov 03 13:30:01.033458 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:01.033425 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895210-8rs5b"] Nov 03 13:30:01.039169 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:30:01.039144 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e268eef_9d3a_4459_a456_381085fdfa63.slice/crio-42b3a657c30bcfd592a508cc0115e90f5b33cdb8dc2b5da471be78d1a386dcf5 WatchSource:0}: Error finding container 42b3a657c30bcfd592a508cc0115e90f5b33cdb8dc2b5da471be78d1a386dcf5: Status 404 returned error can't find the container with id 42b3a657c30bcfd592a508cc0115e90f5b33cdb8dc2b5da471be78d1a386dcf5 Nov 03 13:30:01.102036 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:01.102002 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895210-bml7d"] Nov 03 13:30:01.106990 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:30:01.106961 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c1ef57f_1651_4415_a8b1_1f1dff93e5d4.slice/crio-c4bdc2ec9338b264a0df40d010cf66106da4e31ab6c075d0111022268a40cbb1 WatchSource:0}: Error finding container c4bdc2ec9338b264a0df40d010cf66106da4e31ab6c075d0111022268a40cbb1: Status 404 returned error can't find the container with id c4bdc2ec9338b264a0df40d010cf66106da4e31ab6c075d0111022268a40cbb1 Nov 03 13:30:01.114200 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:01.114173 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-abe0e0-29895210-fb7wr"] Nov 03 13:30:01.119257 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:01.119235 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895210-hlf2l"] Nov 03 13:30:01.122504 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:30:01.122479 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ab44829_1b15_408c_b6fd_647ba93cac2e.slice/crio-4eda872172effc1d816c4e373b7005c43e2fe28a5b61874cac26f6674258ff8d WatchSource:0}: Error finding container 4eda872172effc1d816c4e373b7005c43e2fe28a5b61874cac26f6674258ff8d: Status 404 returned error can't find the container with id 4eda872172effc1d816c4e373b7005c43e2fe28a5b61874cac26f6674258ff8d Nov 03 13:30:01.123839 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:30:01.123818 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod088feb81_52ce_4afc_8d07_aa9d2a094442.slice/crio-1fdbeeefe1f7de0094972810db5863c4f300feee67efd2ab5cc7d377c6dd5cde WatchSource:0}: Error finding container 1fdbeeefe1f7de0094972810db5863c4f300feee67efd2ab5cc7d377c6dd5cde: Status 404 returned error can't find the container with id 1fdbeeefe1f7de0094972810db5863c4f300feee67efd2ab5cc7d377c6dd5cde Nov 03 13:30:01.230799 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:01.230713 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-461b2d-7d984ffbf9-262hb" Nov 03 13:30:01.230799 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:01.230780 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-461b2d-6466bf6b8f-48fhn" Nov 03 13:30:01.570051 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:01.570009 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895210-bml7d" event={"ID":"0c1ef57f-1651-4415-a8b1-1f1dff93e5d4","Type":"ContainerStarted","Data":"6b9b6b25a39b58368b1446e8c7263bd5b7649722b6afa70c0b3d3f345d815bdf"} Nov 03 13:30:01.570550 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:01.570088 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895210-bml7d" event={"ID":"0c1ef57f-1651-4415-a8b1-1f1dff93e5d4","Type":"ContainerStarted","Data":"c4bdc2ec9338b264a0df40d010cf66106da4e31ab6c075d0111022268a40cbb1"} Nov 03 13:30:01.571808 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:01.571783 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895210-8rs5b" event={"ID":"8e268eef-9d3a-4459-a456-381085fdfa63","Type":"ContainerStarted","Data":"33e899665c6f08e3c0ea309f6c831fd6d8b18d89a31e61d28d63703b91d83e3a"} Nov 03 13:30:01.571901 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:01.571823 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895210-8rs5b" event={"ID":"8e268eef-9d3a-4459-a456-381085fdfa63","Type":"ContainerStarted","Data":"42b3a657c30bcfd592a508cc0115e90f5b33cdb8dc2b5da471be78d1a386dcf5"} Nov 03 13:30:01.573160 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:01.573138 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-abe0e0-29895210-fb7wr" event={"ID":"9ab44829-1b15-408c-b6fd-647ba93cac2e","Type":"ContainerStarted","Data":"4eda872172effc1d816c4e373b7005c43e2fe28a5b61874cac26f6674258ff8d"} Nov 03 13:30:01.574733 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:01.574708 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5295ab29-29895210-tt82x" event={"ID":"54703132-f756-4072-bb59-170c003ac16b","Type":"ContainerStarted","Data":"faf1f0d04e0c3aaa15fde474c7598f659d314f5ca1eca0cc998adb9fc954b14a"} Nov 03 13:30:01.574888 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:01.574753 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5295ab29-29895210-tt82x" event={"ID":"54703132-f756-4072-bb59-170c003ac16b","Type":"ContainerStarted","Data":"4562538bb8b605242579e84756212e0f6959497e0592cd03696522bed73cd633"} Nov 03 13:30:01.575979 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:01.575954 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895210-hlf2l" event={"ID":"088feb81-52ce-4afc-8d07-aa9d2a094442","Type":"ContainerStarted","Data":"1fdbeeefe1f7de0094972810db5863c4f300feee67efd2ab5cc7d377c6dd5cde"} Nov 03 13:30:01.577559 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:01.577537 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895210-b26x7" event={"ID":"2666652f-479a-44fe-8f0c-17bc354aaaec","Type":"ContainerStarted","Data":"0fe8e1316f415cf8f5701d7986af6b15a798c13e7591b1b27dcd834e2b8703dd"} Nov 03 13:30:01.577649 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:01.577577 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895210-b26x7" event={"ID":"2666652f-479a-44fe-8f0c-17bc354aaaec","Type":"ContainerStarted","Data":"4eb5f6fe3a61f86a3b52952281ff558df2f2ce2256acab76335639f6a8db5214"} Nov 03 13:30:01.684886 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:01.684827 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895210-bml7d" podStartSLOduration=1.684780602 podStartE2EDuration="1.684780602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:30:01.680544802 +0000 UTC m=+1483.691233070" watchObservedRunningTime="2026-11-03 13:30:01.684780602 +0000 UTC m=+1483.695468845" Nov 03 13:30:01.756804 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:01.756778 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a111273-pdcvs" Nov 03 13:30:01.795418 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:01.795356 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895210-b26x7" podStartSLOduration=1.795343594 podStartE2EDuration="1.795343594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:30:01.79189817 +0000 UTC m=+1483.802586389" watchObservedRunningTime="2026-11-03 13:30:01.795343594 +0000 UTC m=+1483.806031836" Nov 03 13:30:01.802663 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:01.802572 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01e5557b-1b62-4d56-843e-1685de2da7f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01e5557b-1b62-4d56-843e-1685de2da7f1-maas-api-xks-ca\") pod \"01e5557b-1b62-4d56-843e-1685de2da7f1\" (UID: \"01e5557b-1b62-4d56-843e-1685de2da7f1\") " Nov 03 13:30:01.802823 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:01.802700 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01e5557b-1b62-4d56-843e-1685de2da7f1-kube-api-access-9d4hg\" (UniqueName: \"kubernetes.io/projected/01e5557b-1b62-4d56-843e-1685de2da7f1-kube-api-access-9d4hg\") pod \"01e5557b-1b62-4d56-843e-1685de2da7f1\" (UID: \"01e5557b-1b62-4d56-843e-1685de2da7f1\") " Nov 03 13:30:01.802930 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:01.802913 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01e5557b-1b62-4d56-843e-1685de2da7f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01e5557b-1b62-4d56-843e-1685de2da7f1-maas-api-openshift-ca\") pod \"01e5557b-1b62-4d56-843e-1685de2da7f1\" (UID: \"01e5557b-1b62-4d56-843e-1685de2da7f1\") " Nov 03 13:30:01.803788 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:01.803762 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01e5557b-1b62-4d56-843e-1685de2da7f1-maas-api-openshift-ca" pod "01e5557b-1b62-4d56-843e-1685de2da7f1" (UID: "01e5557b-1b62-4d56-843e-1685de2da7f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:30:01.805853 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:01.805827 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01e5557b-1b62-4d56-843e-1685de2da7f1-maas-api-xks-ca" pod "01e5557b-1b62-4d56-843e-1685de2da7f1" (UID: "01e5557b-1b62-4d56-843e-1685de2da7f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:30:01.805951 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:01.805919 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01e5557b-1b62-4d56-843e-1685de2da7f1-kube-api-access-9d4hg" pod "01e5557b-1b62-4d56-843e-1685de2da7f1" (UID: "01e5557b-1b62-4d56-843e-1685de2da7f1"). InnerVolumeSpecName "kube-api-access-9d4hg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:30:01.899082 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:01.898974 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895210-8rs5b" podStartSLOduration=1.89895493 podStartE2EDuration="1.89895493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:30:01.898404847 +0000 UTC m=+1483.909093088" watchObservedRunningTime="2026-11-03 13:30:01.89895493 +0000 UTC m=+1483.909643173" Nov 03 13:30:01.904770 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:01.904742 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01e5557b-1b62-4d56-843e-1685de2da7f1-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:01.904770 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:01.904769 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01e5557b-1b62-4d56-843e-1685de2da7f1-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:01.904957 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:01.904779 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d4hg\" (UniqueName: \"kubernetes.io/projected/01e5557b-1b62-4d56-843e-1685de2da7f1-kube-api-access-9d4hg\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:02.009424 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:02.009369 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5295ab29-29895210-tt82x" podStartSLOduration=2.00935592 podStartE2EDuration="2.00935592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:30:02.006152041 +0000 UTC m=+1484.016840284" watchObservedRunningTime="2026-11-03 13:30:02.00935592 +0000 UTC m=+1484.020044189" Nov 03 13:30:02.586602 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:02.586514 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a111273-pdcvs" Nov 03 13:30:02.587019 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:02.586519 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a111273-pdcvs" event={"ID":"01e5557b-1b62-4d56-843e-1685de2da7f1","Type":"ContainerDied","Data":"ffcd3efdf7181589086794ff370928a352e35e6c4ce1f45d7f36ee9257f413df"} Nov 03 13:30:02.587019 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:02.586664 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffcd3efdf7181589086794ff370928a352e35e6c4ce1f45d7f36ee9257f413df" Nov 03 13:30:02.588305 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:02.588278 5979 generic.go:356] "Generic (PLEG): container finished" podID="8e268eef-9d3a-4459-a456-381085fdfa63" containerID="33e899665c6f08e3c0ea309f6c831fd6d8b18d89a31e61d28d63703b91d83e3a" exitCode=60 Nov 03 13:30:02.588450 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:02.588354 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895210-8rs5b" event={"ID":"8e268eef-9d3a-4459-a456-381085fdfa63","Type":"ContainerDied","Data":"33e899665c6f08e3c0ea309f6c831fd6d8b18d89a31e61d28d63703b91d83e3a"} Nov 03 13:30:02.588714 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:02.588696 5979 scope.go:122] "RemoveContainer" containerID="33e899665c6f08e3c0ea309f6c831fd6d8b18d89a31e61d28d63703b91d83e3a" Nov 03 13:30:02.590632 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:02.590611 5979 generic.go:356] "Generic (PLEG): container finished" podID="9ab44829-1b15-408c-b6fd-647ba93cac2e" containerID="7d23a6bb71938eac5aac4357309f4fad2515ee2824dc457e103c29baef1d8ff6" exitCode=60 Nov 03 13:30:02.590729 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:02.590663 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-abe0e0-29895210-fb7wr" event={"ID":"9ab44829-1b15-408c-b6fd-647ba93cac2e","Type":"ContainerDied","Data":"7d23a6bb71938eac5aac4357309f4fad2515ee2824dc457e103c29baef1d8ff6"} Nov 03 13:30:02.590920 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:02.590906 5979 scope.go:122] "RemoveContainer" containerID="7d23a6bb71938eac5aac4357309f4fad2515ee2824dc457e103c29baef1d8ff6" Nov 03 13:30:02.592358 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:02.592333 5979 generic.go:356] "Generic (PLEG): container finished" podID="54703132-f756-4072-bb59-170c003ac16b" containerID="faf1f0d04e0c3aaa15fde474c7598f659d314f5ca1eca0cc998adb9fc954b14a" exitCode=60 Nov 03 13:30:02.592459 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:02.592376 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5295ab29-29895210-tt82x" event={"ID":"54703132-f756-4072-bb59-170c003ac16b","Type":"ContainerDied","Data":"faf1f0d04e0c3aaa15fde474c7598f659d314f5ca1eca0cc998adb9fc954b14a"} Nov 03 13:30:02.592654 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:02.592637 5979 scope.go:122] "RemoveContainer" containerID="faf1f0d04e0c3aaa15fde474c7598f659d314f5ca1eca0cc998adb9fc954b14a" Nov 03 13:30:02.594103 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:02.594084 5979 generic.go:356] "Generic (PLEG): container finished" podID="088feb81-52ce-4afc-8d07-aa9d2a094442" containerID="2ee37b110efac45e6eed23de15048982d1ce8b9180ad7d44e78b0d44a6593ded" exitCode=60 Nov 03 13:30:02.594180 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:02.594161 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895210-hlf2l" event={"ID":"088feb81-52ce-4afc-8d07-aa9d2a094442","Type":"ContainerDied","Data":"2ee37b110efac45e6eed23de15048982d1ce8b9180ad7d44e78b0d44a6593ded"} Nov 03 13:30:02.594361 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:02.594345 5979 scope.go:122] "RemoveContainer" containerID="2ee37b110efac45e6eed23de15048982d1ce8b9180ad7d44e78b0d44a6593ded" Nov 03 13:30:02.595817 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:02.595796 5979 generic.go:356] "Generic (PLEG): container finished" podID="2666652f-479a-44fe-8f0c-17bc354aaaec" containerID="0fe8e1316f415cf8f5701d7986af6b15a798c13e7591b1b27dcd834e2b8703dd" exitCode=60 Nov 03 13:30:02.595927 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:02.595872 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895210-b26x7" event={"ID":"2666652f-479a-44fe-8f0c-17bc354aaaec","Type":"ContainerDied","Data":"0fe8e1316f415cf8f5701d7986af6b15a798c13e7591b1b27dcd834e2b8703dd"} Nov 03 13:30:02.596047 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:02.596033 5979 scope.go:122] "RemoveContainer" containerID="0fe8e1316f415cf8f5701d7986af6b15a798c13e7591b1b27dcd834e2b8703dd" Nov 03 13:30:02.597451 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:02.597432 5979 generic.go:356] "Generic (PLEG): container finished" podID="0c1ef57f-1651-4415-a8b1-1f1dff93e5d4" containerID="6b9b6b25a39b58368b1446e8c7263bd5b7649722b6afa70c0b3d3f345d815bdf" exitCode=60 Nov 03 13:30:02.597563 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:02.597506 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895210-bml7d" event={"ID":"0c1ef57f-1651-4415-a8b1-1f1dff93e5d4","Type":"ContainerDied","Data":"6b9b6b25a39b58368b1446e8c7263bd5b7649722b6afa70c0b3d3f345d815bdf"} Nov 03 13:30:02.597752 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:02.597737 5979 scope.go:122] "RemoveContainer" containerID="6b9b6b25a39b58368b1446e8c7263bd5b7649722b6afa70c0b3d3f345d815bdf" Nov 03 13:30:03.606856 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:03.606752 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-abe0e0-29895210-fb7wr" event={"ID":"9ab44829-1b15-408c-b6fd-647ba93cac2e","Type":"ContainerStarted","Data":"6e46e3b73c4c6067c5562cc9fee86a5ce936a0859aef1ce920e4d3524b705fea"} Nov 03 13:30:03.608706 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:03.608672 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5295ab29-29895210-tt82x" event={"ID":"54703132-f756-4072-bb59-170c003ac16b","Type":"ContainerStarted","Data":"85e8f21ec295232583d02bcd1b0b18322ac641a174acfadc204685e4e585b62e"} Nov 03 13:30:03.610481 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:03.610446 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895210-hlf2l" event={"ID":"088feb81-52ce-4afc-8d07-aa9d2a094442","Type":"ContainerStarted","Data":"45efc88d6ce4707b3f2034640312972728f9f6c22a053796dab238de26e6e9d2"} Nov 03 13:30:03.612302 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:03.612281 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895210-b26x7" event={"ID":"2666652f-479a-44fe-8f0c-17bc354aaaec","Type":"ContainerStarted","Data":"dc3fbda233f701bf52de08812affdcd4363b68f13e867a56c7ca329186d513c0"} Nov 03 13:30:03.614187 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:03.614167 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895210-bml7d" event={"ID":"0c1ef57f-1651-4415-a8b1-1f1dff93e5d4","Type":"ContainerStarted","Data":"8a4920a54ab0e42557366c2f108a760e668cd321ec21b5db3adad55bfe8702f9"} Nov 03 13:30:03.616196 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:03.616176 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895210-8rs5b" event={"ID":"8e268eef-9d3a-4459-a456-381085fdfa63","Type":"ContainerStarted","Data":"5b92af8f4bb41c2ed146ecf09afc4e98ae48f9253e3873c3e683a454e79bdc89"} Nov 03 13:30:03.718458 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:03.718398 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-abe0e0-29895210-fb7wr" podStartSLOduration=3.718384662 podStartE2EDuration="3.718384662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:30:03.714386684 +0000 UTC m=+1485.725074928" watchObservedRunningTime="2026-11-03 13:30:03.718384662 +0000 UTC m=+1485.729072904" Nov 03 13:30:04.259301 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:04.259247 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895210-hlf2l" podStartSLOduration=4.25923213 podStartE2EDuration="4.25923213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:30:04.258005336 +0000 UTC m=+1486.268693580" watchObservedRunningTime="2026-11-03 13:30:04.25923213 +0000 UTC m=+1486.269920371" Nov 03 13:30:04.661573 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:04.661537 5979 generic.go:356] "Generic (PLEG): container finished" podID="8e268eef-9d3a-4459-a456-381085fdfa63" containerID="5b92af8f4bb41c2ed146ecf09afc4e98ae48f9253e3873c3e683a454e79bdc89" exitCode=60 Nov 03 13:30:04.662181 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:04.662158 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895210-8rs5b" event={"ID":"8e268eef-9d3a-4459-a456-381085fdfa63","Type":"ContainerDied","Data":"5b92af8f4bb41c2ed146ecf09afc4e98ae48f9253e3873c3e683a454e79bdc89"} Nov 03 13:30:04.662342 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:04.662329 5979 scope.go:122] "RemoveContainer" containerID="33e899665c6f08e3c0ea309f6c831fd6d8b18d89a31e61d28d63703b91d83e3a" Nov 03 13:30:04.663250 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:04.662966 5979 scope.go:122] "RemoveContainer" containerID="5b92af8f4bb41c2ed146ecf09afc4e98ae48f9253e3873c3e683a454e79bdc89" Nov 03 13:30:04.664372 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:30:04.664346 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5bb463-29895210-8rs5b_odh-ai-gateway-infra(8e268eef-9d3a-4459-a456-381085fdfa63)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895210-8rs5b" podUID="8e268eef-9d3a-4459-a456-381085fdfa63" Nov 03 13:30:04.671890 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:04.671856 5979 generic.go:356] "Generic (PLEG): container finished" podID="9ab44829-1b15-408c-b6fd-647ba93cac2e" containerID="6e46e3b73c4c6067c5562cc9fee86a5ce936a0859aef1ce920e4d3524b705fea" exitCode=60 Nov 03 13:30:04.672175 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:04.672151 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-abe0e0-29895210-fb7wr" event={"ID":"9ab44829-1b15-408c-b6fd-647ba93cac2e","Type":"ContainerDied","Data":"6e46e3b73c4c6067c5562cc9fee86a5ce936a0859aef1ce920e4d3524b705fea"} Nov 03 13:30:04.672904 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:04.672884 5979 scope.go:122] "RemoveContainer" containerID="6e46e3b73c4c6067c5562cc9fee86a5ce936a0859aef1ce920e4d3524b705fea" Nov 03 13:30:04.673354 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:30:04.673331 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-abe0e0-29895210-fb7wr_odh-ai-gateway-infra(9ab44829-1b15-408c-b6fd-647ba93cac2e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-abe0e0-29895210-fb7wr" podUID="9ab44829-1b15-408c-b6fd-647ba93cac2e" Nov 03 13:30:04.677815 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:04.677791 5979 generic.go:356] "Generic (PLEG): container finished" podID="54703132-f756-4072-bb59-170c003ac16b" containerID="85e8f21ec295232583d02bcd1b0b18322ac641a174acfadc204685e4e585b62e" exitCode=60 Nov 03 13:30:04.678215 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:04.678200 5979 scope.go:122] "RemoveContainer" containerID="85e8f21ec295232583d02bcd1b0b18322ac641a174acfadc204685e4e585b62e" Nov 03 13:30:04.678312 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:04.678293 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5295ab29-29895210-tt82x" event={"ID":"54703132-f756-4072-bb59-170c003ac16b","Type":"ContainerDied","Data":"85e8f21ec295232583d02bcd1b0b18322ac641a174acfadc204685e4e585b62e"} Nov 03 13:30:04.678496 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:30:04.678464 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-5295ab29-29895210-tt82x_odh-ai-gateway-infra(54703132-f756-4072-bb59-170c003ac16b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5295ab29-29895210-tt82x" podUID="54703132-f756-4072-bb59-170c003ac16b" Nov 03 13:30:04.680412 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:04.680365 5979 generic.go:356] "Generic (PLEG): container finished" podID="088feb81-52ce-4afc-8d07-aa9d2a094442" containerID="45efc88d6ce4707b3f2034640312972728f9f6c22a053796dab238de26e6e9d2" exitCode=60 Nov 03 13:30:04.680535 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:04.680511 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895210-hlf2l" event={"ID":"088feb81-52ce-4afc-8d07-aa9d2a094442","Type":"ContainerDied","Data":"45efc88d6ce4707b3f2034640312972728f9f6c22a053796dab238de26e6e9d2"} Nov 03 13:30:04.680737 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:04.680717 5979 scope.go:122] "RemoveContainer" containerID="45efc88d6ce4707b3f2034640312972728f9f6c22a053796dab238de26e6e9d2" Nov 03 13:30:04.681053 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:30:04.681029 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895210-hlf2l_odh-ai-gateway-infra(088feb81-52ce-4afc-8d07-aa9d2a094442)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895210-hlf2l" podUID="088feb81-52ce-4afc-8d07-aa9d2a094442" Nov 03 13:30:04.682857 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:04.682838 5979 generic.go:356] "Generic (PLEG): container finished" podID="2666652f-479a-44fe-8f0c-17bc354aaaec" containerID="dc3fbda233f701bf52de08812affdcd4363b68f13e867a56c7ca329186d513c0" exitCode=60 Nov 03 13:30:04.682962 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:04.682879 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895210-b26x7" event={"ID":"2666652f-479a-44fe-8f0c-17bc354aaaec","Type":"ContainerDied","Data":"dc3fbda233f701bf52de08812affdcd4363b68f13e867a56c7ca329186d513c0"} Nov 03 13:30:04.683207 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:04.683186 5979 scope.go:122] "RemoveContainer" containerID="dc3fbda233f701bf52de08812affdcd4363b68f13e867a56c7ca329186d513c0" Nov 03 13:30:04.683597 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:30:04.683570 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-34b13c-29895210-b26x7_odh-ai-gateway-infra(2666652f-479a-44fe-8f0c-17bc354aaaec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895210-b26x7" podUID="2666652f-479a-44fe-8f0c-17bc354aaaec" Nov 03 13:30:04.684755 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:04.684738 5979 scope.go:122] "RemoveContainer" containerID="7d23a6bb71938eac5aac4357309f4fad2515ee2824dc457e103c29baef1d8ff6" Nov 03 13:30:04.684977 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:04.684957 5979 generic.go:356] "Generic (PLEG): container finished" podID="0c1ef57f-1651-4415-a8b1-1f1dff93e5d4" containerID="8a4920a54ab0e42557366c2f108a760e668cd321ec21b5db3adad55bfe8702f9" exitCode=60 Nov 03 13:30:04.685053 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:04.685000 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895210-bml7d" event={"ID":"0c1ef57f-1651-4415-a8b1-1f1dff93e5d4","Type":"ContainerDied","Data":"8a4920a54ab0e42557366c2f108a760e668cd321ec21b5db3adad55bfe8702f9"} Nov 03 13:30:04.685278 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:04.685262 5979 scope.go:122] "RemoveContainer" containerID="8a4920a54ab0e42557366c2f108a760e668cd321ec21b5db3adad55bfe8702f9" Nov 03 13:30:04.685570 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:30:04.685535 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-cdb532-29895210-bml7d_odh-ai-gateway-infra(0c1ef57f-1651-4415-a8b1-1f1dff93e5d4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895210-bml7d" podUID="0c1ef57f-1651-4415-a8b1-1f1dff93e5d4" Nov 03 13:30:04.702118 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:04.702089 5979 scope.go:122] "RemoveContainer" containerID="faf1f0d04e0c3aaa15fde474c7598f659d314f5ca1eca0cc998adb9fc954b14a" Nov 03 13:30:04.716982 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:04.716958 5979 scope.go:122] "RemoveContainer" containerID="2ee37b110efac45e6eed23de15048982d1ce8b9180ad7d44e78b0d44a6593ded" Nov 03 13:30:04.730986 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:04.730964 5979 scope.go:122] "RemoveContainer" containerID="0fe8e1316f415cf8f5701d7986af6b15a798c13e7591b1b27dcd834e2b8703dd" Nov 03 13:30:04.744430 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:04.744412 5979 scope.go:122] "RemoveContainer" containerID="6b9b6b25a39b58368b1446e8c7263bd5b7649722b6afa70c0b3d3f345d815bdf" Nov 03 13:30:05.695122 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:05.695092 5979 scope.go:122] "RemoveContainer" containerID="5b92af8f4bb41c2ed146ecf09afc4e98ae48f9253e3873c3e683a454e79bdc89" Nov 03 13:30:05.695579 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:30:05.695428 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5bb463-29895210-8rs5b_odh-ai-gateway-infra(8e268eef-9d3a-4459-a456-381085fdfa63)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895210-8rs5b" podUID="8e268eef-9d3a-4459-a456-381085fdfa63" Nov 03 13:30:05.696680 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:05.696662 5979 scope.go:122] "RemoveContainer" containerID="6e46e3b73c4c6067c5562cc9fee86a5ce936a0859aef1ce920e4d3524b705fea" Nov 03 13:30:05.696928 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:30:05.696910 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-abe0e0-29895210-fb7wr_odh-ai-gateway-infra(9ab44829-1b15-408c-b6fd-647ba93cac2e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-abe0e0-29895210-fb7wr" podUID="9ab44829-1b15-408c-b6fd-647ba93cac2e" Nov 03 13:30:05.698211 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:05.698193 5979 scope.go:122] "RemoveContainer" containerID="85e8f21ec295232583d02bcd1b0b18322ac641a174acfadc204685e4e585b62e" Nov 03 13:30:05.698443 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:30:05.698418 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-5295ab29-29895210-tt82x_odh-ai-gateway-infra(54703132-f756-4072-bb59-170c003ac16b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5295ab29-29895210-tt82x" podUID="54703132-f756-4072-bb59-170c003ac16b" Nov 03 13:30:05.699730 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:05.699712 5979 scope.go:122] "RemoveContainer" containerID="45efc88d6ce4707b3f2034640312972728f9f6c22a053796dab238de26e6e9d2" Nov 03 13:30:05.699956 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:30:05.699933 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895210-hlf2l_odh-ai-gateway-infra(088feb81-52ce-4afc-8d07-aa9d2a094442)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895210-hlf2l" podUID="088feb81-52ce-4afc-8d07-aa9d2a094442" Nov 03 13:30:05.701317 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:05.701297 5979 scope.go:122] "RemoveContainer" containerID="dc3fbda233f701bf52de08812affdcd4363b68f13e867a56c7ca329186d513c0" Nov 03 13:30:05.701550 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:30:05.701531 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-34b13c-29895210-b26x7_odh-ai-gateway-infra(2666652f-479a-44fe-8f0c-17bc354aaaec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895210-b26x7" podUID="2666652f-479a-44fe-8f0c-17bc354aaaec" Nov 03 13:30:05.702842 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:05.702826 5979 scope.go:122] "RemoveContainer" containerID="8a4920a54ab0e42557366c2f108a760e668cd321ec21b5db3adad55bfe8702f9" Nov 03 13:30:05.703021 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:30:05.703006 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-cdb532-29895210-bml7d_odh-ai-gateway-infra(0c1ef57f-1651-4415-a8b1-1f1dff93e5d4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895210-bml7d" podUID="0c1ef57f-1651-4415-a8b1-1f1dff93e5d4" Nov 03 13:30:06.761239 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:06.761203 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:30:07.234518 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:30:07.234465 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:30:07.528136 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:07.528051 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-28d87c-6c79c6f6dc-pvhkm" Nov 03 13:30:07.528136 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:07.528119 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-28d87c-f8886f745-ggq7c" Nov 03 13:30:10.049178 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:10.049143 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a821e75-8tt4c"] Nov 03 13:30:10.095600 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:10.095565 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9a821e75-8tt4c"] Nov 03 13:30:10.118466 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:10.118429 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0746779-2ad5-4002-b404-a62c1e171da0" path="/var/lib/kubelet/pods/d0746779-2ad5-4002-b404-a62c1e171da0/volumes" Nov 03 13:30:14.108244 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:14.108207 5979 scope.go:122] "RemoveContainer" containerID="85e8f21ec295232583d02bcd1b0b18322ac641a174acfadc204685e4e585b62e" Nov 03 13:30:14.777571 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:14.777530 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5295ab29-29895210-tt82x" event={"ID":"54703132-f756-4072-bb59-170c003ac16b","Type":"ContainerStarted","Data":"8bbe3298483c76400cec4f7b3df62d78bed2f180f8e6ad769a07726954f1e928"} Nov 03 13:30:15.787237 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:15.787202 5979 generic.go:356] "Generic (PLEG): container finished" podID="54703132-f756-4072-bb59-170c003ac16b" containerID="8bbe3298483c76400cec4f7b3df62d78bed2f180f8e6ad769a07726954f1e928" exitCode=60 Nov 03 13:30:15.787744 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:15.787245 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5295ab29-29895210-tt82x" event={"ID":"54703132-f756-4072-bb59-170c003ac16b","Type":"ContainerDied","Data":"8bbe3298483c76400cec4f7b3df62d78bed2f180f8e6ad769a07726954f1e928"} Nov 03 13:30:15.787744 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:15.787290 5979 scope.go:122] "RemoveContainer" containerID="85e8f21ec295232583d02bcd1b0b18322ac641a174acfadc204685e4e585b62e" Nov 03 13:30:15.787866 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:15.787748 5979 scope.go:122] "RemoveContainer" containerID="8bbe3298483c76400cec4f7b3df62d78bed2f180f8e6ad769a07726954f1e928" Nov 03 13:30:15.788129 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:30:15.788103 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-5295ab29-29895210-tt82x_odh-ai-gateway-infra(54703132-f756-4072-bb59-170c003ac16b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5295ab29-29895210-tt82x" podUID="54703132-f756-4072-bb59-170c003ac16b" Nov 03 13:30:15.908092 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:15.908061 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5295ab29-29895210-tt82x"] Nov 03 13:30:16.046564 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:16.046453 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2620536c-tnzcp"] Nov 03 13:30:16.051564 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:16.051540 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2620536c-tnzcp"] Nov 03 13:30:16.116892 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:16.116860 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b95dcf2b-e814-4b75-a234-82b5b765e52a" path="/var/lib/kubelet/pods/b95dcf2b-e814-4b75-a234-82b5b765e52a/volumes" Nov 03 13:30:16.117814 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:16.117786 5979 scope.go:122] "RemoveContainer" containerID="45efc88d6ce4707b3f2034640312972728f9f6c22a053796dab238de26e6e9d2" Nov 03 13:30:16.117942 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:16.117923 5979 scope.go:122] "RemoveContainer" containerID="8a4920a54ab0e42557366c2f108a760e668cd321ec21b5db3adad55bfe8702f9" Nov 03 13:30:16.118066 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:16.118049 5979 scope.go:122] "RemoveContainer" containerID="5b92af8f4bb41c2ed146ecf09afc4e98ae48f9253e3873c3e683a454e79bdc89" Nov 03 13:30:16.797735 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:16.797702 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895210-bml7d" event={"ID":"0c1ef57f-1651-4415-a8b1-1f1dff93e5d4","Type":"ContainerStarted","Data":"496f1c21193a570c1bbb67bfe004185547fd808dec319acc786f31e99f2394a0"} Nov 03 13:30:16.799612 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:16.799587 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895210-8rs5b" event={"ID":"8e268eef-9d3a-4459-a456-381085fdfa63","Type":"ContainerStarted","Data":"a0c7f6598456f3bd148027e9de4f3f5a04319f7b3f60e865facb45e1cdcf577a"} Nov 03 13:30:16.803208 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:16.803182 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895210-hlf2l" event={"ID":"088feb81-52ce-4afc-8d07-aa9d2a094442","Type":"ContainerStarted","Data":"85b4ea0cbd890139511b8279470cb73bf445a68e9c0f66c2c16a5c3df325e4d6"} Nov 03 13:30:16.972265 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:16.972238 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5295ab29-29895210-tt82x" Nov 03 13:30:17.097068 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:17.097033 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54703132-f756-4072-bb59-170c003ac16b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54703132-f756-4072-bb59-170c003ac16b-maas-api-service-ca\") pod \"54703132-f756-4072-bb59-170c003ac16b\" (UID: \"54703132-f756-4072-bb59-170c003ac16b\") " Nov 03 13:30:17.097399 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:17.097374 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54703132-f756-4072-bb59-170c003ac16b-maas-api-service-ca" pod "54703132-f756-4072-bb59-170c003ac16b" (UID: "54703132-f756-4072-bb59-170c003ac16b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:30:17.097467 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:17.097433 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54703132-f756-4072-bb59-170c003ac16b-kube-api-access-dndql\" (UniqueName: \"kubernetes.io/projected/54703132-f756-4072-bb59-170c003ac16b-kube-api-access-dndql\") pod \"54703132-f756-4072-bb59-170c003ac16b\" (UID: \"54703132-f756-4072-bb59-170c003ac16b\") " Nov 03 13:30:17.098019 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:17.097999 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54703132-f756-4072-bb59-170c003ac16b-maas-api-service-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:17.099440 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:17.099420 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54703132-f756-4072-bb59-170c003ac16b-kube-api-access-dndql" pod "54703132-f756-4072-bb59-170c003ac16b" (UID: "54703132-f756-4072-bb59-170c003ac16b"). InnerVolumeSpecName "kube-api-access-dndql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:30:17.199071 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:17.198988 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dndql\" (UniqueName: \"kubernetes.io/projected/54703132-f756-4072-bb59-170c003ac16b-kube-api-access-dndql\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:17.812204 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:17.812175 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5295ab29-29895210-tt82x" Nov 03 13:30:17.812693 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:17.812223 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5295ab29-29895210-tt82x" event={"ID":"54703132-f756-4072-bb59-170c003ac16b","Type":"ContainerDied","Data":"4562538bb8b605242579e84756212e0f6959497e0592cd03696522bed73cd633"} Nov 03 13:30:17.812693 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:17.812310 5979 scope.go:122] "RemoveContainer" containerID="8bbe3298483c76400cec4f7b3df62d78bed2f180f8e6ad769a07726954f1e928" Nov 03 13:30:17.814587 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:17.814560 5979 generic.go:356] "Generic (PLEG): container finished" podID="088feb81-52ce-4afc-8d07-aa9d2a094442" containerID="85b4ea0cbd890139511b8279470cb73bf445a68e9c0f66c2c16a5c3df325e4d6" exitCode=60 Nov 03 13:30:17.814725 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:17.814651 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895210-hlf2l" event={"ID":"088feb81-52ce-4afc-8d07-aa9d2a094442","Type":"ContainerDied","Data":"85b4ea0cbd890139511b8279470cb73bf445a68e9c0f66c2c16a5c3df325e4d6"} Nov 03 13:30:17.815082 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:17.815059 5979 scope.go:122] "RemoveContainer" containerID="85b4ea0cbd890139511b8279470cb73bf445a68e9c0f66c2c16a5c3df325e4d6" Nov 03 13:30:17.815414 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:30:17.815392 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895210-hlf2l_odh-ai-gateway-infra(088feb81-52ce-4afc-8d07-aa9d2a094442)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895210-hlf2l" podUID="088feb81-52ce-4afc-8d07-aa9d2a094442" Nov 03 13:30:17.816987 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:17.816962 5979 generic.go:356] "Generic (PLEG): container finished" podID="0c1ef57f-1651-4415-a8b1-1f1dff93e5d4" containerID="496f1c21193a570c1bbb67bfe004185547fd808dec319acc786f31e99f2394a0" exitCode=60 Nov 03 13:30:17.817094 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:17.817036 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895210-bml7d" event={"ID":"0c1ef57f-1651-4415-a8b1-1f1dff93e5d4","Type":"ContainerDied","Data":"496f1c21193a570c1bbb67bfe004185547fd808dec319acc786f31e99f2394a0"} Nov 03 13:30:17.817442 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:17.817422 5979 scope.go:122] "RemoveContainer" containerID="496f1c21193a570c1bbb67bfe004185547fd808dec319acc786f31e99f2394a0" Nov 03 13:30:17.817766 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:30:17.817745 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-cdb532-29895210-bml7d_odh-ai-gateway-infra(0c1ef57f-1651-4415-a8b1-1f1dff93e5d4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895210-bml7d" podUID="0c1ef57f-1651-4415-a8b1-1f1dff93e5d4" Nov 03 13:30:17.819409 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:17.819381 5979 generic.go:356] "Generic (PLEG): container finished" podID="8e268eef-9d3a-4459-a456-381085fdfa63" containerID="a0c7f6598456f3bd148027e9de4f3f5a04319f7b3f60e865facb45e1cdcf577a" exitCode=60 Nov 03 13:30:17.819508 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:17.819455 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895210-8rs5b" event={"ID":"8e268eef-9d3a-4459-a456-381085fdfa63","Type":"ContainerDied","Data":"a0c7f6598456f3bd148027e9de4f3f5a04319f7b3f60e865facb45e1cdcf577a"} Nov 03 13:30:17.819874 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:17.819857 5979 scope.go:122] "RemoveContainer" containerID="a0c7f6598456f3bd148027e9de4f3f5a04319f7b3f60e865facb45e1cdcf577a" Nov 03 13:30:17.820201 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:30:17.820181 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5bb463-29895210-8rs5b_odh-ai-gateway-infra(8e268eef-9d3a-4459-a456-381085fdfa63)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895210-8rs5b" podUID="8e268eef-9d3a-4459-a456-381085fdfa63" Nov 03 13:30:17.824592 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:17.824573 5979 scope.go:122] "RemoveContainer" containerID="45efc88d6ce4707b3f2034640312972728f9f6c22a053796dab238de26e6e9d2" Nov 03 13:30:17.839926 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:17.839904 5979 scope.go:122] "RemoveContainer" containerID="8a4920a54ab0e42557366c2f108a760e668cd321ec21b5db3adad55bfe8702f9" Nov 03 13:30:17.858947 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:17.858927 5979 scope.go:122] "RemoveContainer" containerID="5b92af8f4bb41c2ed146ecf09afc4e98ae48f9253e3873c3e683a454e79bdc89" Nov 03 13:30:17.918694 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:17.918667 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895210-bml7d"] Nov 03 13:30:18.076557 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:18.076461 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5295ab29-29895210-tt82x"] Nov 03 13:30:18.080321 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:18.080293 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-5295ab29-29895210-tt82x"] Nov 03 13:30:18.117331 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:18.117301 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54703132-f756-4072-bb59-170c003ac16b" path="/var/lib/kubelet/pods/54703132-f756-4072-bb59-170c003ac16b/volumes" Nov 03 13:30:18.118141 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:18.118120 5979 scope.go:122] "RemoveContainer" containerID="dc3fbda233f701bf52de08812affdcd4363b68f13e867a56c7ca329186d513c0" Nov 03 13:30:18.118268 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:18.118224 5979 scope.go:122] "RemoveContainer" containerID="6e46e3b73c4c6067c5562cc9fee86a5ce936a0859aef1ce920e4d3524b705fea" Nov 03 13:30:18.200753 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:18.200719 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895210-hlf2l"] Nov 03 13:30:18.313636 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:18.313605 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895210-8rs5b"] Nov 03 13:30:18.692573 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:18.692544 5979 scope.go:122] "RemoveContainer" containerID="a0c8051819b8aa5e8529d622d95453fc39b8a0e3cc6ff8c6270a3da4c1596244" Nov 03 13:30:18.702733 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:18.702709 5979 scope.go:122] "RemoveContainer" containerID="88cf2463e3b603591a6c0d94d849071003887187674cee56a829f61fbb2ac24d" Nov 03 13:30:18.714357 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:18.714334 5979 scope.go:122] "RemoveContainer" containerID="e24901793e81dd52699a4f7934a6d11fd6d7bdfb051e59bb0ce92aa3533d3b5c" Nov 03 13:30:18.723793 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:18.723773 5979 scope.go:122] "RemoveContainer" containerID="f624f368d24560733813bc4f4a5e7ff2345915d65c55b32448ccba40df88a211" Nov 03 13:30:18.838345 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:18.838308 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895210-b26x7" event={"ID":"2666652f-479a-44fe-8f0c-17bc354aaaec","Type":"ContainerStarted","Data":"8f5f16727cec4150aa9bb90afebd7dd3c6e3eab4120dfa30046db73d2f1532b2"} Nov 03 13:30:18.844432 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:18.844405 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-abe0e0-29895210-fb7wr" event={"ID":"9ab44829-1b15-408c-b6fd-647ba93cac2e","Type":"ContainerStarted","Data":"705b9cc370b6d0c5dc3fd29d9c8fb1ddbbfbfe4b693adf31a27010209b0f4663"} Nov 03 13:30:19.022260 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:19.022229 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895210-hlf2l" Nov 03 13:30:19.093940 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:19.093905 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895210-8rs5b" Nov 03 13:30:19.104127 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:19.104104 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895210-bml7d" Nov 03 13:30:19.133677 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:19.133643 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/088feb81-52ce-4afc-8d07-aa9d2a094442-kube-api-access-fgk2h\" (UniqueName: \"kubernetes.io/projected/088feb81-52ce-4afc-8d07-aa9d2a094442-kube-api-access-fgk2h\") pod \"088feb81-52ce-4afc-8d07-aa9d2a094442\" (UID: \"088feb81-52ce-4afc-8d07-aa9d2a094442\") " Nov 03 13:30:19.133970 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:19.133938 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/088feb81-52ce-4afc-8d07-aa9d2a094442-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/088feb81-52ce-4afc-8d07-aa9d2a094442-maas-api-service-ca\") pod \"088feb81-52ce-4afc-8d07-aa9d2a094442\" (UID: \"088feb81-52ce-4afc-8d07-aa9d2a094442\") " Nov 03 13:30:19.134315 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:19.134286 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/088feb81-52ce-4afc-8d07-aa9d2a094442-maas-api-service-ca" pod "088feb81-52ce-4afc-8d07-aa9d2a094442" (UID: "088feb81-52ce-4afc-8d07-aa9d2a094442"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:30:19.134583 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:19.134561 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/088feb81-52ce-4afc-8d07-aa9d2a094442-maas-api-service-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:19.136167 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:19.136143 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/088feb81-52ce-4afc-8d07-aa9d2a094442-kube-api-access-fgk2h" pod "088feb81-52ce-4afc-8d07-aa9d2a094442" (UID: "088feb81-52ce-4afc-8d07-aa9d2a094442"). InnerVolumeSpecName "kube-api-access-fgk2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:30:19.235377 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:19.235311 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e268eef-9d3a-4459-a456-381085fdfa63-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e268eef-9d3a-4459-a456-381085fdfa63-maas-api-service-ca\") pod \"8e268eef-9d3a-4459-a456-381085fdfa63\" (UID: \"8e268eef-9d3a-4459-a456-381085fdfa63\") " Nov 03 13:30:19.235515 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:19.235391 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c1ef57f-1651-4415-a8b1-1f1dff93e5d4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c1ef57f-1651-4415-a8b1-1f1dff93e5d4-maas-api-service-ca\") pod \"0c1ef57f-1651-4415-a8b1-1f1dff93e5d4\" (UID: \"0c1ef57f-1651-4415-a8b1-1f1dff93e5d4\") " Nov 03 13:30:19.235515 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:19.235423 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c1ef57f-1651-4415-a8b1-1f1dff93e5d4-kube-api-access-fxvz5\" (UniqueName: \"kubernetes.io/projected/0c1ef57f-1651-4415-a8b1-1f1dff93e5d4-kube-api-access-fxvz5\") pod \"0c1ef57f-1651-4415-a8b1-1f1dff93e5d4\" (UID: \"0c1ef57f-1651-4415-a8b1-1f1dff93e5d4\") " Nov 03 13:30:19.235515 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:19.235497 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e268eef-9d3a-4459-a456-381085fdfa63-kube-api-access-nw8jg\" (UniqueName: \"kubernetes.io/projected/8e268eef-9d3a-4459-a456-381085fdfa63-kube-api-access-nw8jg\") pod \"8e268eef-9d3a-4459-a456-381085fdfa63\" (UID: \"8e268eef-9d3a-4459-a456-381085fdfa63\") " Nov 03 13:30:19.235736 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:19.235713 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e268eef-9d3a-4459-a456-381085fdfa63-maas-api-service-ca" pod "8e268eef-9d3a-4459-a456-381085fdfa63" (UID: "8e268eef-9d3a-4459-a456-381085fdfa63"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:30:19.235846 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:19.235821 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c1ef57f-1651-4415-a8b1-1f1dff93e5d4-maas-api-service-ca" pod "0c1ef57f-1651-4415-a8b1-1f1dff93e5d4" (UID: "0c1ef57f-1651-4415-a8b1-1f1dff93e5d4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:30:19.236123 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:19.236102 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgk2h\" (UniqueName: \"kubernetes.io/projected/088feb81-52ce-4afc-8d07-aa9d2a094442-kube-api-access-fgk2h\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:19.236211 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:19.236122 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e268eef-9d3a-4459-a456-381085fdfa63-maas-api-service-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:19.236211 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:19.236133 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c1ef57f-1651-4415-a8b1-1f1dff93e5d4-maas-api-service-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:19.244189 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:19.243883 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e268eef-9d3a-4459-a456-381085fdfa63-kube-api-access-nw8jg" pod "8e268eef-9d3a-4459-a456-381085fdfa63" (UID: "8e268eef-9d3a-4459-a456-381085fdfa63"). InnerVolumeSpecName "kube-api-access-nw8jg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:30:19.244189 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:19.243918 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c1ef57f-1651-4415-a8b1-1f1dff93e5d4-kube-api-access-fxvz5" pod "0c1ef57f-1651-4415-a8b1-1f1dff93e5d4" (UID: "0c1ef57f-1651-4415-a8b1-1f1dff93e5d4"). InnerVolumeSpecName "kube-api-access-fxvz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:30:19.338143 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:19.338110 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxvz5\" (UniqueName: \"kubernetes.io/projected/0c1ef57f-1651-4415-a8b1-1f1dff93e5d4-kube-api-access-fxvz5\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:19.338143 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:19.338141 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw8jg\" (UniqueName: \"kubernetes.io/projected/8e268eef-9d3a-4459-a456-381085fdfa63-kube-api-access-nw8jg\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:19.853406 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:19.853371 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895210-hlf2l" Nov 03 13:30:19.853406 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:19.853380 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895210-hlf2l" event={"ID":"088feb81-52ce-4afc-8d07-aa9d2a094442","Type":"ContainerDied","Data":"1fdbeeefe1f7de0094972810db5863c4f300feee67efd2ab5cc7d377c6dd5cde"} Nov 03 13:30:19.853994 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:19.853449 5979 scope.go:122] "RemoveContainer" containerID="85b4ea0cbd890139511b8279470cb73bf445a68e9c0f66c2c16a5c3df325e4d6" Nov 03 13:30:19.855464 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:19.855437 5979 generic.go:356] "Generic (PLEG): container finished" podID="2666652f-479a-44fe-8f0c-17bc354aaaec" containerID="8f5f16727cec4150aa9bb90afebd7dd3c6e3eab4120dfa30046db73d2f1532b2" exitCode=60 Nov 03 13:30:19.855592 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:19.855568 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895210-b26x7" event={"ID":"2666652f-479a-44fe-8f0c-17bc354aaaec","Type":"ContainerDied","Data":"8f5f16727cec4150aa9bb90afebd7dd3c6e3eab4120dfa30046db73d2f1532b2"} Nov 03 13:30:19.856082 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:19.856059 5979 scope.go:122] "RemoveContainer" containerID="8f5f16727cec4150aa9bb90afebd7dd3c6e3eab4120dfa30046db73d2f1532b2" Nov 03 13:30:19.856506 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:30:19.856452 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-34b13c-29895210-b26x7_odh-ai-gateway-infra(2666652f-479a-44fe-8f0c-17bc354aaaec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895210-b26x7" podUID="2666652f-479a-44fe-8f0c-17bc354aaaec" Nov 03 13:30:19.856983 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:19.856959 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895210-bml7d" event={"ID":"0c1ef57f-1651-4415-a8b1-1f1dff93e5d4","Type":"ContainerDied","Data":"c4bdc2ec9338b264a0df40d010cf66106da4e31ab6c075d0111022268a40cbb1"} Nov 03 13:30:19.857082 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:19.856967 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895210-bml7d" Nov 03 13:30:19.858557 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:19.858535 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895210-8rs5b" Nov 03 13:30:19.858686 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:19.858554 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895210-8rs5b" event={"ID":"8e268eef-9d3a-4459-a456-381085fdfa63","Type":"ContainerDied","Data":"42b3a657c30bcfd592a508cc0115e90f5b33cdb8dc2b5da471be78d1a386dcf5"} Nov 03 13:30:19.860698 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:19.860674 5979 generic.go:356] "Generic (PLEG): container finished" podID="9ab44829-1b15-408c-b6fd-647ba93cac2e" containerID="705b9cc370b6d0c5dc3fd29d9c8fb1ddbbfbfe4b693adf31a27010209b0f4663" exitCode=60 Nov 03 13:30:19.860785 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:19.860732 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-abe0e0-29895210-fb7wr" event={"ID":"9ab44829-1b15-408c-b6fd-647ba93cac2e","Type":"ContainerDied","Data":"705b9cc370b6d0c5dc3fd29d9c8fb1ddbbfbfe4b693adf31a27010209b0f4663"} Nov 03 13:30:19.861147 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:19.861129 5979 scope.go:122] "RemoveContainer" containerID="705b9cc370b6d0c5dc3fd29d9c8fb1ddbbfbfe4b693adf31a27010209b0f4663" Nov 03 13:30:19.861444 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:30:19.861423 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-abe0e0-29895210-fb7wr_odh-ai-gateway-infra(9ab44829-1b15-408c-b6fd-647ba93cac2e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-abe0e0-29895210-fb7wr" podUID="9ab44829-1b15-408c-b6fd-647ba93cac2e" Nov 03 13:30:19.866158 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:19.866141 5979 scope.go:122] "RemoveContainer" containerID="dc3fbda233f701bf52de08812affdcd4363b68f13e867a56c7ca329186d513c0" Nov 03 13:30:19.880968 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:19.880947 5979 scope.go:122] "RemoveContainer" containerID="496f1c21193a570c1bbb67bfe004185547fd808dec319acc786f31e99f2394a0" Nov 03 13:30:19.889849 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:19.889833 5979 scope.go:122] "RemoveContainer" containerID="a0c7f6598456f3bd148027e9de4f3f5a04319f7b3f60e865facb45e1cdcf577a" Nov 03 13:30:19.898916 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:19.898899 5979 scope.go:122] "RemoveContainer" containerID="6e46e3b73c4c6067c5562cc9fee86a5ce936a0859aef1ce920e4d3524b705fea" Nov 03 13:30:20.058394 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:20.058357 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895210-b26x7"] Nov 03 13:30:20.119826 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:20.119734 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895210-bml7d"] Nov 03 13:30:20.120848 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:20.120821 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-cdb532-29895210-bml7d"] Nov 03 13:30:20.219626 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:20.219590 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895210-hlf2l"] Nov 03 13:30:20.223638 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:20.223614 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895210-hlf2l"] Nov 03 13:30:20.270706 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:20.270674 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-abe0e0-29895210-fb7wr"] Nov 03 13:30:20.470722 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:20.467399 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895210-8rs5b"] Nov 03 13:30:20.508573 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:20.496994 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895210-8rs5b"] Nov 03 13:30:21.081188 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:21.081162 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895210-b26x7" Nov 03 13:30:21.083845 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:21.083824 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-abe0e0-29895210-fb7wr" Nov 03 13:30:21.204366 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:21.204336 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2666652f-479a-44fe-8f0c-17bc354aaaec-kube-api-access-mvkmf\" (UniqueName: \"kubernetes.io/projected/2666652f-479a-44fe-8f0c-17bc354aaaec-kube-api-access-mvkmf\") pod \"2666652f-479a-44fe-8f0c-17bc354aaaec\" (UID: \"2666652f-479a-44fe-8f0c-17bc354aaaec\") " Nov 03 13:30:21.204622 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:21.204599 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ab44829-1b15-408c-b6fd-647ba93cac2e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ab44829-1b15-408c-b6fd-647ba93cac2e-maas-api-service-ca\") pod \"9ab44829-1b15-408c-b6fd-647ba93cac2e\" (UID: \"9ab44829-1b15-408c-b6fd-647ba93cac2e\") " Nov 03 13:30:21.204786 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:21.204643 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ab44829-1b15-408c-b6fd-647ba93cac2e-kube-api-access-bbfdw\" (UniqueName: \"kubernetes.io/projected/9ab44829-1b15-408c-b6fd-647ba93cac2e-kube-api-access-bbfdw\") pod \"9ab44829-1b15-408c-b6fd-647ba93cac2e\" (UID: \"9ab44829-1b15-408c-b6fd-647ba93cac2e\") " Nov 03 13:30:21.204786 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:21.204687 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2666652f-479a-44fe-8f0c-17bc354aaaec-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2666652f-479a-44fe-8f0c-17bc354aaaec-maas-api-service-ca\") pod \"2666652f-479a-44fe-8f0c-17bc354aaaec\" (UID: \"2666652f-479a-44fe-8f0c-17bc354aaaec\") " Nov 03 13:30:21.205046 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:21.204955 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ab44829-1b15-408c-b6fd-647ba93cac2e-maas-api-service-ca" pod "9ab44829-1b15-408c-b6fd-647ba93cac2e" (UID: "9ab44829-1b15-408c-b6fd-647ba93cac2e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:30:21.205208 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:21.205182 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2666652f-479a-44fe-8f0c-17bc354aaaec-maas-api-service-ca" pod "2666652f-479a-44fe-8f0c-17bc354aaaec" (UID: "2666652f-479a-44fe-8f0c-17bc354aaaec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:30:21.205381 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:21.205357 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ab44829-1b15-408c-b6fd-647ba93cac2e-maas-api-service-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:21.205537 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:21.205384 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2666652f-479a-44fe-8f0c-17bc354aaaec-maas-api-service-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:21.206934 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:21.206913 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ab44829-1b15-408c-b6fd-647ba93cac2e-kube-api-access-bbfdw" pod "9ab44829-1b15-408c-b6fd-647ba93cac2e" (UID: "9ab44829-1b15-408c-b6fd-647ba93cac2e"). InnerVolumeSpecName "kube-api-access-bbfdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:30:21.207168 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:21.207151 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2666652f-479a-44fe-8f0c-17bc354aaaec-kube-api-access-mvkmf" pod "2666652f-479a-44fe-8f0c-17bc354aaaec" (UID: "2666652f-479a-44fe-8f0c-17bc354aaaec"). InnerVolumeSpecName "kube-api-access-mvkmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:30:21.307287 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:21.307251 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbfdw\" (UniqueName: \"kubernetes.io/projected/9ab44829-1b15-408c-b6fd-647ba93cac2e-kube-api-access-bbfdw\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:21.307287 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:21.307287 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvkmf\" (UniqueName: \"kubernetes.io/projected/2666652f-479a-44fe-8f0c-17bc354aaaec-kube-api-access-mvkmf\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:21.884611 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:21.884580 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895210-b26x7" Nov 03 13:30:21.884814 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:21.884615 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895210-b26x7" event={"ID":"2666652f-479a-44fe-8f0c-17bc354aaaec","Type":"ContainerDied","Data":"4eb5f6fe3a61f86a3b52952281ff558df2f2ce2256acab76335639f6a8db5214"} Nov 03 13:30:21.884814 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:21.884677 5979 scope.go:122] "RemoveContainer" containerID="8f5f16727cec4150aa9bb90afebd7dd3c6e3eab4120dfa30046db73d2f1532b2" Nov 03 13:30:21.886291 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:21.886263 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-abe0e0-29895210-fb7wr" Nov 03 13:30:21.886411 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:21.886287 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-abe0e0-29895210-fb7wr" event={"ID":"9ab44829-1b15-408c-b6fd-647ba93cac2e","Type":"ContainerDied","Data":"4eda872172effc1d816c4e373b7005c43e2fe28a5b61874cac26f6674258ff8d"} Nov 03 13:30:21.900210 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:21.899197 5979 scope.go:122] "RemoveContainer" containerID="705b9cc370b6d0c5dc3fd29d9c8fb1ddbbfbfe4b693adf31a27010209b0f4663" Nov 03 13:30:22.077494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:22.077411 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895210-b26x7"] Nov 03 13:30:22.124133 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:22.124102 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="088feb81-52ce-4afc-8d07-aa9d2a094442" path="/var/lib/kubelet/pods/088feb81-52ce-4afc-8d07-aa9d2a094442/volumes" Nov 03 13:30:22.124607 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:22.124591 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c1ef57f-1651-4415-a8b1-1f1dff93e5d4" path="/var/lib/kubelet/pods/0c1ef57f-1651-4415-a8b1-1f1dff93e5d4/volumes" Nov 03 13:30:22.125002 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:22.124989 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e268eef-9d3a-4459-a456-381085fdfa63" path="/var/lib/kubelet/pods/8e268eef-9d3a-4459-a456-381085fdfa63/volumes" Nov 03 13:30:22.125619 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:22.125605 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895210-b26x7"] Nov 03 13:30:22.180557 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:22.180442 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-abe0e0-29895210-fb7wr"] Nov 03 13:30:22.184198 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:22.184171 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-abe0e0-29895210-fb7wr"] Nov 03 13:30:24.117025 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:24.116992 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2666652f-479a-44fe-8f0c-17bc354aaaec" path="/var/lib/kubelet/pods/2666652f-479a-44fe-8f0c-17bc354aaaec/volumes" Nov 03 13:30:24.117521 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:24.117505 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ab44829-1b15-408c-b6fd-647ba93cac2e" path="/var/lib/kubelet/pods/9ab44829-1b15-408c-b6fd-647ba93cac2e/volumes" Nov 03 13:30:27.048096 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:27.048059 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f67e1b40-bdr95"] Nov 03 13:30:27.094849 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:27.094813 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3dc9b19-p5jwq"] Nov 03 13:30:27.098131 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:27.098112 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f67e1b40-bdr95"] Nov 03 13:30:27.101320 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:27.101286 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3dc9b19-p5jwq"] Nov 03 13:30:28.117153 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:28.117117 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ab6d0b5-79bc-45d5-8b77-75f5d5fd49cd" path="/var/lib/kubelet/pods/2ab6d0b5-79bc-45d5-8b77-75f5d5fd49cd/volumes" Nov 03 13:30:28.117958 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:28.117935 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f23a7bba-1d47-4cb9-853a-5adbe0e34430" path="/var/lib/kubelet/pods/f23a7bba-1d47-4cb9-853a-5adbe0e34430/volumes" Nov 03 13:30:30.386320 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:30.386290 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-30074e-7986f65ffc-82gzw"] Nov 03 13:30:30.510106 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:30.510029 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-30074e-5dd5666c67-gkb7j"] Nov 03 13:30:35.387027 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:35.386985 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-30074e-7986f65ffc-82gzw" podUID="ef6ffd56-1244-4430-9740-436407eb0ebe" containerName="payload-processing" containerID="cri-o://42309357fb4917487bee8cb84fd479bdc1b9af23e1de9bbb36d3fafc9159ffb6" gracePeriod=25 Nov 03 13:30:35.511214 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:35.511177 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-30074e-5dd5666c67-gkb7j" podUID="f59b1527-f24d-47bb-bbd2-3de97661eeee" containerName="payload-pre-processing" containerID="cri-o://7e50e4e5c0ffc192c0511b98f2edba69f71ed1588d20e2c55cb51ea21b896a79" gracePeriod=25 Nov 03 13:30:35.683350 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:35.683322 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-30074e-7986f65ffc-82gzw" Nov 03 13:30:35.758412 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:35.758374 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef6ffd56-1244-4430-9740-436407eb0ebe-kube-api-access-m5hrt\" (UniqueName: \"kubernetes.io/projected/ef6ffd56-1244-4430-9740-436407eb0ebe-kube-api-access-m5hrt\") pod \"ef6ffd56-1244-4430-9740-436407eb0ebe\" (UID: \"ef6ffd56-1244-4430-9740-436407eb0ebe\") " Nov 03 13:30:35.758672 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:35.758635 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef6ffd56-1244-4430-9740-436407eb0ebe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef6ffd56-1244-4430-9740-436407eb0ebe-plugins-config-volume\") pod \"ef6ffd56-1244-4430-9740-436407eb0ebe\" (UID: \"ef6ffd56-1244-4430-9740-436407eb0ebe\") " Nov 03 13:30:35.759302 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:35.759258 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef6ffd56-1244-4430-9740-436407eb0ebe-plugins-config-volume" pod "ef6ffd56-1244-4430-9740-436407eb0ebe" (UID: "ef6ffd56-1244-4430-9740-436407eb0ebe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:30:35.763282 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:35.763254 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef6ffd56-1244-4430-9740-436407eb0ebe-kube-api-access-m5hrt" pod "ef6ffd56-1244-4430-9740-436407eb0ebe" (UID: "ef6ffd56-1244-4430-9740-436407eb0ebe"). InnerVolumeSpecName "kube-api-access-m5hrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:30:35.787918 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:35.787898 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-30074e-5dd5666c67-gkb7j" Nov 03 13:30:35.859992 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:35.859956 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f59b1527-f24d-47bb-bbd2-3de97661eeee-kube-api-access-p68b6\" (UniqueName: \"kubernetes.io/projected/f59b1527-f24d-47bb-bbd2-3de97661eeee-kube-api-access-p68b6\") pod \"f59b1527-f24d-47bb-bbd2-3de97661eeee\" (UID: \"f59b1527-f24d-47bb-bbd2-3de97661eeee\") " Nov 03 13:30:35.860159 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:35.860137 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f59b1527-f24d-47bb-bbd2-3de97661eeee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f59b1527-f24d-47bb-bbd2-3de97661eeee-plugins-config-volume\") pod \"f59b1527-f24d-47bb-bbd2-3de97661eeee\" (UID: \"f59b1527-f24d-47bb-bbd2-3de97661eeee\") " Nov 03 13:30:35.860552 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:35.860529 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f59b1527-f24d-47bb-bbd2-3de97661eeee-plugins-config-volume" pod "f59b1527-f24d-47bb-bbd2-3de97661eeee" (UID: "f59b1527-f24d-47bb-bbd2-3de97661eeee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:30:35.860666 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:35.860651 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef6ffd56-1244-4430-9740-436407eb0ebe-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:35.860712 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:35.860669 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5hrt\" (UniqueName: \"kubernetes.io/projected/ef6ffd56-1244-4430-9740-436407eb0ebe-kube-api-access-m5hrt\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:35.860712 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:35.860679 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f59b1527-f24d-47bb-bbd2-3de97661eeee-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:35.862032 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:35.862016 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f59b1527-f24d-47bb-bbd2-3de97661eeee-kube-api-access-p68b6" pod "f59b1527-f24d-47bb-bbd2-3de97661eeee" (UID: "f59b1527-f24d-47bb-bbd2-3de97661eeee"). InnerVolumeSpecName "kube-api-access-p68b6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:30:35.962224 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:35.962149 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p68b6\" (UniqueName: \"kubernetes.io/projected/f59b1527-f24d-47bb-bbd2-3de97661eeee-kube-api-access-p68b6\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:30:36.053527 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:36.053488 5979 generic.go:356] "Generic (PLEG): container finished" podID="ef6ffd56-1244-4430-9740-436407eb0ebe" containerID="42309357fb4917487bee8cb84fd479bdc1b9af23e1de9bbb36d3fafc9159ffb6" exitCode=0 Nov 03 13:30:36.053735 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:36.053570 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-30074e-7986f65ffc-82gzw" Nov 03 13:30:36.053735 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:36.053583 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-30074e-7986f65ffc-82gzw" event={"ID":"ef6ffd56-1244-4430-9740-436407eb0ebe","Type":"ContainerDied","Data":"42309357fb4917487bee8cb84fd479bdc1b9af23e1de9bbb36d3fafc9159ffb6"} Nov 03 13:30:36.053735 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:36.053661 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-30074e-7986f65ffc-82gzw" event={"ID":"ef6ffd56-1244-4430-9740-436407eb0ebe","Type":"ContainerDied","Data":"82bbb7d740d1280fd9e2905f897109d5fcc8bd6b3e348c6f5c9dd87fcb1f6fe2"} Nov 03 13:30:36.053735 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:36.053683 5979 scope.go:122] "RemoveContainer" containerID="42309357fb4917487bee8cb84fd479bdc1b9af23e1de9bbb36d3fafc9159ffb6" Nov 03 13:30:36.055334 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:36.055312 5979 generic.go:356] "Generic (PLEG): container finished" podID="f59b1527-f24d-47bb-bbd2-3de97661eeee" containerID="7e50e4e5c0ffc192c0511b98f2edba69f71ed1588d20e2c55cb51ea21b896a79" exitCode=0 Nov 03 13:30:36.055494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:36.055362 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-30074e-5dd5666c67-gkb7j" Nov 03 13:30:36.055494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:36.055377 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-30074e-5dd5666c67-gkb7j" event={"ID":"f59b1527-f24d-47bb-bbd2-3de97661eeee","Type":"ContainerDied","Data":"7e50e4e5c0ffc192c0511b98f2edba69f71ed1588d20e2c55cb51ea21b896a79"} Nov 03 13:30:36.055494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:36.055408 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-30074e-5dd5666c67-gkb7j" event={"ID":"f59b1527-f24d-47bb-bbd2-3de97661eeee","Type":"ContainerDied","Data":"b0235964855c9593a0ab5f2ad5a56d5587916905753c2b97bcb6d91096c3a4fb"} Nov 03 13:30:36.064991 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:36.064851 5979 scope.go:122] "RemoveContainer" containerID="42309357fb4917487bee8cb84fd479bdc1b9af23e1de9bbb36d3fafc9159ffb6" Nov 03 13:30:36.065208 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:30:36.065177 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42309357fb4917487bee8cb84fd479bdc1b9af23e1de9bbb36d3fafc9159ffb6\": container with ID starting with 42309357fb4917487bee8cb84fd479bdc1b9af23e1de9bbb36d3fafc9159ffb6 not found: ID does not exist" containerID="42309357fb4917487bee8cb84fd479bdc1b9af23e1de9bbb36d3fafc9159ffb6" Nov 03 13:30:36.065307 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:36.065219 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42309357fb4917487bee8cb84fd479bdc1b9af23e1de9bbb36d3fafc9159ffb6"} err="failed to get container status \"42309357fb4917487bee8cb84fd479bdc1b9af23e1de9bbb36d3fafc9159ffb6\": rpc error: code = NotFound desc = could not find container \"42309357fb4917487bee8cb84fd479bdc1b9af23e1de9bbb36d3fafc9159ffb6\": container with ID starting with 42309357fb4917487bee8cb84fd479bdc1b9af23e1de9bbb36d3fafc9159ffb6 not found: ID does not exist" Nov 03 13:30:36.065307 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:36.065243 5979 scope.go:122] "RemoveContainer" containerID="7e50e4e5c0ffc192c0511b98f2edba69f71ed1588d20e2c55cb51ea21b896a79" Nov 03 13:30:36.076949 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:36.076932 5979 scope.go:122] "RemoveContainer" containerID="7e50e4e5c0ffc192c0511b98f2edba69f71ed1588d20e2c55cb51ea21b896a79" Nov 03 13:30:36.077203 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:30:36.077185 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e50e4e5c0ffc192c0511b98f2edba69f71ed1588d20e2c55cb51ea21b896a79\": container with ID starting with 7e50e4e5c0ffc192c0511b98f2edba69f71ed1588d20e2c55cb51ea21b896a79 not found: ID does not exist" containerID="7e50e4e5c0ffc192c0511b98f2edba69f71ed1588d20e2c55cb51ea21b896a79" Nov 03 13:30:36.077262 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:36.077210 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e50e4e5c0ffc192c0511b98f2edba69f71ed1588d20e2c55cb51ea21b896a79"} err="failed to get container status \"7e50e4e5c0ffc192c0511b98f2edba69f71ed1588d20e2c55cb51ea21b896a79\": rpc error: code = NotFound desc = could not find container \"7e50e4e5c0ffc192c0511b98f2edba69f71ed1588d20e2c55cb51ea21b896a79\": container with ID starting with 7e50e4e5c0ffc192c0511b98f2edba69f71ed1588d20e2c55cb51ea21b896a79 not found: ID does not exist" Nov 03 13:30:36.210791 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:36.210754 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-30074e-5dd5666c67-gkb7j"] Nov 03 13:30:36.214436 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:36.214378 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-30074e-5dd5666c67-gkb7j"] Nov 03 13:30:36.316122 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:36.316086 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-30074e-7986f65ffc-82gzw"] Nov 03 13:30:36.320378 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:36.320348 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-30074e-7986f65ffc-82gzw"] Nov 03 13:30:37.236588 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:30:37.236547 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:30:38.117079 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:38.117049 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef6ffd56-1244-4430-9740-436407eb0ebe" path="/var/lib/kubelet/pods/ef6ffd56-1244-4430-9740-436407eb0ebe/volumes" Nov 03 13:30:38.117606 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:38.117591 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f59b1527-f24d-47bb-bbd2-3de97661eeee" path="/var/lib/kubelet/pods/f59b1527-f24d-47bb-bbd2-3de97661eeee/volumes" Nov 03 13:30:40.054772 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:40.054738 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c33b5d69-fh6bc"] Nov 03 13:30:40.101442 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:40.101410 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c33b5d69-fh6bc"] Nov 03 13:30:40.116792 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:40.116763 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be076614-167d-49f2-850a-8649432fb5dc" path="/var/lib/kubelet/pods/be076614-167d-49f2-850a-8649432fb5dc/volumes" Nov 03 13:30:48.049362 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:48.049315 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8575f973-dm7n6"] Nov 03 13:30:48.095181 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:48.095148 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8575f973-dm7n6"] Nov 03 13:30:48.116282 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:48.116251 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="731dc017-0497-42c3-9a66-001519ede38b" path="/var/lib/kubelet/pods/731dc017-0497-42c3-9a66-001519ede38b/volumes" Nov 03 13:30:52.269770 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:52.269730 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:30:58.075506 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.073641 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d567b768-j8mp7"] Nov 03 13:30:58.104751 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.104704 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d567b768-j8mp7"] Nov 03 13:30:58.175938 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.175899 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ef15fe1-f088-46d7-adce-da164a1b0b77" path="/var/lib/kubelet/pods/9ef15fe1-f088-46d7-adce-da164a1b0b77/volumes" Nov 03 13:30:58.190147 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.190076 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18f090c4-z927b"] Nov 03 13:30:58.190862 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.190840 5979 cpu_manager.go:414] "containerMap: removing container" podUID="2666652f-479a-44fe-8f0c-17bc354aaaec" containerName="cleanup" Nov 03 13:30:58.191040 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.191021 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2666652f-479a-44fe-8f0c-17bc354aaaec" containerName="cleanup" Nov 03 13:30:58.191229 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.191211 5979 cpu_manager.go:414] "containerMap: removing container" podUID="2666652f-479a-44fe-8f0c-17bc354aaaec" containerName="cleanup" Nov 03 13:30:58.191340 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.191326 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2666652f-479a-44fe-8f0c-17bc354aaaec" containerName="cleanup" Nov 03 13:30:58.191414 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.191382 5979 cpu_manager.go:414] "containerMap: removing container" podUID="8e268eef-9d3a-4459-a456-381085fdfa63" containerName="cleanup" Nov 03 13:30:58.191414 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.191395 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e268eef-9d3a-4459-a456-381085fdfa63" containerName="cleanup" Nov 03 13:30:58.191414 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.191412 5979 cpu_manager.go:414] "containerMap: removing container" podUID="8e268eef-9d3a-4459-a456-381085fdfa63" containerName="cleanup" Nov 03 13:30:58.191766 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.191420 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e268eef-9d3a-4459-a456-381085fdfa63" containerName="cleanup" Nov 03 13:30:58.191766 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.191435 5979 cpu_manager.go:414] "containerMap: removing container" podUID="088feb81-52ce-4afc-8d07-aa9d2a094442" containerName="cleanup" Nov 03 13:30:58.191766 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.191443 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="088feb81-52ce-4afc-8d07-aa9d2a094442" containerName="cleanup" Nov 03 13:30:58.191766 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.191461 5979 cpu_manager.go:414] "containerMap: removing container" podUID="9ab44829-1b15-408c-b6fd-647ba93cac2e" containerName="cleanup" Nov 03 13:30:58.191766 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.191488 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ab44829-1b15-408c-b6fd-647ba93cac2e" containerName="cleanup" Nov 03 13:30:58.191766 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.191504 5979 cpu_manager.go:414] "containerMap: removing container" podUID="0c1ef57f-1651-4415-a8b1-1f1dff93e5d4" containerName="cleanup" Nov 03 13:30:58.191766 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.191511 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c1ef57f-1651-4415-a8b1-1f1dff93e5d4" containerName="cleanup" Nov 03 13:30:58.191766 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.191528 5979 cpu_manager.go:414] "containerMap: removing container" podUID="ef6ffd56-1244-4430-9740-436407eb0ebe" containerName="payload-processing" Nov 03 13:30:58.191766 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.191535 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef6ffd56-1244-4430-9740-436407eb0ebe" containerName="payload-processing" Nov 03 13:30:58.191766 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.191549 5979 cpu_manager.go:414] "containerMap: removing container" podUID="0c1ef57f-1651-4415-a8b1-1f1dff93e5d4" containerName="cleanup" Nov 03 13:30:58.191766 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.191556 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c1ef57f-1651-4415-a8b1-1f1dff93e5d4" containerName="cleanup" Nov 03 13:30:58.191766 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.191567 5979 cpu_manager.go:414] "containerMap: removing container" podUID="9ab44829-1b15-408c-b6fd-647ba93cac2e" containerName="cleanup" Nov 03 13:30:58.191766 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.191573 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ab44829-1b15-408c-b6fd-647ba93cac2e" containerName="cleanup" Nov 03 13:30:58.191766 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.191587 5979 cpu_manager.go:414] "containerMap: removing container" podUID="54703132-f756-4072-bb59-170c003ac16b" containerName="cleanup" Nov 03 13:30:58.191766 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.191606 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54703132-f756-4072-bb59-170c003ac16b" containerName="cleanup" Nov 03 13:30:58.191766 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.191627 5979 cpu_manager.go:414] "containerMap: removing container" podUID="54703132-f756-4072-bb59-170c003ac16b" containerName="cleanup" Nov 03 13:30:58.191766 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.191634 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54703132-f756-4072-bb59-170c003ac16b" containerName="cleanup" Nov 03 13:30:58.191766 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.191661 5979 cpu_manager.go:414] "containerMap: removing container" podUID="01e5557b-1b62-4d56-843e-1685de2da7f1" containerName="revoke-keys" Nov 03 13:30:58.191766 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.191669 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01e5557b-1b62-4d56-843e-1685de2da7f1" containerName="revoke-keys" Nov 03 13:30:58.191766 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.191682 5979 cpu_manager.go:414] "containerMap: removing container" podUID="f59b1527-f24d-47bb-bbd2-3de97661eeee" containerName="payload-pre-processing" Nov 03 13:30:58.191766 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.191690 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f59b1527-f24d-47bb-bbd2-3de97661eeee" containerName="payload-pre-processing" Nov 03 13:30:58.191766 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.191714 5979 cpu_manager.go:414] "containerMap: removing container" podUID="088feb81-52ce-4afc-8d07-aa9d2a094442" containerName="cleanup" Nov 03 13:30:58.191766 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.191721 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="088feb81-52ce-4afc-8d07-aa9d2a094442" containerName="cleanup" Nov 03 13:30:58.192971 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.191907 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c1ef57f-1651-4415-a8b1-1f1dff93e5d4" containerName="cleanup" Nov 03 13:30:58.192971 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.191920 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="2666652f-479a-44fe-8f0c-17bc354aaaec" containerName="cleanup" Nov 03 13:30:58.192971 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.191929 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef6ffd56-1244-4430-9740-436407eb0ebe" containerName="payload-processing" Nov 03 13:30:58.192971 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.191942 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c1ef57f-1651-4415-a8b1-1f1dff93e5d4" containerName="cleanup" Nov 03 13:30:58.192971 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.191953 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ab44829-1b15-408c-b6fd-647ba93cac2e" containerName="cleanup" Nov 03 13:30:58.192971 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.191963 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="54703132-f756-4072-bb59-170c003ac16b" containerName="cleanup" Nov 03 13:30:58.192971 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.191973 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="54703132-f756-4072-bb59-170c003ac16b" containerName="cleanup" Nov 03 13:30:58.192971 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.191984 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ab44829-1b15-408c-b6fd-647ba93cac2e" containerName="cleanup" Nov 03 13:30:58.192971 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.191994 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="01e5557b-1b62-4d56-843e-1685de2da7f1" containerName="revoke-keys" Nov 03 13:30:58.192971 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.192006 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="f59b1527-f24d-47bb-bbd2-3de97661eeee" containerName="payload-pre-processing" Nov 03 13:30:58.192971 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.192078 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="54703132-f756-4072-bb59-170c003ac16b" containerName="cleanup" Nov 03 13:30:58.192971 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.192091 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c1ef57f-1651-4415-a8b1-1f1dff93e5d4" containerName="cleanup" Nov 03 13:30:58.192971 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.192101 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="088feb81-52ce-4afc-8d07-aa9d2a094442" containerName="cleanup" Nov 03 13:30:58.192971 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.192112 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e268eef-9d3a-4459-a456-381085fdfa63" containerName="cleanup" Nov 03 13:30:58.192971 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.192121 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="088feb81-52ce-4afc-8d07-aa9d2a094442" containerName="cleanup" Nov 03 13:30:58.192971 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.192137 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="2666652f-479a-44fe-8f0c-17bc354aaaec" containerName="cleanup" Nov 03 13:30:58.192971 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.192146 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="2666652f-479a-44fe-8f0c-17bc354aaaec" containerName="cleanup" Nov 03 13:30:58.192971 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.192159 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e268eef-9d3a-4459-a456-381085fdfa63" containerName="cleanup" Nov 03 13:30:58.192971 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.192170 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e268eef-9d3a-4459-a456-381085fdfa63" containerName="cleanup" Nov 03 13:30:58.192971 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.192180 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="088feb81-52ce-4afc-8d07-aa9d2a094442" containerName="cleanup" Nov 03 13:30:58.198534 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.198513 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f090c4-z927b" Nov 03 13:30:58.234332 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.229563 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18f090c4-z927b"] Nov 03 13:30:58.267493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.254834 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 13:30:58.267493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.254947 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 13:30:58.267493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.255671 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:30:58.274517 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.271879 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:30:58.283828 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.283640 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76db9c1e-ad76-412e-b373-13cce12b2f0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18f090c4-z927b\" (UID: \"76db9c1e-ad76-412e-b373-13cce12b2f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f090c4-z927b" Nov 03 13:30:58.283828 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.283741 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76db9c1e-ad76-412e-b373-13cce12b2f0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18f090c4-z927b\" (UID: \"76db9c1e-ad76-412e-b373-13cce12b2f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f090c4-z927b" Nov 03 13:30:58.284732 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.284708 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmjcx\" (UniqueName: \"kubernetes.io/projected/76db9c1e-ad76-412e-b373-13cce12b2f0c-kube-api-access-cmjcx\") pod \"maas-api-revoke-sub-18f090c4-z927b\" (UID: \"76db9c1e-ad76-412e-b373-13cce12b2f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f090c4-z927b" Nov 03 13:30:58.387834 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.386569 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76db9c1e-ad76-412e-b373-13cce12b2f0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18f090c4-z927b\" (UID: \"76db9c1e-ad76-412e-b373-13cce12b2f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f090c4-z927b" Nov 03 13:30:58.387834 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.386678 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76db9c1e-ad76-412e-b373-13cce12b2f0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18f090c4-z927b\" (UID: \"76db9c1e-ad76-412e-b373-13cce12b2f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f090c4-z927b" Nov 03 13:30:58.387834 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.386883 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmjcx\" (UniqueName: \"kubernetes.io/projected/76db9c1e-ad76-412e-b373-13cce12b2f0c-kube-api-access-cmjcx\") pod \"maas-api-revoke-sub-18f090c4-z927b\" (UID: \"76db9c1e-ad76-412e-b373-13cce12b2f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f090c4-z927b" Nov 03 13:30:58.388913 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.388849 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76db9c1e-ad76-412e-b373-13cce12b2f0c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18f090c4-z927b\" (UID: \"76db9c1e-ad76-412e-b373-13cce12b2f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f090c4-z927b" Nov 03 13:30:58.391902 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.391868 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76db9c1e-ad76-412e-b373-13cce12b2f0c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18f090c4-z927b\" (UID: \"76db9c1e-ad76-412e-b373-13cce12b2f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f090c4-z927b" Nov 03 13:30:58.473502 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.471687 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmjcx\" (UniqueName: \"kubernetes.io/projected/76db9c1e-ad76-412e-b373-13cce12b2f0c-kube-api-access-cmjcx\") pod \"maas-api-revoke-sub-18f090c4-z927b\" (UID: \"76db9c1e-ad76-412e-b373-13cce12b2f0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f090c4-z927b" Nov 03 13:30:58.515414 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:58.515375 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f090c4-z927b" Nov 03 13:30:59.375565 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:30:59.372258 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18f090c4-z927b"] Nov 03 13:31:00.385183 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:00.385143 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f090c4-z927b" event={"ID":"76db9c1e-ad76-412e-b373-13cce12b2f0c","Type":"ContainerStarted","Data":"76eeebf7398bc59db926c5f1d80974c839f9e1eece61f8808de63574975c4b5c"} Nov 03 13:31:00.385670 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:00.385206 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f090c4-z927b" event={"ID":"76db9c1e-ad76-412e-b373-13cce12b2f0c","Type":"ContainerStarted","Data":"67b61698cabd852e3784c0077258f05ec24bc886d0df4beebd701b0411eed8f2"} Nov 03 13:31:00.513910 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:00.512627 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f090c4-z927b" podStartSLOduration=2.5126040659999997 podStartE2EDuration="2.512604066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:30:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:31:00.501075203 +0000 UTC m=+1542.511763446" watchObservedRunningTime="2026-11-03 13:31:00.512604066 +0000 UTC m=+1542.523292305" Nov 03 13:31:00.624114 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:00.624080 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5295ab29-zqqnv"] Nov 03 13:31:00.629484 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:00.629435 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5295ab29-zqqnv"] Nov 03 13:31:01.396445 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:01.396415 5979 generic.go:356] "Generic (PLEG): container finished" podID="76db9c1e-ad76-412e-b373-13cce12b2f0c" containerID="76eeebf7398bc59db926c5f1d80974c839f9e1eece61f8808de63574975c4b5c" exitCode=0 Nov 03 13:31:01.396911 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:01.396465 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f090c4-z927b" event={"ID":"76db9c1e-ad76-412e-b373-13cce12b2f0c","Type":"ContainerDied","Data":"76eeebf7398bc59db926c5f1d80974c839f9e1eece61f8808de63574975c4b5c"} Nov 03 13:31:02.119327 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:02.119297 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="416c0968-c727-4bbc-996a-a63c72331f51" path="/var/lib/kubelet/pods/416c0968-c727-4bbc-996a-a63c72331f51/volumes" Nov 03 13:31:02.586376 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:02.586342 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f090c4-z927b" Nov 03 13:31:02.710159 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:02.710064 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76db9c1e-ad76-412e-b373-13cce12b2f0c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76db9c1e-ad76-412e-b373-13cce12b2f0c-maas-api-xks-ca\") pod \"76db9c1e-ad76-412e-b373-13cce12b2f0c\" (UID: \"76db9c1e-ad76-412e-b373-13cce12b2f0c\") " Nov 03 13:31:02.710329 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:02.710216 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76db9c1e-ad76-412e-b373-13cce12b2f0c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76db9c1e-ad76-412e-b373-13cce12b2f0c-maas-api-openshift-ca\") pod \"76db9c1e-ad76-412e-b373-13cce12b2f0c\" (UID: \"76db9c1e-ad76-412e-b373-13cce12b2f0c\") " Nov 03 13:31:02.710460 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:02.710441 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76db9c1e-ad76-412e-b373-13cce12b2f0c-kube-api-access-cmjcx\" (UniqueName: \"kubernetes.io/projected/76db9c1e-ad76-412e-b373-13cce12b2f0c-kube-api-access-cmjcx\") pod \"76db9c1e-ad76-412e-b373-13cce12b2f0c\" (UID: \"76db9c1e-ad76-412e-b373-13cce12b2f0c\") " Nov 03 13:31:02.710619 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:02.710591 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76db9c1e-ad76-412e-b373-13cce12b2f0c-maas-api-openshift-ca" pod "76db9c1e-ad76-412e-b373-13cce12b2f0c" (UID: "76db9c1e-ad76-412e-b373-13cce12b2f0c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:31:02.711334 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:02.711305 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76db9c1e-ad76-412e-b373-13cce12b2f0c-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:02.712435 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:02.712403 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76db9c1e-ad76-412e-b373-13cce12b2f0c-maas-api-xks-ca" pod "76db9c1e-ad76-412e-b373-13cce12b2f0c" (UID: "76db9c1e-ad76-412e-b373-13cce12b2f0c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:31:02.713345 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:02.713319 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76db9c1e-ad76-412e-b373-13cce12b2f0c-kube-api-access-cmjcx" pod "76db9c1e-ad76-412e-b373-13cce12b2f0c" (UID: "76db9c1e-ad76-412e-b373-13cce12b2f0c"). InnerVolumeSpecName "kube-api-access-cmjcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:31:02.836494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:02.824150 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmjcx\" (UniqueName: \"kubernetes.io/projected/76db9c1e-ad76-412e-b373-13cce12b2f0c-kube-api-access-cmjcx\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:02.836494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:02.824200 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76db9c1e-ad76-412e-b373-13cce12b2f0c-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:03.422278 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:03.422240 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8"] Nov 03 13:31:03.425063 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:03.425031 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8" podUID="53fab3c0-d97b-4c5d-ad31-798afd7c5b6c" containerName="istio-proxy" containerID="cri-o://cce14b17abb038ab6260fc4aff6cffa4ae4ffe694ede6d492e2b8fa06d62b478" gracePeriod=30 Nov 03 13:31:03.436762 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:03.430055 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f090c4-z927b" event={"ID":"76db9c1e-ad76-412e-b373-13cce12b2f0c","Type":"ContainerDied","Data":"67b61698cabd852e3784c0077258f05ec24bc886d0df4beebd701b0411eed8f2"} Nov 03 13:31:03.436762 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:03.430107 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67b61698cabd852e3784c0077258f05ec24bc886d0df4beebd701b0411eed8f2" Nov 03 13:31:03.436762 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:03.430181 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18f090c4-z927b" Nov 03 13:31:03.439601 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:03.439568 5979 generic.go:356] "Generic (PLEG): container finished" podID="e5478d2d-1afc-4d18-9e91-e1965f3f51c0" containerID="4c37c527438e7778be040ed075fd4a776f6721feb812f6539f6939850020aba2" exitCode=0 Nov 03 13:31:03.439733 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:03.439664 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw" event={"ID":"e5478d2d-1afc-4d18-9e91-e1965f3f51c0","Type":"ContainerDied","Data":"4c37c527438e7778be040ed075fd4a776f6721feb812f6539f6939850020aba2"} Nov 03 13:31:03.440200 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:03.440177 5979 scope.go:122] "RemoveContainer" containerID="4c37c527438e7778be040ed075fd4a776f6721feb812f6539f6939850020aba2" Nov 03 13:31:03.442284 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:03.442245 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:31:03.442364 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:03.442346 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:31:03.442417 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:03.442400 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:31:04.466495 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:04.458660 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw" event={"ID":"e5478d2d-1afc-4d18-9e91-e1965f3f51c0","Type":"ContainerStarted","Data":"be2293debe68f293b71ce940c4694059c4861b51eabf8ea268630439a72c0a2a"} Nov 03 13:31:04.759002 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:04.758899 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw" Nov 03 13:31:04.760407 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:04.759307 5979 patch_prober.go:29] interesting pod/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 03 13:31:04.760661 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:04.760636 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw" podUID="e5478d2d-1afc-4d18-9e91-e1965f3f51c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 03 13:31:05.757879 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:05.757840 5979 patch_prober.go:29] interesting pod/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 03 13:31:05.758537 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:05.758502 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw" podUID="e5478d2d-1afc-4d18-9e91-e1965f3f51c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 03 13:31:06.768683 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:06.768640 5979 patch_prober.go:29] interesting pod/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 03 13:31:06.769189 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:06.768703 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw" podUID="e5478d2d-1afc-4d18-9e91-e1965f3f51c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 03 13:31:07.234774 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:31:07.234731 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:31:07.574694 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:07.574603 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk"] Nov 03 13:31:07.574921 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:07.574896 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk" podUID="5211f1d9-4181-4ad0-b83a-4e4e7a556be8" containerName="istio-proxy" containerID="cri-o://95d28afb5b88bab8ea99934962ff3b61b20bda95db7fa5baff3736a423a418b8" gracePeriod=30 Nov 03 13:31:07.768592 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:07.758847 5979 patch_prober.go:29] interesting pod/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 03 13:31:07.768592 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:07.758911 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw" podUID="e5478d2d-1afc-4d18-9e91-e1965f3f51c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 03 13:31:08.121058 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:08.121026 5979 patch_prober.go:29] interesting pod/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 13:31:08.121224 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:08.121075 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8" podUID="53fab3c0-d97b-4c5d-ad31-798afd7c5b6c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 13:31:08.769542 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:08.768680 5979 patch_prober.go:29] interesting pod/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 03 13:31:08.770078 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:08.770046 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw" podUID="e5478d2d-1afc-4d18-9e91-e1965f3f51c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 03 13:31:08.783322 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:08.783299 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8" Nov 03 13:31:08.796460 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:08.796429 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-kube-api-access-wxl67\" (UniqueName: \"kubernetes.io/projected/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-kube-api-access-wxl67\") pod \"53fab3c0-d97b-4c5d-ad31-798afd7c5b6c\" (UID: \"53fab3c0-d97b-4c5d-ad31-798afd7c5b6c\") " Nov 03 13:31:08.796623 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:08.796500 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-istio-podinfo\") pod \"53fab3c0-d97b-4c5d-ad31-798afd7c5b6c\" (UID: \"53fab3c0-d97b-4c5d-ad31-798afd7c5b6c\") " Nov 03 13:31:08.796623 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:08.796561 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-istiod-ca-cert\") pod \"53fab3c0-d97b-4c5d-ad31-798afd7c5b6c\" (UID: \"53fab3c0-d97b-4c5d-ad31-798afd7c5b6c\") " Nov 03 13:31:08.796749 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:08.796717 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-workload-socket\") pod \"53fab3c0-d97b-4c5d-ad31-798afd7c5b6c\" (UID: \"53fab3c0-d97b-4c5d-ad31-798afd7c5b6c\") " Nov 03 13:31:08.796809 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:08.796791 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-istio-data\") pod \"53fab3c0-d97b-4c5d-ad31-798afd7c5b6c\" (UID: \"53fab3c0-d97b-4c5d-ad31-798afd7c5b6c\") " Nov 03 13:31:08.796868 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:08.796829 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-istio-envoy\") pod \"53fab3c0-d97b-4c5d-ad31-798afd7c5b6c\" (UID: \"53fab3c0-d97b-4c5d-ad31-798afd7c5b6c\") " Nov 03 13:31:08.796868 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:08.796862 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-credential-socket\") pod \"53fab3c0-d97b-4c5d-ad31-798afd7c5b6c\" (UID: \"53fab3c0-d97b-4c5d-ad31-798afd7c5b6c\") " Nov 03 13:31:08.796977 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:08.796910 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-workload-certs\") pod \"53fab3c0-d97b-4c5d-ad31-798afd7c5b6c\" (UID: \"53fab3c0-d97b-4c5d-ad31-798afd7c5b6c\") " Nov 03 13:31:08.797026 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:08.796974 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-istio-token\" (UniqueName: \"kubernetes.io/projected/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-istio-token\") pod \"53fab3c0-d97b-4c5d-ad31-798afd7c5b6c\" (UID: \"53fab3c0-d97b-4c5d-ad31-798afd7c5b6c\") " Nov 03 13:31:08.797499 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:08.797451 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-istio-data" pod "53fab3c0-d97b-4c5d-ad31-798afd7c5b6c" (UID: "53fab3c0-d97b-4c5d-ad31-798afd7c5b6c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:08.798363 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:08.797763 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-workload-socket" pod "53fab3c0-d97b-4c5d-ad31-798afd7c5b6c" (UID: "53fab3c0-d97b-4c5d-ad31-798afd7c5b6c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:08.798363 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:08.797823 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-credential-socket" pod "53fab3c0-d97b-4c5d-ad31-798afd7c5b6c" (UID: "53fab3c0-d97b-4c5d-ad31-798afd7c5b6c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:08.798363 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:08.797876 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-workload-certs" pod "53fab3c0-d97b-4c5d-ad31-798afd7c5b6c" (UID: "53fab3c0-d97b-4c5d-ad31-798afd7c5b6c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:08.798363 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:08.798274 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:08.798363 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:08.798299 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:08.798363 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:08.798312 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:08.798363 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:08.798327 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:08.798363 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:08.798330 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-istiod-ca-cert" pod "53fab3c0-d97b-4c5d-ad31-798afd7c5b6c" (UID: "53fab3c0-d97b-4c5d-ad31-798afd7c5b6c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:31:08.799259 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:08.799221 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-kube-api-access-wxl67" pod "53fab3c0-d97b-4c5d-ad31-798afd7c5b6c" (UID: "53fab3c0-d97b-4c5d-ad31-798afd7c5b6c"). InnerVolumeSpecName "kube-api-access-wxl67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:31:08.799741 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:08.799721 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-istio-envoy" pod "53fab3c0-d97b-4c5d-ad31-798afd7c5b6c" (UID: "53fab3c0-d97b-4c5d-ad31-798afd7c5b6c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:08.799843 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:08.799828 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-istio-token" pod "53fab3c0-d97b-4c5d-ad31-798afd7c5b6c" (UID: "53fab3c0-d97b-4c5d-ad31-798afd7c5b6c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:31:08.800918 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:08.800900 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-istio-podinfo" pod "53fab3c0-d97b-4c5d-ad31-798afd7c5b6c" (UID: "53fab3c0-d97b-4c5d-ad31-798afd7c5b6c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:31:08.899902 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:08.899866 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:08.899902 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:08.899902 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:08.900118 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:08.899914 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxl67\" (UniqueName: \"kubernetes.io/projected/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-kube-api-access-wxl67\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:08.900118 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:08.899923 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:08.900118 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:08.899932 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:09.073676 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:09.073595 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51b4988d-24mw5"] Nov 03 13:31:09.074393 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:09.074365 5979 cpu_manager.go:414] "containerMap: removing container" podUID="8e268eef-9d3a-4459-a456-381085fdfa63" containerName="cleanup" Nov 03 13:31:09.074393 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:09.074388 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e268eef-9d3a-4459-a456-381085fdfa63" containerName="cleanup" Nov 03 13:31:09.074393 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:09.074398 5979 cpu_manager.go:414] "containerMap: removing container" podUID="088feb81-52ce-4afc-8d07-aa9d2a094442" containerName="cleanup" Nov 03 13:31:09.074676 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:09.074406 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="088feb81-52ce-4afc-8d07-aa9d2a094442" containerName="cleanup" Nov 03 13:31:09.074676 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:09.074421 5979 cpu_manager.go:414] "containerMap: removing container" podUID="76db9c1e-ad76-412e-b373-13cce12b2f0c" containerName="revoke-keys" Nov 03 13:31:09.074676 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:09.074427 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76db9c1e-ad76-412e-b373-13cce12b2f0c" containerName="revoke-keys" Nov 03 13:31:09.074676 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:09.074438 5979 cpu_manager.go:414] "containerMap: removing container" podUID="0c1ef57f-1651-4415-a8b1-1f1dff93e5d4" containerName="cleanup" Nov 03 13:31:09.074676 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:09.074444 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c1ef57f-1651-4415-a8b1-1f1dff93e5d4" containerName="cleanup" Nov 03 13:31:09.074676 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:09.074456 5979 cpu_manager.go:414] "containerMap: removing container" podUID="53fab3c0-d97b-4c5d-ad31-798afd7c5b6c" containerName="istio-proxy" Nov 03 13:31:09.074676 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:09.074461 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53fab3c0-d97b-4c5d-ad31-798afd7c5b6c" containerName="istio-proxy" Nov 03 13:31:09.074676 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:09.074482 5979 cpu_manager.go:414] "containerMap: removing container" podUID="9ab44829-1b15-408c-b6fd-647ba93cac2e" containerName="cleanup" Nov 03 13:31:09.074676 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:09.074490 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ab44829-1b15-408c-b6fd-647ba93cac2e" containerName="cleanup" Nov 03 13:31:09.074676 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:09.074501 5979 cpu_manager.go:414] "containerMap: removing container" podUID="53fab3c0-d97b-4c5d-ad31-798afd7c5b6c" containerName="istio-proxy" Nov 03 13:31:09.074676 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:09.074506 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53fab3c0-d97b-4c5d-ad31-798afd7c5b6c" containerName="istio-proxy" Nov 03 13:31:09.074676 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:09.074541 5979 cpu_manager.go:414] "containerMap: removing container" podUID="54703132-f756-4072-bb59-170c003ac16b" containerName="cleanup" Nov 03 13:31:09.074676 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:09.074549 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54703132-f756-4072-bb59-170c003ac16b" containerName="cleanup" Nov 03 13:31:09.074676 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:09.074616 5979 cpu_manager.go:414] "containerMap: removing container" podUID="2666652f-479a-44fe-8f0c-17bc354aaaec" containerName="cleanup" Nov 03 13:31:09.074676 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:09.074624 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2666652f-479a-44fe-8f0c-17bc354aaaec" containerName="cleanup" Nov 03 13:31:09.075233 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:09.074758 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="53fab3c0-d97b-4c5d-ad31-798afd7c5b6c" containerName="istio-proxy" Nov 03 13:31:09.075233 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:09.074773 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="53fab3c0-d97b-4c5d-ad31-798afd7c5b6c" containerName="istio-proxy" Nov 03 13:31:09.075233 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:09.074789 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ab44829-1b15-408c-b6fd-647ba93cac2e" containerName="cleanup" Nov 03 13:31:09.075233 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:09.074808 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="76db9c1e-ad76-412e-b373-13cce12b2f0c" containerName="revoke-keys" Nov 03 13:31:09.077933 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:09.077916 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51b4988d-24mw5" Nov 03 13:31:09.102389 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:09.102358 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f24a588-62a8-43fa-aa2a-40c6f1e55097-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51b4988d-24mw5\" (UID: \"3f24a588-62a8-43fa-aa2a-40c6f1e55097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51b4988d-24mw5" Nov 03 13:31:09.102599 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:09.102572 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f45m8\" (UniqueName: \"kubernetes.io/projected/3f24a588-62a8-43fa-aa2a-40c6f1e55097-kube-api-access-f45m8\") pod \"maas-api-revoke-sub-51b4988d-24mw5\" (UID: \"3f24a588-62a8-43fa-aa2a-40c6f1e55097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51b4988d-24mw5" Nov 03 13:31:09.102718 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:09.102650 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f24a588-62a8-43fa-aa2a-40c6f1e55097-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51b4988d-24mw5\" (UID: \"3f24a588-62a8-43fa-aa2a-40c6f1e55097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51b4988d-24mw5" Nov 03 13:31:09.124331 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:09.124304 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51b4988d-24mw5"] Nov 03 13:31:09.127049 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:09.127023 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:31:09.127929 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:09.127912 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:31:09.139975 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:09.139954 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 13:31:09.139975 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:09.139965 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 13:31:09.203920 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:09.203884 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f45m8\" (UniqueName: \"kubernetes.io/projected/3f24a588-62a8-43fa-aa2a-40c6f1e55097-kube-api-access-f45m8\") pod \"maas-api-revoke-sub-51b4988d-24mw5\" (UID: \"3f24a588-62a8-43fa-aa2a-40c6f1e55097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51b4988d-24mw5" Nov 03 13:31:09.203920 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:09.203924 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f24a588-62a8-43fa-aa2a-40c6f1e55097-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51b4988d-24mw5\" (UID: \"3f24a588-62a8-43fa-aa2a-40c6f1e55097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51b4988d-24mw5" Nov 03 13:31:09.204165 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:09.204087 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f24a588-62a8-43fa-aa2a-40c6f1e55097-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51b4988d-24mw5\" (UID: \"3f24a588-62a8-43fa-aa2a-40c6f1e55097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51b4988d-24mw5" Nov 03 13:31:09.204639 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:09.204615 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f24a588-62a8-43fa-aa2a-40c6f1e55097-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51b4988d-24mw5\" (UID: \"3f24a588-62a8-43fa-aa2a-40c6f1e55097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51b4988d-24mw5" Nov 03 13:31:09.206329 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:09.206311 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f24a588-62a8-43fa-aa2a-40c6f1e55097-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51b4988d-24mw5\" (UID: \"3f24a588-62a8-43fa-aa2a-40c6f1e55097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51b4988d-24mw5" Nov 03 13:31:09.271594 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:09.271558 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f45m8\" (UniqueName: \"kubernetes.io/projected/3f24a588-62a8-43fa-aa2a-40c6f1e55097-kube-api-access-f45m8\") pod \"maas-api-revoke-sub-51b4988d-24mw5\" (UID: \"3f24a588-62a8-43fa-aa2a-40c6f1e55097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51b4988d-24mw5" Nov 03 13:31:09.393929 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:09.393896 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51b4988d-24mw5" Nov 03 13:31:09.514311 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:09.514281 5979 generic.go:356] "Generic (PLEG): container finished" podID="53fab3c0-d97b-4c5d-ad31-798afd7c5b6c" containerID="cce14b17abb038ab6260fc4aff6cffa4ae4ffe694ede6d492e2b8fa06d62b478" exitCode=0 Nov 03 13:31:09.514746 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:09.514365 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8" event={"ID":"53fab3c0-d97b-4c5d-ad31-798afd7c5b6c","Type":"ContainerDied","Data":"cce14b17abb038ab6260fc4aff6cffa4ae4ffe694ede6d492e2b8fa06d62b478"} Nov 03 13:31:09.514746 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:09.514614 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8" event={"ID":"53fab3c0-d97b-4c5d-ad31-798afd7c5b6c","Type":"ContainerDied","Data":"c6baab293447a30adf71f78758975b18d45ff29684cbc73f8a62a200e600f7c4"} Nov 03 13:31:09.514746 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:09.514639 5979 scope.go:122] "RemoveContainer" containerID="cce14b17abb038ab6260fc4aff6cffa4ae4ffe694ede6d492e2b8fa06d62b478" Nov 03 13:31:09.514746 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:09.514385 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8" Nov 03 13:31:09.563690 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:09.550644 5979 scope.go:122] "RemoveContainer" containerID="ff7a2dcb437a5a04242e4663480438260cbdca082dd0eb14c0da32c2e7fee8f7" Nov 03 13:31:09.588881 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:09.588853 5979 scope.go:122] "RemoveContainer" containerID="cce14b17abb038ab6260fc4aff6cffa4ae4ffe694ede6d492e2b8fa06d62b478" Nov 03 13:31:09.589337 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:31:09.589196 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cce14b17abb038ab6260fc4aff6cffa4ae4ffe694ede6d492e2b8fa06d62b478\": container with ID starting with cce14b17abb038ab6260fc4aff6cffa4ae4ffe694ede6d492e2b8fa06d62b478 not found: ID does not exist" containerID="cce14b17abb038ab6260fc4aff6cffa4ae4ffe694ede6d492e2b8fa06d62b478" Nov 03 13:31:09.589337 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:09.589235 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cce14b17abb038ab6260fc4aff6cffa4ae4ffe694ede6d492e2b8fa06d62b478"} err="failed to get container status \"cce14b17abb038ab6260fc4aff6cffa4ae4ffe694ede6d492e2b8fa06d62b478\": rpc error: code = NotFound desc = could not find container \"cce14b17abb038ab6260fc4aff6cffa4ae4ffe694ede6d492e2b8fa06d62b478\": container with ID starting with cce14b17abb038ab6260fc4aff6cffa4ae4ffe694ede6d492e2b8fa06d62b478 not found: ID does not exist" Nov 03 13:31:09.589337 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:09.589260 5979 scope.go:122] "RemoveContainer" containerID="ff7a2dcb437a5a04242e4663480438260cbdca082dd0eb14c0da32c2e7fee8f7" Nov 03 13:31:09.589610 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:31:09.589585 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff7a2dcb437a5a04242e4663480438260cbdca082dd0eb14c0da32c2e7fee8f7\": container with ID starting with ff7a2dcb437a5a04242e4663480438260cbdca082dd0eb14c0da32c2e7fee8f7 not found: ID does not exist" containerID="ff7a2dcb437a5a04242e4663480438260cbdca082dd0eb14c0da32c2e7fee8f7" Nov 03 13:31:09.589689 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:09.589620 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff7a2dcb437a5a04242e4663480438260cbdca082dd0eb14c0da32c2e7fee8f7"} err="failed to get container status \"ff7a2dcb437a5a04242e4663480438260cbdca082dd0eb14c0da32c2e7fee8f7\": rpc error: code = NotFound desc = could not find container \"ff7a2dcb437a5a04242e4663480438260cbdca082dd0eb14c0da32c2e7fee8f7\": container with ID starting with ff7a2dcb437a5a04242e4663480438260cbdca082dd0eb14c0da32c2e7fee8f7 not found: ID does not exist" Nov 03 13:31:09.675599 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:09.675515 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8"] Nov 03 13:31:09.721729 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:09.721692 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-30074e-openshift-default-79c784cd47-5nhh8"] Nov 03 13:31:09.758646 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:09.758607 5979 patch_prober.go:29] interesting pod/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Nov 03 13:31:09.758847 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:09.758665 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw" podUID="e5478d2d-1afc-4d18-9e91-e1965f3f51c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Nov 03 13:31:09.791512 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:09.791467 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51b4988d-24mw5"] Nov 03 13:31:09.799247 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:31:09.799220 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f24a588_62a8_43fa_aa2a_40c6f1e55097.slice/crio-695202a40d034ff427b90ff6368767c81790e88bf9e80657a2da397af5e446a7 WatchSource:0}: Error finding container 695202a40d034ff427b90ff6368767c81790e88bf9e80657a2da397af5e446a7: Status 404 returned error can't find the container with id 695202a40d034ff427b90ff6368767c81790e88bf9e80657a2da397af5e446a7 Nov 03 13:31:10.116580 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:10.116546 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53fab3c0-d97b-4c5d-ad31-798afd7c5b6c" path="/var/lib/kubelet/pods/53fab3c0-d97b-4c5d-ad31-798afd7c5b6c/volumes" Nov 03 13:31:10.524525 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:10.523898 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51b4988d-24mw5" event={"ID":"3f24a588-62a8-43fa-aa2a-40c6f1e55097","Type":"ContainerStarted","Data":"89e0e2d1884b48753ce12355a85f607bd069a97190b666803bca5f1ed5225809"} Nov 03 13:31:10.524525 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:10.523967 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51b4988d-24mw5" event={"ID":"3f24a588-62a8-43fa-aa2a-40c6f1e55097","Type":"ContainerStarted","Data":"695202a40d034ff427b90ff6368767c81790e88bf9e80657a2da397af5e446a7"} Nov 03 13:31:10.635146 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:10.635087 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51b4988d-24mw5" podStartSLOduration=1.635069436 podStartE2EDuration="1.635069436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:31:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:31:10.631605944 +0000 UTC m=+1552.642294183" watchObservedRunningTime="2026-11-03 13:31:10.635069436 +0000 UTC m=+1552.645757680" Nov 03 13:31:10.762822 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:10.762791 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw" Nov 03 13:31:10.763224 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:10.763200 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw" Nov 03 13:31:10.764011 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:10.763994 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw" Nov 03 13:31:11.534829 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:11.534794 5979 generic.go:356] "Generic (PLEG): container finished" podID="3f24a588-62a8-43fa-aa2a-40c6f1e55097" containerID="89e0e2d1884b48753ce12355a85f607bd069a97190b666803bca5f1ed5225809" exitCode=0 Nov 03 13:31:11.535282 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:11.534880 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51b4988d-24mw5" event={"ID":"3f24a588-62a8-43fa-aa2a-40c6f1e55097","Type":"ContainerDied","Data":"89e0e2d1884b48753ce12355a85f607bd069a97190b666803bca5f1ed5225809"} Nov 03 13:31:11.746494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:11.727348 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5295ab29-6d58657867-ls4rq"] Nov 03 13:31:11.796795 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:11.796717 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5295ab29-6d58657867-ls4rq"] Nov 03 13:31:11.799423 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:11.799397 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-abe0e0-6d6c498cf9-lktgw"] Nov 03 13:31:11.799863 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:11.798677 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5295ab29-6d58657867-ls4rq" Nov 03 13:31:11.850504 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:11.850454 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-abe0e0-766d597f55-t5mk2"] Nov 03 13:31:11.850893 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:11.850865 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e421da61-cb60-47e5-8da3-936ec53af260-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5295ab29-6d58657867-ls4rq\" (UID: \"e421da61-cb60-47e5-8da3-936ec53af260\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5295ab29-6d58657867-ls4rq" Nov 03 13:31:11.851405 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:11.851376 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdn2k\" (UniqueName: \"kubernetes.io/projected/e421da61-cb60-47e5-8da3-936ec53af260-kube-api-access-jdn2k\") pod \"payload-pre-processing-e2e-mt-5295ab29-6d58657867-ls4rq\" (UID: \"e421da61-cb60-47e5-8da3-936ec53af260\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5295ab29-6d58657867-ls4rq" Nov 03 13:31:11.853076 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:11.853054 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-5295ab29-dockercfg-hjdsj\"" Nov 03 13:31:11.853194 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:11.853116 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-5295ab29\"" Nov 03 13:31:11.872880 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:11.872853 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5295ab29-79c7656b49-gxvjq"] Nov 03 13:31:11.876285 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:11.876264 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5295ab29-79c7656b49-gxvjq" Nov 03 13:31:11.934394 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:11.934358 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5295ab29-79c7656b49-gxvjq"] Nov 03 13:31:11.953216 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:11.953179 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvhff\" (UniqueName: \"kubernetes.io/projected/96afb289-d271-43d2-aeb8-3deaa841af80-kube-api-access-qvhff\") pod \"payload-processing-e2e-mt-5295ab29-79c7656b49-gxvjq\" (UID: \"96afb289-d271-43d2-aeb8-3deaa841af80\") " pod="openshift-ingress/payload-processing-e2e-mt-5295ab29-79c7656b49-gxvjq" Nov 03 13:31:11.953375 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:11.953334 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96afb289-d271-43d2-aeb8-3deaa841af80-plugins-config-volume\") pod \"payload-processing-e2e-mt-5295ab29-79c7656b49-gxvjq\" (UID: \"96afb289-d271-43d2-aeb8-3deaa841af80\") " pod="openshift-ingress/payload-processing-e2e-mt-5295ab29-79c7656b49-gxvjq" Nov 03 13:31:11.953506 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:11.953455 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdn2k\" (UniqueName: \"kubernetes.io/projected/e421da61-cb60-47e5-8da3-936ec53af260-kube-api-access-jdn2k\") pod \"payload-pre-processing-e2e-mt-5295ab29-6d58657867-ls4rq\" (UID: \"e421da61-cb60-47e5-8da3-936ec53af260\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5295ab29-6d58657867-ls4rq" Nov 03 13:31:11.953658 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:11.953552 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e421da61-cb60-47e5-8da3-936ec53af260-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5295ab29-6d58657867-ls4rq\" (UID: \"e421da61-cb60-47e5-8da3-936ec53af260\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5295ab29-6d58657867-ls4rq" Nov 03 13:31:11.954692 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:11.954567 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e421da61-cb60-47e5-8da3-936ec53af260-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5295ab29-6d58657867-ls4rq\" (UID: \"e421da61-cb60-47e5-8da3-936ec53af260\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5295ab29-6d58657867-ls4rq" Nov 03 13:31:12.011433 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:12.011393 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdn2k\" (UniqueName: \"kubernetes.io/projected/e421da61-cb60-47e5-8da3-936ec53af260-kube-api-access-jdn2k\") pod \"payload-pre-processing-e2e-mt-5295ab29-6d58657867-ls4rq\" (UID: \"e421da61-cb60-47e5-8da3-936ec53af260\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5295ab29-6d58657867-ls4rq" Nov 03 13:31:12.054799 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:12.054728 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvhff\" (UniqueName: \"kubernetes.io/projected/96afb289-d271-43d2-aeb8-3deaa841af80-kube-api-access-qvhff\") pod \"payload-processing-e2e-mt-5295ab29-79c7656b49-gxvjq\" (UID: \"96afb289-d271-43d2-aeb8-3deaa841af80\") " pod="openshift-ingress/payload-processing-e2e-mt-5295ab29-79c7656b49-gxvjq" Nov 03 13:31:12.054799 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:12.054796 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96afb289-d271-43d2-aeb8-3deaa841af80-plugins-config-volume\") pod \"payload-processing-e2e-mt-5295ab29-79c7656b49-gxvjq\" (UID: \"96afb289-d271-43d2-aeb8-3deaa841af80\") " pod="openshift-ingress/payload-processing-e2e-mt-5295ab29-79c7656b49-gxvjq" Nov 03 13:31:12.055442 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:12.055423 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96afb289-d271-43d2-aeb8-3deaa841af80-plugins-config-volume\") pod \"payload-processing-e2e-mt-5295ab29-79c7656b49-gxvjq\" (UID: \"96afb289-d271-43d2-aeb8-3deaa841af80\") " pod="openshift-ingress/payload-processing-e2e-mt-5295ab29-79c7656b49-gxvjq" Nov 03 13:31:12.110858 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:12.110831 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvhff\" (UniqueName: \"kubernetes.io/projected/96afb289-d271-43d2-aeb8-3deaa841af80-kube-api-access-qvhff\") pod \"payload-processing-e2e-mt-5295ab29-79c7656b49-gxvjq\" (UID: \"96afb289-d271-43d2-aeb8-3deaa841af80\") " pod="openshift-ingress/payload-processing-e2e-mt-5295ab29-79c7656b49-gxvjq" Nov 03 13:31:12.121924 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:12.121902 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5295ab29-6d58657867-ls4rq" Nov 03 13:31:12.195803 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:12.195537 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5295ab29-79c7656b49-gxvjq" Nov 03 13:31:12.654141 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:12.654111 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5295ab29-6d58657867-ls4rq"] Nov 03 13:31:12.657249 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:31:12.657217 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode421da61_cb60_47e5_8da3_936ec53af260.slice/crio-6d49123dbc365325d405f7d3dca7fb829a9df731535e0db1bb6a58d466de2cb0 WatchSource:0}: Error finding container 6d49123dbc365325d405f7d3dca7fb829a9df731535e0db1bb6a58d466de2cb0: Status 404 returned error can't find the container with id 6d49123dbc365325d405f7d3dca7fb829a9df731535e0db1bb6a58d466de2cb0 Nov 03 13:31:12.685606 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:12.685565 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5295ab29-79c7656b49-gxvjq"] Nov 03 13:31:12.715053 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:31:12.715025 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96afb289_d271_43d2_aeb8_3deaa841af80.slice/crio-f6976eedb8d63a6627b14e8831b73349b69783dbfe97335578e0a6fe3120a6f6 WatchSource:0}: Error finding container f6976eedb8d63a6627b14e8831b73349b69783dbfe97335578e0a6fe3120a6f6: Status 404 returned error can't find the container with id f6976eedb8d63a6627b14e8831b73349b69783dbfe97335578e0a6fe3120a6f6 Nov 03 13:31:12.849144 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:12.849119 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51b4988d-24mw5" Nov 03 13:31:13.040066 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:13.040033 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f24a588-62a8-43fa-aa2a-40c6f1e55097-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f24a588-62a8-43fa-aa2a-40c6f1e55097-maas-api-openshift-ca\") pod \"3f24a588-62a8-43fa-aa2a-40c6f1e55097\" (UID: \"3f24a588-62a8-43fa-aa2a-40c6f1e55097\") " Nov 03 13:31:13.040205 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:13.040182 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f24a588-62a8-43fa-aa2a-40c6f1e55097-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f24a588-62a8-43fa-aa2a-40c6f1e55097-maas-api-xks-ca\") pod \"3f24a588-62a8-43fa-aa2a-40c6f1e55097\" (UID: \"3f24a588-62a8-43fa-aa2a-40c6f1e55097\") " Nov 03 13:31:13.040315 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:13.040298 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f24a588-62a8-43fa-aa2a-40c6f1e55097-kube-api-access-f45m8\" (UniqueName: \"kubernetes.io/projected/3f24a588-62a8-43fa-aa2a-40c6f1e55097-kube-api-access-f45m8\") pod \"3f24a588-62a8-43fa-aa2a-40c6f1e55097\" (UID: \"3f24a588-62a8-43fa-aa2a-40c6f1e55097\") " Nov 03 13:31:13.040593 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:13.040547 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f24a588-62a8-43fa-aa2a-40c6f1e55097-maas-api-openshift-ca" pod "3f24a588-62a8-43fa-aa2a-40c6f1e55097" (UID: "3f24a588-62a8-43fa-aa2a-40c6f1e55097"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:31:13.041465 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:13.041435 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f24a588-62a8-43fa-aa2a-40c6f1e55097-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:13.042859 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:13.042831 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f24a588-62a8-43fa-aa2a-40c6f1e55097-maas-api-xks-ca" pod "3f24a588-62a8-43fa-aa2a-40c6f1e55097" (UID: "3f24a588-62a8-43fa-aa2a-40c6f1e55097"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:31:13.042965 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:13.042942 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f24a588-62a8-43fa-aa2a-40c6f1e55097-kube-api-access-f45m8" pod "3f24a588-62a8-43fa-aa2a-40c6f1e55097" (UID: "3f24a588-62a8-43fa-aa2a-40c6f1e55097"). InnerVolumeSpecName "kube-api-access-f45m8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:31:13.143249 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:13.143212 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f24a588-62a8-43fa-aa2a-40c6f1e55097-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:13.143249 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:13.143243 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f45m8\" (UniqueName: \"kubernetes.io/projected/3f24a588-62a8-43fa-aa2a-40c6f1e55097-kube-api-access-f45m8\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:13.333662 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:13.333635 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk" Nov 03 13:31:13.448083 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:13.448040 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-workload-certs\") pod \"5211f1d9-4181-4ad0-b83a-4e4e7a556be8\" (UID: \"5211f1d9-4181-4ad0-b83a-4e4e7a556be8\") " Nov 03 13:31:13.448271 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:13.448096 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-istiod-ca-cert\") pod \"5211f1d9-4181-4ad0-b83a-4e4e7a556be8\" (UID: \"5211f1d9-4181-4ad0-b83a-4e4e7a556be8\") " Nov 03 13:31:13.448271 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:13.448118 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-istio-data\") pod \"5211f1d9-4181-4ad0-b83a-4e4e7a556be8\" (UID: \"5211f1d9-4181-4ad0-b83a-4e4e7a556be8\") " Nov 03 13:31:13.448271 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:13.448237 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-credential-socket\") pod \"5211f1d9-4181-4ad0-b83a-4e4e7a556be8\" (UID: \"5211f1d9-4181-4ad0-b83a-4e4e7a556be8\") " Nov 03 13:31:13.448271 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:13.448265 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-workload-socket\") pod \"5211f1d9-4181-4ad0-b83a-4e4e7a556be8\" (UID: \"5211f1d9-4181-4ad0-b83a-4e4e7a556be8\") " Nov 03 13:31:13.448562 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:13.448381 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-kube-api-access-2k8dt\" (UniqueName: \"kubernetes.io/projected/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-kube-api-access-2k8dt\") pod \"5211f1d9-4181-4ad0-b83a-4e4e7a556be8\" (UID: \"5211f1d9-4181-4ad0-b83a-4e4e7a556be8\") " Nov 03 13:31:13.448562 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:13.448391 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-workload-certs" pod "5211f1d9-4181-4ad0-b83a-4e4e7a556be8" (UID: "5211f1d9-4181-4ad0-b83a-4e4e7a556be8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:13.448562 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:13.448490 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-istio-podinfo\") pod \"5211f1d9-4181-4ad0-b83a-4e4e7a556be8\" (UID: \"5211f1d9-4181-4ad0-b83a-4e4e7a556be8\") " Nov 03 13:31:13.448562 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:13.448544 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-istio-envoy\") pod \"5211f1d9-4181-4ad0-b83a-4e4e7a556be8\" (UID: \"5211f1d9-4181-4ad0-b83a-4e4e7a556be8\") " Nov 03 13:31:13.448785 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:13.448568 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-istio-data" pod "5211f1d9-4181-4ad0-b83a-4e4e7a556be8" (UID: "5211f1d9-4181-4ad0-b83a-4e4e7a556be8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:13.448785 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:13.448615 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-istio-token\" (UniqueName: \"kubernetes.io/projected/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-istio-token\") pod \"5211f1d9-4181-4ad0-b83a-4e4e7a556be8\" (UID: \"5211f1d9-4181-4ad0-b83a-4e4e7a556be8\") " Nov 03 13:31:13.449116 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:13.449093 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-credential-socket" pod "5211f1d9-4181-4ad0-b83a-4e4e7a556be8" (UID: "5211f1d9-4181-4ad0-b83a-4e4e7a556be8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:13.449521 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:13.449118 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-istiod-ca-cert" pod "5211f1d9-4181-4ad0-b83a-4e4e7a556be8" (UID: "5211f1d9-4181-4ad0-b83a-4e4e7a556be8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:31:13.449634 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:13.449555 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-workload-socket" pod "5211f1d9-4181-4ad0-b83a-4e4e7a556be8" (UID: "5211f1d9-4181-4ad0-b83a-4e4e7a556be8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:13.451452 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:13.451428 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:13.451604 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:13.451456 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:13.451604 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:13.451483 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:13.451604 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:13.451497 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:13.451604 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:13.451511 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:13.451872 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:13.451847 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-istio-token" pod "5211f1d9-4181-4ad0-b83a-4e4e7a556be8" (UID: "5211f1d9-4181-4ad0-b83a-4e4e7a556be8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:31:13.451984 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:13.451934 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-istio-podinfo" pod "5211f1d9-4181-4ad0-b83a-4e4e7a556be8" (UID: "5211f1d9-4181-4ad0-b83a-4e4e7a556be8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:31:13.453027 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:13.453003 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-kube-api-access-2k8dt" pod "5211f1d9-4181-4ad0-b83a-4e4e7a556be8" (UID: "5211f1d9-4181-4ad0-b83a-4e4e7a556be8"). InnerVolumeSpecName "kube-api-access-2k8dt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:31:13.453101 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:13.453046 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-istio-envoy" pod "5211f1d9-4181-4ad0-b83a-4e4e7a556be8" (UID: "5211f1d9-4181-4ad0-b83a-4e4e7a556be8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:13.556270 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:13.554498 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:13.556270 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:13.554533 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:13.556270 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:13.554555 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:13.556270 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:13.554570 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k8dt\" (UniqueName: \"kubernetes.io/projected/5211f1d9-4181-4ad0-b83a-4e4e7a556be8-kube-api-access-2k8dt\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:13.710574 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:13.698627 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51b4988d-24mw5" event={"ID":"3f24a588-62a8-43fa-aa2a-40c6f1e55097","Type":"ContainerDied","Data":"695202a40d034ff427b90ff6368767c81790e88bf9e80657a2da397af5e446a7"} Nov 03 13:31:13.710574 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:13.698727 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="695202a40d034ff427b90ff6368767c81790e88bf9e80657a2da397af5e446a7" Nov 03 13:31:13.710574 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:13.698810 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51b4988d-24mw5" Nov 03 13:31:13.733245 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:13.733207 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5295ab29-6d58657867-ls4rq" event={"ID":"e421da61-cb60-47e5-8da3-936ec53af260","Type":"ContainerStarted","Data":"2b6c90cf03d523392a7d8de0ad54a095b9eaeb08e33d728fe5ca2557a1378487"} Nov 03 13:31:13.733517 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:13.733455 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5295ab29-6d58657867-ls4rq" event={"ID":"e421da61-cb60-47e5-8da3-936ec53af260","Type":"ContainerStarted","Data":"6d49123dbc365325d405f7d3dca7fb829a9df731535e0db1bb6a58d466de2cb0"} Nov 03 13:31:13.735084 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:13.734815 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-5295ab29-6d58657867-ls4rq" Nov 03 13:31:13.741850 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:13.741822 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5295ab29-79c7656b49-gxvjq" event={"ID":"96afb289-d271-43d2-aeb8-3deaa841af80","Type":"ContainerStarted","Data":"27846f2111f3bc6e6d58007b420245f9bc9f09118d311e667cb2e99893b49bf7"} Nov 03 13:31:13.742052 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:13.742031 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5295ab29-79c7656b49-gxvjq" event={"ID":"96afb289-d271-43d2-aeb8-3deaa841af80","Type":"ContainerStarted","Data":"f6976eedb8d63a6627b14e8831b73349b69783dbfe97335578e0a6fe3120a6f6"} Nov 03 13:31:13.742980 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:13.742958 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-5295ab29-79c7656b49-gxvjq" Nov 03 13:31:13.745014 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:13.744987 5979 generic.go:356] "Generic (PLEG): container finished" podID="5211f1d9-4181-4ad0-b83a-4e4e7a556be8" containerID="95d28afb5b88bab8ea99934962ff3b61b20bda95db7fa5baff3736a423a418b8" exitCode=0 Nov 03 13:31:13.745128 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:13.745023 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk" event={"ID":"5211f1d9-4181-4ad0-b83a-4e4e7a556be8","Type":"ContainerDied","Data":"95d28afb5b88bab8ea99934962ff3b61b20bda95db7fa5baff3736a423a418b8"} Nov 03 13:31:13.745128 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:13.745083 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk" event={"ID":"5211f1d9-4181-4ad0-b83a-4e4e7a556be8","Type":"ContainerDied","Data":"004028a21494b5acfd90ae1a7b19f0c014a7b1a1a0c3efffa8d681cc709b042e"} Nov 03 13:31:13.745128 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:13.745108 5979 scope.go:122] "RemoveContainer" containerID="95d28afb5b88bab8ea99934962ff3b61b20bda95db7fa5baff3736a423a418b8" Nov 03 13:31:13.745704 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:13.745679 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk" Nov 03 13:31:13.756946 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:13.756916 5979 scope.go:122] "RemoveContainer" containerID="95d28afb5b88bab8ea99934962ff3b61b20bda95db7fa5baff3736a423a418b8" Nov 03 13:31:13.757254 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:31:13.757229 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95d28afb5b88bab8ea99934962ff3b61b20bda95db7fa5baff3736a423a418b8\": container with ID starting with 95d28afb5b88bab8ea99934962ff3b61b20bda95db7fa5baff3736a423a418b8 not found: ID does not exist" containerID="95d28afb5b88bab8ea99934962ff3b61b20bda95db7fa5baff3736a423a418b8" Nov 03 13:31:13.757316 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:13.757266 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95d28afb5b88bab8ea99934962ff3b61b20bda95db7fa5baff3736a423a418b8"} err="failed to get container status \"95d28afb5b88bab8ea99934962ff3b61b20bda95db7fa5baff3736a423a418b8\": rpc error: code = NotFound desc = could not find container \"95d28afb5b88bab8ea99934962ff3b61b20bda95db7fa5baff3736a423a418b8\": container with ID starting with 95d28afb5b88bab8ea99934962ff3b61b20bda95db7fa5baff3736a423a418b8 not found: ID does not exist" Nov 03 13:31:13.845725 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:13.845625 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-5295ab29-6d58657867-ls4rq" podStartSLOduration=2.845609767 podStartE2EDuration="2.845609767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:31:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:31:13.843190019 +0000 UTC m=+1555.853878261" watchObservedRunningTime="2026-11-03 13:31:13.845609767 +0000 UTC m=+1555.856298009" Nov 03 13:31:13.951067 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:13.950846 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk"] Nov 03 13:31:13.954482 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:13.954452 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-klqdk"] Nov 03 13:31:14.069973 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:14.069920 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-5295ab29-79c7656b49-gxvjq" podStartSLOduration=3.069906817 podStartE2EDuration="3.069906817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:31:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:31:14.067221928 +0000 UTC m=+1556.077910172" watchObservedRunningTime="2026-11-03 13:31:14.069906817 +0000 UTC m=+1556.080595061" Nov 03 13:31:14.116825 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:14.116753 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5211f1d9-4181-4ad0-b83a-4e4e7a556be8" path="/var/lib/kubelet/pods/5211f1d9-4181-4ad0-b83a-4e4e7a556be8/volumes" Nov 03 13:31:16.800823 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:16.800784 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-abe0e0-6d6c498cf9-lktgw" podUID="1b8fe7cb-3ae1-474f-a573-cea5cae96854" containerName="payload-processing" containerID="cri-o://9d66874e1f30507d5ad763160d3ff3c1d16c6dbdc3fcf770836add1f044fece8" gracePeriod=25 Nov 03 13:31:16.851519 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:16.851443 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-abe0e0-766d597f55-t5mk2" podUID="f266d718-6f0c-4972-b21b-db7533013291" containerName="payload-pre-processing" containerID="cri-o://094bb5adb816224db5f9bc89202c0038d5aea0443b79f5b769f7f09aa071158d" gracePeriod=25 Nov 03 13:31:17.156041 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:17.156005 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-abe0e0-6d6c498cf9-lktgw" Nov 03 13:31:17.180547 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:17.180510 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-abe0e0-766d597f55-t5mk2" Nov 03 13:31:17.240696 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:17.240663 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b8fe7cb-3ae1-474f-a573-cea5cae96854-kube-api-access-t54jr\" (UniqueName: \"kubernetes.io/projected/1b8fe7cb-3ae1-474f-a573-cea5cae96854-kube-api-access-t54jr\") pod \"1b8fe7cb-3ae1-474f-a573-cea5cae96854\" (UID: \"1b8fe7cb-3ae1-474f-a573-cea5cae96854\") " Nov 03 13:31:17.240870 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:17.240712 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b8fe7cb-3ae1-474f-a573-cea5cae96854-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b8fe7cb-3ae1-474f-a573-cea5cae96854-plugins-config-volume\") pod \"1b8fe7cb-3ae1-474f-a573-cea5cae96854\" (UID: \"1b8fe7cb-3ae1-474f-a573-cea5cae96854\") " Nov 03 13:31:17.240870 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:17.240839 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f266d718-6f0c-4972-b21b-db7533013291-kube-api-access-wprxj\" (UniqueName: \"kubernetes.io/projected/f266d718-6f0c-4972-b21b-db7533013291-kube-api-access-wprxj\") pod \"f266d718-6f0c-4972-b21b-db7533013291\" (UID: \"f266d718-6f0c-4972-b21b-db7533013291\") " Nov 03 13:31:17.240984 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:17.240939 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f266d718-6f0c-4972-b21b-db7533013291-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f266d718-6f0c-4972-b21b-db7533013291-plugins-config-volume\") pod \"f266d718-6f0c-4972-b21b-db7533013291\" (UID: \"f266d718-6f0c-4972-b21b-db7533013291\") " Nov 03 13:31:17.241100 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:17.241077 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b8fe7cb-3ae1-474f-a573-cea5cae96854-plugins-config-volume" pod "1b8fe7cb-3ae1-474f-a573-cea5cae96854" (UID: "1b8fe7cb-3ae1-474f-a573-cea5cae96854"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:31:17.241373 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:17.241349 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f266d718-6f0c-4972-b21b-db7533013291-plugins-config-volume" pod "f266d718-6f0c-4972-b21b-db7533013291" (UID: "f266d718-6f0c-4972-b21b-db7533013291"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:31:17.241576 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:17.241559 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1b8fe7cb-3ae1-474f-a573-cea5cae96854-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:17.241630 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:17.241578 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f266d718-6f0c-4972-b21b-db7533013291-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:17.242857 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:17.242839 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b8fe7cb-3ae1-474f-a573-cea5cae96854-kube-api-access-t54jr" pod "1b8fe7cb-3ae1-474f-a573-cea5cae96854" (UID: "1b8fe7cb-3ae1-474f-a573-cea5cae96854"). InnerVolumeSpecName "kube-api-access-t54jr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:31:17.242929 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:17.242854 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f266d718-6f0c-4972-b21b-db7533013291-kube-api-access-wprxj" pod "f266d718-6f0c-4972-b21b-db7533013291" (UID: "f266d718-6f0c-4972-b21b-db7533013291"). InnerVolumeSpecName "kube-api-access-wprxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:31:17.343010 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:17.342929 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t54jr\" (UniqueName: \"kubernetes.io/projected/1b8fe7cb-3ae1-474f-a573-cea5cae96854-kube-api-access-t54jr\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:17.343010 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:17.342958 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wprxj\" (UniqueName: \"kubernetes.io/projected/f266d718-6f0c-4972-b21b-db7533013291-kube-api-access-wprxj\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:17.779170 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:17.779131 5979 generic.go:356] "Generic (PLEG): container finished" podID="1b8fe7cb-3ae1-474f-a573-cea5cae96854" containerID="9d66874e1f30507d5ad763160d3ff3c1d16c6dbdc3fcf770836add1f044fece8" exitCode=0 Nov 03 13:31:17.779343 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:17.779174 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-abe0e0-6d6c498cf9-lktgw" Nov 03 13:31:17.779343 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:17.779215 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-abe0e0-6d6c498cf9-lktgw" event={"ID":"1b8fe7cb-3ae1-474f-a573-cea5cae96854","Type":"ContainerDied","Data":"9d66874e1f30507d5ad763160d3ff3c1d16c6dbdc3fcf770836add1f044fece8"} Nov 03 13:31:17.779343 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:17.779265 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-abe0e0-6d6c498cf9-lktgw" event={"ID":"1b8fe7cb-3ae1-474f-a573-cea5cae96854","Type":"ContainerDied","Data":"14e8b1f8b584d1551ddfc0a1ea2dc15c504bd7a8c99b556a88be2a1bff2b7612"} Nov 03 13:31:17.779343 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:17.779288 5979 scope.go:122] "RemoveContainer" containerID="9d66874e1f30507d5ad763160d3ff3c1d16c6dbdc3fcf770836add1f044fece8" Nov 03 13:31:17.786306 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:17.786283 5979 generic.go:356] "Generic (PLEG): container finished" podID="f266d718-6f0c-4972-b21b-db7533013291" containerID="094bb5adb816224db5f9bc89202c0038d5aea0443b79f5b769f7f09aa071158d" exitCode=0 Nov 03 13:31:17.786419 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:17.786333 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-abe0e0-766d597f55-t5mk2" event={"ID":"f266d718-6f0c-4972-b21b-db7533013291","Type":"ContainerDied","Data":"094bb5adb816224db5f9bc89202c0038d5aea0443b79f5b769f7f09aa071158d"} Nov 03 13:31:17.786419 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:17.786373 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-abe0e0-766d597f55-t5mk2" event={"ID":"f266d718-6f0c-4972-b21b-db7533013291","Type":"ContainerDied","Data":"19e504de467b9753f70a2bb2e10d02d1c23cc06ab4539d6e7c0344fd637a74d7"} Nov 03 13:31:17.786419 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:17.786382 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-abe0e0-766d597f55-t5mk2" Nov 03 13:31:17.795397 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:17.795377 5979 scope.go:122] "RemoveContainer" containerID="9d66874e1f30507d5ad763160d3ff3c1d16c6dbdc3fcf770836add1f044fece8" Nov 03 13:31:17.795740 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:31:17.795706 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d66874e1f30507d5ad763160d3ff3c1d16c6dbdc3fcf770836add1f044fece8\": container with ID starting with 9d66874e1f30507d5ad763160d3ff3c1d16c6dbdc3fcf770836add1f044fece8 not found: ID does not exist" containerID="9d66874e1f30507d5ad763160d3ff3c1d16c6dbdc3fcf770836add1f044fece8" Nov 03 13:31:17.795824 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:17.795745 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d66874e1f30507d5ad763160d3ff3c1d16c6dbdc3fcf770836add1f044fece8"} err="failed to get container status \"9d66874e1f30507d5ad763160d3ff3c1d16c6dbdc3fcf770836add1f044fece8\": rpc error: code = NotFound desc = could not find container \"9d66874e1f30507d5ad763160d3ff3c1d16c6dbdc3fcf770836add1f044fece8\": container with ID starting with 9d66874e1f30507d5ad763160d3ff3c1d16c6dbdc3fcf770836add1f044fece8 not found: ID does not exist" Nov 03 13:31:17.795824 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:17.795770 5979 scope.go:122] "RemoveContainer" containerID="094bb5adb816224db5f9bc89202c0038d5aea0443b79f5b769f7f09aa071158d" Nov 03 13:31:17.805637 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:17.805393 5979 scope.go:122] "RemoveContainer" containerID="094bb5adb816224db5f9bc89202c0038d5aea0443b79f5b769f7f09aa071158d" Nov 03 13:31:17.805860 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:31:17.805664 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"094bb5adb816224db5f9bc89202c0038d5aea0443b79f5b769f7f09aa071158d\": container with ID starting with 094bb5adb816224db5f9bc89202c0038d5aea0443b79f5b769f7f09aa071158d not found: ID does not exist" containerID="094bb5adb816224db5f9bc89202c0038d5aea0443b79f5b769f7f09aa071158d" Nov 03 13:31:17.805860 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:17.805690 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"094bb5adb816224db5f9bc89202c0038d5aea0443b79f5b769f7f09aa071158d"} err="failed to get container status \"094bb5adb816224db5f9bc89202c0038d5aea0443b79f5b769f7f09aa071158d\": rpc error: code = NotFound desc = could not find container \"094bb5adb816224db5f9bc89202c0038d5aea0443b79f5b769f7f09aa071158d\": container with ID starting with 094bb5adb816224db5f9bc89202c0038d5aea0443b79f5b769f7f09aa071158d not found: ID does not exist" Nov 03 13:31:17.936273 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:17.936242 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-abe0e0-6d6c498cf9-lktgw"] Nov 03 13:31:17.941804 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:17.941775 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-abe0e0-6d6c498cf9-lktgw"] Nov 03 13:31:18.045528 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:18.045450 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-abe0e0-766d597f55-t5mk2"] Nov 03 13:31:18.050827 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:18.050799 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-abe0e0-766d597f55-t5mk2"] Nov 03 13:31:18.118514 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:18.118463 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b8fe7cb-3ae1-474f-a573-cea5cae96854" path="/var/lib/kubelet/pods/1b8fe7cb-3ae1-474f-a573-cea5cae96854/volumes" Nov 03 13:31:18.119022 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:18.119006 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f266d718-6f0c-4972-b21b-db7533013291" path="/var/lib/kubelet/pods/f266d718-6f0c-4972-b21b-db7533013291/volumes" Nov 03 13:31:18.782149 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:18.782118 5979 scope.go:122] "RemoveContainer" containerID="7648ebd2f307d4fdf1953fa81f86a36878b3e7fe4f32570569e93b4cf9969aae" Nov 03 13:31:18.793821 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:18.793799 5979 scope.go:122] "RemoveContainer" containerID="a60acca48c729c78b87b02f02addbcd386cc86001ba1ff6b4dfee6a82c79f3ef" Nov 03 13:31:18.803952 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:18.803923 5979 scope.go:122] "RemoveContainer" containerID="63b9f7499141642d39b9c9bb582b347f48ec38c7fa641252f7a5782b5eece4e1" Nov 03 13:31:18.813269 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:18.813088 5979 scope.go:122] "RemoveContainer" containerID="2dbf5b669b9dcc29da5e3afa79f1ce43d953d669b791f906c8c62da72d2af378" Nov 03 13:31:18.822171 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:18.822152 5979 scope.go:122] "RemoveContainer" containerID="f05588e47270503226b9fc1fca1ba41f9490bf8a6a7136c3709544f2470e5978" Nov 03 13:31:18.831340 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:18.831315 5979 scope.go:122] "RemoveContainer" containerID="35e639b60d996109594af2e74b5f956fe01570a669ee4bdfdb87b103dbe24d73" Nov 03 13:31:21.044870 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:21.044834 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60a9f79b-2kx52"] Nov 03 13:31:21.091540 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:21.091499 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60a9f79b-2kx52"] Nov 03 13:31:22.115880 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:22.115850 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a8f32fb-a894-49b1-8da8-3cccbca598cc" path="/var/lib/kubelet/pods/1a8f32fb-a894-49b1-8da8-3cccbca598cc/volumes" Nov 03 13:31:22.341421 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:22.341383 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:31:23.970538 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:23.970502 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf"] Nov 03 13:31:23.970988 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:23.970792 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf" podUID="fa17ed42-8832-4fea-b962-9029c6e6dbb9" containerName="istio-proxy" containerID="cri-o://453ebeeb01d5af571ca02703d1b423cda52203f86f2355ad7249ba0d400804f1" gracePeriod=30 Nov 03 13:31:25.774793 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:25.774761 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-5295ab29-6d58657867-ls4rq" Nov 03 13:31:25.776509 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:25.776455 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-5295ab29-79c7656b49-gxvjq" Nov 03 13:31:29.285795 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:29.285764 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf" Nov 03 13:31:29.391525 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:29.391460 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa17ed42-8832-4fea-b962-9029c6e6dbb9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa17ed42-8832-4fea-b962-9029c6e6dbb9-credential-socket\") pod \"fa17ed42-8832-4fea-b962-9029c6e6dbb9\" (UID: \"fa17ed42-8832-4fea-b962-9029c6e6dbb9\") " Nov 03 13:31:29.391729 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:29.391544 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa17ed42-8832-4fea-b962-9029c6e6dbb9-kube-api-access-8h4bp\" (UniqueName: \"kubernetes.io/projected/fa17ed42-8832-4fea-b962-9029c6e6dbb9-kube-api-access-8h4bp\") pod \"fa17ed42-8832-4fea-b962-9029c6e6dbb9\" (UID: \"fa17ed42-8832-4fea-b962-9029c6e6dbb9\") " Nov 03 13:31:29.391729 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:29.391580 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa17ed42-8832-4fea-b962-9029c6e6dbb9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa17ed42-8832-4fea-b962-9029c6e6dbb9-istio-envoy\") pod \"fa17ed42-8832-4fea-b962-9029c6e6dbb9\" (UID: \"fa17ed42-8832-4fea-b962-9029c6e6dbb9\") " Nov 03 13:31:29.391729 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:29.391682 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa17ed42-8832-4fea-b962-9029c6e6dbb9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa17ed42-8832-4fea-b962-9029c6e6dbb9-istiod-ca-cert\") pod \"fa17ed42-8832-4fea-b962-9029c6e6dbb9\" (UID: \"fa17ed42-8832-4fea-b962-9029c6e6dbb9\") " Nov 03 13:31:29.391916 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:29.391747 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa17ed42-8832-4fea-b962-9029c6e6dbb9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa17ed42-8832-4fea-b962-9029c6e6dbb9-workload-certs\") pod \"fa17ed42-8832-4fea-b962-9029c6e6dbb9\" (UID: \"fa17ed42-8832-4fea-b962-9029c6e6dbb9\") " Nov 03 13:31:29.391916 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:29.391837 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa17ed42-8832-4fea-b962-9029c6e6dbb9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa17ed42-8832-4fea-b962-9029c6e6dbb9-istio-data\") pod \"fa17ed42-8832-4fea-b962-9029c6e6dbb9\" (UID: \"fa17ed42-8832-4fea-b962-9029c6e6dbb9\") " Nov 03 13:31:29.392023 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:29.391946 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa17ed42-8832-4fea-b962-9029c6e6dbb9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa17ed42-8832-4fea-b962-9029c6e6dbb9-istio-podinfo\") pod \"fa17ed42-8832-4fea-b962-9029c6e6dbb9\" (UID: \"fa17ed42-8832-4fea-b962-9029c6e6dbb9\") " Nov 03 13:31:29.392077 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:29.392064 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa17ed42-8832-4fea-b962-9029c6e6dbb9-istio-token\" (UniqueName: \"kubernetes.io/projected/fa17ed42-8832-4fea-b962-9029c6e6dbb9-istio-token\") pod \"fa17ed42-8832-4fea-b962-9029c6e6dbb9\" (UID: \"fa17ed42-8832-4fea-b962-9029c6e6dbb9\") " Nov 03 13:31:29.392182 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:29.392156 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa17ed42-8832-4fea-b962-9029c6e6dbb9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa17ed42-8832-4fea-b962-9029c6e6dbb9-workload-socket\") pod \"fa17ed42-8832-4fea-b962-9029c6e6dbb9\" (UID: \"fa17ed42-8832-4fea-b962-9029c6e6dbb9\") " Nov 03 13:31:29.392448 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:29.392418 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa17ed42-8832-4fea-b962-9029c6e6dbb9-istio-data" pod "fa17ed42-8832-4fea-b962-9029c6e6dbb9" (UID: "fa17ed42-8832-4fea-b962-9029c6e6dbb9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:29.392585 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:29.392535 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa17ed42-8832-4fea-b962-9029c6e6dbb9-workload-certs" pod "fa17ed42-8832-4fea-b962-9029c6e6dbb9" (UID: "fa17ed42-8832-4fea-b962-9029c6e6dbb9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:29.392663 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:29.392635 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa17ed42-8832-4fea-b962-9029c6e6dbb9-credential-socket" pod "fa17ed42-8832-4fea-b962-9029c6e6dbb9" (UID: "fa17ed42-8832-4fea-b962-9029c6e6dbb9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:29.392729 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:29.392674 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa17ed42-8832-4fea-b962-9029c6e6dbb9-workload-socket" pod "fa17ed42-8832-4fea-b962-9029c6e6dbb9" (UID: "fa17ed42-8832-4fea-b962-9029c6e6dbb9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:29.392729 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:29.392695 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa17ed42-8832-4fea-b962-9029c6e6dbb9-istiod-ca-cert" pod "fa17ed42-8832-4fea-b962-9029c6e6dbb9" (UID: "fa17ed42-8832-4fea-b962-9029c6e6dbb9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:31:29.392861 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:29.392838 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa17ed42-8832-4fea-b962-9029c6e6dbb9-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:29.392861 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:29.392857 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa17ed42-8832-4fea-b962-9029c6e6dbb9-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:29.392949 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:29.392865 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa17ed42-8832-4fea-b962-9029c6e6dbb9-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:29.392949 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:29.392879 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa17ed42-8832-4fea-b962-9029c6e6dbb9-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:29.392949 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:29.392898 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa17ed42-8832-4fea-b962-9029c6e6dbb9-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:29.394668 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:29.394644 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa17ed42-8832-4fea-b962-9029c6e6dbb9-istio-envoy" pod "fa17ed42-8832-4fea-b962-9029c6e6dbb9" (UID: "fa17ed42-8832-4fea-b962-9029c6e6dbb9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:29.394773 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:29.394658 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa17ed42-8832-4fea-b962-9029c6e6dbb9-istio-podinfo" pod "fa17ed42-8832-4fea-b962-9029c6e6dbb9" (UID: "fa17ed42-8832-4fea-b962-9029c6e6dbb9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:31:29.395065 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:29.395046 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa17ed42-8832-4fea-b962-9029c6e6dbb9-kube-api-access-8h4bp" pod "fa17ed42-8832-4fea-b962-9029c6e6dbb9" (UID: "fa17ed42-8832-4fea-b962-9029c6e6dbb9"). InnerVolumeSpecName "kube-api-access-8h4bp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:31:29.395103 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:29.395076 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa17ed42-8832-4fea-b962-9029c6e6dbb9-istio-token" pod "fa17ed42-8832-4fea-b962-9029c6e6dbb9" (UID: "fa17ed42-8832-4fea-b962-9029c6e6dbb9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:31:29.495003 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:29.494924 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa17ed42-8832-4fea-b962-9029c6e6dbb9-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:29.495003 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:29.494968 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h4bp\" (UniqueName: \"kubernetes.io/projected/fa17ed42-8832-4fea-b962-9029c6e6dbb9-kube-api-access-8h4bp\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:29.495003 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:29.494985 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa17ed42-8832-4fea-b962-9029c6e6dbb9-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:29.495003 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:29.494998 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa17ed42-8832-4fea-b962-9029c6e6dbb9-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:29.902776 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:29.902738 5979 generic.go:356] "Generic (PLEG): container finished" podID="fa17ed42-8832-4fea-b962-9029c6e6dbb9" containerID="453ebeeb01d5af571ca02703d1b423cda52203f86f2355ad7249ba0d400804f1" exitCode=0 Nov 03 13:31:29.902972 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:29.902777 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf" event={"ID":"fa17ed42-8832-4fea-b962-9029c6e6dbb9","Type":"ContainerDied","Data":"453ebeeb01d5af571ca02703d1b423cda52203f86f2355ad7249ba0d400804f1"} Nov 03 13:31:29.902972 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:29.902815 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf" Nov 03 13:31:29.902972 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:29.902859 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf" event={"ID":"fa17ed42-8832-4fea-b962-9029c6e6dbb9","Type":"ContainerDied","Data":"9220fa135b50f1d093d7152926f14cc29b384849f8b32e593595b58bc0cefc01"} Nov 03 13:31:29.902972 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:29.902881 5979 scope.go:122] "RemoveContainer" containerID="453ebeeb01d5af571ca02703d1b423cda52203f86f2355ad7249ba0d400804f1" Nov 03 13:31:29.914240 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:29.914224 5979 scope.go:122] "RemoveContainer" containerID="0638dfc254a429df1d30b3e1bd1175552ef5d69f8b77c529acf1f3fa2970ab8b" Nov 03 13:31:29.923775 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:29.923752 5979 scope.go:122] "RemoveContainer" containerID="453ebeeb01d5af571ca02703d1b423cda52203f86f2355ad7249ba0d400804f1" Nov 03 13:31:29.924042 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:31:29.924021 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"453ebeeb01d5af571ca02703d1b423cda52203f86f2355ad7249ba0d400804f1\": container with ID starting with 453ebeeb01d5af571ca02703d1b423cda52203f86f2355ad7249ba0d400804f1 not found: ID does not exist" containerID="453ebeeb01d5af571ca02703d1b423cda52203f86f2355ad7249ba0d400804f1" Nov 03 13:31:29.924127 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:29.924052 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"453ebeeb01d5af571ca02703d1b423cda52203f86f2355ad7249ba0d400804f1"} err="failed to get container status \"453ebeeb01d5af571ca02703d1b423cda52203f86f2355ad7249ba0d400804f1\": rpc error: code = NotFound desc = could not find container \"453ebeeb01d5af571ca02703d1b423cda52203f86f2355ad7249ba0d400804f1\": container with ID starting with 453ebeeb01d5af571ca02703d1b423cda52203f86f2355ad7249ba0d400804f1 not found: ID does not exist" Nov 03 13:31:29.924127 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:29.924077 5979 scope.go:122] "RemoveContainer" containerID="0638dfc254a429df1d30b3e1bd1175552ef5d69f8b77c529acf1f3fa2970ab8b" Nov 03 13:31:29.924365 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:31:29.924347 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0638dfc254a429df1d30b3e1bd1175552ef5d69f8b77c529acf1f3fa2970ab8b\": container with ID starting with 0638dfc254a429df1d30b3e1bd1175552ef5d69f8b77c529acf1f3fa2970ab8b not found: ID does not exist" containerID="0638dfc254a429df1d30b3e1bd1175552ef5d69f8b77c529acf1f3fa2970ab8b" Nov 03 13:31:29.924405 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:29.924372 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0638dfc254a429df1d30b3e1bd1175552ef5d69f8b77c529acf1f3fa2970ab8b"} err="failed to get container status \"0638dfc254a429df1d30b3e1bd1175552ef5d69f8b77c529acf1f3fa2970ab8b\": rpc error: code = NotFound desc = could not find container \"0638dfc254a429df1d30b3e1bd1175552ef5d69f8b77c529acf1f3fa2970ab8b\": container with ID starting with 0638dfc254a429df1d30b3e1bd1175552ef5d69f8b77c529acf1f3fa2970ab8b not found: ID does not exist" Nov 03 13:31:30.118985 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:30.118952 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf"] Nov 03 13:31:30.157300 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:30.157223 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf"] Nov 03 13:31:30.243246 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:30.243211 5979 patch_prober.go:29] interesting pod/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:31:30.243410 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:30.243269 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-abe0e0-openshift-default-599b59b6fc-fn6qf" podUID="fa17ed42-8832-4fea-b962-9029c6e6dbb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:31:32.118449 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:32.118416 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa17ed42-8832-4fea-b962-9029c6e6dbb9" path="/var/lib/kubelet/pods/fa17ed42-8832-4fea-b962-9029c6e6dbb9/volumes" Nov 03 13:31:32.453580 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:32.453543 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd"] Nov 03 13:31:32.454162 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:32.454150 5979 cpu_manager.go:414] "containerMap: removing container" podUID="fa17ed42-8832-4fea-b962-9029c6e6dbb9" containerName="istio-proxy" Nov 03 13:31:32.454210 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:32.454163 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa17ed42-8832-4fea-b962-9029c6e6dbb9" containerName="istio-proxy" Nov 03 13:31:32.454210 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:32.454178 5979 cpu_manager.go:414] "containerMap: removing container" podUID="1b8fe7cb-3ae1-474f-a573-cea5cae96854" containerName="payload-processing" Nov 03 13:31:32.454210 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:32.454183 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b8fe7cb-3ae1-474f-a573-cea5cae96854" containerName="payload-processing" Nov 03 13:31:32.454210 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:32.454193 5979 cpu_manager.go:414] "containerMap: removing container" podUID="fa17ed42-8832-4fea-b962-9029c6e6dbb9" containerName="istio-proxy" Nov 03 13:31:32.454210 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:32.454198 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa17ed42-8832-4fea-b962-9029c6e6dbb9" containerName="istio-proxy" Nov 03 13:31:32.454377 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:32.454220 5979 cpu_manager.go:414] "containerMap: removing container" podUID="f266d718-6f0c-4972-b21b-db7533013291" containerName="payload-pre-processing" Nov 03 13:31:32.454377 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:32.454226 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f266d718-6f0c-4972-b21b-db7533013291" containerName="payload-pre-processing" Nov 03 13:31:32.454377 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:32.454232 5979 cpu_manager.go:414] "containerMap: removing container" podUID="3f24a588-62a8-43fa-aa2a-40c6f1e55097" containerName="revoke-keys" Nov 03 13:31:32.454377 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:32.454237 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f24a588-62a8-43fa-aa2a-40c6f1e55097" containerName="revoke-keys" Nov 03 13:31:32.454377 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:32.454263 5979 cpu_manager.go:414] "containerMap: removing container" podUID="5211f1d9-4181-4ad0-b83a-4e4e7a556be8" containerName="istio-proxy" Nov 03 13:31:32.454377 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:32.454269 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5211f1d9-4181-4ad0-b83a-4e4e7a556be8" containerName="istio-proxy" Nov 03 13:31:32.454377 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:32.454375 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="f266d718-6f0c-4972-b21b-db7533013291" containerName="payload-pre-processing" Nov 03 13:31:32.454751 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:32.454391 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f24a588-62a8-43fa-aa2a-40c6f1e55097" containerName="revoke-keys" Nov 03 13:31:32.454751 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:32.454409 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="5211f1d9-4181-4ad0-b83a-4e4e7a556be8" containerName="istio-proxy" Nov 03 13:31:32.454751 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:32.454423 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa17ed42-8832-4fea-b962-9029c6e6dbb9" containerName="istio-proxy" Nov 03 13:31:32.454751 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:32.454434 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b8fe7cb-3ae1-474f-a573-cea5cae96854" containerName="payload-processing" Nov 03 13:31:32.454751 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:32.454443 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa17ed42-8832-4fea-b962-9029c6e6dbb9" containerName="istio-proxy" Nov 03 13:31:32.458141 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:32.458122 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd" Nov 03 13:31:32.538631 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:32.538598 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abf1926b-6e99-4b17-846d-8c2e9c924d9f-istio-token\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd\" (UID: \"abf1926b-6e99-4b17-846d-8c2e9c924d9f\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd" Nov 03 13:31:32.538631 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:32.538697 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abf1926b-6e99-4b17-846d-8c2e9c924d9f-istio-podinfo\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd\" (UID: \"abf1926b-6e99-4b17-846d-8c2e9c924d9f\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd" Nov 03 13:31:32.538987 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:32.538832 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf1926b-6e99-4b17-846d-8c2e9c924d9f-credential-socket\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd\" (UID: \"abf1926b-6e99-4b17-846d-8c2e9c924d9f\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd" Nov 03 13:31:32.538987 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:32.538901 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abf1926b-6e99-4b17-846d-8c2e9c924d9f-istiod-ca-cert\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd\" (UID: \"abf1926b-6e99-4b17-846d-8c2e9c924d9f\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd" Nov 03 13:31:32.538987 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:32.538930 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abf1926b-6e99-4b17-846d-8c2e9c924d9f-istio-data\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd\" (UID: \"abf1926b-6e99-4b17-846d-8c2e9c924d9f\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd" Nov 03 13:31:32.538987 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:32.538954 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4shkz\" (UniqueName: \"kubernetes.io/projected/abf1926b-6e99-4b17-846d-8c2e9c924d9f-kube-api-access-4shkz\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd\" (UID: \"abf1926b-6e99-4b17-846d-8c2e9c924d9f\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd" Nov 03 13:31:32.538987 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:32.538985 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf1926b-6e99-4b17-846d-8c2e9c924d9f-workload-socket\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd\" (UID: \"abf1926b-6e99-4b17-846d-8c2e9c924d9f\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd" Nov 03 13:31:32.539239 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:32.539062 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abf1926b-6e99-4b17-846d-8c2e9c924d9f-istio-envoy\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd\" (UID: \"abf1926b-6e99-4b17-846d-8c2e9c924d9f\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd" Nov 03 13:31:32.539239 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:32.539125 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abf1926b-6e99-4b17-846d-8c2e9c924d9f-workload-certs\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd\" (UID: \"abf1926b-6e99-4b17-846d-8c2e9c924d9f\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd" Nov 03 13:31:32.640281 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:32.640248 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abf1926b-6e99-4b17-846d-8c2e9c924d9f-istio-podinfo\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd\" (UID: \"abf1926b-6e99-4b17-846d-8c2e9c924d9f\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd" Nov 03 13:31:32.640465 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:32.640368 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf1926b-6e99-4b17-846d-8c2e9c924d9f-credential-socket\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd\" (UID: \"abf1926b-6e99-4b17-846d-8c2e9c924d9f\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd" Nov 03 13:31:32.640465 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:32.640431 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abf1926b-6e99-4b17-846d-8c2e9c924d9f-istiod-ca-cert\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd\" (UID: \"abf1926b-6e99-4b17-846d-8c2e9c924d9f\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd" Nov 03 13:31:32.640465 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:32.640455 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abf1926b-6e99-4b17-846d-8c2e9c924d9f-istio-data\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd\" (UID: \"abf1926b-6e99-4b17-846d-8c2e9c924d9f\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd" Nov 03 13:31:32.640672 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:32.640494 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4shkz\" (UniqueName: \"kubernetes.io/projected/abf1926b-6e99-4b17-846d-8c2e9c924d9f-kube-api-access-4shkz\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd\" (UID: \"abf1926b-6e99-4b17-846d-8c2e9c924d9f\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd" Nov 03 13:31:32.640672 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:32.640523 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf1926b-6e99-4b17-846d-8c2e9c924d9f-workload-socket\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd\" (UID: \"abf1926b-6e99-4b17-846d-8c2e9c924d9f\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd" Nov 03 13:31:32.640672 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:32.640596 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abf1926b-6e99-4b17-846d-8c2e9c924d9f-istio-envoy\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd\" (UID: \"abf1926b-6e99-4b17-846d-8c2e9c924d9f\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd" Nov 03 13:31:32.640672 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:32.640657 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abf1926b-6e99-4b17-846d-8c2e9c924d9f-workload-certs\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd\" (UID: \"abf1926b-6e99-4b17-846d-8c2e9c924d9f\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd" Nov 03 13:31:32.640883 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:32.640722 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abf1926b-6e99-4b17-846d-8c2e9c924d9f-istio-token\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd\" (UID: \"abf1926b-6e99-4b17-846d-8c2e9c924d9f\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd" Nov 03 13:31:32.640883 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:32.640797 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf1926b-6e99-4b17-846d-8c2e9c924d9f-credential-socket\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd\" (UID: \"abf1926b-6e99-4b17-846d-8c2e9c924d9f\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd" Nov 03 13:31:32.640883 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:32.640831 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abf1926b-6e99-4b17-846d-8c2e9c924d9f-istio-data\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd\" (UID: \"abf1926b-6e99-4b17-846d-8c2e9c924d9f\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd" Nov 03 13:31:32.641034 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:32.640937 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf1926b-6e99-4b17-846d-8c2e9c924d9f-workload-socket\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd\" (UID: \"abf1926b-6e99-4b17-846d-8c2e9c924d9f\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd" Nov 03 13:31:32.641177 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:32.641157 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abf1926b-6e99-4b17-846d-8c2e9c924d9f-workload-certs\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd\" (UID: \"abf1926b-6e99-4b17-846d-8c2e9c924d9f\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd" Nov 03 13:31:32.641329 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:32.641308 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abf1926b-6e99-4b17-846d-8c2e9c924d9f-istiod-ca-cert\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd\" (UID: \"abf1926b-6e99-4b17-846d-8c2e9c924d9f\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd" Nov 03 13:31:32.642980 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:32.642953 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abf1926b-6e99-4b17-846d-8c2e9c924d9f-istio-envoy\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd\" (UID: \"abf1926b-6e99-4b17-846d-8c2e9c924d9f\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd" Nov 03 13:31:32.643218 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:32.643198 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abf1926b-6e99-4b17-846d-8c2e9c924d9f-istio-podinfo\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd\" (UID: \"abf1926b-6e99-4b17-846d-8c2e9c924d9f\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd" Nov 03 13:31:32.659131 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:32.659098 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd"] Nov 03 13:31:32.694747 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:32.694705 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4shkz\" (UniqueName: \"kubernetes.io/projected/abf1926b-6e99-4b17-846d-8c2e9c924d9f-kube-api-access-4shkz\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd\" (UID: \"abf1926b-6e99-4b17-846d-8c2e9c924d9f\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd" Nov 03 13:31:32.695927 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:32.695886 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abf1926b-6e99-4b17-846d-8c2e9c924d9f-istio-token\") pod \"e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd\" (UID: \"abf1926b-6e99-4b17-846d-8c2e9c924d9f\") " pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd" Nov 03 13:31:32.772023 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:32.771947 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd" Nov 03 13:31:33.208045 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:33.208006 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd"] Nov 03 13:31:33.250684 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:31:33.250646 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabf1926b_6e99_4b17_846d_8c2e9c924d9f.slice/crio-18b8b44197dff8ee96a79035599651f8948c92d7047e7cb7906b0b2633e74299 WatchSource:0}: Error finding container 18b8b44197dff8ee96a79035599651f8948c92d7047e7cb7906b0b2633e74299: Status 404 returned error can't find the container with id 18b8b44197dff8ee96a79035599651f8948c92d7047e7cb7906b0b2633e74299 Nov 03 13:31:33.253307 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:33.253274 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:31:33.253399 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:33.253343 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:31:33.253399 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:33.253373 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:31:33.942596 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:33.942511 5979 generic.go:356] "Generic (PLEG): container finished" podID="dd4c2ad4-8ff1-40ba-91e5-5e4556213933" containerID="7da7678ef1c981b9e5f62d2acaf668e2134b7db5116739fde82b81dffa45b13d" exitCode=0 Nov 03 13:31:33.942596 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:33.942573 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7" event={"ID":"dd4c2ad4-8ff1-40ba-91e5-5e4556213933","Type":"ContainerDied","Data":"7da7678ef1c981b9e5f62d2acaf668e2134b7db5116739fde82b81dffa45b13d"} Nov 03 13:31:33.943071 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:33.943053 5979 scope.go:122] "RemoveContainer" containerID="7da7678ef1c981b9e5f62d2acaf668e2134b7db5116739fde82b81dffa45b13d" Nov 03 13:31:33.946176 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:33.946137 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:31:33.946404 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:33.946382 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:31:33.946579 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:33.946560 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:31:33.977527 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:33.977484 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd" event={"ID":"abf1926b-6e99-4b17-846d-8c2e9c924d9f","Type":"ContainerStarted","Data":"e3e0a46dbb0df4c84f8db0b7b74830383b859441ebfbf7778e1a938b65cabcd4"} Nov 03 13:31:33.977724 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:33.977544 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd" event={"ID":"abf1926b-6e99-4b17-846d-8c2e9c924d9f","Type":"ContainerStarted","Data":"18b8b44197dff8ee96a79035599651f8948c92d7047e7cb7906b0b2633e74299"} Nov 03 13:31:34.238699 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:34.238649 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd" podStartSLOduration=2.238636303 podStartE2EDuration="2.238636303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:31:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:31:34.234926804 +0000 UTC m=+1576.245615046" watchObservedRunningTime="2026-11-03 13:31:34.238636303 +0000 UTC m=+1576.249324544" Nov 03 13:31:34.776921 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:34.776880 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd" Nov 03 13:31:34.780667 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:34.780628 5979 patch_prober.go:29] interesting pod/e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 03 13:31:34.780918 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:34.780892 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd" podUID="abf1926b-6e99-4b17-846d-8c2e9c924d9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 03 13:31:35.002990 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:35.002955 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7" event={"ID":"dd4c2ad4-8ff1-40ba-91e5-5e4556213933","Type":"ContainerStarted","Data":"d3068b8ae120b3170cc8fcc8321f4b752fd67bf29eb9021e5e34bced70e469da"} Nov 03 13:31:35.219544 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:35.219504 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7" Nov 03 13:31:35.428281 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:35.428249 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7" Nov 03 13:31:35.429518 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:35.429492 5979 patch_prober.go:29] interesting pod/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Nov 03 13:31:35.429653 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:35.429539 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7" podUID="dd4c2ad4-8ff1-40ba-91e5-5e4556213933" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Nov 03 13:31:35.778497 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:35.778005 5979 patch_prober.go:29] interesting pod/e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 03 13:31:35.778497 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:35.778063 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd" podUID="abf1926b-6e99-4b17-846d-8c2e9c924d9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 03 13:31:36.428296 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:36.428257 5979 patch_prober.go:29] interesting pod/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Nov 03 13:31:36.428849 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:36.428317 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7" podUID="dd4c2ad4-8ff1-40ba-91e5-5e4556213933" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Nov 03 13:31:36.781550 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:36.780907 5979 patch_prober.go:29] interesting pod/e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 03 13:31:36.781550 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:36.780971 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd" podUID="abf1926b-6e99-4b17-846d-8c2e9c924d9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 03 13:31:37.326634 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:31:37.326600 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:31:37.402428 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:37.402386 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8a2e426-dxzqm"] Nov 03 13:31:37.415188 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:37.415151 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8a2e426-dxzqm" Nov 03 13:31:37.428253 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:37.428199 5979 patch_prober.go:29] interesting pod/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Nov 03 13:31:37.428253 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:37.428244 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7" podUID="dd4c2ad4-8ff1-40ba-91e5-5e4556213933" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Nov 03 13:31:37.443060 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:37.443036 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8a2e426-dxzqm"] Nov 03 13:31:37.499973 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:37.499940 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:31:37.500368 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:37.500332 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:31:37.501354 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:37.501334 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 13:31:37.501644 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:37.501628 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 13:31:37.504461 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:37.504437 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thm84\" (UniqueName: \"kubernetes.io/projected/9f5db4ce-3fef-4033-8d83-71495a0b6373-kube-api-access-thm84\") pod \"maas-api-revoke-sub-b8a2e426-dxzqm\" (UID: \"9f5db4ce-3fef-4033-8d83-71495a0b6373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8a2e426-dxzqm" Nov 03 13:31:37.504679 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:37.504649 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f5db4ce-3fef-4033-8d83-71495a0b6373-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8a2e426-dxzqm\" (UID: \"9f5db4ce-3fef-4033-8d83-71495a0b6373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8a2e426-dxzqm" Nov 03 13:31:37.504806 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:37.504764 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f5db4ce-3fef-4033-8d83-71495a0b6373-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8a2e426-dxzqm\" (UID: \"9f5db4ce-3fef-4033-8d83-71495a0b6373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8a2e426-dxzqm" Nov 03 13:31:37.613741 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:37.613633 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f5db4ce-3fef-4033-8d83-71495a0b6373-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8a2e426-dxzqm\" (UID: \"9f5db4ce-3fef-4033-8d83-71495a0b6373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8a2e426-dxzqm" Nov 03 13:31:37.614671 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:37.614644 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f5db4ce-3fef-4033-8d83-71495a0b6373-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8a2e426-dxzqm\" (UID: \"9f5db4ce-3fef-4033-8d83-71495a0b6373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8a2e426-dxzqm" Nov 03 13:31:37.617973 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:37.617014 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f5db4ce-3fef-4033-8d83-71495a0b6373-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b8a2e426-dxzqm\" (UID: \"9f5db4ce-3fef-4033-8d83-71495a0b6373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8a2e426-dxzqm" Nov 03 13:31:37.618301 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:37.618280 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thm84\" (UniqueName: \"kubernetes.io/projected/9f5db4ce-3fef-4033-8d83-71495a0b6373-kube-api-access-thm84\") pod \"maas-api-revoke-sub-b8a2e426-dxzqm\" (UID: \"9f5db4ce-3fef-4033-8d83-71495a0b6373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8a2e426-dxzqm" Nov 03 13:31:37.620770 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:37.620749 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f5db4ce-3fef-4033-8d83-71495a0b6373-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b8a2e426-dxzqm\" (UID: \"9f5db4ce-3fef-4033-8d83-71495a0b6373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8a2e426-dxzqm" Nov 03 13:31:37.623120 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:37.623098 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759"] Nov 03 13:31:37.623548 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:37.623520 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759" podUID="dc5abfae-7745-4857-b1f1-ff7d91a65ddd" containerName="istio-proxy" containerID="cri-o://38ee84a1ef40cc675de03cf7b86ef2a79808c566217ca0644e0c478c3482ee45" gracePeriod=30 Nov 03 13:31:37.628546 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:37.628524 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m"] Nov 03 13:31:37.628902 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:37.628876 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m" podUID="d8a57a81-8b6f-4049-9c5a-9884e968fee7" containerName="istio-proxy" containerID="cri-o://805c85f533d3da1367d1a1bed99c666311e802eeffa0006313025278dd1f26f2" gracePeriod=30 Nov 03 13:31:37.632541 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:37.632517 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf"] Nov 03 13:31:37.632857 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:37.632834 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf" podUID="620d1fb2-9a2b-4e36-819e-6997985fbc1b" containerName="istio-proxy" containerID="cri-o://034a9e2816fda727fe7e62094aca88d7049e132615fc3cdd1ec916d33882dc92" gracePeriod=30 Nov 03 13:31:37.636611 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:37.636589 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4"] Nov 03 13:31:37.636806 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:37.636774 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4" podUID="0a4f947e-dd99-4332-95dc-c41c040733b7" containerName="istio-proxy" containerID="cri-o://b59c381b93fa9c8d3329dc5260151cf8162f092a87241d3305f9d4acf0ca40cf" gracePeriod=30 Nov 03 13:31:37.715106 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:37.715073 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thm84\" (UniqueName: \"kubernetes.io/projected/9f5db4ce-3fef-4033-8d83-71495a0b6373-kube-api-access-thm84\") pod \"maas-api-revoke-sub-b8a2e426-dxzqm\" (UID: \"9f5db4ce-3fef-4033-8d83-71495a0b6373\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8a2e426-dxzqm" Nov 03 13:31:37.729700 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:37.729512 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8a2e426-dxzqm" Nov 03 13:31:37.828317 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:37.827744 5979 patch_prober.go:29] interesting pod/e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 03 13:31:37.828317 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:37.827801 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd" podUID="abf1926b-6e99-4b17-846d-8c2e9c924d9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 03 13:31:38.238839 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:38.238754 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8a2e426-dxzqm"] Nov 03 13:31:38.256491 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:31:38.248001 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f5db4ce_3fef_4033_8d83_71495a0b6373.slice/crio-19c37c809c90e6ce37babf4782e91040d45e7b5a7c0b2a78ab6f79a9689bedaa WatchSource:0}: Error finding container 19c37c809c90e6ce37babf4782e91040d45e7b5a7c0b2a78ab6f79a9689bedaa: Status 404 returned error can't find the container with id 19c37c809c90e6ce37babf4782e91040d45e7b5a7c0b2a78ab6f79a9689bedaa Nov 03 13:31:38.428232 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:38.428193 5979 patch_prober.go:29] interesting pod/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Nov 03 13:31:38.428403 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:38.428248 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7" podUID="dd4c2ad4-8ff1-40ba-91e5-5e4556213933" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Nov 03 13:31:38.772368 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:38.772329 5979 patch_prober.go:29] interesting pod/e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Nov 03 13:31:38.772682 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:38.772648 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd" podUID="abf1926b-6e99-4b17-846d-8c2e9c924d9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Nov 03 13:31:38.799989 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:38.786543 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd"] Nov 03 13:31:38.799989 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:38.798839 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd" podUID="3e80110d-950d-452f-9fdf-eecb56840a3c" containerName="istio-proxy" containerID="cri-o://bb0eeff0d2aa46c774377d182ba268af2f4372bbb98a29e51607aa1949913f6d" gracePeriod=30 Nov 03 13:31:38.801174 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:38.801149 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b"] Nov 03 13:31:38.801371 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:38.801347 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b" podUID="7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48" containerName="istio-proxy" containerID="cri-o://3a45e1eb1ce8fc3d09630da3da69fcedea2efa4460dfac8c37d65d275ec2f47b" gracePeriod=30 Nov 03 13:31:38.806106 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:38.806069 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9"] Nov 03 13:31:38.806291 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:38.806266 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9" podUID="26360457-2ef0-4c64-9a1c-4d62a783dc49" containerName="istio-proxy" containerID="cri-o://494e19cb04491dbd8aea6b37136258d4636937646ec2d53aaee8af9e85fff96d" gracePeriod=30 Nov 03 13:31:38.809570 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:38.809547 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh"] Nov 03 13:31:38.809938 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:38.809870 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh" podUID="821c73f7-cf2f-414c-8aea-1aff84e017af" containerName="istio-proxy" containerID="cri-o://c51d402870f6fc59222f6ef744e7c8e0e2d58ffbab0a0eefc227009683c30dc8" gracePeriod=30 Nov 03 13:31:39.118537 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:39.118501 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8a2e426-dxzqm" event={"ID":"9f5db4ce-3fef-4033-8d83-71495a0b6373","Type":"ContainerStarted","Data":"ba3b18c0434f4adb2184bed7e48c444e620195f7c39a2800171d59fd8bc8c4f5"} Nov 03 13:31:39.118733 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:39.118564 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8a2e426-dxzqm" event={"ID":"9f5db4ce-3fef-4033-8d83-71495a0b6373","Type":"ContainerStarted","Data":"19c37c809c90e6ce37babf4782e91040d45e7b5a7c0b2a78ab6f79a9689bedaa"} Nov 03 13:31:39.428181 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:39.428149 5979 patch_prober.go:29] interesting pod/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Nov 03 13:31:39.428355 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:39.428202 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7" podUID="dd4c2ad4-8ff1-40ba-91e5-5e4556213933" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Nov 03 13:31:39.440093 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:39.440046 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8a2e426-dxzqm" podStartSLOduration=2.440033889 podStartE2EDuration="2.440033889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:31:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:31:39.436407949 +0000 UTC m=+1581.447096191" watchObservedRunningTime="2026-11-03 13:31:39.440033889 +0000 UTC m=+1581.450722173" Nov 03 13:31:39.777443 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:39.777351 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd" Nov 03 13:31:39.777919 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:39.777890 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd" Nov 03 13:31:39.778539 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:39.778520 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd" Nov 03 13:31:40.136729 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:40.136696 5979 generic.go:356] "Generic (PLEG): container finished" podID="9f5db4ce-3fef-4033-8d83-71495a0b6373" containerID="ba3b18c0434f4adb2184bed7e48c444e620195f7c39a2800171d59fd8bc8c4f5" exitCode=0 Nov 03 13:31:40.136873 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:40.136783 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8a2e426-dxzqm" event={"ID":"9f5db4ce-3fef-4033-8d83-71495a0b6373","Type":"ContainerDied","Data":"ba3b18c0434f4adb2184bed7e48c444e620195f7c39a2800171d59fd8bc8c4f5"} Nov 03 13:31:40.491805 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:40.491699 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7" Nov 03 13:31:40.494153 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:40.494129 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7" Nov 03 13:31:41.295718 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:41.295689 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8a2e426-dxzqm" Nov 03 13:31:41.385973 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:41.385936 5979 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 13:31:41.386121 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:41.385989 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4" podUID="0a4f947e-dd99-4332-95dc-c41c040733b7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 13:31:41.400727 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:41.400671 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f5db4ce-3fef-4033-8d83-71495a0b6373-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f5db4ce-3fef-4033-8d83-71495a0b6373-maas-api-xks-ca\") pod \"9f5db4ce-3fef-4033-8d83-71495a0b6373\" (UID: \"9f5db4ce-3fef-4033-8d83-71495a0b6373\") " Nov 03 13:31:41.400727 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:41.400707 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f5db4ce-3fef-4033-8d83-71495a0b6373-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f5db4ce-3fef-4033-8d83-71495a0b6373-maas-api-openshift-ca\") pod \"9f5db4ce-3fef-4033-8d83-71495a0b6373\" (UID: \"9f5db4ce-3fef-4033-8d83-71495a0b6373\") " Nov 03 13:31:41.400943 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:41.400929 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f5db4ce-3fef-4033-8d83-71495a0b6373-kube-api-access-thm84\" (UniqueName: \"kubernetes.io/projected/9f5db4ce-3fef-4033-8d83-71495a0b6373-kube-api-access-thm84\") pod \"9f5db4ce-3fef-4033-8d83-71495a0b6373\" (UID: \"9f5db4ce-3fef-4033-8d83-71495a0b6373\") " Nov 03 13:31:41.401141 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:41.401118 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f5db4ce-3fef-4033-8d83-71495a0b6373-maas-api-openshift-ca" pod "9f5db4ce-3fef-4033-8d83-71495a0b6373" (UID: "9f5db4ce-3fef-4033-8d83-71495a0b6373"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:31:41.401506 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:41.401489 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f5db4ce-3fef-4033-8d83-71495a0b6373-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:41.402846 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:41.402825 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f5db4ce-3fef-4033-8d83-71495a0b6373-maas-api-xks-ca" pod "9f5db4ce-3fef-4033-8d83-71495a0b6373" (UID: "9f5db4ce-3fef-4033-8d83-71495a0b6373"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:31:41.402949 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:41.402926 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f5db4ce-3fef-4033-8d83-71495a0b6373-kube-api-access-thm84" pod "9f5db4ce-3fef-4033-8d83-71495a0b6373" (UID: "9f5db4ce-3fef-4033-8d83-71495a0b6373"). InnerVolumeSpecName "kube-api-access-thm84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:31:41.502946 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:41.502907 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thm84\" (UniqueName: \"kubernetes.io/projected/9f5db4ce-3fef-4033-8d83-71495a0b6373-kube-api-access-thm84\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:41.502946 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:41.502940 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f5db4ce-3fef-4033-8d83-71495a0b6373-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:41.890171 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:41.890129 5979 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 13:31:41.890334 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:41.890200 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m" podUID="d8a57a81-8b6f-4049-9c5a-9884e968fee7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 13:31:42.151996 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:42.151913 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8a2e426-dxzqm" Nov 03 13:31:42.151996 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:42.151939 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b8a2e426-dxzqm" event={"ID":"9f5db4ce-3fef-4033-8d83-71495a0b6373","Type":"ContainerDied","Data":"19c37c809c90e6ce37babf4782e91040d45e7b5a7c0b2a78ab6f79a9689bedaa"} Nov 03 13:31:42.151996 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:42.151990 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19c37c809c90e6ce37babf4782e91040d45e7b5a7c0b2a78ab6f79a9689bedaa" Nov 03 13:31:42.929728 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:42.929699 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m" Nov 03 13:31:42.940675 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:42.940643 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759" Nov 03 13:31:43.002039 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.001998 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf" Nov 03 13:31:43.017585 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.017560 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4" Nov 03 13:31:43.025078 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.025041 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8a57a81-8b6f-4049-9c5a-9884e968fee7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8a57a81-8b6f-4049-9c5a-9884e968fee7-workload-certs\") pod \"d8a57a81-8b6f-4049-9c5a-9884e968fee7\" (UID: \"d8a57a81-8b6f-4049-9c5a-9884e968fee7\") " Nov 03 13:31:43.025206 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.025088 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/620d1fb2-9a2b-4e36-819e-6997985fbc1b-kube-api-access-cs9rz\" (UniqueName: \"kubernetes.io/projected/620d1fb2-9a2b-4e36-819e-6997985fbc1b-kube-api-access-cs9rz\") pod \"620d1fb2-9a2b-4e36-819e-6997985fbc1b\" (UID: \"620d1fb2-9a2b-4e36-819e-6997985fbc1b\") " Nov 03 13:31:43.025206 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.025123 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/620d1fb2-9a2b-4e36-819e-6997985fbc1b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/620d1fb2-9a2b-4e36-819e-6997985fbc1b-istio-envoy\") pod \"620d1fb2-9a2b-4e36-819e-6997985fbc1b\" (UID: \"620d1fb2-9a2b-4e36-819e-6997985fbc1b\") " Nov 03 13:31:43.025403 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.025238 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8a57a81-8b6f-4049-9c5a-9884e968fee7-workload-certs" pod "d8a57a81-8b6f-4049-9c5a-9884e968fee7" (UID: "d8a57a81-8b6f-4049-9c5a-9884e968fee7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:43.025403 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.025277 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-credential-socket\") pod \"dc5abfae-7745-4857-b1f1-ff7d91a65ddd\" (UID: \"dc5abfae-7745-4857-b1f1-ff7d91a65ddd\") " Nov 03 13:31:43.025684 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.025339 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/620d1fb2-9a2b-4e36-819e-6997985fbc1b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/620d1fb2-9a2b-4e36-819e-6997985fbc1b-istiod-ca-cert\") pod \"620d1fb2-9a2b-4e36-819e-6997985fbc1b\" (UID: \"620d1fb2-9a2b-4e36-819e-6997985fbc1b\") " Nov 03 13:31:43.025766 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.025710 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8a57a81-8b6f-4049-9c5a-9884e968fee7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8a57a81-8b6f-4049-9c5a-9884e968fee7-istio-data\") pod \"d8a57a81-8b6f-4049-9c5a-9884e968fee7\" (UID: \"d8a57a81-8b6f-4049-9c5a-9884e968fee7\") " Nov 03 13:31:43.025766 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.025741 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/620d1fb2-9a2b-4e36-819e-6997985fbc1b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/620d1fb2-9a2b-4e36-819e-6997985fbc1b-workload-certs\") pod \"620d1fb2-9a2b-4e36-819e-6997985fbc1b\" (UID: \"620d1fb2-9a2b-4e36-819e-6997985fbc1b\") " Nov 03 13:31:43.025766 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.025592 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-credential-socket" pod "dc5abfae-7745-4857-b1f1-ff7d91a65ddd" (UID: "dc5abfae-7745-4857-b1f1-ff7d91a65ddd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:43.025766 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.025742 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/620d1fb2-9a2b-4e36-819e-6997985fbc1b-istiod-ca-cert" pod "620d1fb2-9a2b-4e36-819e-6997985fbc1b" (UID: "620d1fb2-9a2b-4e36-819e-6997985fbc1b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:31:43.025987 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.025775 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-istio-podinfo\") pod \"dc5abfae-7745-4857-b1f1-ff7d91a65ddd\" (UID: \"dc5abfae-7745-4857-b1f1-ff7d91a65ddd\") " Nov 03 13:31:43.025987 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.025822 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8a57a81-8b6f-4049-9c5a-9884e968fee7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a57a81-8b6f-4049-9c5a-9884e968fee7-credential-socket\") pod \"d8a57a81-8b6f-4049-9c5a-9884e968fee7\" (UID: \"d8a57a81-8b6f-4049-9c5a-9884e968fee7\") " Nov 03 13:31:43.025987 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.025856 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8a57a81-8b6f-4049-9c5a-9884e968fee7-kube-api-access-86mhx\" (UniqueName: \"kubernetes.io/projected/d8a57a81-8b6f-4049-9c5a-9884e968fee7-kube-api-access-86mhx\") pod \"d8a57a81-8b6f-4049-9c5a-9884e968fee7\" (UID: \"d8a57a81-8b6f-4049-9c5a-9884e968fee7\") " Nov 03 13:31:43.025987 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.025879 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-istio-token\" (UniqueName: \"kubernetes.io/projected/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-istio-token\") pod \"dc5abfae-7745-4857-b1f1-ff7d91a65ddd\" (UID: \"dc5abfae-7745-4857-b1f1-ff7d91a65ddd\") " Nov 03 13:31:43.025987 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.025919 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8a57a81-8b6f-4049-9c5a-9884e968fee7-istio-data" pod "d8a57a81-8b6f-4049-9c5a-9884e968fee7" (UID: "d8a57a81-8b6f-4049-9c5a-9884e968fee7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:43.025987 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.025920 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8a57a81-8b6f-4049-9c5a-9884e968fee7-istio-token\" (UniqueName: \"kubernetes.io/projected/d8a57a81-8b6f-4049-9c5a-9884e968fee7-istio-token\") pod \"d8a57a81-8b6f-4049-9c5a-9884e968fee7\" (UID: \"d8a57a81-8b6f-4049-9c5a-9884e968fee7\") " Nov 03 13:31:43.025987 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.025986 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d8a57a81-8b6f-4049-9c5a-9884e968fee7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8a57a81-8b6f-4049-9c5a-9884e968fee7-istio-podinfo\") pod \"d8a57a81-8b6f-4049-9c5a-9884e968fee7\" (UID: \"d8a57a81-8b6f-4049-9c5a-9884e968fee7\") " Nov 03 13:31:43.026374 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.026019 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/620d1fb2-9a2b-4e36-819e-6997985fbc1b-istio-token\" (UniqueName: \"kubernetes.io/projected/620d1fb2-9a2b-4e36-819e-6997985fbc1b-istio-token\") pod \"620d1fb2-9a2b-4e36-819e-6997985fbc1b\" (UID: \"620d1fb2-9a2b-4e36-819e-6997985fbc1b\") " Nov 03 13:31:43.026374 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.026064 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8a57a81-8b6f-4049-9c5a-9884e968fee7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8a57a81-8b6f-4049-9c5a-9884e968fee7-istio-envoy\") pod \"d8a57a81-8b6f-4049-9c5a-9884e968fee7\" (UID: \"d8a57a81-8b6f-4049-9c5a-9884e968fee7\") " Nov 03 13:31:43.026374 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.026104 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/620d1fb2-9a2b-4e36-819e-6997985fbc1b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/620d1fb2-9a2b-4e36-819e-6997985fbc1b-workload-socket\") pod \"620d1fb2-9a2b-4e36-819e-6997985fbc1b\" (UID: \"620d1fb2-9a2b-4e36-819e-6997985fbc1b\") " Nov 03 13:31:43.026374 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.026210 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/620d1fb2-9a2b-4e36-819e-6997985fbc1b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/620d1fb2-9a2b-4e36-819e-6997985fbc1b-istio-podinfo\") pod \"620d1fb2-9a2b-4e36-819e-6997985fbc1b\" (UID: \"620d1fb2-9a2b-4e36-819e-6997985fbc1b\") " Nov 03 13:31:43.026374 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.026262 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-kube-api-access-2shnh\" (UniqueName: \"kubernetes.io/projected/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-kube-api-access-2shnh\") pod \"dc5abfae-7745-4857-b1f1-ff7d91a65ddd\" (UID: \"dc5abfae-7745-4857-b1f1-ff7d91a65ddd\") " Nov 03 13:31:43.026374 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.026307 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-workload-certs\") pod \"dc5abfae-7745-4857-b1f1-ff7d91a65ddd\" (UID: \"dc5abfae-7745-4857-b1f1-ff7d91a65ddd\") " Nov 03 13:31:43.026374 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.026344 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-istio-data\") pod \"dc5abfae-7745-4857-b1f1-ff7d91a65ddd\" (UID: \"dc5abfae-7745-4857-b1f1-ff7d91a65ddd\") " Nov 03 13:31:43.026374 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.026374 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/620d1fb2-9a2b-4e36-819e-6997985fbc1b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/620d1fb2-9a2b-4e36-819e-6997985fbc1b-istio-data\") pod \"620d1fb2-9a2b-4e36-819e-6997985fbc1b\" (UID: \"620d1fb2-9a2b-4e36-819e-6997985fbc1b\") " Nov 03 13:31:43.026823 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.026420 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-istio-envoy\") pod \"dc5abfae-7745-4857-b1f1-ff7d91a65ddd\" (UID: \"dc5abfae-7745-4857-b1f1-ff7d91a65ddd\") " Nov 03 13:31:43.026823 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.026455 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8a57a81-8b6f-4049-9c5a-9884e968fee7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8a57a81-8b6f-4049-9c5a-9884e968fee7-istiod-ca-cert\") pod \"d8a57a81-8b6f-4049-9c5a-9884e968fee7\" (UID: \"d8a57a81-8b6f-4049-9c5a-9884e968fee7\") " Nov 03 13:31:43.026823 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.026508 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-istiod-ca-cert\") pod \"dc5abfae-7745-4857-b1f1-ff7d91a65ddd\" (UID: \"dc5abfae-7745-4857-b1f1-ff7d91a65ddd\") " Nov 03 13:31:43.026823 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.026556 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8a57a81-8b6f-4049-9c5a-9884e968fee7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a57a81-8b6f-4049-9c5a-9884e968fee7-workload-socket\") pod \"d8a57a81-8b6f-4049-9c5a-9884e968fee7\" (UID: \"d8a57a81-8b6f-4049-9c5a-9884e968fee7\") " Nov 03 13:31:43.026823 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.026616 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/620d1fb2-9a2b-4e36-819e-6997985fbc1b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/620d1fb2-9a2b-4e36-819e-6997985fbc1b-credential-socket\") pod \"620d1fb2-9a2b-4e36-819e-6997985fbc1b\" (UID: \"620d1fb2-9a2b-4e36-819e-6997985fbc1b\") " Nov 03 13:31:43.026823 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.026673 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-workload-socket\") pod \"dc5abfae-7745-4857-b1f1-ff7d91a65ddd\" (UID: \"dc5abfae-7745-4857-b1f1-ff7d91a65ddd\") " Nov 03 13:31:43.027252 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.026210 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8a57a81-8b6f-4049-9c5a-9884e968fee7-credential-socket" pod "d8a57a81-8b6f-4049-9c5a-9884e968fee7" (UID: "d8a57a81-8b6f-4049-9c5a-9884e968fee7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:43.027252 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.026872 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-workload-certs" pod "dc5abfae-7745-4857-b1f1-ff7d91a65ddd" (UID: "dc5abfae-7745-4857-b1f1-ff7d91a65ddd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:43.027362 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.027144 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/620d1fb2-9a2b-4e36-819e-6997985fbc1b-workload-certs" pod "620d1fb2-9a2b-4e36-819e-6997985fbc1b" (UID: "620d1fb2-9a2b-4e36-819e-6997985fbc1b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:43.027571 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.027545 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-istio-data" pod "dc5abfae-7745-4857-b1f1-ff7d91a65ddd" (UID: "dc5abfae-7745-4857-b1f1-ff7d91a65ddd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:43.028686 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.027889 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8a57a81-8b6f-4049-9c5a-9884e968fee7-workload-socket" pod "d8a57a81-8b6f-4049-9c5a-9884e968fee7" (UID: "d8a57a81-8b6f-4049-9c5a-9884e968fee7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:43.028686 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.028164 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-workload-socket" pod "dc5abfae-7745-4857-b1f1-ff7d91a65ddd" (UID: "dc5abfae-7745-4857-b1f1-ff7d91a65ddd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:43.028686 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.028194 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/620d1fb2-9a2b-4e36-819e-6997985fbc1b-kube-api-access-cs9rz" pod "620d1fb2-9a2b-4e36-819e-6997985fbc1b" (UID: "620d1fb2-9a2b-4e36-819e-6997985fbc1b"). InnerVolumeSpecName "kube-api-access-cs9rz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:31:43.028686 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.028243 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/620d1fb2-9a2b-4e36-819e-6997985fbc1b-workload-socket" pod "620d1fb2-9a2b-4e36-819e-6997985fbc1b" (UID: "620d1fb2-9a2b-4e36-819e-6997985fbc1b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:43.028686 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.028260 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/620d1fb2-9a2b-4e36-819e-6997985fbc1b-credential-socket" pod "620d1fb2-9a2b-4e36-819e-6997985fbc1b" (UID: "620d1fb2-9a2b-4e36-819e-6997985fbc1b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:43.028686 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.028324 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8a57a81-8b6f-4049-9c5a-9884e968fee7-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:43.028686 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.028350 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:43.028686 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.028367 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/620d1fb2-9a2b-4e36-819e-6997985fbc1b-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:43.028686 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.028382 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8a57a81-8b6f-4049-9c5a-9884e968fee7-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:43.028686 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.028399 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/620d1fb2-9a2b-4e36-819e-6997985fbc1b-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:43.028686 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.028415 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a57a81-8b6f-4049-9c5a-9884e968fee7-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:43.028686 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.028431 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:43.028686 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.028446 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:43.028686 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.028460 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8a57a81-8b6f-4049-9c5a-9884e968fee7-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:43.028686 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.028509 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:43.028686 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.028592 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-istiod-ca-cert" pod "dc5abfae-7745-4857-b1f1-ff7d91a65ddd" (UID: "dc5abfae-7745-4857-b1f1-ff7d91a65ddd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:31:43.028686 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.028593 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8a57a81-8b6f-4049-9c5a-9884e968fee7-istiod-ca-cert" pod "d8a57a81-8b6f-4049-9c5a-9884e968fee7" (UID: "d8a57a81-8b6f-4049-9c5a-9884e968fee7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:31:43.029785 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.029731 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/620d1fb2-9a2b-4e36-819e-6997985fbc1b-istio-podinfo" pod "620d1fb2-9a2b-4e36-819e-6997985fbc1b" (UID: "620d1fb2-9a2b-4e36-819e-6997985fbc1b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:31:43.029785 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.029752 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/620d1fb2-9a2b-4e36-819e-6997985fbc1b-istio-data" pod "620d1fb2-9a2b-4e36-819e-6997985fbc1b" (UID: "620d1fb2-9a2b-4e36-819e-6997985fbc1b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:43.029919 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.029887 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-kube-api-access-2shnh" pod "dc5abfae-7745-4857-b1f1-ff7d91a65ddd" (UID: "dc5abfae-7745-4857-b1f1-ff7d91a65ddd"). InnerVolumeSpecName "kube-api-access-2shnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:31:43.030775 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.030747 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8a57a81-8b6f-4049-9c5a-9884e968fee7-istio-envoy" pod "d8a57a81-8b6f-4049-9c5a-9884e968fee7" (UID: "d8a57a81-8b6f-4049-9c5a-9884e968fee7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:43.030775 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.030766 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-istio-token" pod "dc5abfae-7745-4857-b1f1-ff7d91a65ddd" (UID: "dc5abfae-7745-4857-b1f1-ff7d91a65ddd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:31:43.030914 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.030789 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-istio-podinfo" pod "dc5abfae-7745-4857-b1f1-ff7d91a65ddd" (UID: "dc5abfae-7745-4857-b1f1-ff7d91a65ddd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:31:43.030914 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.030795 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-istio-envoy" pod "dc5abfae-7745-4857-b1f1-ff7d91a65ddd" (UID: "dc5abfae-7745-4857-b1f1-ff7d91a65ddd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:43.030914 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.030792 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8a57a81-8b6f-4049-9c5a-9884e968fee7-kube-api-access-86mhx" pod "d8a57a81-8b6f-4049-9c5a-9884e968fee7" (UID: "d8a57a81-8b6f-4049-9c5a-9884e968fee7"). InnerVolumeSpecName "kube-api-access-86mhx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:31:43.030914 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.030799 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/620d1fb2-9a2b-4e36-819e-6997985fbc1b-istio-token" pod "620d1fb2-9a2b-4e36-819e-6997985fbc1b" (UID: "620d1fb2-9a2b-4e36-819e-6997985fbc1b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:31:43.030914 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.030813 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/620d1fb2-9a2b-4e36-819e-6997985fbc1b-istio-envoy" pod "620d1fb2-9a2b-4e36-819e-6997985fbc1b" (UID: "620d1fb2-9a2b-4e36-819e-6997985fbc1b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:43.031092 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.030940 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8a57a81-8b6f-4049-9c5a-9884e968fee7-istio-podinfo" pod "d8a57a81-8b6f-4049-9c5a-9884e968fee7" (UID: "d8a57a81-8b6f-4049-9c5a-9884e968fee7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:31:43.032059 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.032044 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8a57a81-8b6f-4049-9c5a-9884e968fee7-istio-token" pod "d8a57a81-8b6f-4049-9c5a-9884e968fee7" (UID: "d8a57a81-8b6f-4049-9c5a-9884e968fee7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:31:43.044447 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.044419 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e11b3dc0-flp98"] Nov 03 13:31:43.051934 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.051909 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e11b3dc0-flp98"] Nov 03 13:31:43.129497 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.129453 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a4f947e-dd99-4332-95dc-c41c040733b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a4f947e-dd99-4332-95dc-c41c040733b7-istio-envoy\") pod \"0a4f947e-dd99-4332-95dc-c41c040733b7\" (UID: \"0a4f947e-dd99-4332-95dc-c41c040733b7\") " Nov 03 13:31:43.129660 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.129583 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a4f947e-dd99-4332-95dc-c41c040733b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a4f947e-dd99-4332-95dc-c41c040733b7-workload-socket\") pod \"0a4f947e-dd99-4332-95dc-c41c040733b7\" (UID: \"0a4f947e-dd99-4332-95dc-c41c040733b7\") " Nov 03 13:31:43.129660 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.129640 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a4f947e-dd99-4332-95dc-c41c040733b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a4f947e-dd99-4332-95dc-c41c040733b7-credential-socket\") pod \"0a4f947e-dd99-4332-95dc-c41c040733b7\" (UID: \"0a4f947e-dd99-4332-95dc-c41c040733b7\") " Nov 03 13:31:43.129782 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.129701 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a4f947e-dd99-4332-95dc-c41c040733b7-istio-token\" (UniqueName: \"kubernetes.io/projected/0a4f947e-dd99-4332-95dc-c41c040733b7-istio-token\") pod \"0a4f947e-dd99-4332-95dc-c41c040733b7\" (UID: \"0a4f947e-dd99-4332-95dc-c41c040733b7\") " Nov 03 13:31:43.129782 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.129739 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a4f947e-dd99-4332-95dc-c41c040733b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a4f947e-dd99-4332-95dc-c41c040733b7-workload-certs\") pod \"0a4f947e-dd99-4332-95dc-c41c040733b7\" (UID: \"0a4f947e-dd99-4332-95dc-c41c040733b7\") " Nov 03 13:31:43.129913 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.129806 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a4f947e-dd99-4332-95dc-c41c040733b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a4f947e-dd99-4332-95dc-c41c040733b7-istio-podinfo\") pod \"0a4f947e-dd99-4332-95dc-c41c040733b7\" (UID: \"0a4f947e-dd99-4332-95dc-c41c040733b7\") " Nov 03 13:31:43.129913 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.129843 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a4f947e-dd99-4332-95dc-c41c040733b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a4f947e-dd99-4332-95dc-c41c040733b7-istiod-ca-cert\") pod \"0a4f947e-dd99-4332-95dc-c41c040733b7\" (UID: \"0a4f947e-dd99-4332-95dc-c41c040733b7\") " Nov 03 13:31:43.129913 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.129908 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a4f947e-dd99-4332-95dc-c41c040733b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a4f947e-dd99-4332-95dc-c41c040733b7-istio-data\") pod \"0a4f947e-dd99-4332-95dc-c41c040733b7\" (UID: \"0a4f947e-dd99-4332-95dc-c41c040733b7\") " Nov 03 13:31:43.130066 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.129962 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a4f947e-dd99-4332-95dc-c41c040733b7-kube-api-access-7vlpc\" (UniqueName: \"kubernetes.io/projected/0a4f947e-dd99-4332-95dc-c41c040733b7-kube-api-access-7vlpc\") pod \"0a4f947e-dd99-4332-95dc-c41c040733b7\" (UID: \"0a4f947e-dd99-4332-95dc-c41c040733b7\") " Nov 03 13:31:43.130120 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.130062 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a4f947e-dd99-4332-95dc-c41c040733b7-credential-socket" pod "0a4f947e-dd99-4332-95dc-c41c040733b7" (UID: "0a4f947e-dd99-4332-95dc-c41c040733b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:43.132715 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.130316 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a4f947e-dd99-4332-95dc-c41c040733b7-workload-socket" pod "0a4f947e-dd99-4332-95dc-c41c040733b7" (UID: "0a4f947e-dd99-4332-95dc-c41c040733b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:43.132715 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.130330 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a4f947e-dd99-4332-95dc-c41c040733b7-workload-certs" pod "0a4f947e-dd99-4332-95dc-c41c040733b7" (UID: "0a4f947e-dd99-4332-95dc-c41c040733b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:43.132715 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.131034 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs9rz\" (UniqueName: \"kubernetes.io/projected/620d1fb2-9a2b-4e36-819e-6997985fbc1b-kube-api-access-cs9rz\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:43.132715 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.131282 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/620d1fb2-9a2b-4e36-819e-6997985fbc1b-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:43.132715 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.131303 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a4f947e-dd99-4332-95dc-c41c040733b7-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:43.132715 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.131317 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a4f947e-dd99-4332-95dc-c41c040733b7-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:43.132715 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.131332 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:43.132715 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.131346 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a4f947e-dd99-4332-95dc-c41c040733b7-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:43.132715 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.131360 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86mhx\" (UniqueName: \"kubernetes.io/projected/d8a57a81-8b6f-4049-9c5a-9884e968fee7-kube-api-access-86mhx\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:43.132715 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.131374 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:43.132715 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.131388 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8a57a81-8b6f-4049-9c5a-9884e968fee7-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:43.132715 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.131401 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8a57a81-8b6f-4049-9c5a-9884e968fee7-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:43.132715 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.131414 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/620d1fb2-9a2b-4e36-819e-6997985fbc1b-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:43.132715 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.131428 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8a57a81-8b6f-4049-9c5a-9884e968fee7-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:43.132715 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.132160 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a4f947e-dd99-4332-95dc-c41c040733b7-istiod-ca-cert" pod "0a4f947e-dd99-4332-95dc-c41c040733b7" (UID: "0a4f947e-dd99-4332-95dc-c41c040733b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:31:43.132715 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.132421 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/620d1fb2-9a2b-4e36-819e-6997985fbc1b-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:43.132715 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.132446 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/620d1fb2-9a2b-4e36-819e-6997985fbc1b-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:43.132715 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.132466 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2shnh\" (UniqueName: \"kubernetes.io/projected/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-kube-api-access-2shnh\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:43.132715 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.132502 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/620d1fb2-9a2b-4e36-819e-6997985fbc1b-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:43.132715 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.132516 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:43.132715 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.132531 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8a57a81-8b6f-4049-9c5a-9884e968fee7-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:43.132715 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.132545 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc5abfae-7745-4857-b1f1-ff7d91a65ddd-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:43.132715 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.132559 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/620d1fb2-9a2b-4e36-819e-6997985fbc1b-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:43.132715 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.132676 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a4f947e-dd99-4332-95dc-c41c040733b7-kube-api-access-7vlpc" pod "0a4f947e-dd99-4332-95dc-c41c040733b7" (UID: "0a4f947e-dd99-4332-95dc-c41c040733b7"). InnerVolumeSpecName "kube-api-access-7vlpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:31:43.134435 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.133425 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a4f947e-dd99-4332-95dc-c41c040733b7-istio-data" pod "0a4f947e-dd99-4332-95dc-c41c040733b7" (UID: "0a4f947e-dd99-4332-95dc-c41c040733b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:43.136364 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.136327 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a4f947e-dd99-4332-95dc-c41c040733b7-istio-envoy" pod "0a4f947e-dd99-4332-95dc-c41c040733b7" (UID: "0a4f947e-dd99-4332-95dc-c41c040733b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:43.137902 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.136708 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a4f947e-dd99-4332-95dc-c41c040733b7-istio-token" pod "0a4f947e-dd99-4332-95dc-c41c040733b7" (UID: "0a4f947e-dd99-4332-95dc-c41c040733b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:31:43.137902 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.136770 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a4f947e-dd99-4332-95dc-c41c040733b7-istio-podinfo" pod "0a4f947e-dd99-4332-95dc-c41c040733b7" (UID: "0a4f947e-dd99-4332-95dc-c41c040733b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:31:43.165489 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.165391 5979 generic.go:356] "Generic (PLEG): container finished" podID="0a4f947e-dd99-4332-95dc-c41c040733b7" containerID="b59c381b93fa9c8d3329dc5260151cf8162f092a87241d3305f9d4acf0ca40cf" exitCode=0 Nov 03 13:31:43.165489 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.165451 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4" Nov 03 13:31:43.165705 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.165481 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4" event={"ID":"0a4f947e-dd99-4332-95dc-c41c040733b7","Type":"ContainerDied","Data":"b59c381b93fa9c8d3329dc5260151cf8162f092a87241d3305f9d4acf0ca40cf"} Nov 03 13:31:43.165705 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.165675 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4" event={"ID":"0a4f947e-dd99-4332-95dc-c41c040733b7","Type":"ContainerDied","Data":"745cb3f242784008e96da88ac43b8a545cda62a317620c9f1fd5ebe409ca0ff4"} Nov 03 13:31:43.165820 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.165713 5979 scope.go:122] "RemoveContainer" containerID="b59c381b93fa9c8d3329dc5260151cf8162f092a87241d3305f9d4acf0ca40cf" Nov 03 13:31:43.167151 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.167128 5979 generic.go:356] "Generic (PLEG): container finished" podID="620d1fb2-9a2b-4e36-819e-6997985fbc1b" containerID="034a9e2816fda727fe7e62094aca88d7049e132615fc3cdd1ec916d33882dc92" exitCode=0 Nov 03 13:31:43.167260 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.167178 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf" Nov 03 13:31:43.167322 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.167293 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf" event={"ID":"620d1fb2-9a2b-4e36-819e-6997985fbc1b","Type":"ContainerDied","Data":"034a9e2816fda727fe7e62094aca88d7049e132615fc3cdd1ec916d33882dc92"} Nov 03 13:31:43.167384 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.167329 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf" event={"ID":"620d1fb2-9a2b-4e36-819e-6997985fbc1b","Type":"ContainerDied","Data":"3dd7e62c0a6bf29bb265c538a2dd0d47f4232abd5726e83e136f70bd32fc4c7c"} Nov 03 13:31:43.172849 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.172791 5979 generic.go:356] "Generic (PLEG): container finished" podID="d8a57a81-8b6f-4049-9c5a-9884e968fee7" containerID="805c85f533d3da1367d1a1bed99c666311e802eeffa0006313025278dd1f26f2" exitCode=0 Nov 03 13:31:43.173073 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.173044 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m" event={"ID":"d8a57a81-8b6f-4049-9c5a-9884e968fee7","Type":"ContainerDied","Data":"805c85f533d3da1367d1a1bed99c666311e802eeffa0006313025278dd1f26f2"} Nov 03 13:31:43.173172 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.173097 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m" event={"ID":"d8a57a81-8b6f-4049-9c5a-9884e968fee7","Type":"ContainerDied","Data":"039d35a97d82e80edab224d8171c22432978580f41e7c2d095a43cf6cbafa820"} Nov 03 13:31:43.173247 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.173178 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m" Nov 03 13:31:43.177911 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.177643 5979 scope.go:122] "RemoveContainer" containerID="b59c381b93fa9c8d3329dc5260151cf8162f092a87241d3305f9d4acf0ca40cf" Nov 03 13:31:43.177911 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.177808 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759" Nov 03 13:31:43.177911 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.177763 5979 generic.go:356] "Generic (PLEG): container finished" podID="dc5abfae-7745-4857-b1f1-ff7d91a65ddd" containerID="38ee84a1ef40cc675de03cf7b86ef2a79808c566217ca0644e0c478c3482ee45" exitCode=0 Nov 03 13:31:43.177911 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.177847 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759" event={"ID":"dc5abfae-7745-4857-b1f1-ff7d91a65ddd","Type":"ContainerDied","Data":"38ee84a1ef40cc675de03cf7b86ef2a79808c566217ca0644e0c478c3482ee45"} Nov 03 13:31:43.177911 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.177886 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759" event={"ID":"dc5abfae-7745-4857-b1f1-ff7d91a65ddd","Type":"ContainerDied","Data":"f8d2769f98f5443810d1295886aa52ddcfc418b8cd93ccdbc0a12e4b5ad5956a"} Nov 03 13:31:43.178174 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:31:43.177954 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b59c381b93fa9c8d3329dc5260151cf8162f092a87241d3305f9d4acf0ca40cf\": container with ID starting with b59c381b93fa9c8d3329dc5260151cf8162f092a87241d3305f9d4acf0ca40cf not found: ID does not exist" containerID="b59c381b93fa9c8d3329dc5260151cf8162f092a87241d3305f9d4acf0ca40cf" Nov 03 13:31:43.178174 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.177990 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b59c381b93fa9c8d3329dc5260151cf8162f092a87241d3305f9d4acf0ca40cf"} err="failed to get container status \"b59c381b93fa9c8d3329dc5260151cf8162f092a87241d3305f9d4acf0ca40cf\": rpc error: code = NotFound desc = could not find container \"b59c381b93fa9c8d3329dc5260151cf8162f092a87241d3305f9d4acf0ca40cf\": container with ID starting with b59c381b93fa9c8d3329dc5260151cf8162f092a87241d3305f9d4acf0ca40cf not found: ID does not exist" Nov 03 13:31:43.178174 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.178013 5979 scope.go:122] "RemoveContainer" containerID="034a9e2816fda727fe7e62094aca88d7049e132615fc3cdd1ec916d33882dc92" Nov 03 13:31:43.189969 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.189951 5979 scope.go:122] "RemoveContainer" containerID="034a9e2816fda727fe7e62094aca88d7049e132615fc3cdd1ec916d33882dc92" Nov 03 13:31:43.190263 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:31:43.190236 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"034a9e2816fda727fe7e62094aca88d7049e132615fc3cdd1ec916d33882dc92\": container with ID starting with 034a9e2816fda727fe7e62094aca88d7049e132615fc3cdd1ec916d33882dc92 not found: ID does not exist" containerID="034a9e2816fda727fe7e62094aca88d7049e132615fc3cdd1ec916d33882dc92" Nov 03 13:31:43.190367 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.190270 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"034a9e2816fda727fe7e62094aca88d7049e132615fc3cdd1ec916d33882dc92"} err="failed to get container status \"034a9e2816fda727fe7e62094aca88d7049e132615fc3cdd1ec916d33882dc92\": rpc error: code = NotFound desc = could not find container \"034a9e2816fda727fe7e62094aca88d7049e132615fc3cdd1ec916d33882dc92\": container with ID starting with 034a9e2816fda727fe7e62094aca88d7049e132615fc3cdd1ec916d33882dc92 not found: ID does not exist" Nov 03 13:31:43.190367 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.190291 5979 scope.go:122] "RemoveContainer" containerID="805c85f533d3da1367d1a1bed99c666311e802eeffa0006313025278dd1f26f2" Nov 03 13:31:43.199393 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.199364 5979 scope.go:122] "RemoveContainer" containerID="5b71848d06a9680f1862c89aedcf2da53f592a49762b808ecb20a5c7195c4795" Nov 03 13:31:43.213539 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.213508 5979 scope.go:122] "RemoveContainer" containerID="805c85f533d3da1367d1a1bed99c666311e802eeffa0006313025278dd1f26f2" Nov 03 13:31:43.213927 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:31:43.213901 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"805c85f533d3da1367d1a1bed99c666311e802eeffa0006313025278dd1f26f2\": container with ID starting with 805c85f533d3da1367d1a1bed99c666311e802eeffa0006313025278dd1f26f2 not found: ID does not exist" containerID="805c85f533d3da1367d1a1bed99c666311e802eeffa0006313025278dd1f26f2" Nov 03 13:31:43.214047 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.213931 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"805c85f533d3da1367d1a1bed99c666311e802eeffa0006313025278dd1f26f2"} err="failed to get container status \"805c85f533d3da1367d1a1bed99c666311e802eeffa0006313025278dd1f26f2\": rpc error: code = NotFound desc = could not find container \"805c85f533d3da1367d1a1bed99c666311e802eeffa0006313025278dd1f26f2\": container with ID starting with 805c85f533d3da1367d1a1bed99c666311e802eeffa0006313025278dd1f26f2 not found: ID does not exist" Nov 03 13:31:43.214047 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.213949 5979 scope.go:122] "RemoveContainer" containerID="5b71848d06a9680f1862c89aedcf2da53f592a49762b808ecb20a5c7195c4795" Nov 03 13:31:43.214264 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:31:43.214239 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b71848d06a9680f1862c89aedcf2da53f592a49762b808ecb20a5c7195c4795\": container with ID starting with 5b71848d06a9680f1862c89aedcf2da53f592a49762b808ecb20a5c7195c4795 not found: ID does not exist" containerID="5b71848d06a9680f1862c89aedcf2da53f592a49762b808ecb20a5c7195c4795" Nov 03 13:31:43.214328 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.214271 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b71848d06a9680f1862c89aedcf2da53f592a49762b808ecb20a5c7195c4795"} err="failed to get container status \"5b71848d06a9680f1862c89aedcf2da53f592a49762b808ecb20a5c7195c4795\": rpc error: code = NotFound desc = could not find container \"5b71848d06a9680f1862c89aedcf2da53f592a49762b808ecb20a5c7195c4795\": container with ID starting with 5b71848d06a9680f1862c89aedcf2da53f592a49762b808ecb20a5c7195c4795 not found: ID does not exist" Nov 03 13:31:43.214328 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.214288 5979 scope.go:122] "RemoveContainer" containerID="38ee84a1ef40cc675de03cf7b86ef2a79808c566217ca0644e0c478c3482ee45" Nov 03 13:31:43.228148 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.228127 5979 scope.go:122] "RemoveContainer" containerID="38ee84a1ef40cc675de03cf7b86ef2a79808c566217ca0644e0c478c3482ee45" Nov 03 13:31:43.228610 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:31:43.228445 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38ee84a1ef40cc675de03cf7b86ef2a79808c566217ca0644e0c478c3482ee45\": container with ID starting with 38ee84a1ef40cc675de03cf7b86ef2a79808c566217ca0644e0c478c3482ee45 not found: ID does not exist" containerID="38ee84a1ef40cc675de03cf7b86ef2a79808c566217ca0644e0c478c3482ee45" Nov 03 13:31:43.228610 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.228495 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38ee84a1ef40cc675de03cf7b86ef2a79808c566217ca0644e0c478c3482ee45"} err="failed to get container status \"38ee84a1ef40cc675de03cf7b86ef2a79808c566217ca0644e0c478c3482ee45\": rpc error: code = NotFound desc = could not find container \"38ee84a1ef40cc675de03cf7b86ef2a79808c566217ca0644e0c478c3482ee45\": container with ID starting with 38ee84a1ef40cc675de03cf7b86ef2a79808c566217ca0644e0c478c3482ee45 not found: ID does not exist" Nov 03 13:31:43.234726 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.234693 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a4f947e-dd99-4332-95dc-c41c040733b7-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:43.234835 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.234729 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a4f947e-dd99-4332-95dc-c41c040733b7-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:43.234835 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.234746 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a4f947e-dd99-4332-95dc-c41c040733b7-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:43.234835 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.234760 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a4f947e-dd99-4332-95dc-c41c040733b7-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:43.234835 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.234774 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vlpc\" (UniqueName: \"kubernetes.io/projected/0a4f947e-dd99-4332-95dc-c41c040733b7-kube-api-access-7vlpc\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:43.234835 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.234787 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a4f947e-dd99-4332-95dc-c41c040733b7-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:43.550835 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.550750 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4"] Nov 03 13:31:43.555870 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.555843 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-vtcz4"] Nov 03 13:31:43.572234 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.572202 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg"] Nov 03 13:31:43.573589 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.573526 5979 cpu_manager.go:414] "containerMap: removing container" podUID="9f5db4ce-3fef-4033-8d83-71495a0b6373" containerName="revoke-keys" Nov 03 13:31:43.573589 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.573550 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f5db4ce-3fef-4033-8d83-71495a0b6373" containerName="revoke-keys" Nov 03 13:31:43.573755 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.573628 5979 cpu_manager.go:414] "containerMap: removing container" podUID="620d1fb2-9a2b-4e36-819e-6997985fbc1b" containerName="istio-proxy" Nov 03 13:31:43.573755 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.573636 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="620d1fb2-9a2b-4e36-819e-6997985fbc1b" containerName="istio-proxy" Nov 03 13:31:43.573755 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.573657 5979 cpu_manager.go:414] "containerMap: removing container" podUID="dc5abfae-7745-4857-b1f1-ff7d91a65ddd" containerName="istio-proxy" Nov 03 13:31:43.573755 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.573664 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc5abfae-7745-4857-b1f1-ff7d91a65ddd" containerName="istio-proxy" Nov 03 13:31:43.573755 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.573690 5979 cpu_manager.go:414] "containerMap: removing container" podUID="d8a57a81-8b6f-4049-9c5a-9884e968fee7" containerName="istio-proxy" Nov 03 13:31:43.573755 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.573696 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8a57a81-8b6f-4049-9c5a-9884e968fee7" containerName="istio-proxy" Nov 03 13:31:43.573755 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.573745 5979 cpu_manager.go:414] "containerMap: removing container" podUID="0a4f947e-dd99-4332-95dc-c41c040733b7" containerName="istio-proxy" Nov 03 13:31:43.573755 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.573752 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a4f947e-dd99-4332-95dc-c41c040733b7" containerName="istio-proxy" Nov 03 13:31:43.574147 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.573799 5979 cpu_manager.go:414] "containerMap: removing container" podUID="d8a57a81-8b6f-4049-9c5a-9884e968fee7" containerName="istio-proxy" Nov 03 13:31:43.574147 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.573806 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8a57a81-8b6f-4049-9c5a-9884e968fee7" containerName="istio-proxy" Nov 03 13:31:43.574147 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.574075 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="620d1fb2-9a2b-4e36-819e-6997985fbc1b" containerName="istio-proxy" Nov 03 13:31:43.574147 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.574089 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc5abfae-7745-4857-b1f1-ff7d91a65ddd" containerName="istio-proxy" Nov 03 13:31:43.574147 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.574097 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8a57a81-8b6f-4049-9c5a-9884e968fee7" containerName="istio-proxy" Nov 03 13:31:43.574147 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.574118 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a4f947e-dd99-4332-95dc-c41c040733b7" containerName="istio-proxy" Nov 03 13:31:43.574147 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.574138 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8a57a81-8b6f-4049-9c5a-9884e968fee7" containerName="istio-proxy" Nov 03 13:31:43.574147 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.574146 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f5db4ce-3fef-4033-8d83-71495a0b6373" containerName="revoke-keys" Nov 03 13:31:43.579702 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.579681 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg" Nov 03 13:31:43.630891 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.630831 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg"] Nov 03 13:31:43.631198 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.631162 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-e7ca74-openshift-default-dockercfg-kps6d\"" Nov 03 13:31:43.640646 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.640614 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec489f9b-9c50-45d9-aefb-1da511837bae-workload-certs\") pod \"e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg\" (UID: \"ec489f9b-9c50-45d9-aefb-1da511837bae\") " pod="openshift-ingress/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg" Nov 03 13:31:43.640767 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.640724 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec489f9b-9c50-45d9-aefb-1da511837bae-workload-socket\") pod \"e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg\" (UID: \"ec489f9b-9c50-45d9-aefb-1da511837bae\") " pod="openshift-ingress/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg" Nov 03 13:31:43.640767 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.640759 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec489f9b-9c50-45d9-aefb-1da511837bae-istio-data\") pod \"e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg\" (UID: \"ec489f9b-9c50-45d9-aefb-1da511837bae\") " pod="openshift-ingress/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg" Nov 03 13:31:43.640899 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.640791 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zppmj\" (UniqueName: \"kubernetes.io/projected/ec489f9b-9c50-45d9-aefb-1da511837bae-kube-api-access-zppmj\") pod \"e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg\" (UID: \"ec489f9b-9c50-45d9-aefb-1da511837bae\") " pod="openshift-ingress/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg" Nov 03 13:31:43.640899 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.640854 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec489f9b-9c50-45d9-aefb-1da511837bae-istio-token\") pod \"e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg\" (UID: \"ec489f9b-9c50-45d9-aefb-1da511837bae\") " pod="openshift-ingress/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg" Nov 03 13:31:43.641018 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.640964 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec489f9b-9c50-45d9-aefb-1da511837bae-istio-podinfo\") pod \"e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg\" (UID: \"ec489f9b-9c50-45d9-aefb-1da511837bae\") " pod="openshift-ingress/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg" Nov 03 13:31:43.641018 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.640989 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec489f9b-9c50-45d9-aefb-1da511837bae-istio-envoy\") pod \"e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg\" (UID: \"ec489f9b-9c50-45d9-aefb-1da511837bae\") " pod="openshift-ingress/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg" Nov 03 13:31:43.641128 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.641020 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec489f9b-9c50-45d9-aefb-1da511837bae-istiod-ca-cert\") pod \"e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg\" (UID: \"ec489f9b-9c50-45d9-aefb-1da511837bae\") " pod="openshift-ingress/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg" Nov 03 13:31:43.641128 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.641048 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec489f9b-9c50-45d9-aefb-1da511837bae-credential-socket\") pod \"e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg\" (UID: \"ec489f9b-9c50-45d9-aefb-1da511837bae\") " pod="openshift-ingress/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg" Nov 03 13:31:43.661792 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.661756 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m"] Nov 03 13:31:43.669360 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.669325 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-blq4m"] Nov 03 13:31:43.783495 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.779860 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec489f9b-9c50-45d9-aefb-1da511837bae-istio-data\") pod \"e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg\" (UID: \"ec489f9b-9c50-45d9-aefb-1da511837bae\") " pod="openshift-ingress/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg" Nov 03 13:31:43.783495 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.779927 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zppmj\" (UniqueName: \"kubernetes.io/projected/ec489f9b-9c50-45d9-aefb-1da511837bae-kube-api-access-zppmj\") pod \"e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg\" (UID: \"ec489f9b-9c50-45d9-aefb-1da511837bae\") " pod="openshift-ingress/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg" Nov 03 13:31:43.783495 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.779997 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec489f9b-9c50-45d9-aefb-1da511837bae-istio-token\") pod \"e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg\" (UID: \"ec489f9b-9c50-45d9-aefb-1da511837bae\") " pod="openshift-ingress/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg" Nov 03 13:31:43.783495 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.780104 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec489f9b-9c50-45d9-aefb-1da511837bae-istio-podinfo\") pod \"e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg\" (UID: \"ec489f9b-9c50-45d9-aefb-1da511837bae\") " pod="openshift-ingress/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg" Nov 03 13:31:43.783495 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.780131 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec489f9b-9c50-45d9-aefb-1da511837bae-istio-envoy\") pod \"e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg\" (UID: \"ec489f9b-9c50-45d9-aefb-1da511837bae\") " pod="openshift-ingress/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg" Nov 03 13:31:43.783495 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.780164 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec489f9b-9c50-45d9-aefb-1da511837bae-istiod-ca-cert\") pod \"e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg\" (UID: \"ec489f9b-9c50-45d9-aefb-1da511837bae\") " pod="openshift-ingress/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg" Nov 03 13:31:43.783495 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.780195 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec489f9b-9c50-45d9-aefb-1da511837bae-credential-socket\") pod \"e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg\" (UID: \"ec489f9b-9c50-45d9-aefb-1da511837bae\") " pod="openshift-ingress/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg" Nov 03 13:31:43.783495 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.780340 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec489f9b-9c50-45d9-aefb-1da511837bae-workload-certs\") pod \"e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg\" (UID: \"ec489f9b-9c50-45d9-aefb-1da511837bae\") " pod="openshift-ingress/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg" Nov 03 13:31:43.783495 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.780436 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec489f9b-9c50-45d9-aefb-1da511837bae-workload-socket\") pod \"e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg\" (UID: \"ec489f9b-9c50-45d9-aefb-1da511837bae\") " pod="openshift-ingress/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg" Nov 03 13:31:43.783495 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.780933 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec489f9b-9c50-45d9-aefb-1da511837bae-workload-socket\") pod \"e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg\" (UID: \"ec489f9b-9c50-45d9-aefb-1da511837bae\") " pod="openshift-ingress/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg" Nov 03 13:31:43.783495 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.781300 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec489f9b-9c50-45d9-aefb-1da511837bae-istio-data\") pod \"e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg\" (UID: \"ec489f9b-9c50-45d9-aefb-1da511837bae\") " pod="openshift-ingress/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg" Nov 03 13:31:43.802226 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.798654 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec489f9b-9c50-45d9-aefb-1da511837bae-istiod-ca-cert\") pod \"e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg\" (UID: \"ec489f9b-9c50-45d9-aefb-1da511837bae\") " pod="openshift-ingress/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg" Nov 03 13:31:43.802226 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.799322 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec489f9b-9c50-45d9-aefb-1da511837bae-workload-certs\") pod \"e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg\" (UID: \"ec489f9b-9c50-45d9-aefb-1da511837bae\") " pod="openshift-ingress/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg" Nov 03 13:31:43.810360 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.809339 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec489f9b-9c50-45d9-aefb-1da511837bae-credential-socket\") pod \"e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg\" (UID: \"ec489f9b-9c50-45d9-aefb-1da511837bae\") " pod="openshift-ingress/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg" Nov 03 13:31:43.843733 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.839759 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec489f9b-9c50-45d9-aefb-1da511837bae-istio-envoy\") pod \"e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg\" (UID: \"ec489f9b-9c50-45d9-aefb-1da511837bae\") " pod="openshift-ingress/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg" Nov 03 13:31:43.843733 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.840266 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec489f9b-9c50-45d9-aefb-1da511837bae-istio-podinfo\") pod \"e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg\" (UID: \"ec489f9b-9c50-45d9-aefb-1da511837bae\") " pod="openshift-ingress/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg" Nov 03 13:31:43.878513 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.878214 5979 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Nov 03 13:31:43.878513 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.878289 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh" podUID="821c73f7-cf2f-414c-8aea-1aff84e017af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Nov 03 13:31:43.914211 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.914164 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec489f9b-9c50-45d9-aefb-1da511837bae-istio-token\") pod \"e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg\" (UID: \"ec489f9b-9c50-45d9-aefb-1da511837bae\") " pod="openshift-ingress/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg" Nov 03 13:31:43.935756 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.935700 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf"] Nov 03 13:31:43.936334 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.936304 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zppmj\" (UniqueName: \"kubernetes.io/projected/ec489f9b-9c50-45d9-aefb-1da511837bae-kube-api-access-zppmj\") pod \"e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg\" (UID: \"ec489f9b-9c50-45d9-aefb-1da511837bae\") " pod="openshift-ingress/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg" Nov 03 13:31:43.939607 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.937966 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-fp6zf"] Nov 03 13:31:43.988231 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.978855 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759"] Nov 03 13:31:43.988231 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:43.981589 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-hq759"] Nov 03 13:31:44.119050 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.119010 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a4f947e-dd99-4332-95dc-c41c040733b7" path="/var/lib/kubelet/pods/0a4f947e-dd99-4332-95dc-c41c040733b7/volumes" Nov 03 13:31:44.119710 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.119671 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="620d1fb2-9a2b-4e36-819e-6997985fbc1b" path="/var/lib/kubelet/pods/620d1fb2-9a2b-4e36-819e-6997985fbc1b/volumes" Nov 03 13:31:44.120211 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.120193 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccdc18d4-45e3-497e-9f79-454392831e04" path="/var/lib/kubelet/pods/ccdc18d4-45e3-497e-9f79-454392831e04/volumes" Nov 03 13:31:44.121159 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.121136 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8a57a81-8b6f-4049-9c5a-9884e968fee7" path="/var/lib/kubelet/pods/d8a57a81-8b6f-4049-9c5a-9884e968fee7/volumes" Nov 03 13:31:44.121687 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.121668 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc5abfae-7745-4857-b1f1-ff7d91a65ddd" path="/var/lib/kubelet/pods/dc5abfae-7745-4857-b1f1-ff7d91a65ddd/volumes" Nov 03 13:31:44.140678 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.140610 5979 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Nov 03 13:31:44.140946 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.140920 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9" podUID="26360457-2ef0-4c64-9a1c-4d62a783dc49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Nov 03 13:31:44.174464 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.174438 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh" Nov 03 13:31:44.192299 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.192262 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg" Nov 03 13:31:44.196231 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.196201 5979 generic.go:356] "Generic (PLEG): container finished" podID="3e80110d-950d-452f-9fdf-eecb56840a3c" containerID="bb0eeff0d2aa46c774377d182ba268af2f4372bbb98a29e51607aa1949913f6d" exitCode=0 Nov 03 13:31:44.196387 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.196365 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd" event={"ID":"3e80110d-950d-452f-9fdf-eecb56840a3c","Type":"ContainerDied","Data":"bb0eeff0d2aa46c774377d182ba268af2f4372bbb98a29e51607aa1949913f6d"} Nov 03 13:31:44.199736 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.199704 5979 generic.go:356] "Generic (PLEG): container finished" podID="7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48" containerID="3a45e1eb1ce8fc3d09630da3da69fcedea2efa4460dfac8c37d65d275ec2f47b" exitCode=0 Nov 03 13:31:44.199891 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.199867 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b" event={"ID":"7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48","Type":"ContainerDied","Data":"3a45e1eb1ce8fc3d09630da3da69fcedea2efa4460dfac8c37d65d275ec2f47b"} Nov 03 13:31:44.200016 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.199979 5979 scope.go:122] "RemoveContainer" containerID="3b0bb741dc94ba17beaa92dee8f2d1ae54bb513ae3a6dbe9734b2ee5d01627a9" Nov 03 13:31:44.202949 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.202913 5979 generic.go:356] "Generic (PLEG): container finished" podID="26360457-2ef0-4c64-9a1c-4d62a783dc49" containerID="494e19cb04491dbd8aea6b37136258d4636937646ec2d53aaee8af9e85fff96d" exitCode=0 Nov 03 13:31:44.203062 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.203013 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9" event={"ID":"26360457-2ef0-4c64-9a1c-4d62a783dc49","Type":"ContainerDied","Data":"494e19cb04491dbd8aea6b37136258d4636937646ec2d53aaee8af9e85fff96d"} Nov 03 13:31:44.209790 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.209580 5979 generic.go:356] "Generic (PLEG): container finished" podID="821c73f7-cf2f-414c-8aea-1aff84e017af" containerID="c51d402870f6fc59222f6ef744e7c8e0e2d58ffbab0a0eefc227009683c30dc8" exitCode=0 Nov 03 13:31:44.209790 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.209650 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh" event={"ID":"821c73f7-cf2f-414c-8aea-1aff84e017af","Type":"ContainerDied","Data":"c51d402870f6fc59222f6ef744e7c8e0e2d58ffbab0a0eefc227009683c30dc8"} Nov 03 13:31:44.209790 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.209689 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh" event={"ID":"821c73f7-cf2f-414c-8aea-1aff84e017af","Type":"ContainerDied","Data":"85f6599716f9fbac7b9f2e78dbaf7cd6091655d2588f8a0b9b4da3f5d926413b"} Nov 03 13:31:44.209790 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.209768 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh" Nov 03 13:31:44.225992 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.225969 5979 scope.go:122] "RemoveContainer" containerID="c51d402870f6fc59222f6ef744e7c8e0e2d58ffbab0a0eefc227009683c30dc8" Nov 03 13:31:44.235119 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.235086 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd" Nov 03 13:31:44.247225 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.247200 5979 scope.go:122] "RemoveContainer" containerID="c51d402870f6fc59222f6ef744e7c8e0e2d58ffbab0a0eefc227009683c30dc8" Nov 03 13:31:44.247838 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:31:44.247807 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c51d402870f6fc59222f6ef744e7c8e0e2d58ffbab0a0eefc227009683c30dc8\": container with ID starting with c51d402870f6fc59222f6ef744e7c8e0e2d58ffbab0a0eefc227009683c30dc8 not found: ID does not exist" containerID="c51d402870f6fc59222f6ef744e7c8e0e2d58ffbab0a0eefc227009683c30dc8" Nov 03 13:31:44.248047 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.248007 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c51d402870f6fc59222f6ef744e7c8e0e2d58ffbab0a0eefc227009683c30dc8"} err="failed to get container status \"c51d402870f6fc59222f6ef744e7c8e0e2d58ffbab0a0eefc227009683c30dc8\": rpc error: code = NotFound desc = could not find container \"c51d402870f6fc59222f6ef744e7c8e0e2d58ffbab0a0eefc227009683c30dc8\": container with ID starting with c51d402870f6fc59222f6ef744e7c8e0e2d58ffbab0a0eefc227009683c30dc8 not found: ID does not exist" Nov 03 13:31:44.264151 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.264117 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9" Nov 03 13:31:44.339600 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.339013 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/821c73f7-cf2f-414c-8aea-1aff84e017af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/821c73f7-cf2f-414c-8aea-1aff84e017af-credential-socket\") pod \"821c73f7-cf2f-414c-8aea-1aff84e017af\" (UID: \"821c73f7-cf2f-414c-8aea-1aff84e017af\") " Nov 03 13:31:44.339600 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.339086 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/821c73f7-cf2f-414c-8aea-1aff84e017af-kube-api-access-cm5hk\" (UniqueName: \"kubernetes.io/projected/821c73f7-cf2f-414c-8aea-1aff84e017af-kube-api-access-cm5hk\") pod \"821c73f7-cf2f-414c-8aea-1aff84e017af\" (UID: \"821c73f7-cf2f-414c-8aea-1aff84e017af\") " Nov 03 13:31:44.339600 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.339130 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e80110d-950d-452f-9fdf-eecb56840a3c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e80110d-950d-452f-9fdf-eecb56840a3c-istiod-ca-cert\") pod \"3e80110d-950d-452f-9fdf-eecb56840a3c\" (UID: \"3e80110d-950d-452f-9fdf-eecb56840a3c\") " Nov 03 13:31:44.339600 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.339185 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/821c73f7-cf2f-414c-8aea-1aff84e017af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/821c73f7-cf2f-414c-8aea-1aff84e017af-workload-certs\") pod \"821c73f7-cf2f-414c-8aea-1aff84e017af\" (UID: \"821c73f7-cf2f-414c-8aea-1aff84e017af\") " Nov 03 13:31:44.339600 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.339210 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/821c73f7-cf2f-414c-8aea-1aff84e017af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/821c73f7-cf2f-414c-8aea-1aff84e017af-istio-envoy\") pod \"821c73f7-cf2f-414c-8aea-1aff84e017af\" (UID: \"821c73f7-cf2f-414c-8aea-1aff84e017af\") " Nov 03 13:31:44.339600 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.339227 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e80110d-950d-452f-9fdf-eecb56840a3c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e80110d-950d-452f-9fdf-eecb56840a3c-credential-socket\") pod \"3e80110d-950d-452f-9fdf-eecb56840a3c\" (UID: \"3e80110d-950d-452f-9fdf-eecb56840a3c\") " Nov 03 13:31:44.339600 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.339246 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/821c73f7-cf2f-414c-8aea-1aff84e017af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/821c73f7-cf2f-414c-8aea-1aff84e017af-istio-data\") pod \"821c73f7-cf2f-414c-8aea-1aff84e017af\" (UID: \"821c73f7-cf2f-414c-8aea-1aff84e017af\") " Nov 03 13:31:44.339600 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.339296 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/821c73f7-cf2f-414c-8aea-1aff84e017af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/821c73f7-cf2f-414c-8aea-1aff84e017af-istiod-ca-cert\") pod \"821c73f7-cf2f-414c-8aea-1aff84e017af\" (UID: \"821c73f7-cf2f-414c-8aea-1aff84e017af\") " Nov 03 13:31:44.339600 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.339324 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e80110d-950d-452f-9fdf-eecb56840a3c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e80110d-950d-452f-9fdf-eecb56840a3c-workload-socket\") pod \"3e80110d-950d-452f-9fdf-eecb56840a3c\" (UID: \"3e80110d-950d-452f-9fdf-eecb56840a3c\") " Nov 03 13:31:44.339600 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.339350 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/821c73f7-cf2f-414c-8aea-1aff84e017af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/821c73f7-cf2f-414c-8aea-1aff84e017af-istio-podinfo\") pod \"821c73f7-cf2f-414c-8aea-1aff84e017af\" (UID: \"821c73f7-cf2f-414c-8aea-1aff84e017af\") " Nov 03 13:31:44.339600 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.339410 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e80110d-950d-452f-9fdf-eecb56840a3c-kube-api-access-nktj8\" (UniqueName: \"kubernetes.io/projected/3e80110d-950d-452f-9fdf-eecb56840a3c-kube-api-access-nktj8\") pod \"3e80110d-950d-452f-9fdf-eecb56840a3c\" (UID: \"3e80110d-950d-452f-9fdf-eecb56840a3c\") " Nov 03 13:31:44.339600 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.339447 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e80110d-950d-452f-9fdf-eecb56840a3c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e80110d-950d-452f-9fdf-eecb56840a3c-istio-podinfo\") pod \"3e80110d-950d-452f-9fdf-eecb56840a3c\" (UID: \"3e80110d-950d-452f-9fdf-eecb56840a3c\") " Nov 03 13:31:44.339600 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.339533 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e80110d-950d-452f-9fdf-eecb56840a3c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e80110d-950d-452f-9fdf-eecb56840a3c-istio-envoy\") pod \"3e80110d-950d-452f-9fdf-eecb56840a3c\" (UID: \"3e80110d-950d-452f-9fdf-eecb56840a3c\") " Nov 03 13:31:44.339600 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.339584 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/821c73f7-cf2f-414c-8aea-1aff84e017af-istio-token\" (UniqueName: \"kubernetes.io/projected/821c73f7-cf2f-414c-8aea-1aff84e017af-istio-token\") pod \"821c73f7-cf2f-414c-8aea-1aff84e017af\" (UID: \"821c73f7-cf2f-414c-8aea-1aff84e017af\") " Nov 03 13:31:44.339600 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.339617 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/821c73f7-cf2f-414c-8aea-1aff84e017af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/821c73f7-cf2f-414c-8aea-1aff84e017af-workload-socket\") pod \"821c73f7-cf2f-414c-8aea-1aff84e017af\" (UID: \"821c73f7-cf2f-414c-8aea-1aff84e017af\") " Nov 03 13:31:44.341094 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.339644 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e80110d-950d-452f-9fdf-eecb56840a3c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e80110d-950d-452f-9fdf-eecb56840a3c-istio-data\") pod \"3e80110d-950d-452f-9fdf-eecb56840a3c\" (UID: \"3e80110d-950d-452f-9fdf-eecb56840a3c\") " Nov 03 13:31:44.341094 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.339747 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e80110d-950d-452f-9fdf-eecb56840a3c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e80110d-950d-452f-9fdf-eecb56840a3c-workload-certs\") pod \"3e80110d-950d-452f-9fdf-eecb56840a3c\" (UID: \"3e80110d-950d-452f-9fdf-eecb56840a3c\") " Nov 03 13:31:44.341094 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.339929 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e80110d-950d-452f-9fdf-eecb56840a3c-istio-token\" (UniqueName: \"kubernetes.io/projected/3e80110d-950d-452f-9fdf-eecb56840a3c-istio-token\") pod \"3e80110d-950d-452f-9fdf-eecb56840a3c\" (UID: \"3e80110d-950d-452f-9fdf-eecb56840a3c\") " Nov 03 13:31:44.342654 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.342631 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e80110d-950d-452f-9fdf-eecb56840a3c-workload-socket" pod "3e80110d-950d-452f-9fdf-eecb56840a3c" (UID: "3e80110d-950d-452f-9fdf-eecb56840a3c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:44.342769 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.342751 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/821c73f7-cf2f-414c-8aea-1aff84e017af-workload-certs" pod "821c73f7-cf2f-414c-8aea-1aff84e017af" (UID: "821c73f7-cf2f-414c-8aea-1aff84e017af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:44.343305 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.343286 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/821c73f7-cf2f-414c-8aea-1aff84e017af-istiod-ca-cert" pod "821c73f7-cf2f-414c-8aea-1aff84e017af" (UID: "821c73f7-cf2f-414c-8aea-1aff84e017af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:31:44.343590 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.343460 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e80110d-950d-452f-9fdf-eecb56840a3c-istio-data" pod "3e80110d-950d-452f-9fdf-eecb56840a3c" (UID: "3e80110d-950d-452f-9fdf-eecb56840a3c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:44.343706 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.343579 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e80110d-950d-452f-9fdf-eecb56840a3c-credential-socket" pod "3e80110d-950d-452f-9fdf-eecb56840a3c" (UID: "3e80110d-950d-452f-9fdf-eecb56840a3c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:44.343796 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.343717 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e80110d-950d-452f-9fdf-eecb56840a3c-istio-token" pod "3e80110d-950d-452f-9fdf-eecb56840a3c" (UID: "3e80110d-950d-452f-9fdf-eecb56840a3c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:31:44.343879 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.343732 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e80110d-950d-452f-9fdf-eecb56840a3c-workload-certs" pod "3e80110d-950d-452f-9fdf-eecb56840a3c" (UID: "3e80110d-950d-452f-9fdf-eecb56840a3c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:44.344376 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.343751 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/821c73f7-cf2f-414c-8aea-1aff84e017af-workload-socket" pod "821c73f7-cf2f-414c-8aea-1aff84e017af" (UID: "821c73f7-cf2f-414c-8aea-1aff84e017af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:44.344376 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.343920 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/821c73f7-cf2f-414c-8aea-1aff84e017af-credential-socket" pod "821c73f7-cf2f-414c-8aea-1aff84e017af" (UID: "821c73f7-cf2f-414c-8aea-1aff84e017af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:44.344376 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.342990 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/821c73f7-cf2f-414c-8aea-1aff84e017af-istio-data" pod "821c73f7-cf2f-414c-8aea-1aff84e017af" (UID: "821c73f7-cf2f-414c-8aea-1aff84e017af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:44.344376 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.344208 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e80110d-950d-452f-9fdf-eecb56840a3c-istiod-ca-cert" pod "3e80110d-950d-452f-9fdf-eecb56840a3c" (UID: "3e80110d-950d-452f-9fdf-eecb56840a3c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:31:44.346574 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.346548 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/821c73f7-cf2f-414c-8aea-1aff84e017af-istio-podinfo" pod "821c73f7-cf2f-414c-8aea-1aff84e017af" (UID: "821c73f7-cf2f-414c-8aea-1aff84e017af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:31:44.346798 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.346775 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e80110d-950d-452f-9fdf-eecb56840a3c-istio-envoy" pod "3e80110d-950d-452f-9fdf-eecb56840a3c" (UID: "3e80110d-950d-452f-9fdf-eecb56840a3c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:44.346798 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.346788 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/821c73f7-cf2f-414c-8aea-1aff84e017af-istio-envoy" pod "821c73f7-cf2f-414c-8aea-1aff84e017af" (UID: "821c73f7-cf2f-414c-8aea-1aff84e017af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:44.346911 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.346820 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/821c73f7-cf2f-414c-8aea-1aff84e017af-istio-token" pod "821c73f7-cf2f-414c-8aea-1aff84e017af" (UID: "821c73f7-cf2f-414c-8aea-1aff84e017af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:31:44.347015 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.346993 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e80110d-950d-452f-9fdf-eecb56840a3c-istio-podinfo" pod "3e80110d-950d-452f-9fdf-eecb56840a3c" (UID: "3e80110d-950d-452f-9fdf-eecb56840a3c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:31:44.347854 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.347779 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/821c73f7-cf2f-414c-8aea-1aff84e017af-kube-api-access-cm5hk" pod "821c73f7-cf2f-414c-8aea-1aff84e017af" (UID: "821c73f7-cf2f-414c-8aea-1aff84e017af"). InnerVolumeSpecName "kube-api-access-cm5hk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:31:44.348661 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.348636 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e80110d-950d-452f-9fdf-eecb56840a3c-kube-api-access-nktj8" pod "3e80110d-950d-452f-9fdf-eecb56840a3c" (UID: "3e80110d-950d-452f-9fdf-eecb56840a3c"). InnerVolumeSpecName "kube-api-access-nktj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:31:44.441525 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.441485 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26360457-2ef0-4c64-9a1c-4d62a783dc49-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26360457-2ef0-4c64-9a1c-4d62a783dc49-workload-socket\") pod \"26360457-2ef0-4c64-9a1c-4d62a783dc49\" (UID: \"26360457-2ef0-4c64-9a1c-4d62a783dc49\") " Nov 03 13:31:44.441719 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.441552 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26360457-2ef0-4c64-9a1c-4d62a783dc49-istio-token\" (UniqueName: \"kubernetes.io/projected/26360457-2ef0-4c64-9a1c-4d62a783dc49-istio-token\") pod \"26360457-2ef0-4c64-9a1c-4d62a783dc49\" (UID: \"26360457-2ef0-4c64-9a1c-4d62a783dc49\") " Nov 03 13:31:44.441719 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.441681 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26360457-2ef0-4c64-9a1c-4d62a783dc49-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26360457-2ef0-4c64-9a1c-4d62a783dc49-credential-socket\") pod \"26360457-2ef0-4c64-9a1c-4d62a783dc49\" (UID: \"26360457-2ef0-4c64-9a1c-4d62a783dc49\") " Nov 03 13:31:44.441858 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.441764 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26360457-2ef0-4c64-9a1c-4d62a783dc49-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26360457-2ef0-4c64-9a1c-4d62a783dc49-workload-certs\") pod \"26360457-2ef0-4c64-9a1c-4d62a783dc49\" (UID: \"26360457-2ef0-4c64-9a1c-4d62a783dc49\") " Nov 03 13:31:44.441858 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.441835 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26360457-2ef0-4c64-9a1c-4d62a783dc49-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26360457-2ef0-4c64-9a1c-4d62a783dc49-istio-envoy\") pod \"26360457-2ef0-4c64-9a1c-4d62a783dc49\" (UID: \"26360457-2ef0-4c64-9a1c-4d62a783dc49\") " Nov 03 13:31:44.441976 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.441931 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26360457-2ef0-4c64-9a1c-4d62a783dc49-kube-api-access-kq9hl\" (UniqueName: \"kubernetes.io/projected/26360457-2ef0-4c64-9a1c-4d62a783dc49-kube-api-access-kq9hl\") pod \"26360457-2ef0-4c64-9a1c-4d62a783dc49\" (UID: \"26360457-2ef0-4c64-9a1c-4d62a783dc49\") " Nov 03 13:31:44.441976 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.441966 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26360457-2ef0-4c64-9a1c-4d62a783dc49-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26360457-2ef0-4c64-9a1c-4d62a783dc49-istio-data\") pod \"26360457-2ef0-4c64-9a1c-4d62a783dc49\" (UID: \"26360457-2ef0-4c64-9a1c-4d62a783dc49\") " Nov 03 13:31:44.442065 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.442012 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26360457-2ef0-4c64-9a1c-4d62a783dc49-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26360457-2ef0-4c64-9a1c-4d62a783dc49-istiod-ca-cert\") pod \"26360457-2ef0-4c64-9a1c-4d62a783dc49\" (UID: \"26360457-2ef0-4c64-9a1c-4d62a783dc49\") " Nov 03 13:31:44.442119 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.442081 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/26360457-2ef0-4c64-9a1c-4d62a783dc49-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26360457-2ef0-4c64-9a1c-4d62a783dc49-istio-podinfo\") pod \"26360457-2ef0-4c64-9a1c-4d62a783dc49\" (UID: \"26360457-2ef0-4c64-9a1c-4d62a783dc49\") " Nov 03 13:31:44.442789 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.442759 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26360457-2ef0-4c64-9a1c-4d62a783dc49-workload-certs" pod "26360457-2ef0-4c64-9a1c-4d62a783dc49" (UID: "26360457-2ef0-4c64-9a1c-4d62a783dc49"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:44.442927 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.442832 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e80110d-950d-452f-9fdf-eecb56840a3c-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:44.442927 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.442859 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e80110d-950d-452f-9fdf-eecb56840a3c-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:44.442927 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.442874 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/821c73f7-cf2f-414c-8aea-1aff84e017af-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:44.442927 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.442889 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/821c73f7-cf2f-414c-8aea-1aff84e017af-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:44.442927 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.442902 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e80110d-950d-452f-9fdf-eecb56840a3c-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:44.442927 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.442916 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e80110d-950d-452f-9fdf-eecb56840a3c-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:44.442927 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.442928 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e80110d-950d-452f-9fdf-eecb56840a3c-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:44.443293 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.442941 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/821c73f7-cf2f-414c-8aea-1aff84e017af-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:44.443293 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.442956 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm5hk\" (UniqueName: \"kubernetes.io/projected/821c73f7-cf2f-414c-8aea-1aff84e017af-kube-api-access-cm5hk\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:44.443293 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.442970 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e80110d-950d-452f-9fdf-eecb56840a3c-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:44.443293 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.442983 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/821c73f7-cf2f-414c-8aea-1aff84e017af-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:44.443293 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.442988 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26360457-2ef0-4c64-9a1c-4d62a783dc49-workload-socket" pod "26360457-2ef0-4c64-9a1c-4d62a783dc49" (UID: "26360457-2ef0-4c64-9a1c-4d62a783dc49"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:44.443293 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.442998 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/821c73f7-cf2f-414c-8aea-1aff84e017af-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:44.443293 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.443013 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e80110d-950d-452f-9fdf-eecb56840a3c-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:44.443293 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.443028 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/821c73f7-cf2f-414c-8aea-1aff84e017af-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:44.443293 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.443041 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/821c73f7-cf2f-414c-8aea-1aff84e017af-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:44.443293 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.443054 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e80110d-950d-452f-9fdf-eecb56840a3c-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:44.443293 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.443068 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/821c73f7-cf2f-414c-8aea-1aff84e017af-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:44.443293 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.443084 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nktj8\" (UniqueName: \"kubernetes.io/projected/3e80110d-950d-452f-9fdf-eecb56840a3c-kube-api-access-nktj8\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:44.443918 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.443452 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26360457-2ef0-4c64-9a1c-4d62a783dc49-istio-data" pod "26360457-2ef0-4c64-9a1c-4d62a783dc49" (UID: "26360457-2ef0-4c64-9a1c-4d62a783dc49"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:44.444555 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.444517 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26360457-2ef0-4c64-9a1c-4d62a783dc49-istiod-ca-cert" pod "26360457-2ef0-4c64-9a1c-4d62a783dc49" (UID: "26360457-2ef0-4c64-9a1c-4d62a783dc49"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:31:44.444727 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.444695 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26360457-2ef0-4c64-9a1c-4d62a783dc49-credential-socket" pod "26360457-2ef0-4c64-9a1c-4d62a783dc49" (UID: "26360457-2ef0-4c64-9a1c-4d62a783dc49"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:44.446695 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.446658 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26360457-2ef0-4c64-9a1c-4d62a783dc49-istio-podinfo" pod "26360457-2ef0-4c64-9a1c-4d62a783dc49" (UID: "26360457-2ef0-4c64-9a1c-4d62a783dc49"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:31:44.447997 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.447966 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26360457-2ef0-4c64-9a1c-4d62a783dc49-istio-token" pod "26360457-2ef0-4c64-9a1c-4d62a783dc49" (UID: "26360457-2ef0-4c64-9a1c-4d62a783dc49"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:31:44.449069 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.449035 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26360457-2ef0-4c64-9a1c-4d62a783dc49-istio-envoy" pod "26360457-2ef0-4c64-9a1c-4d62a783dc49" (UID: "26360457-2ef0-4c64-9a1c-4d62a783dc49"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:44.449229 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.449203 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26360457-2ef0-4c64-9a1c-4d62a783dc49-kube-api-access-kq9hl" pod "26360457-2ef0-4c64-9a1c-4d62a783dc49" (UID: "26360457-2ef0-4c64-9a1c-4d62a783dc49"). InnerVolumeSpecName "kube-api-access-kq9hl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:31:44.478903 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.478881 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b" Nov 03 13:31:44.562492 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.560982 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-istio-token\" (UniqueName: \"kubernetes.io/projected/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-istio-token\") pod \"7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48\" (UID: \"7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48\") " Nov 03 13:31:44.562492 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.561042 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-istio-podinfo\") pod \"7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48\" (UID: \"7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48\") " Nov 03 13:31:44.562492 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.561082 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-credential-socket\") pod \"7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48\" (UID: \"7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48\") " Nov 03 13:31:44.562492 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.561125 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-istiod-ca-cert\") pod \"7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48\" (UID: \"7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48\") " Nov 03 13:31:44.562492 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.561167 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-istio-data\") pod \"7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48\" (UID: \"7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48\") " Nov 03 13:31:44.562492 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.561255 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-workload-certs\") pod \"7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48\" (UID: \"7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48\") " Nov 03 13:31:44.562492 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.561290 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-kube-api-access-q8nld\" (UniqueName: \"kubernetes.io/projected/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-kube-api-access-q8nld\") pod \"7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48\" (UID: \"7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48\") " Nov 03 13:31:44.562492 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.561329 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-istio-envoy\") pod \"7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48\" (UID: \"7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48\") " Nov 03 13:31:44.562492 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.561382 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-workload-socket\") pod \"7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48\" (UID: \"7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48\") " Nov 03 13:31:44.562492 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.562102 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26360457-2ef0-4c64-9a1c-4d62a783dc49-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:44.562492 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.562125 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26360457-2ef0-4c64-9a1c-4d62a783dc49-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:44.562492 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.562140 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26360457-2ef0-4c64-9a1c-4d62a783dc49-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:44.562492 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.562154 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26360457-2ef0-4c64-9a1c-4d62a783dc49-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:44.562492 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.562168 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26360457-2ef0-4c64-9a1c-4d62a783dc49-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:44.562492 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.562184 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kq9hl\" (UniqueName: \"kubernetes.io/projected/26360457-2ef0-4c64-9a1c-4d62a783dc49-kube-api-access-kq9hl\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:44.562492 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.562199 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26360457-2ef0-4c64-9a1c-4d62a783dc49-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:44.562492 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.562213 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26360457-2ef0-4c64-9a1c-4d62a783dc49-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:44.562492 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.562227 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26360457-2ef0-4c64-9a1c-4d62a783dc49-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:44.582236 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.581759 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-istio-data" pod "7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48" (UID: "7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:44.582236 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.581954 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-workload-certs" pod "7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48" (UID: "7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:44.612314 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.593696 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-istiod-ca-cert" pod "7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48" (UID: "7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:31:44.612314 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.596929 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-credential-socket" pod "7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48" (UID: "7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:44.621044 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.612645 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-workload-socket" pod "7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48" (UID: "7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:44.621044 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.613085 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-kube-api-access-q8nld" pod "7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48" (UID: "7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48"). InnerVolumeSpecName "kube-api-access-q8nld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:31:44.630756 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.630693 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-istio-token" pod "7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48" (UID: "7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:31:44.658667 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.658607 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-istio-podinfo" pod "7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48" (UID: "7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:31:44.735662 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.663558 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:44.735662 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.663590 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:44.735662 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.663606 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:44.735662 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.663620 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:44.735662 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.663636 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:44.735662 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.663651 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:44.735662 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.663666 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8nld\" (UniqueName: \"kubernetes.io/projected/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-kube-api-access-q8nld\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:44.735662 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.663681 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:44.735662 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.722602 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg"] Nov 03 13:31:44.735662 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.727790 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-istio-envoy" pod "7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48" (UID: "7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:31:44.735662 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.730396 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:31:44.735662 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.730559 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:31:44.735662 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.730607 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:31:44.764765 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.764738 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:31:44.833464 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.833435 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh"] Nov 03 13:31:44.840196 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:44.840170 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-4zpmh"] Nov 03 13:31:45.219941 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.219889 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg" event={"ID":"ec489f9b-9c50-45d9-aefb-1da511837bae","Type":"ContainerStarted","Data":"899494b34430f7ca67b7b29888b402f22dd9dcf680dbf73212bb456dd2c2b313"} Nov 03 13:31:45.220424 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.219972 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg" event={"ID":"ec489f9b-9c50-45d9-aefb-1da511837bae","Type":"ContainerStarted","Data":"6b3875fbab3e145130f022452fc33059fc64f0b503b26768eea960cc3460732e"} Nov 03 13:31:45.221341 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.221320 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd" Nov 03 13:31:45.221465 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.221351 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd" event={"ID":"3e80110d-950d-452f-9fdf-eecb56840a3c","Type":"ContainerDied","Data":"684b3ce026539ac2b84ba20efbf741b721766e8d2fa3a52c6b4997a56299ac9a"} Nov 03 13:31:45.221465 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.221399 5979 scope.go:122] "RemoveContainer" containerID="bb0eeff0d2aa46c774377d182ba268af2f4372bbb98a29e51607aa1949913f6d" Nov 03 13:31:45.222897 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.222877 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b" Nov 03 13:31:45.223014 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.222879 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b" event={"ID":"7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48","Type":"ContainerDied","Data":"892c4273cc12998ec492ffef0b8802e882f74cd75ff404d20d83bf5183a96b98"} Nov 03 13:31:45.224561 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.224526 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9" event={"ID":"26360457-2ef0-4c64-9a1c-4d62a783dc49","Type":"ContainerDied","Data":"3100cc868accee62836bd6b0c9fd084052265266a77c0cdc388ee6126c68d5fa"} Nov 03 13:31:45.224690 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.224547 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9" Nov 03 13:31:45.240583 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.240564 5979 scope.go:122] "RemoveContainer" containerID="3a45e1eb1ce8fc3d09630da3da69fcedea2efa4460dfac8c37d65d275ec2f47b" Nov 03 13:31:45.251134 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.251111 5979 scope.go:122] "RemoveContainer" containerID="494e19cb04491dbd8aea6b37136258d4636937646ec2d53aaee8af9e85fff96d" Nov 03 13:31:45.331525 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.331438 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg" podStartSLOduration=2.33141892 podStartE2EDuration="2.33141892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:31:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:31:45.330830001 +0000 UTC m=+1587.341518243" watchObservedRunningTime="2026-11-03 13:31:45.33141892 +0000 UTC m=+1587.342107163" Nov 03 13:31:45.440462 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.440428 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b"] Nov 03 13:31:45.444682 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.444657 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-nqb7b"] Nov 03 13:31:45.510851 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.510766 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s"] Nov 03 13:31:45.511500 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.511459 5979 cpu_manager.go:414] "containerMap: removing container" podUID="7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48" containerName="istio-proxy" Nov 03 13:31:45.511500 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.511495 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48" containerName="istio-proxy" Nov 03 13:31:45.511670 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.511529 5979 cpu_manager.go:414] "containerMap: removing container" podUID="3e80110d-950d-452f-9fdf-eecb56840a3c" containerName="istio-proxy" Nov 03 13:31:45.511670 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.511534 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e80110d-950d-452f-9fdf-eecb56840a3c" containerName="istio-proxy" Nov 03 13:31:45.511670 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.511555 5979 cpu_manager.go:414] "containerMap: removing container" podUID="821c73f7-cf2f-414c-8aea-1aff84e017af" containerName="istio-proxy" Nov 03 13:31:45.511670 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.511562 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="821c73f7-cf2f-414c-8aea-1aff84e017af" containerName="istio-proxy" Nov 03 13:31:45.511670 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.511590 5979 cpu_manager.go:414] "containerMap: removing container" podUID="26360457-2ef0-4c64-9a1c-4d62a783dc49" containerName="istio-proxy" Nov 03 13:31:45.511670 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.511595 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26360457-2ef0-4c64-9a1c-4d62a783dc49" containerName="istio-proxy" Nov 03 13:31:45.511848 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.511692 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="821c73f7-cf2f-414c-8aea-1aff84e017af" containerName="istio-proxy" Nov 03 13:31:45.511848 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.511703 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48" containerName="istio-proxy" Nov 03 13:31:45.511848 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.511711 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="26360457-2ef0-4c64-9a1c-4d62a783dc49" containerName="istio-proxy" Nov 03 13:31:45.511848 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.511719 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48" containerName="istio-proxy" Nov 03 13:31:45.511848 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.511727 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e80110d-950d-452f-9fdf-eecb56840a3c" containerName="istio-proxy" Nov 03 13:31:45.515672 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.515653 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s" Nov 03 13:31:45.593302 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.591828 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-fa2568-openshift-default-dockercfg-jc76b\"" Nov 03 13:31:45.612694 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.612661 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80fa8184-94c4-4ce1-8354-92681a69a6bf-istio-token\") pod \"e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s\" (UID: \"80fa8184-94c4-4ce1-8354-92681a69a6bf\") " pod="openshift-ingress/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s" Nov 03 13:31:45.613048 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.613021 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80fa8184-94c4-4ce1-8354-92681a69a6bf-workload-certs\") pod \"e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s\" (UID: \"80fa8184-94c4-4ce1-8354-92681a69a6bf\") " pod="openshift-ingress/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s" Nov 03 13:31:45.613295 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.613277 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80fa8184-94c4-4ce1-8354-92681a69a6bf-workload-socket\") pod \"e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s\" (UID: \"80fa8184-94c4-4ce1-8354-92681a69a6bf\") " pod="openshift-ingress/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s" Nov 03 13:31:45.613445 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.613430 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80fa8184-94c4-4ce1-8354-92681a69a6bf-istio-envoy\") pod \"e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s\" (UID: \"80fa8184-94c4-4ce1-8354-92681a69a6bf\") " pod="openshift-ingress/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s" Nov 03 13:31:45.613658 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.613640 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80fa8184-94c4-4ce1-8354-92681a69a6bf-credential-socket\") pod \"e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s\" (UID: \"80fa8184-94c4-4ce1-8354-92681a69a6bf\") " pod="openshift-ingress/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s" Nov 03 13:31:45.613819 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.613805 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80fa8184-94c4-4ce1-8354-92681a69a6bf-istio-data\") pod \"e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s\" (UID: \"80fa8184-94c4-4ce1-8354-92681a69a6bf\") " pod="openshift-ingress/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s" Nov 03 13:31:45.613989 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.613975 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80fa8184-94c4-4ce1-8354-92681a69a6bf-istiod-ca-cert\") pod \"e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s\" (UID: \"80fa8184-94c4-4ce1-8354-92681a69a6bf\") " pod="openshift-ingress/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s" Nov 03 13:31:45.614111 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.614098 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80fa8184-94c4-4ce1-8354-92681a69a6bf-istio-podinfo\") pod \"e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s\" (UID: \"80fa8184-94c4-4ce1-8354-92681a69a6bf\") " pod="openshift-ingress/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s" Nov 03 13:31:45.614284 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.614271 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5c25\" (UniqueName: \"kubernetes.io/projected/80fa8184-94c4-4ce1-8354-92681a69a6bf-kube-api-access-s5c25\") pod \"e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s\" (UID: \"80fa8184-94c4-4ce1-8354-92681a69a6bf\") " pod="openshift-ingress/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s" Nov 03 13:31:45.620766 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.620735 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9"] Nov 03 13:31:45.624779 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.624756 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-9sts9"] Nov 03 13:31:45.637696 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.637659 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s"] Nov 03 13:31:45.669665 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.669633 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd"] Nov 03 13:31:45.675607 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.675566 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lmwd"] Nov 03 13:31:45.718790 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.718742 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80fa8184-94c4-4ce1-8354-92681a69a6bf-istio-envoy\") pod \"e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s\" (UID: \"80fa8184-94c4-4ce1-8354-92681a69a6bf\") " pod="openshift-ingress/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s" Nov 03 13:31:45.719548 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.719517 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80fa8184-94c4-4ce1-8354-92681a69a6bf-credential-socket\") pod \"e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s\" (UID: \"80fa8184-94c4-4ce1-8354-92681a69a6bf\") " pod="openshift-ingress/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s" Nov 03 13:31:45.720244 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.720222 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80fa8184-94c4-4ce1-8354-92681a69a6bf-istio-data\") pod \"e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s\" (UID: \"80fa8184-94c4-4ce1-8354-92681a69a6bf\") " pod="openshift-ingress/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s" Nov 03 13:31:45.721902 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.721878 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80fa8184-94c4-4ce1-8354-92681a69a6bf-istiod-ca-cert\") pod \"e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s\" (UID: \"80fa8184-94c4-4ce1-8354-92681a69a6bf\") " pod="openshift-ingress/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s" Nov 03 13:31:45.722746 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.722724 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80fa8184-94c4-4ce1-8354-92681a69a6bf-istio-podinfo\") pod \"e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s\" (UID: \"80fa8184-94c4-4ce1-8354-92681a69a6bf\") " pod="openshift-ingress/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s" Nov 03 13:31:45.723414 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.723394 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5c25\" (UniqueName: \"kubernetes.io/projected/80fa8184-94c4-4ce1-8354-92681a69a6bf-kube-api-access-s5c25\") pod \"e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s\" (UID: \"80fa8184-94c4-4ce1-8354-92681a69a6bf\") " pod="openshift-ingress/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s" Nov 03 13:31:45.725054 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.725032 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80fa8184-94c4-4ce1-8354-92681a69a6bf-istio-token\") pod \"e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s\" (UID: \"80fa8184-94c4-4ce1-8354-92681a69a6bf\") " pod="openshift-ingress/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s" Nov 03 13:31:45.725765 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.725742 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80fa8184-94c4-4ce1-8354-92681a69a6bf-workload-certs\") pod \"e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s\" (UID: \"80fa8184-94c4-4ce1-8354-92681a69a6bf\") " pod="openshift-ingress/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s" Nov 03 13:31:45.726310 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.726292 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80fa8184-94c4-4ce1-8354-92681a69a6bf-workload-socket\") pod \"e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s\" (UID: \"80fa8184-94c4-4ce1-8354-92681a69a6bf\") " pod="openshift-ingress/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s" Nov 03 13:31:45.726851 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.722675 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80fa8184-94c4-4ce1-8354-92681a69a6bf-istiod-ca-cert\") pod \"e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s\" (UID: \"80fa8184-94c4-4ce1-8354-92681a69a6bf\") " pod="openshift-ingress/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s" Nov 03 13:31:45.726989 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.720630 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80fa8184-94c4-4ce1-8354-92681a69a6bf-istio-data\") pod \"e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s\" (UID: \"80fa8184-94c4-4ce1-8354-92681a69a6bf\") " pod="openshift-ingress/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s" Nov 03 13:31:45.727080 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.726121 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80fa8184-94c4-4ce1-8354-92681a69a6bf-workload-certs\") pod \"e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s\" (UID: \"80fa8184-94c4-4ce1-8354-92681a69a6bf\") " pod="openshift-ingress/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s" Nov 03 13:31:45.727184 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.720146 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80fa8184-94c4-4ce1-8354-92681a69a6bf-credential-socket\") pod \"e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s\" (UID: \"80fa8184-94c4-4ce1-8354-92681a69a6bf\") " pod="openshift-ingress/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s" Nov 03 13:31:45.727291 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.726696 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80fa8184-94c4-4ce1-8354-92681a69a6bf-workload-socket\") pod \"e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s\" (UID: \"80fa8184-94c4-4ce1-8354-92681a69a6bf\") " pod="openshift-ingress/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s" Nov 03 13:31:45.767506 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.767374 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80fa8184-94c4-4ce1-8354-92681a69a6bf-istio-envoy\") pod \"e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s\" (UID: \"80fa8184-94c4-4ce1-8354-92681a69a6bf\") " pod="openshift-ingress/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s" Nov 03 13:31:45.770633 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.770601 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80fa8184-94c4-4ce1-8354-92681a69a6bf-istio-podinfo\") pod \"e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s\" (UID: \"80fa8184-94c4-4ce1-8354-92681a69a6bf\") " pod="openshift-ingress/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s" Nov 03 13:31:45.792433 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.792394 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5c25\" (UniqueName: \"kubernetes.io/projected/80fa8184-94c4-4ce1-8354-92681a69a6bf-kube-api-access-s5c25\") pod \"e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s\" (UID: \"80fa8184-94c4-4ce1-8354-92681a69a6bf\") " pod="openshift-ingress/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s" Nov 03 13:31:45.792673 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.792650 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80fa8184-94c4-4ce1-8354-92681a69a6bf-istio-token\") pod \"e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s\" (UID: \"80fa8184-94c4-4ce1-8354-92681a69a6bf\") " pod="openshift-ingress/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s" Nov 03 13:31:45.829399 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:45.829372 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s" Nov 03 13:31:46.116565 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:46.116453 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26360457-2ef0-4c64-9a1c-4d62a783dc49" path="/var/lib/kubelet/pods/26360457-2ef0-4c64-9a1c-4d62a783dc49/volumes" Nov 03 13:31:46.117002 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:46.116985 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e80110d-950d-452f-9fdf-eecb56840a3c" path="/var/lib/kubelet/pods/3e80110d-950d-452f-9fdf-eecb56840a3c/volumes" Nov 03 13:31:46.117419 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:46.117406 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48" path="/var/lib/kubelet/pods/7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48/volumes" Nov 03 13:31:46.118092 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:46.118070 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="821c73f7-cf2f-414c-8aea-1aff84e017af" path="/var/lib/kubelet/pods/821c73f7-cf2f-414c-8aea-1aff84e017af/volumes" Nov 03 13:31:46.192741 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:46.192704 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg" Nov 03 13:31:46.199568 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:46.199544 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg" Nov 03 13:31:46.235774 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:46.235742 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg" Nov 03 13:31:46.236733 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:46.236712 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg" Nov 03 13:31:46.259832 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:46.259807 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s"] Nov 03 13:31:46.266090 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:31:46.266060 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80fa8184_94c4_4ce1_8354_92681a69a6bf.slice/crio-d2a8feb67827203babc5a40232622c272c897df25aab03e8dbf5d814ebfefa65 WatchSource:0}: Error finding container d2a8feb67827203babc5a40232622c272c897df25aab03e8dbf5d814ebfefa65: Status 404 returned error can't find the container with id d2a8feb67827203babc5a40232622c272c897df25aab03e8dbf5d814ebfefa65 Nov 03 13:31:46.268557 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:46.268530 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:31:46.268642 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:46.268593 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:31:46.268642 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:46.268627 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:31:47.246046 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:47.246001 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s" event={"ID":"80fa8184-94c4-4ce1-8354-92681a69a6bf","Type":"ContainerStarted","Data":"4d964944b8213e907ff08f64d1e48efce13f570076c250aa322688d326ad21c3"} Nov 03 13:31:47.246509 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:47.246065 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s" event={"ID":"80fa8184-94c4-4ce1-8354-92681a69a6bf","Type":"ContainerStarted","Data":"d2a8feb67827203babc5a40232622c272c897df25aab03e8dbf5d814ebfefa65"} Nov 03 13:31:47.359335 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:47.359274 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s" podStartSLOduration=2.359255159 podStartE2EDuration="2.359255159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:31:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:31:47.355776998 +0000 UTC m=+1589.366465239" watchObservedRunningTime="2026-11-03 13:31:47.359255159 +0000 UTC m=+1589.369943404" Nov 03 13:31:47.843207 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:47.843177 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s" Nov 03 13:31:47.897320 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:47.896497 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s" Nov 03 13:31:48.253891 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:48.253858 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s" Nov 03 13:31:48.254775 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:48.254756 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s" Nov 03 13:31:54.054131 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:54.054095 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s"] Nov 03 13:31:54.054826 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:54.054806 5979 cpu_manager.go:414] "containerMap: removing container" podUID="7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48" containerName="istio-proxy" Nov 03 13:31:54.054875 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:54.054828 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7073d2c6-f0fd-4ed0-b98b-ca65ebddbf48" containerName="istio-proxy" Nov 03 13:31:54.057675 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:54.057656 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s" Nov 03 13:31:54.123406 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:54.118730 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-be8d0a-openshift-default-dockercfg-hnpj6\"" Nov 03 13:31:54.163910 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:54.163863 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80f447c3-327a-4726-b976-8ce7adcd92fa-istio-envoy\") pod \"e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s\" (UID: \"80f447c3-327a-4726-b976-8ce7adcd92fa\") " pod="openshift-ingress/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s" Nov 03 13:31:54.164198 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:54.164171 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80f447c3-327a-4726-b976-8ce7adcd92fa-istio-token\") pod \"e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s\" (UID: \"80f447c3-327a-4726-b976-8ce7adcd92fa\") " pod="openshift-ingress/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s" Nov 03 13:31:54.164391 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:54.164362 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80f447c3-327a-4726-b976-8ce7adcd92fa-workload-socket\") pod \"e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s\" (UID: \"80f447c3-327a-4726-b976-8ce7adcd92fa\") " pod="openshift-ingress/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s" Nov 03 13:31:54.164538 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:54.164450 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80f447c3-327a-4726-b976-8ce7adcd92fa-workload-certs\") pod \"e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s\" (UID: \"80f447c3-327a-4726-b976-8ce7adcd92fa\") " pod="openshift-ingress/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s" Nov 03 13:31:54.164612 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:54.164537 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76t66\" (UniqueName: \"kubernetes.io/projected/80f447c3-327a-4726-b976-8ce7adcd92fa-kube-api-access-76t66\") pod \"e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s\" (UID: \"80f447c3-327a-4726-b976-8ce7adcd92fa\") " pod="openshift-ingress/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s" Nov 03 13:31:54.164612 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:54.164595 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80f447c3-327a-4726-b976-8ce7adcd92fa-istio-data\") pod \"e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s\" (UID: \"80f447c3-327a-4726-b976-8ce7adcd92fa\") " pod="openshift-ingress/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s" Nov 03 13:31:54.164723 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:54.164655 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80f447c3-327a-4726-b976-8ce7adcd92fa-istiod-ca-cert\") pod \"e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s\" (UID: \"80f447c3-327a-4726-b976-8ce7adcd92fa\") " pod="openshift-ingress/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s" Nov 03 13:31:54.164723 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:54.164708 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80f447c3-327a-4726-b976-8ce7adcd92fa-credential-socket\") pod \"e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s\" (UID: \"80f447c3-327a-4726-b976-8ce7adcd92fa\") " pod="openshift-ingress/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s" Nov 03 13:31:54.164853 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:54.164745 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80f447c3-327a-4726-b976-8ce7adcd92fa-istio-podinfo\") pod \"e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s\" (UID: \"80f447c3-327a-4726-b976-8ce7adcd92fa\") " pod="openshift-ingress/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s" Nov 03 13:31:54.170552 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:54.170525 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s"] Nov 03 13:31:54.266936 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:54.266902 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80f447c3-327a-4726-b976-8ce7adcd92fa-istiod-ca-cert\") pod \"e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s\" (UID: \"80f447c3-327a-4726-b976-8ce7adcd92fa\") " pod="openshift-ingress/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s" Nov 03 13:31:54.267139 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:54.266952 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80f447c3-327a-4726-b976-8ce7adcd92fa-credential-socket\") pod \"e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s\" (UID: \"80f447c3-327a-4726-b976-8ce7adcd92fa\") " pod="openshift-ingress/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s" Nov 03 13:31:54.267139 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:54.266976 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80f447c3-327a-4726-b976-8ce7adcd92fa-istio-podinfo\") pod \"e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s\" (UID: \"80f447c3-327a-4726-b976-8ce7adcd92fa\") " pod="openshift-ingress/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s" Nov 03 13:31:54.267139 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:54.267053 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80f447c3-327a-4726-b976-8ce7adcd92fa-istio-envoy\") pod \"e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s\" (UID: \"80f447c3-327a-4726-b976-8ce7adcd92fa\") " pod="openshift-ingress/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s" Nov 03 13:31:54.267139 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:54.267100 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80f447c3-327a-4726-b976-8ce7adcd92fa-istio-token\") pod \"e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s\" (UID: \"80f447c3-327a-4726-b976-8ce7adcd92fa\") " pod="openshift-ingress/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s" Nov 03 13:31:54.267139 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:54.267131 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80f447c3-327a-4726-b976-8ce7adcd92fa-workload-socket\") pod \"e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s\" (UID: \"80f447c3-327a-4726-b976-8ce7adcd92fa\") " pod="openshift-ingress/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s" Nov 03 13:31:54.267430 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:54.267169 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80f447c3-327a-4726-b976-8ce7adcd92fa-workload-certs\") pod \"e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s\" (UID: \"80f447c3-327a-4726-b976-8ce7adcd92fa\") " pod="openshift-ingress/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s" Nov 03 13:31:54.267430 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:54.267227 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76t66\" (UniqueName: \"kubernetes.io/projected/80f447c3-327a-4726-b976-8ce7adcd92fa-kube-api-access-76t66\") pod \"e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s\" (UID: \"80f447c3-327a-4726-b976-8ce7adcd92fa\") " pod="openshift-ingress/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s" Nov 03 13:31:54.267430 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:54.267380 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80f447c3-327a-4726-b976-8ce7adcd92fa-credential-socket\") pod \"e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s\" (UID: \"80f447c3-327a-4726-b976-8ce7adcd92fa\") " pod="openshift-ingress/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s" Nov 03 13:31:54.267769 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:54.267743 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80f447c3-327a-4726-b976-8ce7adcd92fa-workload-certs\") pod \"e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s\" (UID: \"80f447c3-327a-4726-b976-8ce7adcd92fa\") " pod="openshift-ingress/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s" Nov 03 13:31:54.267930 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:54.267844 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80f447c3-327a-4726-b976-8ce7adcd92fa-istiod-ca-cert\") pod \"e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s\" (UID: \"80f447c3-327a-4726-b976-8ce7adcd92fa\") " pod="openshift-ingress/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s" Nov 03 13:31:54.268062 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:54.268033 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80f447c3-327a-4726-b976-8ce7adcd92fa-istio-data\") pod \"e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s\" (UID: \"80f447c3-327a-4726-b976-8ce7adcd92fa\") " pod="openshift-ingress/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s" Nov 03 13:31:54.268183 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:54.268119 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80f447c3-327a-4726-b976-8ce7adcd92fa-workload-socket\") pod \"e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s\" (UID: \"80f447c3-327a-4726-b976-8ce7adcd92fa\") " pod="openshift-ingress/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s" Nov 03 13:31:54.268380 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:54.268356 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80f447c3-327a-4726-b976-8ce7adcd92fa-istio-data\") pod \"e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s\" (UID: \"80f447c3-327a-4726-b976-8ce7adcd92fa\") " pod="openshift-ingress/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s" Nov 03 13:31:54.269657 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:54.269638 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80f447c3-327a-4726-b976-8ce7adcd92fa-istio-envoy\") pod \"e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s\" (UID: \"80f447c3-327a-4726-b976-8ce7adcd92fa\") " pod="openshift-ingress/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s" Nov 03 13:31:54.269869 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:54.269853 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80f447c3-327a-4726-b976-8ce7adcd92fa-istio-podinfo\") pod \"e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s\" (UID: \"80f447c3-327a-4726-b976-8ce7adcd92fa\") " pod="openshift-ingress/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s" Nov 03 13:31:54.355821 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:54.353065 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76t66\" (UniqueName: \"kubernetes.io/projected/80f447c3-327a-4726-b976-8ce7adcd92fa-kube-api-access-76t66\") pod \"e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s\" (UID: \"80f447c3-327a-4726-b976-8ce7adcd92fa\") " pod="openshift-ingress/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s" Nov 03 13:31:54.358348 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:54.358313 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80f447c3-327a-4726-b976-8ce7adcd92fa-istio-token\") pod \"e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s\" (UID: \"80f447c3-327a-4726-b976-8ce7adcd92fa\") " pod="openshift-ingress/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s" Nov 03 13:31:54.409505 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:54.407799 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s" Nov 03 13:31:54.907645 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:54.907610 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s"] Nov 03 13:31:54.914242 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:31:54.914213 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80f447c3_327a_4726_b976_8ce7adcd92fa.slice/crio-14f13776cfca46fb84d4fc1270efa0248036247334865bf8e42d597d1c9db941 WatchSource:0}: Error finding container 14f13776cfca46fb84d4fc1270efa0248036247334865bf8e42d597d1c9db941: Status 404 returned error can't find the container with id 14f13776cfca46fb84d4fc1270efa0248036247334865bf8e42d597d1c9db941 Nov 03 13:31:54.917259 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:54.917225 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:31:54.917351 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:54.917302 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:31:54.917351 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:54.917331 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:31:55.018175 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:55.018141 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e7ca74-74db7b7bdc-gz58m"] Nov 03 13:31:55.022372 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:55.022340 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e7ca74-74db7b7bdc-gz58m" Nov 03 13:31:55.079230 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:55.075037 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-e7ca74\"" Nov 03 13:31:55.079230 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:55.075057 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-e7ca74-dockercfg-fw67s\"" Nov 03 13:31:55.079230 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:55.079169 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e7ca74-74db7b7bdc-gz58m"] Nov 03 13:31:55.106978 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:55.096557 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/211eda79-4ce3-4b71-a4e5-73b86f72ac15-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e7ca74-74db7b7bdc-gz58m\" (UID: \"211eda79-4ce3-4b71-a4e5-73b86f72ac15\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e7ca74-74db7b7bdc-gz58m" Nov 03 13:31:55.106978 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:55.096613 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmjhb\" (UniqueName: \"kubernetes.io/projected/211eda79-4ce3-4b71-a4e5-73b86f72ac15-kube-api-access-pmjhb\") pod \"payload-pre-processing-e2e-api-b-e7ca74-74db7b7bdc-gz58m\" (UID: \"211eda79-4ce3-4b71-a4e5-73b86f72ac15\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e7ca74-74db7b7bdc-gz58m" Nov 03 13:31:55.242520 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:55.224529 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/211eda79-4ce3-4b71-a4e5-73b86f72ac15-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e7ca74-74db7b7bdc-gz58m\" (UID: \"211eda79-4ce3-4b71-a4e5-73b86f72ac15\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e7ca74-74db7b7bdc-gz58m" Nov 03 13:31:55.242520 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:55.224584 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmjhb\" (UniqueName: \"kubernetes.io/projected/211eda79-4ce3-4b71-a4e5-73b86f72ac15-kube-api-access-pmjhb\") pod \"payload-pre-processing-e2e-api-b-e7ca74-74db7b7bdc-gz58m\" (UID: \"211eda79-4ce3-4b71-a4e5-73b86f72ac15\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e7ca74-74db7b7bdc-gz58m" Nov 03 13:31:55.242520 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:55.228514 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/211eda79-4ce3-4b71-a4e5-73b86f72ac15-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e7ca74-74db7b7bdc-gz58m\" (UID: \"211eda79-4ce3-4b71-a4e5-73b86f72ac15\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e7ca74-74db7b7bdc-gz58m" Nov 03 13:31:55.256759 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:55.256714 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e7ca74-848576db5f-kgxq7"] Nov 03 13:31:55.312893 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:55.312863 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e7ca74-848576db5f-kgxq7"] Nov 03 13:31:55.313230 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:55.313209 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e7ca74-848576db5f-kgxq7" Nov 03 13:31:55.327590 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:55.323574 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmjhb\" (UniqueName: \"kubernetes.io/projected/211eda79-4ce3-4b71-a4e5-73b86f72ac15-kube-api-access-pmjhb\") pod \"payload-pre-processing-e2e-api-b-e7ca74-74db7b7bdc-gz58m\" (UID: \"211eda79-4ce3-4b71-a4e5-73b86f72ac15\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e7ca74-74db7b7bdc-gz58m" Nov 03 13:31:55.341139 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:55.341106 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e7ca74-74db7b7bdc-gz58m" Nov 03 13:31:55.393515 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:55.392668 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s" event={"ID":"80f447c3-327a-4726-b976-8ce7adcd92fa","Type":"ContainerStarted","Data":"0d6b04afa5b09b783bbb268c1ce484933b3ea0b2010040be34269f66ef17ef95"} Nov 03 13:31:55.393515 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:55.392720 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s" event={"ID":"80f447c3-327a-4726-b976-8ce7adcd92fa","Type":"ContainerStarted","Data":"14f13776cfca46fb84d4fc1270efa0248036247334865bf8e42d597d1c9db941"} Nov 03 13:31:55.409017 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:55.408986 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s" Nov 03 13:31:55.411076 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:55.410738 5979 patch_prober.go:29] interesting pod/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 03 13:31:55.411076 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:55.410791 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s" podUID="80f447c3-327a-4726-b976-8ce7adcd92fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 03 13:31:55.445176 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:55.443258 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5d98b25-2641-4cbb-b288-38142a3869fd-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e7ca74-848576db5f-kgxq7\" (UID: \"f5d98b25-2641-4cbb-b288-38142a3869fd\") " pod="openshift-ingress/payload-processing-e2e-api-b-e7ca74-848576db5f-kgxq7" Nov 03 13:31:55.445176 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:55.443316 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrxmn\" (UniqueName: \"kubernetes.io/projected/f5d98b25-2641-4cbb-b288-38142a3869fd-kube-api-access-vrxmn\") pod \"payload-processing-e2e-api-b-e7ca74-848576db5f-kgxq7\" (UID: \"f5d98b25-2641-4cbb-b288-38142a3869fd\") " pod="openshift-ingress/payload-processing-e2e-api-b-e7ca74-848576db5f-kgxq7" Nov 03 13:31:55.544328 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:55.544244 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5d98b25-2641-4cbb-b288-38142a3869fd-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e7ca74-848576db5f-kgxq7\" (UID: \"f5d98b25-2641-4cbb-b288-38142a3869fd\") " pod="openshift-ingress/payload-processing-e2e-api-b-e7ca74-848576db5f-kgxq7" Nov 03 13:31:55.544328 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:55.544293 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrxmn\" (UniqueName: \"kubernetes.io/projected/f5d98b25-2641-4cbb-b288-38142a3869fd-kube-api-access-vrxmn\") pod \"payload-processing-e2e-api-b-e7ca74-848576db5f-kgxq7\" (UID: \"f5d98b25-2641-4cbb-b288-38142a3869fd\") " pod="openshift-ingress/payload-processing-e2e-api-b-e7ca74-848576db5f-kgxq7" Nov 03 13:31:55.545174 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:55.545150 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5d98b25-2641-4cbb-b288-38142a3869fd-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e7ca74-848576db5f-kgxq7\" (UID: \"f5d98b25-2641-4cbb-b288-38142a3869fd\") " pod="openshift-ingress/payload-processing-e2e-api-b-e7ca74-848576db5f-kgxq7" Nov 03 13:31:55.555419 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:55.555372 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s" podStartSLOduration=2.555359866 podStartE2EDuration="2.555359866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:31:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:31:55.552348349 +0000 UTC m=+1597.563036591" watchObservedRunningTime="2026-11-03 13:31:55.555359866 +0000 UTC m=+1597.566048108" Nov 03 13:31:55.598419 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:55.598386 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrxmn\" (UniqueName: \"kubernetes.io/projected/f5d98b25-2641-4cbb-b288-38142a3869fd-kube-api-access-vrxmn\") pod \"payload-processing-e2e-api-b-e7ca74-848576db5f-kgxq7\" (UID: \"f5d98b25-2641-4cbb-b288-38142a3869fd\") " pod="openshift-ingress/payload-processing-e2e-api-b-e7ca74-848576db5f-kgxq7" Nov 03 13:31:55.654519 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:55.654486 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e7ca74-848576db5f-kgxq7" Nov 03 13:31:55.746286 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:55.746246 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e7ca74-74db7b7bdc-gz58m"] Nov 03 13:31:55.766689 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:31:55.766653 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod211eda79_4ce3_4b71_a4e5_73b86f72ac15.slice/crio-a7ad9c4e56276cef8159f13b2a37a59bba4183583575952e59d646708e46f54d WatchSource:0}: Error finding container a7ad9c4e56276cef8159f13b2a37a59bba4183583575952e59d646708e46f54d: Status 404 returned error can't find the container with id a7ad9c4e56276cef8159f13b2a37a59bba4183583575952e59d646708e46f54d Nov 03 13:31:56.116950 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:56.116921 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e7ca74-848576db5f-kgxq7"] Nov 03 13:31:56.117509 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:31:56.117459 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5d98b25_2641_4cbb_b288_38142a3869fd.slice/crio-3a7ff767a6d59ce78aaae29666ac66398e82935cd57ba101a9577b1c70543128 WatchSource:0}: Error finding container 3a7ff767a6d59ce78aaae29666ac66398e82935cd57ba101a9577b1c70543128: Status 404 returned error can't find the container with id 3a7ff767a6d59ce78aaae29666ac66398e82935cd57ba101a9577b1c70543128 Nov 03 13:31:56.401641 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:56.401601 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e7ca74-74db7b7bdc-gz58m" event={"ID":"211eda79-4ce3-4b71-a4e5-73b86f72ac15","Type":"ContainerStarted","Data":"e3fa584e570991631115cdb20fbe5cdf02ffd94ba654ba9f66051d2790adaeb7"} Nov 03 13:31:56.401848 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:56.401686 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e7ca74-74db7b7bdc-gz58m" event={"ID":"211eda79-4ce3-4b71-a4e5-73b86f72ac15","Type":"ContainerStarted","Data":"a7ad9c4e56276cef8159f13b2a37a59bba4183583575952e59d646708e46f54d"} Nov 03 13:31:56.401848 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:56.401802 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e7ca74-74db7b7bdc-gz58m" Nov 03 13:31:56.403420 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:56.403394 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e7ca74-848576db5f-kgxq7" event={"ID":"f5d98b25-2641-4cbb-b288-38142a3869fd","Type":"ContainerStarted","Data":"27bcbe9d87b6e4e661a5d465e456afadcb58f9843d5d0b84ab1cb6825a93df78"} Nov 03 13:31:56.403553 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:56.403438 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e7ca74-848576db5f-kgxq7" event={"ID":"f5d98b25-2641-4cbb-b288-38142a3869fd","Type":"ContainerStarted","Data":"3a7ff767a6d59ce78aaae29666ac66398e82935cd57ba101a9577b1c70543128"} Nov 03 13:31:56.403650 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:56.403634 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-e7ca74-848576db5f-kgxq7" Nov 03 13:31:56.413344 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:56.413321 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s" Nov 03 13:31:56.511243 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:56.511186 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e7ca74-74db7b7bdc-gz58m" podStartSLOduration=2.5111720440000003 podStartE2EDuration="2.511172044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:31:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:31:56.508577652 +0000 UTC m=+1598.519265893" watchObservedRunningTime="2026-11-03 13:31:56.511172044 +0000 UTC m=+1598.521860286" Nov 03 13:31:56.739647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:56.739542 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-e7ca74-848576db5f-kgxq7" podStartSLOduration=1.739526515 podStartE2EDuration="1.739526515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:31:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:31:56.735309612 +0000 UTC m=+1598.745997855" watchObservedRunningTime="2026-11-03 13:31:56.739526515 +0000 UTC m=+1598.750214757" Nov 03 13:31:57.411293 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:57.411253 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s" Nov 03 13:31:57.412011 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:57.411982 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s" Nov 03 13:31:58.912217 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:58.912181 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st"] Nov 03 13:31:58.919863 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:58.919827 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st" Nov 03 13:31:59.005758 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:59.005722 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be094f6c-c6d9-4a53-8c71-33acdca60856-istio-podinfo\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st\" (UID: \"be094f6c-c6d9-4a53-8c71-33acdca60856\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st" Nov 03 13:31:59.006013 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:59.005774 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be094f6c-c6d9-4a53-8c71-33acdca60856-istio-data\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st\" (UID: \"be094f6c-c6d9-4a53-8c71-33acdca60856\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st" Nov 03 13:31:59.006013 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:59.005969 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be094f6c-c6d9-4a53-8c71-33acdca60856-workload-certs\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st\" (UID: \"be094f6c-c6d9-4a53-8c71-33acdca60856\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st" Nov 03 13:31:59.006123 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:59.006013 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxvjl\" (UniqueName: \"kubernetes.io/projected/be094f6c-c6d9-4a53-8c71-33acdca60856-kube-api-access-wxvjl\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st\" (UID: \"be094f6c-c6d9-4a53-8c71-33acdca60856\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st" Nov 03 13:31:59.006176 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:59.006114 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be094f6c-c6d9-4a53-8c71-33acdca60856-istio-token\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st\" (UID: \"be094f6c-c6d9-4a53-8c71-33acdca60856\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st" Nov 03 13:31:59.006237 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:59.006195 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be094f6c-c6d9-4a53-8c71-33acdca60856-workload-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st\" (UID: \"be094f6c-c6d9-4a53-8c71-33acdca60856\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st" Nov 03 13:31:59.006325 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:59.006303 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be094f6c-c6d9-4a53-8c71-33acdca60856-credential-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st\" (UID: \"be094f6c-c6d9-4a53-8c71-33acdca60856\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st" Nov 03 13:31:59.006416 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:59.006392 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be094f6c-c6d9-4a53-8c71-33acdca60856-istiod-ca-cert\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st\" (UID: \"be094f6c-c6d9-4a53-8c71-33acdca60856\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st" Nov 03 13:31:59.006506 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:59.006436 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be094f6c-c6d9-4a53-8c71-33acdca60856-istio-envoy\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st\" (UID: \"be094f6c-c6d9-4a53-8c71-33acdca60856\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st" Nov 03 13:31:59.022134 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:59.021917 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st"] Nov 03 13:31:59.107778 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:59.107742 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be094f6c-c6d9-4a53-8c71-33acdca60856-istio-podinfo\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st\" (UID: \"be094f6c-c6d9-4a53-8c71-33acdca60856\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st" Nov 03 13:31:59.107977 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:59.107804 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be094f6c-c6d9-4a53-8c71-33acdca60856-istio-data\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st\" (UID: \"be094f6c-c6d9-4a53-8c71-33acdca60856\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st" Nov 03 13:31:59.107977 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:59.107861 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be094f6c-c6d9-4a53-8c71-33acdca60856-workload-certs\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st\" (UID: \"be094f6c-c6d9-4a53-8c71-33acdca60856\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st" Nov 03 13:31:59.107977 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:59.107878 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxvjl\" (UniqueName: \"kubernetes.io/projected/be094f6c-c6d9-4a53-8c71-33acdca60856-kube-api-access-wxvjl\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st\" (UID: \"be094f6c-c6d9-4a53-8c71-33acdca60856\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st" Nov 03 13:31:59.107977 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:59.107917 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be094f6c-c6d9-4a53-8c71-33acdca60856-istio-token\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st\" (UID: \"be094f6c-c6d9-4a53-8c71-33acdca60856\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st" Nov 03 13:31:59.108208 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:59.108138 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be094f6c-c6d9-4a53-8c71-33acdca60856-workload-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st\" (UID: \"be094f6c-c6d9-4a53-8c71-33acdca60856\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st" Nov 03 13:31:59.108263 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:59.108210 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be094f6c-c6d9-4a53-8c71-33acdca60856-credential-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st\" (UID: \"be094f6c-c6d9-4a53-8c71-33acdca60856\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st" Nov 03 13:31:59.108315 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:59.108271 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be094f6c-c6d9-4a53-8c71-33acdca60856-istiod-ca-cert\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st\" (UID: \"be094f6c-c6d9-4a53-8c71-33acdca60856\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st" Nov 03 13:31:59.108380 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:59.108312 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be094f6c-c6d9-4a53-8c71-33acdca60856-workload-certs\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st\" (UID: \"be094f6c-c6d9-4a53-8c71-33acdca60856\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st" Nov 03 13:31:59.108380 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:59.108317 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be094f6c-c6d9-4a53-8c71-33acdca60856-istio-envoy\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st\" (UID: \"be094f6c-c6d9-4a53-8c71-33acdca60856\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st" Nov 03 13:31:59.108380 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:59.108320 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be094f6c-c6d9-4a53-8c71-33acdca60856-istio-data\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st\" (UID: \"be094f6c-c6d9-4a53-8c71-33acdca60856\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st" Nov 03 13:31:59.108667 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:59.108623 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be094f6c-c6d9-4a53-8c71-33acdca60856-credential-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st\" (UID: \"be094f6c-c6d9-4a53-8c71-33acdca60856\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st" Nov 03 13:31:59.108790 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:59.108775 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be094f6c-c6d9-4a53-8c71-33acdca60856-workload-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st\" (UID: \"be094f6c-c6d9-4a53-8c71-33acdca60856\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st" Nov 03 13:31:59.109068 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:59.109044 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be094f6c-c6d9-4a53-8c71-33acdca60856-istiod-ca-cert\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st\" (UID: \"be094f6c-c6d9-4a53-8c71-33acdca60856\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st" Nov 03 13:31:59.110876 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:59.110853 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be094f6c-c6d9-4a53-8c71-33acdca60856-istio-podinfo\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st\" (UID: \"be094f6c-c6d9-4a53-8c71-33acdca60856\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st" Nov 03 13:31:59.110978 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:59.110939 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be094f6c-c6d9-4a53-8c71-33acdca60856-istio-envoy\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st\" (UID: \"be094f6c-c6d9-4a53-8c71-33acdca60856\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st" Nov 03 13:31:59.162663 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:59.162591 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxvjl\" (UniqueName: \"kubernetes.io/projected/be094f6c-c6d9-4a53-8c71-33acdca60856-kube-api-access-wxvjl\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st\" (UID: \"be094f6c-c6d9-4a53-8c71-33acdca60856\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st" Nov 03 13:31:59.162794 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:59.162660 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be094f6c-c6d9-4a53-8c71-33acdca60856-istio-token\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st\" (UID: \"be094f6c-c6d9-4a53-8c71-33acdca60856\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st" Nov 03 13:31:59.248851 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:59.248808 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st" Nov 03 13:31:59.713548 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:59.701438 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st"] Nov 03 13:31:59.956012 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:59.955956 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:31:59.956432 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:59.956123 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:31:59.956432 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:31:59.956197 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:32:00.035098 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:00.034973 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fa2568-6cb8658547-kqzwp"] Nov 03 13:32:00.042228 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:00.042191 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa2568-6cb8658547-kqzwp" Nov 03 13:32:00.045926 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:00.045898 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fa2568-6cb8658547-kqzwp"] Nov 03 13:32:00.242683 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:00.161884 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-fa2568\"" Nov 03 13:32:00.242683 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:00.163411 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecd0ef06-5ba4-496f-9470-43a606f8c687-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fa2568-6cb8658547-kqzwp\" (UID: \"ecd0ef06-5ba4-496f-9470-43a606f8c687\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa2568-6cb8658547-kqzwp" Nov 03 13:32:00.242683 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:00.184883 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-fa2568-dockercfg-vsf9t\"" Nov 03 13:32:00.244958 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:00.243399 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5qgm\" (UniqueName: \"kubernetes.io/projected/ecd0ef06-5ba4-496f-9470-43a606f8c687-kube-api-access-c5qgm\") pod \"payload-pre-processing-e2e-ipp-a-fa2568-6cb8658547-kqzwp\" (UID: \"ecd0ef06-5ba4-496f-9470-43a606f8c687\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa2568-6cb8658547-kqzwp" Nov 03 13:32:00.252067 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:00.245236 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fa2568-794d67c965-wnxdb"] Nov 03 13:32:00.286658 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:00.286609 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa2568-794d67c965-wnxdb" Nov 03 13:32:00.302507 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:00.302459 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fa2568-794d67c965-wnxdb"] Nov 03 13:32:00.344722 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:00.344688 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecd0ef06-5ba4-496f-9470-43a606f8c687-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fa2568-6cb8658547-kqzwp\" (UID: \"ecd0ef06-5ba4-496f-9470-43a606f8c687\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa2568-6cb8658547-kqzwp" Nov 03 13:32:00.344870 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:00.344858 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5qgm\" (UniqueName: \"kubernetes.io/projected/ecd0ef06-5ba4-496f-9470-43a606f8c687-kube-api-access-c5qgm\") pod \"payload-pre-processing-e2e-ipp-a-fa2568-6cb8658547-kqzwp\" (UID: \"ecd0ef06-5ba4-496f-9470-43a606f8c687\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa2568-6cb8658547-kqzwp" Nov 03 13:32:00.345647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:00.345618 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecd0ef06-5ba4-496f-9470-43a606f8c687-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fa2568-6cb8658547-kqzwp\" (UID: \"ecd0ef06-5ba4-496f-9470-43a606f8c687\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa2568-6cb8658547-kqzwp" Nov 03 13:32:00.399953 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:00.399917 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5qgm\" (UniqueName: \"kubernetes.io/projected/ecd0ef06-5ba4-496f-9470-43a606f8c687-kube-api-access-c5qgm\") pod \"payload-pre-processing-e2e-ipp-a-fa2568-6cb8658547-kqzwp\" (UID: \"ecd0ef06-5ba4-496f-9470-43a606f8c687\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa2568-6cb8658547-kqzwp" Nov 03 13:32:00.437941 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:00.437906 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st" event={"ID":"be094f6c-c6d9-4a53-8c71-33acdca60856","Type":"ContainerStarted","Data":"ea98d4a6d893c9e781d1c4307b043cd6d67333b62972779fd65eb089738a092e"} Nov 03 13:32:00.438124 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:00.437964 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st" event={"ID":"be094f6c-c6d9-4a53-8c71-33acdca60856","Type":"ContainerStarted","Data":"76706258da6bb1473e3f53d716b8150a13d083d100098cb85356a706604a20ca"} Nov 03 13:32:00.446853 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:00.446822 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmlc2\" (UniqueName: \"kubernetes.io/projected/2a6a387d-1db5-42e3-8d68-11fec4436ea0-kube-api-access-qmlc2\") pod \"payload-processing-e2e-ipp-a-fa2568-794d67c965-wnxdb\" (UID: \"2a6a387d-1db5-42e3-8d68-11fec4436ea0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fa2568-794d67c965-wnxdb" Nov 03 13:32:00.447061 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:00.447044 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a6a387d-1db5-42e3-8d68-11fec4436ea0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fa2568-794d67c965-wnxdb\" (UID: \"2a6a387d-1db5-42e3-8d68-11fec4436ea0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fa2568-794d67c965-wnxdb" Nov 03 13:32:00.549536 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:00.549078 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmlc2\" (UniqueName: \"kubernetes.io/projected/2a6a387d-1db5-42e3-8d68-11fec4436ea0-kube-api-access-qmlc2\") pod \"payload-processing-e2e-ipp-a-fa2568-794d67c965-wnxdb\" (UID: \"2a6a387d-1db5-42e3-8d68-11fec4436ea0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fa2568-794d67c965-wnxdb" Nov 03 13:32:00.549536 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:00.549287 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a6a387d-1db5-42e3-8d68-11fec4436ea0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fa2568-794d67c965-wnxdb\" (UID: \"2a6a387d-1db5-42e3-8d68-11fec4436ea0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fa2568-794d67c965-wnxdb" Nov 03 13:32:00.550198 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:00.550170 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a6a387d-1db5-42e3-8d68-11fec4436ea0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fa2568-794d67c965-wnxdb\" (UID: \"2a6a387d-1db5-42e3-8d68-11fec4436ea0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fa2568-794d67c965-wnxdb" Nov 03 13:32:00.553750 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:00.553691 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st" podStartSLOduration=2.553675956 podStartE2EDuration="2.553675956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:31:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:32:00.550227976 +0000 UTC m=+1602.560916218" watchObservedRunningTime="2026-11-03 13:32:00.553675956 +0000 UTC m=+1602.564364201" Nov 03 13:32:00.598159 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:00.598122 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa2568-6cb8658547-kqzwp" Nov 03 13:32:00.602914 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:00.602891 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmlc2\" (UniqueName: \"kubernetes.io/projected/2a6a387d-1db5-42e3-8d68-11fec4436ea0-kube-api-access-qmlc2\") pod \"payload-processing-e2e-ipp-a-fa2568-794d67c965-wnxdb\" (UID: \"2a6a387d-1db5-42e3-8d68-11fec4436ea0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fa2568-794d67c965-wnxdb" Nov 03 13:32:00.614793 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:00.614762 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa2568-794d67c965-wnxdb" Nov 03 13:32:00.992045 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:00.992005 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fa2568-6cb8658547-kqzwp"] Nov 03 13:32:01.053495 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:32:01.041087 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecd0ef06_5ba4_496f_9470_43a606f8c687.slice/crio-b88b70fce06227f7c9d00b63aeaf02045c1509663cdc73a2e3678d61059ee83d WatchSource:0}: Error finding container b88b70fce06227f7c9d00b63aeaf02045c1509663cdc73a2e3678d61059ee83d: Status 404 returned error can't find the container with id b88b70fce06227f7c9d00b63aeaf02045c1509663cdc73a2e3678d61059ee83d Nov 03 13:32:01.075143 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:01.069370 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fa2568-794d67c965-wnxdb"] Nov 03 13:32:01.268556 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:01.264863 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st" Nov 03 13:32:01.268738 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:01.268654 5979 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 13:32:01.268803 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:01.268735 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st" podUID="be094f6c-c6d9-4a53-8c71-33acdca60856" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 13:32:01.471865 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:01.471829 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa2568-6cb8658547-kqzwp" event={"ID":"ecd0ef06-5ba4-496f-9470-43a606f8c687","Type":"ContainerStarted","Data":"07c1dd2164c21afe3b068853634f64698faffee285d0a0d8694fcbc760a1c23b"} Nov 03 13:32:01.472092 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:01.471887 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa2568-6cb8658547-kqzwp" event={"ID":"ecd0ef06-5ba4-496f-9470-43a606f8c687","Type":"ContainerStarted","Data":"b88b70fce06227f7c9d00b63aeaf02045c1509663cdc73a2e3678d61059ee83d"} Nov 03 13:32:01.472092 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:01.472062 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa2568-6cb8658547-kqzwp" Nov 03 13:32:01.473943 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:01.473921 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa2568-794d67c965-wnxdb" event={"ID":"2a6a387d-1db5-42e3-8d68-11fec4436ea0","Type":"ContainerStarted","Data":"0aea09ff01ebe71b2ee93515fad85c75f614ffe80f74f4ad58c7afd6bd4a873d"} Nov 03 13:32:01.474054 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:01.473960 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa2568-794d67c965-wnxdb" event={"ID":"2a6a387d-1db5-42e3-8d68-11fec4436ea0","Type":"ContainerStarted","Data":"289f14d04ecd33d6d87b5ef867afdb954873546afcaa473342a2475e288dcca6"} Nov 03 13:32:01.474054 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:01.474012 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa2568-794d67c965-wnxdb" Nov 03 13:32:01.581898 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:01.581781 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa2568-6cb8658547-kqzwp" podStartSLOduration=2.581765 podStartE2EDuration="2.581765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:31:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:32:01.581670428 +0000 UTC m=+1603.592358670" watchObservedRunningTime="2026-11-03 13:32:01.581765 +0000 UTC m=+1603.592453243" Nov 03 13:32:01.697516 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:01.697450 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa2568-794d67c965-wnxdb" podStartSLOduration=1.69743684 podStartE2EDuration="1.69743684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:32:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:32:01.69599264 +0000 UTC m=+1603.706680883" watchObservedRunningTime="2026-11-03 13:32:01.69743684 +0000 UTC m=+1603.708125081" Nov 03 13:32:02.253496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:02.249845 5979 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 13:32:02.253496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:02.249907 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st" podUID="be094f6c-c6d9-4a53-8c71-33acdca60856" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 13:32:03.208781 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:03.208740 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1fdd09-68b57fc9d7-jpz5x"] Nov 03 13:32:03.257496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:03.255127 5979 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 13:32:03.257496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:03.255206 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st" podUID="be094f6c-c6d9-4a53-8c71-33acdca60856" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 13:32:03.326296 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:03.326252 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1fdd09-5f897ffc9f-m85wq"] Nov 03 13:32:04.263507 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:04.262916 5979 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 13:32:04.263507 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:04.262985 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st" podUID="be094f6c-c6d9-4a53-8c71-33acdca60856" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 13:32:05.254496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:05.249849 5979 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 13:32:05.254496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:05.249915 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st" podUID="be094f6c-c6d9-4a53-8c71-33acdca60856" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 13:32:06.265497 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:06.262906 5979 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 13:32:06.265497 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:06.262974 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st" podUID="be094f6c-c6d9-4a53-8c71-33acdca60856" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 13:32:07.235735 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:32:07.235692 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:32:07.250047 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:07.250016 5979 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Nov 03 13:32:07.250205 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:07.250065 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st" podUID="be094f6c-c6d9-4a53-8c71-33acdca60856" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Nov 03 13:32:07.414678 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:07.414648 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-e7ca74-848576db5f-kgxq7" Nov 03 13:32:07.415076 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:07.414718 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e7ca74-74db7b7bdc-gz58m" Nov 03 13:32:08.210045 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:08.210000 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-1fdd09-68b57fc9d7-jpz5x" podUID="a895b353-a4b2-48bc-86ae-236473d418ca" containerName="payload-processing" containerID="cri-o://3d4caaf6e852c139668ccbf822c09fb5bfd97a9e93fe6d6d05fa3dd96f2464c8" gracePeriod=25 Nov 03 13:32:08.254447 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:08.254415 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st" Nov 03 13:32:08.254747 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:08.254713 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st" Nov 03 13:32:08.255490 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:08.255454 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st" Nov 03 13:32:08.326744 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:08.326676 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fdd09-5f897ffc9f-m85wq" podUID="d8de58c5-0f63-43b8-91f1-ce1830f4b438" containerName="payload-pre-processing" containerID="cri-o://523e80ec9b9dcb8cf1afaf9612105a0efbcf77392a685db2db1af2c212311590" gracePeriod=25 Nov 03 13:32:08.501463 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:08.501429 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1fdd09-68b57fc9d7-jpz5x" Nov 03 13:32:08.534182 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:08.534144 5979 generic.go:356] "Generic (PLEG): container finished" podID="a895b353-a4b2-48bc-86ae-236473d418ca" containerID="3d4caaf6e852c139668ccbf822c09fb5bfd97a9e93fe6d6d05fa3dd96f2464c8" exitCode=0 Nov 03 13:32:08.534343 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:08.534212 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1fdd09-68b57fc9d7-jpz5x" Nov 03 13:32:08.534343 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:08.534234 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1fdd09-68b57fc9d7-jpz5x" event={"ID":"a895b353-a4b2-48bc-86ae-236473d418ca","Type":"ContainerDied","Data":"3d4caaf6e852c139668ccbf822c09fb5bfd97a9e93fe6d6d05fa3dd96f2464c8"} Nov 03 13:32:08.534343 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:08.534303 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1fdd09-68b57fc9d7-jpz5x" event={"ID":"a895b353-a4b2-48bc-86ae-236473d418ca","Type":"ContainerDied","Data":"4feccea93c607687863fbba223d7dcb2a6f33684a8d83c0a1c892b2380de2697"} Nov 03 13:32:08.534565 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:08.534337 5979 scope.go:122] "RemoveContainer" containerID="3d4caaf6e852c139668ccbf822c09fb5bfd97a9e93fe6d6d05fa3dd96f2464c8" Nov 03 13:32:08.539894 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:08.537568 5979 generic.go:356] "Generic (PLEG): container finished" podID="d8de58c5-0f63-43b8-91f1-ce1830f4b438" containerID="523e80ec9b9dcb8cf1afaf9612105a0efbcf77392a685db2db1af2c212311590" exitCode=0 Nov 03 13:32:08.539894 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:08.538368 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fdd09-5f897ffc9f-m85wq" event={"ID":"d8de58c5-0f63-43b8-91f1-ce1830f4b438","Type":"ContainerDied","Data":"523e80ec9b9dcb8cf1afaf9612105a0efbcf77392a685db2db1af2c212311590"} Nov 03 13:32:08.558842 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:08.558809 5979 scope.go:122] "RemoveContainer" containerID="3d4caaf6e852c139668ccbf822c09fb5bfd97a9e93fe6d6d05fa3dd96f2464c8" Nov 03 13:32:08.559210 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:32:08.559182 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d4caaf6e852c139668ccbf822c09fb5bfd97a9e93fe6d6d05fa3dd96f2464c8\": container with ID starting with 3d4caaf6e852c139668ccbf822c09fb5bfd97a9e93fe6d6d05fa3dd96f2464c8 not found: ID does not exist" containerID="3d4caaf6e852c139668ccbf822c09fb5bfd97a9e93fe6d6d05fa3dd96f2464c8" Nov 03 13:32:08.559306 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:08.559222 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d4caaf6e852c139668ccbf822c09fb5bfd97a9e93fe6d6d05fa3dd96f2464c8"} err="failed to get container status \"3d4caaf6e852c139668ccbf822c09fb5bfd97a9e93fe6d6d05fa3dd96f2464c8\": rpc error: code = NotFound desc = could not find container \"3d4caaf6e852c139668ccbf822c09fb5bfd97a9e93fe6d6d05fa3dd96f2464c8\": container with ID starting with 3d4caaf6e852c139668ccbf822c09fb5bfd97a9e93fe6d6d05fa3dd96f2464c8 not found: ID does not exist" Nov 03 13:32:08.625685 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:08.620852 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a895b353-a4b2-48bc-86ae-236473d418ca-kube-api-access-whj6p\" (UniqueName: \"kubernetes.io/projected/a895b353-a4b2-48bc-86ae-236473d418ca-kube-api-access-whj6p\") pod \"a895b353-a4b2-48bc-86ae-236473d418ca\" (UID: \"a895b353-a4b2-48bc-86ae-236473d418ca\") " Nov 03 13:32:08.625685 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:08.620907 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a895b353-a4b2-48bc-86ae-236473d418ca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a895b353-a4b2-48bc-86ae-236473d418ca-plugins-config-volume\") pod \"a895b353-a4b2-48bc-86ae-236473d418ca\" (UID: \"a895b353-a4b2-48bc-86ae-236473d418ca\") " Nov 03 13:32:08.625685 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:08.623233 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a895b353-a4b2-48bc-86ae-236473d418ca-plugins-config-volume" pod "a895b353-a4b2-48bc-86ae-236473d418ca" (UID: "a895b353-a4b2-48bc-86ae-236473d418ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:32:08.628621 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:08.628343 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a895b353-a4b2-48bc-86ae-236473d418ca-kube-api-access-whj6p" pod "a895b353-a4b2-48bc-86ae-236473d418ca" (UID: "a895b353-a4b2-48bc-86ae-236473d418ca"). InnerVolumeSpecName "kube-api-access-whj6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:32:08.650702 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:08.650666 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fdd09-5f897ffc9f-m85wq" Nov 03 13:32:08.723446 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:08.723374 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whj6p\" (UniqueName: \"kubernetes.io/projected/a895b353-a4b2-48bc-86ae-236473d418ca-kube-api-access-whj6p\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:08.723446 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:08.723399 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a895b353-a4b2-48bc-86ae-236473d418ca-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:08.824297 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:08.824264 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8de58c5-0f63-43b8-91f1-ce1830f4b438-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8de58c5-0f63-43b8-91f1-ce1830f4b438-plugins-config-volume\") pod \"d8de58c5-0f63-43b8-91f1-ce1830f4b438\" (UID: \"d8de58c5-0f63-43b8-91f1-ce1830f4b438\") " Nov 03 13:32:08.824561 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:08.824539 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8de58c5-0f63-43b8-91f1-ce1830f4b438-kube-api-access-8hbcd\" (UniqueName: \"kubernetes.io/projected/d8de58c5-0f63-43b8-91f1-ce1830f4b438-kube-api-access-8hbcd\") pod \"d8de58c5-0f63-43b8-91f1-ce1830f4b438\" (UID: \"d8de58c5-0f63-43b8-91f1-ce1830f4b438\") " Nov 03 13:32:08.824764 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:08.824744 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8de58c5-0f63-43b8-91f1-ce1830f4b438-plugins-config-volume" pod "d8de58c5-0f63-43b8-91f1-ce1830f4b438" (UID: "d8de58c5-0f63-43b8-91f1-ce1830f4b438"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:32:08.825064 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:08.825037 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8de58c5-0f63-43b8-91f1-ce1830f4b438-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:08.826635 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:08.826611 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8de58c5-0f63-43b8-91f1-ce1830f4b438-kube-api-access-8hbcd" pod "d8de58c5-0f63-43b8-91f1-ce1830f4b438" (UID: "d8de58c5-0f63-43b8-91f1-ce1830f4b438"). InnerVolumeSpecName "kube-api-access-8hbcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:32:08.926243 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:08.926207 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hbcd\" (UniqueName: \"kubernetes.io/projected/d8de58c5-0f63-43b8-91f1-ce1830f4b438-kube-api-access-8hbcd\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:08.949553 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:08.949521 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1fdd09-68b57fc9d7-jpz5x"] Nov 03 13:32:08.953765 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:08.953738 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1fdd09-68b57fc9d7-jpz5x"] Nov 03 13:32:09.713690 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:09.713641 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fdd09-5f897ffc9f-m85wq" Nov 03 13:32:09.717685 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:09.714405 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1fdd09-5f897ffc9f-m85wq" event={"ID":"d8de58c5-0f63-43b8-91f1-ce1830f4b438","Type":"ContainerDied","Data":"d22e1e863b059543d86b130a7713eaa451cb7bd661e2b26cdf789cc0c2d4f28e"} Nov 03 13:32:09.717840 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:09.717778 5979 scope.go:122] "RemoveContainer" containerID="523e80ec9b9dcb8cf1afaf9612105a0efbcf77392a685db2db1af2c212311590" Nov 03 13:32:09.808667 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:09.808631 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:32:09.874704 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:09.874671 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1fdd09-5f897ffc9f-m85wq"] Nov 03 13:32:09.880203 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:09.880172 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1fdd09-5f897ffc9f-m85wq"] Nov 03 13:32:10.116926 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:10.116845 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a895b353-a4b2-48bc-86ae-236473d418ca" path="/var/lib/kubelet/pods/a895b353-a4b2-48bc-86ae-236473d418ca/volumes" Nov 03 13:32:10.117323 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:10.117308 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8de58c5-0f63-43b8-91f1-ce1830f4b438" path="/var/lib/kubelet/pods/d8de58c5-0f63-43b8-91f1-ce1830f4b438/volumes" Nov 03 13:32:12.485055 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:12.485027 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa2568-794d67c965-wnxdb" Nov 03 13:32:12.485591 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:12.485109 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa2568-6cb8658547-kqzwp" Nov 03 13:32:14.998380 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:14.998337 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-be8d0a-689bdb4cf5-l4wtk"] Nov 03 13:32:14.999602 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:14.999574 5979 cpu_manager.go:414] "containerMap: removing container" podUID="d8de58c5-0f63-43b8-91f1-ce1830f4b438" containerName="payload-pre-processing" Nov 03 13:32:14.999602 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:14.999601 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8de58c5-0f63-43b8-91f1-ce1830f4b438" containerName="payload-pre-processing" Nov 03 13:32:14.999820 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:14.999643 5979 cpu_manager.go:414] "containerMap: removing container" podUID="a895b353-a4b2-48bc-86ae-236473d418ca" containerName="payload-processing" Nov 03 13:32:14.999820 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:14.999652 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a895b353-a4b2-48bc-86ae-236473d418ca" containerName="payload-processing" Nov 03 13:32:14.999954 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:14.999832 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="a895b353-a4b2-48bc-86ae-236473d418ca" containerName="payload-processing" Nov 03 13:32:14.999954 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:14.999863 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8de58c5-0f63-43b8-91f1-ce1830f4b438" containerName="payload-pre-processing" Nov 03 13:32:15.003945 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:15.003925 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be8d0a-689bdb4cf5-l4wtk" Nov 03 13:32:15.052687 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:15.052656 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-be8d0a-689bdb4cf5-l4wtk"] Nov 03 13:32:15.055519 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:15.055447 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e533eaa-a35c-475e-a88e-6dbf38f3f37e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-be8d0a-689bdb4cf5-l4wtk\" (UID: \"5e533eaa-a35c-475e-a88e-6dbf38f3f37e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be8d0a-689bdb4cf5-l4wtk" Nov 03 13:32:15.055676 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:15.055661 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5wb4\" (UniqueName: \"kubernetes.io/projected/5e533eaa-a35c-475e-a88e-6dbf38f3f37e-kube-api-access-l5wb4\") pod \"payload-pre-processing-e2e-ipp-a-be8d0a-689bdb4cf5-l4wtk\" (UID: \"5e533eaa-a35c-475e-a88e-6dbf38f3f37e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be8d0a-689bdb4cf5-l4wtk" Nov 03 13:32:15.095028 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:15.094985 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-be8d0a\"" Nov 03 13:32:15.095439 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:15.095365 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-be8d0a-dockercfg-7klr8\"" Nov 03 13:32:15.191821 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:15.189963 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e533eaa-a35c-475e-a88e-6dbf38f3f37e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-be8d0a-689bdb4cf5-l4wtk\" (UID: \"5e533eaa-a35c-475e-a88e-6dbf38f3f37e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be8d0a-689bdb4cf5-l4wtk" Nov 03 13:32:15.191821 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:15.190237 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5wb4\" (UniqueName: \"kubernetes.io/projected/5e533eaa-a35c-475e-a88e-6dbf38f3f37e-kube-api-access-l5wb4\") pod \"payload-pre-processing-e2e-ipp-a-be8d0a-689bdb4cf5-l4wtk\" (UID: \"5e533eaa-a35c-475e-a88e-6dbf38f3f37e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be8d0a-689bdb4cf5-l4wtk" Nov 03 13:32:15.191821 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:15.191715 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e533eaa-a35c-475e-a88e-6dbf38f3f37e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-be8d0a-689bdb4cf5-l4wtk\" (UID: \"5e533eaa-a35c-475e-a88e-6dbf38f3f37e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be8d0a-689bdb4cf5-l4wtk" Nov 03 13:32:15.278542 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:15.276249 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5wb4\" (UniqueName: \"kubernetes.io/projected/5e533eaa-a35c-475e-a88e-6dbf38f3f37e-kube-api-access-l5wb4\") pod \"payload-pre-processing-e2e-ipp-a-be8d0a-689bdb4cf5-l4wtk\" (UID: \"5e533eaa-a35c-475e-a88e-6dbf38f3f37e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be8d0a-689bdb4cf5-l4wtk" Nov 03 13:32:15.278542 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:15.276323 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-be8d0a-5cd4fcdcbf-2kt7h"] Nov 03 13:32:15.283292 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:15.283262 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-be8d0a-5cd4fcdcbf-2kt7h" Nov 03 13:32:15.293303 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:15.293261 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-be8d0a-5cd4fcdcbf-2kt7h"] Nov 03 13:32:15.317368 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:15.317348 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be8d0a-689bdb4cf5-l4wtk" Nov 03 13:32:15.397166 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:15.396779 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43bab3d9-bb26-412c-b00d-3d980b572eeb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-be8d0a-5cd4fcdcbf-2kt7h\" (UID: \"43bab3d9-bb26-412c-b00d-3d980b572eeb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-be8d0a-5cd4fcdcbf-2kt7h" Nov 03 13:32:15.397166 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:15.396913 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnm46\" (UniqueName: \"kubernetes.io/projected/43bab3d9-bb26-412c-b00d-3d980b572eeb-kube-api-access-wnm46\") pod \"payload-processing-e2e-ipp-a-be8d0a-5cd4fcdcbf-2kt7h\" (UID: \"43bab3d9-bb26-412c-b00d-3d980b572eeb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-be8d0a-5cd4fcdcbf-2kt7h" Nov 03 13:32:15.498234 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:15.498200 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43bab3d9-bb26-412c-b00d-3d980b572eeb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-be8d0a-5cd4fcdcbf-2kt7h\" (UID: \"43bab3d9-bb26-412c-b00d-3d980b572eeb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-be8d0a-5cd4fcdcbf-2kt7h" Nov 03 13:32:15.498406 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:15.498272 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnm46\" (UniqueName: \"kubernetes.io/projected/43bab3d9-bb26-412c-b00d-3d980b572eeb-kube-api-access-wnm46\") pod \"payload-processing-e2e-ipp-a-be8d0a-5cd4fcdcbf-2kt7h\" (UID: \"43bab3d9-bb26-412c-b00d-3d980b572eeb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-be8d0a-5cd4fcdcbf-2kt7h" Nov 03 13:32:15.498910 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:15.498889 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43bab3d9-bb26-412c-b00d-3d980b572eeb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-be8d0a-5cd4fcdcbf-2kt7h\" (UID: \"43bab3d9-bb26-412c-b00d-3d980b572eeb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-be8d0a-5cd4fcdcbf-2kt7h" Nov 03 13:32:15.552523 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:15.552434 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnm46\" (UniqueName: \"kubernetes.io/projected/43bab3d9-bb26-412c-b00d-3d980b572eeb-kube-api-access-wnm46\") pod \"payload-processing-e2e-ipp-a-be8d0a-5cd4fcdcbf-2kt7h\" (UID: \"43bab3d9-bb26-412c-b00d-3d980b572eeb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-be8d0a-5cd4fcdcbf-2kt7h" Nov 03 13:32:15.624940 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:15.624905 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-be8d0a-5cd4fcdcbf-2kt7h" Nov 03 13:32:15.702812 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:15.702785 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-be8d0a-689bdb4cf5-l4wtk"] Nov 03 13:32:15.716647 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:32:15.716613 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e533eaa_a35c_475e_a88e_6dbf38f3f37e.slice/crio-1e8be6e5ff77acad1af3625b64416e977a956147e0337e1d7f729dbcaa395d93 WatchSource:0}: Error finding container 1e8be6e5ff77acad1af3625b64416e977a956147e0337e1d7f729dbcaa395d93: Status 404 returned error can't find the container with id 1e8be6e5ff77acad1af3625b64416e977a956147e0337e1d7f729dbcaa395d93 Nov 03 13:32:15.836578 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:15.836463 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be8d0a-689bdb4cf5-l4wtk" event={"ID":"5e533eaa-a35c-475e-a88e-6dbf38f3f37e","Type":"ContainerStarted","Data":"deb208ba8a8e33c6e94679d073c5c15deeefdb7693dfc8b1e0b9364ed5990db1"} Nov 03 13:32:15.836578 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:15.836544 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be8d0a-689bdb4cf5-l4wtk" event={"ID":"5e533eaa-a35c-475e-a88e-6dbf38f3f37e","Type":"ContainerStarted","Data":"1e8be6e5ff77acad1af3625b64416e977a956147e0337e1d7f729dbcaa395d93"} Nov 03 13:32:15.836799 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:15.836605 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be8d0a-689bdb4cf5-l4wtk" Nov 03 13:32:15.944971 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:15.944913 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be8d0a-689bdb4cf5-l4wtk" podStartSLOduration=1.944898537 podStartE2EDuration="1.944898537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:32:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:32:15.943383305 +0000 UTC m=+1617.954071550" watchObservedRunningTime="2026-11-03 13:32:15.944898537 +0000 UTC m=+1617.955586778" Nov 03 13:32:16.004401 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:16.004369 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-be8d0a-5cd4fcdcbf-2kt7h"] Nov 03 13:32:16.010464 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:32:16.010440 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43bab3d9_bb26_412c_b00d_3d980b572eeb.slice/crio-31fc49290e0096bb73dd5cfc83df269af8ecbc421a00c24e2181e514a6936169 WatchSource:0}: Error finding container 31fc49290e0096bb73dd5cfc83df269af8ecbc421a00c24e2181e514a6936169: Status 404 returned error can't find the container with id 31fc49290e0096bb73dd5cfc83df269af8ecbc421a00c24e2181e514a6936169 Nov 03 13:32:16.844916 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:16.844877 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-be8d0a-5cd4fcdcbf-2kt7h" event={"ID":"43bab3d9-bb26-412c-b00d-3d980b572eeb","Type":"ContainerStarted","Data":"db51c05f272c49422799cef406b5b23c1dc45f7f81ddb56e0ae46e115ffd3b6a"} Nov 03 13:32:16.845121 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:16.844931 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-be8d0a-5cd4fcdcbf-2kt7h" event={"ID":"43bab3d9-bb26-412c-b00d-3d980b572eeb","Type":"ContainerStarted","Data":"31fc49290e0096bb73dd5cfc83df269af8ecbc421a00c24e2181e514a6936169"} Nov 03 13:32:16.845180 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:16.845162 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-be8d0a-5cd4fcdcbf-2kt7h" Nov 03 13:32:16.955575 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:16.955522 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-be8d0a-5cd4fcdcbf-2kt7h" podStartSLOduration=1.955450625 podStartE2EDuration="1.955450625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:32:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:32:16.953716299 +0000 UTC m=+1618.964404569" watchObservedRunningTime="2026-11-03 13:32:16.955450625 +0000 UTC m=+1618.966138877" Nov 03 13:32:18.938900 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:18.938866 5979 scope.go:122] "RemoveContainer" containerID="48695bbce91e8b1303288143ad4dc1868cdb7e734e566d92b8401433ffa013e3" Nov 03 13:32:18.949364 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:18.949337 5979 scope.go:122] "RemoveContainer" containerID="0411fea64680cc5a963fe7383b8ff76160cca398c44c627598d8f77b76291621" Nov 03 13:32:18.959616 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:18.959589 5979 scope.go:122] "RemoveContainer" containerID="bae4a2d6284260a520d8bfa3ab90704ebfbf97859eeb5271321c8a0f6e85e2e8" Nov 03 13:32:18.969186 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:18.969167 5979 scope.go:122] "RemoveContainer" containerID="7b26f60f94453ea1e97ed5ae4e3343e50989071b14d629c22ab73c7b529e6254" Nov 03 13:32:18.977696 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:18.977679 5979 scope.go:122] "RemoveContainer" containerID="0aaad4592654f5b7575b9552852d8d6b7fe78830210dca27f0d9ca04247e1c02" Nov 03 13:32:18.986361 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:18.986346 5979 scope.go:122] "RemoveContainer" containerID="5a553281075023fd355f3b59b4919913e05d26e102701d0eb606a96bbd3e1b5b" Nov 03 13:32:25.117071 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.117036 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl"] Nov 03 13:32:25.122226 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.122206 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl" Nov 03 13:32:25.162669 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.162639 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z"] Nov 03 13:32:25.165894 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.165878 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z" Nov 03 13:32:25.180605 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.180586 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-5295ab29-openshift-default-dockercfg-h5sdt\"" Nov 03 13:32:25.197572 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.197548 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b109e1f4-308a-4557-ad21-bb8ae0d75553-istio-data\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z\" (UID: \"b109e1f4-308a-4557-ad21-bb8ae0d75553\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z" Nov 03 13:32:25.197700 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.197589 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b109e1f4-308a-4557-ad21-bb8ae0d75553-istio-token\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z\" (UID: \"b109e1f4-308a-4557-ad21-bb8ae0d75553\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z" Nov 03 13:32:25.197700 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.197629 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc325c76-815f-4a33-b5b6-71421ad64ded-istio-data\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl\" (UID: \"fc325c76-815f-4a33-b5b6-71421ad64ded\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl" Nov 03 13:32:25.197798 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.197719 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc325c76-815f-4a33-b5b6-71421ad64ded-istiod-ca-cert\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl\" (UID: \"fc325c76-815f-4a33-b5b6-71421ad64ded\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl" Nov 03 13:32:25.197798 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.197780 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc325c76-815f-4a33-b5b6-71421ad64ded-istio-podinfo\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl\" (UID: \"fc325c76-815f-4a33-b5b6-71421ad64ded\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl" Nov 03 13:32:25.197879 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.197811 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b109e1f4-308a-4557-ad21-bb8ae0d75553-istio-podinfo\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z\" (UID: \"b109e1f4-308a-4557-ad21-bb8ae0d75553\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z" Nov 03 13:32:25.197879 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.197833 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b109e1f4-308a-4557-ad21-bb8ae0d75553-istiod-ca-cert\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z\" (UID: \"b109e1f4-308a-4557-ad21-bb8ae0d75553\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z" Nov 03 13:32:25.197879 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.197849 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc325c76-815f-4a33-b5b6-71421ad64ded-workload-certs\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl\" (UID: \"fc325c76-815f-4a33-b5b6-71421ad64ded\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl" Nov 03 13:32:25.198002 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.197949 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b109e1f4-308a-4557-ad21-bb8ae0d75553-workload-socket\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z\" (UID: \"b109e1f4-308a-4557-ad21-bb8ae0d75553\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z" Nov 03 13:32:25.198002 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.197968 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b109e1f4-308a-4557-ad21-bb8ae0d75553-istio-envoy\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z\" (UID: \"b109e1f4-308a-4557-ad21-bb8ae0d75553\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z" Nov 03 13:32:25.198002 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.197988 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc325c76-815f-4a33-b5b6-71421ad64ded-istio-envoy\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl\" (UID: \"fc325c76-815f-4a33-b5b6-71421ad64ded\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl" Nov 03 13:32:25.198100 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.198036 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b109e1f4-308a-4557-ad21-bb8ae0d75553-credential-socket\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z\" (UID: \"b109e1f4-308a-4557-ad21-bb8ae0d75553\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z" Nov 03 13:32:25.198100 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.198052 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtt6z\" (UniqueName: \"kubernetes.io/projected/fc325c76-815f-4a33-b5b6-71421ad64ded-kube-api-access-jtt6z\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl\" (UID: \"fc325c76-815f-4a33-b5b6-71421ad64ded\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl" Nov 03 13:32:25.198166 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.198121 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b109e1f4-308a-4557-ad21-bb8ae0d75553-workload-certs\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z\" (UID: \"b109e1f4-308a-4557-ad21-bb8ae0d75553\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z" Nov 03 13:32:25.198166 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.198154 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc325c76-815f-4a33-b5b6-71421ad64ded-istio-token\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl\" (UID: \"fc325c76-815f-4a33-b5b6-71421ad64ded\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl" Nov 03 13:32:25.198229 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.198171 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kxx4\" (UniqueName: \"kubernetes.io/projected/b109e1f4-308a-4557-ad21-bb8ae0d75553-kube-api-access-5kxx4\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z\" (UID: \"b109e1f4-308a-4557-ad21-bb8ae0d75553\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z" Nov 03 13:32:25.198267 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.198235 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc325c76-815f-4a33-b5b6-71421ad64ded-credential-socket\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl\" (UID: \"fc325c76-815f-4a33-b5b6-71421ad64ded\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl" Nov 03 13:32:25.198312 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.198264 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc325c76-815f-4a33-b5b6-71421ad64ded-workload-socket\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl\" (UID: \"fc325c76-815f-4a33-b5b6-71421ad64ded\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl" Nov 03 13:32:25.210030 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.209993 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl"] Nov 03 13:32:25.261361 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.261318 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z"] Nov 03 13:32:25.299630 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.299592 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b109e1f4-308a-4557-ad21-bb8ae0d75553-workload-socket\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z\" (UID: \"b109e1f4-308a-4557-ad21-bb8ae0d75553\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z" Nov 03 13:32:25.299797 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.299643 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b109e1f4-308a-4557-ad21-bb8ae0d75553-istio-envoy\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z\" (UID: \"b109e1f4-308a-4557-ad21-bb8ae0d75553\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z" Nov 03 13:32:25.299797 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.299682 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc325c76-815f-4a33-b5b6-71421ad64ded-istio-envoy\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl\" (UID: \"fc325c76-815f-4a33-b5b6-71421ad64ded\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl" Nov 03 13:32:25.299797 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.299719 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b109e1f4-308a-4557-ad21-bb8ae0d75553-credential-socket\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z\" (UID: \"b109e1f4-308a-4557-ad21-bb8ae0d75553\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z" Nov 03 13:32:25.299797 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.299748 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtt6z\" (UniqueName: \"kubernetes.io/projected/fc325c76-815f-4a33-b5b6-71421ad64ded-kube-api-access-jtt6z\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl\" (UID: \"fc325c76-815f-4a33-b5b6-71421ad64ded\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl" Nov 03 13:32:25.300155 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.299837 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b109e1f4-308a-4557-ad21-bb8ae0d75553-workload-certs\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z\" (UID: \"b109e1f4-308a-4557-ad21-bb8ae0d75553\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z" Nov 03 13:32:25.300155 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.299983 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc325c76-815f-4a33-b5b6-71421ad64ded-istio-token\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl\" (UID: \"fc325c76-815f-4a33-b5b6-71421ad64ded\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl" Nov 03 13:32:25.300155 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.300017 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kxx4\" (UniqueName: \"kubernetes.io/projected/b109e1f4-308a-4557-ad21-bb8ae0d75553-kube-api-access-5kxx4\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z\" (UID: \"b109e1f4-308a-4557-ad21-bb8ae0d75553\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z" Nov 03 13:32:25.300155 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.300065 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc325c76-815f-4a33-b5b6-71421ad64ded-credential-socket\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl\" (UID: \"fc325c76-815f-4a33-b5b6-71421ad64ded\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl" Nov 03 13:32:25.300155 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.300099 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc325c76-815f-4a33-b5b6-71421ad64ded-workload-socket\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl\" (UID: \"fc325c76-815f-4a33-b5b6-71421ad64ded\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl" Nov 03 13:32:25.300155 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.300129 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b109e1f4-308a-4557-ad21-bb8ae0d75553-workload-socket\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z\" (UID: \"b109e1f4-308a-4557-ad21-bb8ae0d75553\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z" Nov 03 13:32:25.300543 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.300423 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b109e1f4-308a-4557-ad21-bb8ae0d75553-istio-data\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z\" (UID: \"b109e1f4-308a-4557-ad21-bb8ae0d75553\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z" Nov 03 13:32:25.300543 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.300451 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc325c76-815f-4a33-b5b6-71421ad64ded-credential-socket\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl\" (UID: \"fc325c76-815f-4a33-b5b6-71421ad64ded\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl" Nov 03 13:32:25.300543 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.300503 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b109e1f4-308a-4557-ad21-bb8ae0d75553-workload-certs\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z\" (UID: \"b109e1f4-308a-4557-ad21-bb8ae0d75553\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z" Nov 03 13:32:25.300543 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.300502 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b109e1f4-308a-4557-ad21-bb8ae0d75553-istio-token\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z\" (UID: \"b109e1f4-308a-4557-ad21-bb8ae0d75553\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z" Nov 03 13:32:25.300543 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.300517 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc325c76-815f-4a33-b5b6-71421ad64ded-workload-socket\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl\" (UID: \"fc325c76-815f-4a33-b5b6-71421ad64ded\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl" Nov 03 13:32:25.300935 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.300855 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b109e1f4-308a-4557-ad21-bb8ae0d75553-istio-data\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z\" (UID: \"b109e1f4-308a-4557-ad21-bb8ae0d75553\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z" Nov 03 13:32:25.301009 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.300960 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc325c76-815f-4a33-b5b6-71421ad64ded-istio-data\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl\" (UID: \"fc325c76-815f-4a33-b5b6-71421ad64ded\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl" Nov 03 13:32:25.301009 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.300996 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc325c76-815f-4a33-b5b6-71421ad64ded-istiod-ca-cert\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl\" (UID: \"fc325c76-815f-4a33-b5b6-71421ad64ded\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl" Nov 03 13:32:25.301147 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.301046 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc325c76-815f-4a33-b5b6-71421ad64ded-istio-podinfo\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl\" (UID: \"fc325c76-815f-4a33-b5b6-71421ad64ded\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl" Nov 03 13:32:25.301147 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.301103 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b109e1f4-308a-4557-ad21-bb8ae0d75553-istio-podinfo\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z\" (UID: \"b109e1f4-308a-4557-ad21-bb8ae0d75553\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z" Nov 03 13:32:25.301147 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.301141 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b109e1f4-308a-4557-ad21-bb8ae0d75553-istiod-ca-cert\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z\" (UID: \"b109e1f4-308a-4557-ad21-bb8ae0d75553\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z" Nov 03 13:32:25.301312 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.301171 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc325c76-815f-4a33-b5b6-71421ad64ded-workload-certs\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl\" (UID: \"fc325c76-815f-4a33-b5b6-71421ad64ded\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl" Nov 03 13:32:25.301312 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.301175 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b109e1f4-308a-4557-ad21-bb8ae0d75553-credential-socket\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z\" (UID: \"b109e1f4-308a-4557-ad21-bb8ae0d75553\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z" Nov 03 13:32:25.301557 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.301530 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc325c76-815f-4a33-b5b6-71421ad64ded-istio-data\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl\" (UID: \"fc325c76-815f-4a33-b5b6-71421ad64ded\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl" Nov 03 13:32:25.301961 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.301918 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc325c76-815f-4a33-b5b6-71421ad64ded-workload-certs\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl\" (UID: \"fc325c76-815f-4a33-b5b6-71421ad64ded\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl" Nov 03 13:32:25.302272 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.302248 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc325c76-815f-4a33-b5b6-71421ad64ded-istiod-ca-cert\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl\" (UID: \"fc325c76-815f-4a33-b5b6-71421ad64ded\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl" Nov 03 13:32:25.302362 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.302249 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b109e1f4-308a-4557-ad21-bb8ae0d75553-istiod-ca-cert\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z\" (UID: \"b109e1f4-308a-4557-ad21-bb8ae0d75553\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z" Nov 03 13:32:25.302858 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.302830 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc325c76-815f-4a33-b5b6-71421ad64ded-istio-envoy\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl\" (UID: \"fc325c76-815f-4a33-b5b6-71421ad64ded\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl" Nov 03 13:32:25.303779 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.303756 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b109e1f4-308a-4557-ad21-bb8ae0d75553-istio-envoy\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z\" (UID: \"b109e1f4-308a-4557-ad21-bb8ae0d75553\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z" Nov 03 13:32:25.304404 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.304381 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b109e1f4-308a-4557-ad21-bb8ae0d75553-istio-podinfo\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z\" (UID: \"b109e1f4-308a-4557-ad21-bb8ae0d75553\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z" Nov 03 13:32:25.304860 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.304838 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc325c76-815f-4a33-b5b6-71421ad64ded-istio-podinfo\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl\" (UID: \"fc325c76-815f-4a33-b5b6-71421ad64ded\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl" Nov 03 13:32:25.359115 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.359048 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtt6z\" (UniqueName: \"kubernetes.io/projected/fc325c76-815f-4a33-b5b6-71421ad64ded-kube-api-access-jtt6z\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl\" (UID: \"fc325c76-815f-4a33-b5b6-71421ad64ded\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl" Nov 03 13:32:25.359822 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.359800 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kxx4\" (UniqueName: \"kubernetes.io/projected/b109e1f4-308a-4557-ad21-bb8ae0d75553-kube-api-access-5kxx4\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z\" (UID: \"b109e1f4-308a-4557-ad21-bb8ae0d75553\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z" Nov 03 13:32:25.360571 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.360541 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc325c76-815f-4a33-b5b6-71421ad64ded-istio-token\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl\" (UID: \"fc325c76-815f-4a33-b5b6-71421ad64ded\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl" Nov 03 13:32:25.360671 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.360650 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b109e1f4-308a-4557-ad21-bb8ae0d75553-istio-token\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z\" (UID: \"b109e1f4-308a-4557-ad21-bb8ae0d75553\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z" Nov 03 13:32:25.435624 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.435595 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl" Nov 03 13:32:25.479519 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.479465 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z" Nov 03 13:32:25.873179 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.873091 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl"] Nov 03 13:32:25.917608 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:32:25.917574 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc325c76_815f_4a33_b5b6_71421ad64ded.slice/crio-c7d0ae7bce8b5873397fec1f92f117add9ed1a1b75d9c89a78306b6c8c0c7844 WatchSource:0}: Error finding container c7d0ae7bce8b5873397fec1f92f117add9ed1a1b75d9c89a78306b6c8c0c7844: Status 404 returned error can't find the container with id c7d0ae7bce8b5873397fec1f92f117add9ed1a1b75d9c89a78306b6c8c0c7844 Nov 03 13:32:25.920170 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.920137 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:32:25.920253 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.920207 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:32:25.920253 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.920237 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:32:25.928871 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.928838 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z"] Nov 03 13:32:25.934649 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:32:25.934624 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb109e1f4_308a_4557_ad21_bb8ae0d75553.slice/crio-fb70cb800393dcf2d4cc6454fc956812ad91c838297da2e232929cbd6ffd6f29 WatchSource:0}: Error finding container fb70cb800393dcf2d4cc6454fc956812ad91c838297da2e232929cbd6ffd6f29: Status 404 returned error can't find the container with id fb70cb800393dcf2d4cc6454fc956812ad91c838297da2e232929cbd6ffd6f29 Nov 03 13:32:25.937369 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.937333 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:32:25.937442 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.937420 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:32:25.937519 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.937501 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:32:25.945754 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.945731 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl" event={"ID":"fc325c76-815f-4a33-b5b6-71421ad64ded","Type":"ContainerStarted","Data":"c7d0ae7bce8b5873397fec1f92f117add9ed1a1b75d9c89a78306b6c8c0c7844"} Nov 03 13:32:25.947064 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:25.947040 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z" event={"ID":"b109e1f4-308a-4557-ad21-bb8ae0d75553","Type":"ContainerStarted","Data":"fb70cb800393dcf2d4cc6454fc956812ad91c838297da2e232929cbd6ffd6f29"} Nov 03 13:32:26.880696 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:26.880656 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be8d0a-689bdb4cf5-l4wtk" Nov 03 13:32:26.999203 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:26.999164 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z" event={"ID":"b109e1f4-308a-4557-ad21-bb8ae0d75553","Type":"ContainerStarted","Data":"6b1c04d6646625b809d10de2d313efd013416a8692895d9bd18da25323a49696"} Nov 03 13:32:27.003589 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:27.003533 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl" event={"ID":"fc325c76-815f-4a33-b5b6-71421ad64ded","Type":"ContainerStarted","Data":"178ac86c8cbdd80437a8bf478eb86f1ad0e1e3816991f8b3ccf0ef1eaf3749f1"} Nov 03 13:32:27.120750 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:27.120700 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z" podStartSLOduration=2.120685553 podStartE2EDuration="2.120685553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:32:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:32:27.116135097 +0000 UTC m=+1629.126823342" watchObservedRunningTime="2026-11-03 13:32:27.120685553 +0000 UTC m=+1629.131373795" Nov 03 13:32:27.235531 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:27.235461 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl" podStartSLOduration=2.235447935 podStartE2EDuration="2.235447935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:32:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:32:27.23236491 +0000 UTC m=+1629.243053154" watchObservedRunningTime="2026-11-03 13:32:27.235447935 +0000 UTC m=+1629.246136176" Nov 03 13:32:27.441341 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:27.440860 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl" Nov 03 13:32:27.441341 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:27.440989 5979 patch_prober.go:29] interesting pod/e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 03 13:32:27.441341 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:27.441034 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl" podUID="fc325c76-815f-4a33-b5b6-71421ad64ded" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 03 13:32:27.481495 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:27.480851 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z" Nov 03 13:32:27.489552 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:27.485830 5979 patch_prober.go:29] interesting pod/e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 03 13:32:27.489552 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:27.485893 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z" podUID="b109e1f4-308a-4557-ad21-bb8ae0d75553" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 03 13:32:27.907524 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:27.907117 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb"] Nov 03 13:32:27.907524 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:27.907348 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb" podUID="e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5" containerName="istio-proxy" containerID="cri-o://d50528b2a85504ba67be25c95423ac67172534ed201e3edef2d05eba08509966" gracePeriod=30 Nov 03 13:32:27.908155 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:27.907992 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-be8d0a-5cd4fcdcbf-2kt7h" Nov 03 13:32:28.436584 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:28.436552 5979 patch_prober.go:29] interesting pod/e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 03 13:32:28.436784 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:28.436616 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl" podUID="fc325c76-815f-4a33-b5b6-71421ad64ded" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 03 13:32:28.489496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:28.488842 5979 patch_prober.go:29] interesting pod/e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 03 13:32:28.489496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:28.488920 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z" podUID="b109e1f4-308a-4557-ad21-bb8ae0d75553" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 03 13:32:29.143361 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:29.143304 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb"] Nov 03 13:32:29.143915 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:29.143628 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb" podUID="cfd3eded-c3de-4d6b-9a1e-c284549c6c18" containerName="istio-proxy" containerID="cri-o://405a543b62ebb99ccd78f604fd577bd100d66d8b417afb35d33618fca28d407a" gracePeriod=30 Nov 03 13:32:29.170974 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:29.169364 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7"] Nov 03 13:32:29.198632 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:29.188616 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn"] Nov 03 13:32:29.198632 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:29.188851 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn" podUID="f156e9bb-8f04-4805-b8ff-ca1561fb598c" containerName="istio-proxy" containerID="cri-o://bef503c247cd6c60e93a5a66dc1ce2f2cf2aee3c879196084c9f5277d53556f1" gracePeriod=30 Nov 03 13:32:29.198632 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:29.197781 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7" podUID="42d9f732-f0cc-4e09-8ef5-cce6fc157dad" containerName="istio-proxy" containerID="cri-o://ab52e4bdbd49931b70d29485333cc0e955891ea8c6700ec49000ecb8f3208308" gracePeriod=30 Nov 03 13:32:29.204436 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:29.204404 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7"] Nov 03 13:32:29.204638 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:29.204614 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7" podUID="1f4a018c-c28f-4a81-b727-e276ae0156d3" containerName="istio-proxy" containerID="cri-o://9edcf518b5926b6444322e10b9d784c097113d24911329ae56638bd93d64f5e2" gracePeriod=30 Nov 03 13:32:29.436791 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:29.436713 5979 patch_prober.go:29] interesting pod/e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 03 13:32:29.436791 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:29.436760 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl" podUID="fc325c76-815f-4a33-b5b6-71421ad64ded" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 03 13:32:29.484496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:29.483915 5979 patch_prober.go:29] interesting pod/e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 03 13:32:29.484496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:29.483980 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z" podUID="b109e1f4-308a-4557-ad21-bb8ae0d75553" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 03 13:32:30.435849 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:30.435811 5979 patch_prober.go:29] interesting pod/e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 03 13:32:30.436249 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:30.435861 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl" podUID="fc325c76-815f-4a33-b5b6-71421ad64ded" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 03 13:32:30.486496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:30.483913 5979 patch_prober.go:29] interesting pod/e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 03 13:32:30.486496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:30.483979 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z" podUID="b109e1f4-308a-4557-ad21-bb8ae0d75553" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 03 13:32:31.033112 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:31.033072 5979 patch_prober.go:29] interesting pod/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 13:32:31.033288 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:31.033126 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb" podUID="e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 13:32:31.436305 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:31.436271 5979 patch_prober.go:29] interesting pod/e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Nov 03 13:32:31.436724 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:31.436323 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl" podUID="fc325c76-815f-4a33-b5b6-71421ad64ded" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Nov 03 13:32:31.481496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:31.480933 5979 patch_prober.go:29] interesting pod/e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 03 13:32:31.481496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:31.481010 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z" podUID="b109e1f4-308a-4557-ad21-bb8ae0d75553" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 03 13:32:31.493496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:31.492203 5979 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 13:32:31.493496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:31.492262 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn" podUID="f156e9bb-8f04-4805-b8ff-ca1561fb598c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 13:32:32.480427 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:32.480378 5979 patch_prober.go:29] interesting pod/e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" start-of-body= Nov 03 13:32:32.480921 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:32.480454 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z" podUID="b109e1f4-308a-4557-ad21-bb8ae0d75553" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.156:15021/healthz/ready\": dial tcp 10.134.0.156:15021: connect: connection refused" Nov 03 13:32:32.650853 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:32.650817 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl" Nov 03 13:32:32.651696 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:32.651672 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl" Nov 03 13:32:32.653017 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:32.652989 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl" Nov 03 13:32:33.130457 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:33.130406 5979 generic.go:356] "Generic (PLEG): container finished" podID="e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5" containerID="d50528b2a85504ba67be25c95423ac67172534ed201e3edef2d05eba08509966" exitCode=0 Nov 03 13:32:33.130687 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:33.130493 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb" event={"ID":"e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5","Type":"ContainerDied","Data":"d50528b2a85504ba67be25c95423ac67172534ed201e3edef2d05eba08509966"} Nov 03 13:32:33.130687 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:33.130586 5979 scope.go:122] "RemoveContainer" containerID="c015efe631d4b0927cfa8378487110d1a80d7128c49f1c5bd103219ffa4cdb3c" Nov 03 13:32:33.191664 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:33.191608 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb" Nov 03 13:32:33.222360 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:33.222326 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-istiod-ca-cert\") pod \"e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5\" (UID: \"e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5\") " Nov 03 13:32:33.222543 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:33.222372 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-istio-envoy\") pod \"e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5\" (UID: \"e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5\") " Nov 03 13:32:33.222543 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:33.222416 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-workload-certs\") pod \"e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5\" (UID: \"e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5\") " Nov 03 13:32:33.222543 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:33.222449 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-workload-socket\") pod \"e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5\" (UID: \"e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5\") " Nov 03 13:32:33.222716 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:33.222565 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-credential-socket\") pod \"e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5\" (UID: \"e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5\") " Nov 03 13:32:33.222797 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:33.222763 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-workload-certs" pod "e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5" (UID: "e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:32:33.222797 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:33.222775 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-istiod-ca-cert" pod "e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5" (UID: "e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:32:33.222922 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:33.222795 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-istio-token\" (UniqueName: \"kubernetes.io/projected/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-istio-token\") pod \"e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5\" (UID: \"e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5\") " Nov 03 13:32:33.222922 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:33.222866 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-istio-podinfo\") pod \"e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5\" (UID: \"e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5\") " Nov 03 13:32:33.222922 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:33.222868 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-credential-socket" pod "e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5" (UID: "e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:32:33.222922 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:33.222877 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-workload-socket" pod "e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5" (UID: "e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:32:33.222922 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:33.222909 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-istio-data\") pod \"e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5\" (UID: \"e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5\") " Nov 03 13:32:33.223170 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:33.222957 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-kube-api-access-qp8xn\" (UniqueName: \"kubernetes.io/projected/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-kube-api-access-qp8xn\") pod \"e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5\" (UID: \"e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5\") " Nov 03 13:32:33.223170 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:33.223133 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-istio-data" pod "e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5" (UID: "e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:32:33.223921 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:33.223884 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:33.224035 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:33.223925 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:33.224035 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:33.223941 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:33.224035 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:33.223954 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:33.224035 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:33.223968 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:33.225256 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:33.225223 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-istio-envoy" pod "e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5" (UID: "e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:32:33.225453 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:33.225428 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-istio-token" pod "e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5" (UID: "e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:32:33.225593 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:33.225430 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-istio-podinfo" pod "e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5" (UID: "e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:32:33.225637 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:33.225602 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-kube-api-access-qp8xn" pod "e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5" (UID: "e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5"). InnerVolumeSpecName "kube-api-access-qp8xn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:32:33.325483 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:33.325435 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:33.325483 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:33.325487 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:33.325713 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:33.325505 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp8xn\" (UniqueName: \"kubernetes.io/projected/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-kube-api-access-qp8xn\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:33.325713 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:33.325522 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:33.484166 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:33.484095 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z" Nov 03 13:32:33.484566 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:33.484522 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z" Nov 03 13:32:33.485075 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:33.485058 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z" Nov 03 13:32:34.138004 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.137979 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb" Nov 03 13:32:34.138150 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.138028 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb" event={"ID":"e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5","Type":"ContainerDied","Data":"799d7f0dbfe8a95fce53c58400d1be47766e2931f875a6c1fb1366c1658fcaa5"} Nov 03 13:32:34.138150 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.138081 5979 scope.go:122] "RemoveContainer" containerID="d50528b2a85504ba67be25c95423ac67172534ed201e3edef2d05eba08509966" Nov 03 13:32:34.297007 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.296937 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb"] Nov 03 13:32:34.307031 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.307001 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-1fdd09-openshift-default-dbf7c4cdf-fgtxb"] Nov 03 13:32:34.401016 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.400896 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb" Nov 03 13:32:34.446214 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.446169 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-istio-envoy\") pod \"cfd3eded-c3de-4d6b-9a1e-c284549c6c18\" (UID: \"cfd3eded-c3de-4d6b-9a1e-c284549c6c18\") " Nov 03 13:32:34.446393 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.446254 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-credential-socket\") pod \"cfd3eded-c3de-4d6b-9a1e-c284549c6c18\" (UID: \"cfd3eded-c3de-4d6b-9a1e-c284549c6c18\") " Nov 03 13:32:34.446393 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.446346 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-istiod-ca-cert\") pod \"cfd3eded-c3de-4d6b-9a1e-c284549c6c18\" (UID: \"cfd3eded-c3de-4d6b-9a1e-c284549c6c18\") " Nov 03 13:32:34.446568 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.446480 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-istio-token\" (UniqueName: \"kubernetes.io/projected/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-istio-token\") pod \"cfd3eded-c3de-4d6b-9a1e-c284549c6c18\" (UID: \"cfd3eded-c3de-4d6b-9a1e-c284549c6c18\") " Nov 03 13:32:34.446568 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.446517 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-istio-data\") pod \"cfd3eded-c3de-4d6b-9a1e-c284549c6c18\" (UID: \"cfd3eded-c3de-4d6b-9a1e-c284549c6c18\") " Nov 03 13:32:34.446685 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.446610 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-workload-socket\") pod \"cfd3eded-c3de-4d6b-9a1e-c284549c6c18\" (UID: \"cfd3eded-c3de-4d6b-9a1e-c284549c6c18\") " Nov 03 13:32:34.446685 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.446641 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-istio-podinfo\") pod \"cfd3eded-c3de-4d6b-9a1e-c284549c6c18\" (UID: \"cfd3eded-c3de-4d6b-9a1e-c284549c6c18\") " Nov 03 13:32:34.446794 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.446735 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-workload-certs\") pod \"cfd3eded-c3de-4d6b-9a1e-c284549c6c18\" (UID: \"cfd3eded-c3de-4d6b-9a1e-c284549c6c18\") " Nov 03 13:32:34.446794 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.446770 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-kube-api-access-d7vql\" (UniqueName: \"kubernetes.io/projected/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-kube-api-access-d7vql\") pod \"cfd3eded-c3de-4d6b-9a1e-c284549c6c18\" (UID: \"cfd3eded-c3de-4d6b-9a1e-c284549c6c18\") " Nov 03 13:32:34.448498 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.447517 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-workload-socket" pod "cfd3eded-c3de-4d6b-9a1e-c284549c6c18" (UID: "cfd3eded-c3de-4d6b-9a1e-c284549c6c18"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:32:34.448498 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.448194 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-workload-certs" pod "cfd3eded-c3de-4d6b-9a1e-c284549c6c18" (UID: "cfd3eded-c3de-4d6b-9a1e-c284549c6c18"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:32:34.448498 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.448352 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-istiod-ca-cert" pod "cfd3eded-c3de-4d6b-9a1e-c284549c6c18" (UID: "cfd3eded-c3de-4d6b-9a1e-c284549c6c18"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:32:34.448498 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.448407 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-istio-data" pod "cfd3eded-c3de-4d6b-9a1e-c284549c6c18" (UID: "cfd3eded-c3de-4d6b-9a1e-c284549c6c18"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:32:34.448498 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.448457 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-credential-socket" pod "cfd3eded-c3de-4d6b-9a1e-c284549c6c18" (UID: "cfd3eded-c3de-4d6b-9a1e-c284549c6c18"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:32:34.450325 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.450293 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-istio-podinfo" pod "cfd3eded-c3de-4d6b-9a1e-c284549c6c18" (UID: "cfd3eded-c3de-4d6b-9a1e-c284549c6c18"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:32:34.454761 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.454734 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-istio-token" pod "cfd3eded-c3de-4d6b-9a1e-c284549c6c18" (UID: "cfd3eded-c3de-4d6b-9a1e-c284549c6c18"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:32:34.454868 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.454843 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-istio-envoy" pod "cfd3eded-c3de-4d6b-9a1e-c284549c6c18" (UID: "cfd3eded-c3de-4d6b-9a1e-c284549c6c18"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:32:34.479823 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.479788 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-kube-api-access-d7vql" pod "cfd3eded-c3de-4d6b-9a1e-c284549c6c18" (UID: "cfd3eded-c3de-4d6b-9a1e-c284549c6c18"). InnerVolumeSpecName "kube-api-access-d7vql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:32:34.507629 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.507596 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7" Nov 03 13:32:34.558713 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.549654 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn" Nov 03 13:32:34.558713 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.550133 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-istio-token\" (UniqueName: \"kubernetes.io/projected/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-istio-token\") pod \"42d9f732-f0cc-4e09-8ef5-cce6fc157dad\" (UID: \"42d9f732-f0cc-4e09-8ef5-cce6fc157dad\") " Nov 03 13:32:34.558713 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.550187 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-istio-podinfo\") pod \"42d9f732-f0cc-4e09-8ef5-cce6fc157dad\" (UID: \"42d9f732-f0cc-4e09-8ef5-cce6fc157dad\") " Nov 03 13:32:34.558713 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.550228 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-workload-certs\") pod \"42d9f732-f0cc-4e09-8ef5-cce6fc157dad\" (UID: \"42d9f732-f0cc-4e09-8ef5-cce6fc157dad\") " Nov 03 13:32:34.558713 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.550295 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-istiod-ca-cert\") pod \"42d9f732-f0cc-4e09-8ef5-cce6fc157dad\" (UID: \"42d9f732-f0cc-4e09-8ef5-cce6fc157dad\") " Nov 03 13:32:34.558713 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.550395 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-workload-socket\") pod \"42d9f732-f0cc-4e09-8ef5-cce6fc157dad\" (UID: \"42d9f732-f0cc-4e09-8ef5-cce6fc157dad\") " Nov 03 13:32:34.558713 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.550493 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-credential-socket\") pod \"42d9f732-f0cc-4e09-8ef5-cce6fc157dad\" (UID: \"42d9f732-f0cc-4e09-8ef5-cce6fc157dad\") " Nov 03 13:32:34.558713 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.550591 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-istio-envoy\") pod \"42d9f732-f0cc-4e09-8ef5-cce6fc157dad\" (UID: \"42d9f732-f0cc-4e09-8ef5-cce6fc157dad\") " Nov 03 13:32:34.558713 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.550721 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-kube-api-access-jqkrg\" (UniqueName: \"kubernetes.io/projected/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-kube-api-access-jqkrg\") pod \"42d9f732-f0cc-4e09-8ef5-cce6fc157dad\" (UID: \"42d9f732-f0cc-4e09-8ef5-cce6fc157dad\") " Nov 03 13:32:34.558713 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.550759 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-istio-data\") pod \"42d9f732-f0cc-4e09-8ef5-cce6fc157dad\" (UID: \"42d9f732-f0cc-4e09-8ef5-cce6fc157dad\") " Nov 03 13:32:34.558713 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.551172 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-workload-socket" pod "42d9f732-f0cc-4e09-8ef5-cce6fc157dad" (UID: "42d9f732-f0cc-4e09-8ef5-cce6fc157dad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:32:34.558713 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.553299 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-istio-token" pod "42d9f732-f0cc-4e09-8ef5-cce6fc157dad" (UID: "42d9f732-f0cc-4e09-8ef5-cce6fc157dad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:32:34.558713 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.553573 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-workload-certs" pod "42d9f732-f0cc-4e09-8ef5-cce6fc157dad" (UID: "42d9f732-f0cc-4e09-8ef5-cce6fc157dad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:32:34.558713 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.557977 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-istio-data" pod "42d9f732-f0cc-4e09-8ef5-cce6fc157dad" (UID: "42d9f732-f0cc-4e09-8ef5-cce6fc157dad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:32:34.558713 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.558288 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:34.558713 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.558306 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:34.558713 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.558321 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:34.558713 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.558337 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:34.558713 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.558351 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:34.558713 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.558365 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:34.558713 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.558379 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:34.558713 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.558394 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:34.558713 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.558407 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:34.558713 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.558420 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:34.558713 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.558434 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:34.558713 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.558447 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:34.558713 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.558461 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7vql\" (UniqueName: \"kubernetes.io/projected/cfd3eded-c3de-4d6b-9a1e-c284549c6c18-kube-api-access-d7vql\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:34.558713 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.558666 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-credential-socket" pod "42d9f732-f0cc-4e09-8ef5-cce6fc157dad" (UID: "42d9f732-f0cc-4e09-8ef5-cce6fc157dad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:32:34.560292 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.558819 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-istio-envoy" pod "42d9f732-f0cc-4e09-8ef5-cce6fc157dad" (UID: "42d9f732-f0cc-4e09-8ef5-cce6fc157dad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:32:34.560292 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.558985 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-istiod-ca-cert" pod "42d9f732-f0cc-4e09-8ef5-cce6fc157dad" (UID: "42d9f732-f0cc-4e09-8ef5-cce6fc157dad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:32:34.561584 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.561555 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-istio-podinfo" pod "42d9f732-f0cc-4e09-8ef5-cce6fc157dad" (UID: "42d9f732-f0cc-4e09-8ef5-cce6fc157dad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:32:34.565831 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.565803 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-kube-api-access-jqkrg" pod "42d9f732-f0cc-4e09-8ef5-cce6fc157dad" (UID: "42d9f732-f0cc-4e09-8ef5-cce6fc157dad"). InnerVolumeSpecName "kube-api-access-jqkrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:32:34.590125 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.590096 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7" Nov 03 13:32:34.660332 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.660297 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f156e9bb-8f04-4805-b8ff-ca1561fb598c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f156e9bb-8f04-4805-b8ff-ca1561fb598c-istio-data\") pod \"f156e9bb-8f04-4805-b8ff-ca1561fb598c\" (UID: \"f156e9bb-8f04-4805-b8ff-ca1561fb598c\") " Nov 03 13:32:34.660609 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.660362 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f4a018c-c28f-4a81-b727-e276ae0156d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f4a018c-c28f-4a81-b727-e276ae0156d3-workload-socket\") pod \"1f4a018c-c28f-4a81-b727-e276ae0156d3\" (UID: \"1f4a018c-c28f-4a81-b727-e276ae0156d3\") " Nov 03 13:32:34.660609 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.660421 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f4a018c-c28f-4a81-b727-e276ae0156d3-kube-api-access-9g74z\" (UniqueName: \"kubernetes.io/projected/1f4a018c-c28f-4a81-b727-e276ae0156d3-kube-api-access-9g74z\") pod \"1f4a018c-c28f-4a81-b727-e276ae0156d3\" (UID: \"1f4a018c-c28f-4a81-b727-e276ae0156d3\") " Nov 03 13:32:34.660609 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.660464 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f156e9bb-8f04-4805-b8ff-ca1561fb598c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f156e9bb-8f04-4805-b8ff-ca1561fb598c-workload-certs\") pod \"f156e9bb-8f04-4805-b8ff-ca1561fb598c\" (UID: \"f156e9bb-8f04-4805-b8ff-ca1561fb598c\") " Nov 03 13:32:34.660609 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.660558 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f4a018c-c28f-4a81-b727-e276ae0156d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f4a018c-c28f-4a81-b727-e276ae0156d3-istio-data\") pod \"1f4a018c-c28f-4a81-b727-e276ae0156d3\" (UID: \"1f4a018c-c28f-4a81-b727-e276ae0156d3\") " Nov 03 13:32:34.660876 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.660644 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f156e9bb-8f04-4805-b8ff-ca1561fb598c-istio-data" pod "f156e9bb-8f04-4805-b8ff-ca1561fb598c" (UID: "f156e9bb-8f04-4805-b8ff-ca1561fb598c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:32:34.662078 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.661992 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f4a018c-c28f-4a81-b727-e276ae0156d3-workload-socket" pod "1f4a018c-c28f-4a81-b727-e276ae0156d3" (UID: "1f4a018c-c28f-4a81-b727-e276ae0156d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:32:34.662751 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.662428 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f4a018c-c28f-4a81-b727-e276ae0156d3-istio-data" pod "1f4a018c-c28f-4a81-b727-e276ae0156d3" (UID: "1f4a018c-c28f-4a81-b727-e276ae0156d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:32:34.663135 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.663101 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f156e9bb-8f04-4805-b8ff-ca1561fb598c-workload-certs" pod "f156e9bb-8f04-4805-b8ff-ca1561fb598c" (UID: "f156e9bb-8f04-4805-b8ff-ca1561fb598c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:32:34.663986 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.663967 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f4a018c-c28f-4a81-b727-e276ae0156d3-istio-token\" (UniqueName: \"kubernetes.io/projected/1f4a018c-c28f-4a81-b727-e276ae0156d3-istio-token\") pod \"1f4a018c-c28f-4a81-b727-e276ae0156d3\" (UID: \"1f4a018c-c28f-4a81-b727-e276ae0156d3\") " Nov 03 13:32:34.664185 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.664135 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f4a018c-c28f-4a81-b727-e276ae0156d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f4a018c-c28f-4a81-b727-e276ae0156d3-workload-certs\") pod \"1f4a018c-c28f-4a81-b727-e276ae0156d3\" (UID: \"1f4a018c-c28f-4a81-b727-e276ae0156d3\") " Nov 03 13:32:34.664361 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.664189 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f4a018c-c28f-4a81-b727-e276ae0156d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f4a018c-c28f-4a81-b727-e276ae0156d3-credential-socket\") pod \"1f4a018c-c28f-4a81-b727-e276ae0156d3\" (UID: \"1f4a018c-c28f-4a81-b727-e276ae0156d3\") " Nov 03 13:32:34.664361 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.664276 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f156e9bb-8f04-4805-b8ff-ca1561fb598c-istio-token\" (UniqueName: \"kubernetes.io/projected/f156e9bb-8f04-4805-b8ff-ca1561fb598c-istio-token\") pod \"f156e9bb-8f04-4805-b8ff-ca1561fb598c\" (UID: \"f156e9bb-8f04-4805-b8ff-ca1561fb598c\") " Nov 03 13:32:34.664361 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.664341 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f156e9bb-8f04-4805-b8ff-ca1561fb598c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f156e9bb-8f04-4805-b8ff-ca1561fb598c-istiod-ca-cert\") pod \"f156e9bb-8f04-4805-b8ff-ca1561fb598c\" (UID: \"f156e9bb-8f04-4805-b8ff-ca1561fb598c\") " Nov 03 13:32:34.664710 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.664378 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f4a018c-c28f-4a81-b727-e276ae0156d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f4a018c-c28f-4a81-b727-e276ae0156d3-istiod-ca-cert\") pod \"1f4a018c-c28f-4a81-b727-e276ae0156d3\" (UID: \"1f4a018c-c28f-4a81-b727-e276ae0156d3\") " Nov 03 13:32:34.664710 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.664417 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f4a018c-c28f-4a81-b727-e276ae0156d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f4a018c-c28f-4a81-b727-e276ae0156d3-istio-podinfo\") pod \"1f4a018c-c28f-4a81-b727-e276ae0156d3\" (UID: \"1f4a018c-c28f-4a81-b727-e276ae0156d3\") " Nov 03 13:32:34.664710 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.664466 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f156e9bb-8f04-4805-b8ff-ca1561fb598c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f156e9bb-8f04-4805-b8ff-ca1561fb598c-credential-socket\") pod \"f156e9bb-8f04-4805-b8ff-ca1561fb598c\" (UID: \"f156e9bb-8f04-4805-b8ff-ca1561fb598c\") " Nov 03 13:32:34.664710 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.664507 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f156e9bb-8f04-4805-b8ff-ca1561fb598c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f156e9bb-8f04-4805-b8ff-ca1561fb598c-istio-podinfo\") pod \"f156e9bb-8f04-4805-b8ff-ca1561fb598c\" (UID: \"f156e9bb-8f04-4805-b8ff-ca1561fb598c\") " Nov 03 13:32:34.664710 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.664568 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f156e9bb-8f04-4805-b8ff-ca1561fb598c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f156e9bb-8f04-4805-b8ff-ca1561fb598c-istio-envoy\") pod \"f156e9bb-8f04-4805-b8ff-ca1561fb598c\" (UID: \"f156e9bb-8f04-4805-b8ff-ca1561fb598c\") " Nov 03 13:32:34.664710 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.664662 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f156e9bb-8f04-4805-b8ff-ca1561fb598c-kube-api-access-r9zpd\" (UniqueName: \"kubernetes.io/projected/f156e9bb-8f04-4805-b8ff-ca1561fb598c-kube-api-access-r9zpd\") pod \"f156e9bb-8f04-4805-b8ff-ca1561fb598c\" (UID: \"f156e9bb-8f04-4805-b8ff-ca1561fb598c\") " Nov 03 13:32:34.664710 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.664698 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f156e9bb-8f04-4805-b8ff-ca1561fb598c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f156e9bb-8f04-4805-b8ff-ca1561fb598c-workload-socket\") pod \"f156e9bb-8f04-4805-b8ff-ca1561fb598c\" (UID: \"f156e9bb-8f04-4805-b8ff-ca1561fb598c\") " Nov 03 13:32:34.670676 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.664791 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f4a018c-c28f-4a81-b727-e276ae0156d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f4a018c-c28f-4a81-b727-e276ae0156d3-istio-envoy\") pod \"1f4a018c-c28f-4a81-b727-e276ae0156d3\" (UID: \"1f4a018c-c28f-4a81-b727-e276ae0156d3\") " Nov 03 13:32:34.670676 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.665745 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:34.670676 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.665767 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f156e9bb-8f04-4805-b8ff-ca1561fb598c-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:34.670676 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.665781 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f4a018c-c28f-4a81-b727-e276ae0156d3-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:34.670676 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.665795 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f156e9bb-8f04-4805-b8ff-ca1561fb598c-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:34.670676 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.665808 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:34.670676 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.665822 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f4a018c-c28f-4a81-b727-e276ae0156d3-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:34.670676 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.665836 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqkrg\" (UniqueName: \"kubernetes.io/projected/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-kube-api-access-jqkrg\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:34.670676 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.665851 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:34.670676 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.665864 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42d9f732-f0cc-4e09-8ef5-cce6fc157dad-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:34.670676 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.666443 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f4a018c-c28f-4a81-b727-e276ae0156d3-credential-socket" pod "1f4a018c-c28f-4a81-b727-e276ae0156d3" (UID: "1f4a018c-c28f-4a81-b727-e276ae0156d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:32:34.670676 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.668754 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f4a018c-c28f-4a81-b727-e276ae0156d3-workload-certs" pod "1f4a018c-c28f-4a81-b727-e276ae0156d3" (UID: "1f4a018c-c28f-4a81-b727-e276ae0156d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:32:34.670676 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.669083 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f4a018c-c28f-4a81-b727-e276ae0156d3-istiod-ca-cert" pod "1f4a018c-c28f-4a81-b727-e276ae0156d3" (UID: "1f4a018c-c28f-4a81-b727-e276ae0156d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:32:34.670676 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.669696 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f156e9bb-8f04-4805-b8ff-ca1561fb598c-istiod-ca-cert" pod "f156e9bb-8f04-4805-b8ff-ca1561fb598c" (UID: "f156e9bb-8f04-4805-b8ff-ca1561fb598c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:32:34.670676 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.669964 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f156e9bb-8f04-4805-b8ff-ca1561fb598c-credential-socket" pod "f156e9bb-8f04-4805-b8ff-ca1561fb598c" (UID: "f156e9bb-8f04-4805-b8ff-ca1561fb598c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:32:34.670676 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.670125 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f156e9bb-8f04-4805-b8ff-ca1561fb598c-workload-socket" pod "f156e9bb-8f04-4805-b8ff-ca1561fb598c" (UID: "f156e9bb-8f04-4805-b8ff-ca1561fb598c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:32:34.673053 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.673030 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f4a018c-c28f-4a81-b727-e276ae0156d3-istio-podinfo" pod "1f4a018c-c28f-4a81-b727-e276ae0156d3" (UID: "1f4a018c-c28f-4a81-b727-e276ae0156d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:32:34.673325 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.673304 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f156e9bb-8f04-4805-b8ff-ca1561fb598c-istio-podinfo" pod "f156e9bb-8f04-4805-b8ff-ca1561fb598c" (UID: "f156e9bb-8f04-4805-b8ff-ca1561fb598c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:32:34.673910 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.673889 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f156e9bb-8f04-4805-b8ff-ca1561fb598c-istio-envoy" pod "f156e9bb-8f04-4805-b8ff-ca1561fb598c" (UID: "f156e9bb-8f04-4805-b8ff-ca1561fb598c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:32:34.674019 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.673904 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f4a018c-c28f-4a81-b727-e276ae0156d3-istio-envoy" pod "1f4a018c-c28f-4a81-b727-e276ae0156d3" (UID: "1f4a018c-c28f-4a81-b727-e276ae0156d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:32:34.674087 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.674026 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f4a018c-c28f-4a81-b727-e276ae0156d3-istio-token" pod "1f4a018c-c28f-4a81-b727-e276ae0156d3" (UID: "1f4a018c-c28f-4a81-b727-e276ae0156d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:32:34.674087 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.674056 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f156e9bb-8f04-4805-b8ff-ca1561fb598c-istio-token" pod "f156e9bb-8f04-4805-b8ff-ca1561fb598c" (UID: "f156e9bb-8f04-4805-b8ff-ca1561fb598c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:32:34.675460 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.675439 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f4a018c-c28f-4a81-b727-e276ae0156d3-kube-api-access-9g74z" pod "1f4a018c-c28f-4a81-b727-e276ae0156d3" (UID: "1f4a018c-c28f-4a81-b727-e276ae0156d3"). InnerVolumeSpecName "kube-api-access-9g74z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:32:34.675661 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.675642 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f156e9bb-8f04-4805-b8ff-ca1561fb598c-kube-api-access-r9zpd" pod "f156e9bb-8f04-4805-b8ff-ca1561fb598c" (UID: "f156e9bb-8f04-4805-b8ff-ca1561fb598c"). InnerVolumeSpecName "kube-api-access-r9zpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:32:34.779660 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.777746 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f4a018c-c28f-4a81-b727-e276ae0156d3-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:34.779660 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.777787 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f4a018c-c28f-4a81-b727-e276ae0156d3-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:34.779660 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.777802 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f4a018c-c28f-4a81-b727-e276ae0156d3-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:34.779660 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.777816 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f156e9bb-8f04-4805-b8ff-ca1561fb598c-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:34.779660 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.777829 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f156e9bb-8f04-4805-b8ff-ca1561fb598c-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:34.779660 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.777845 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f4a018c-c28f-4a81-b727-e276ae0156d3-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:34.779660 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.777860 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f4a018c-c28f-4a81-b727-e276ae0156d3-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:34.779660 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.777875 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f156e9bb-8f04-4805-b8ff-ca1561fb598c-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:34.779660 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.777889 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f156e9bb-8f04-4805-b8ff-ca1561fb598c-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:34.779660 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.777904 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f156e9bb-8f04-4805-b8ff-ca1561fb598c-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:34.779660 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.777919 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9zpd\" (UniqueName: \"kubernetes.io/projected/f156e9bb-8f04-4805-b8ff-ca1561fb598c-kube-api-access-r9zpd\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:34.779660 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.777933 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f156e9bb-8f04-4805-b8ff-ca1561fb598c-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:34.779660 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.777945 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f4a018c-c28f-4a81-b727-e276ae0156d3-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:34.779660 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:34.777960 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g74z\" (UniqueName: \"kubernetes.io/projected/1f4a018c-c28f-4a81-b727-e276ae0156d3-kube-api-access-9g74z\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:35.152220 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:35.152184 5979 generic.go:356] "Generic (PLEG): container finished" podID="f156e9bb-8f04-4805-b8ff-ca1561fb598c" containerID="bef503c247cd6c60e93a5a66dc1ce2f2cf2aee3c879196084c9f5277d53556f1" exitCode=0 Nov 03 13:32:35.152666 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:35.152636 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn" event={"ID":"f156e9bb-8f04-4805-b8ff-ca1561fb598c","Type":"ContainerDied","Data":"bef503c247cd6c60e93a5a66dc1ce2f2cf2aee3c879196084c9f5277d53556f1"} Nov 03 13:32:35.152843 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:35.152830 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn" event={"ID":"f156e9bb-8f04-4805-b8ff-ca1561fb598c","Type":"ContainerDied","Data":"aaf3165c67b38e7f134f22612698c539116224341a4eea752a561fd1b3aec482"} Nov 03 13:32:35.153100 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:35.153076 5979 scope.go:122] "RemoveContainer" containerID="bef503c247cd6c60e93a5a66dc1ce2f2cf2aee3c879196084c9f5277d53556f1" Nov 03 13:32:35.153348 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:35.153332 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn" Nov 03 13:32:35.165951 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:35.158017 5979 generic.go:356] "Generic (PLEG): container finished" podID="cfd3eded-c3de-4d6b-9a1e-c284549c6c18" containerID="405a543b62ebb99ccd78f604fd577bd100d66d8b417afb35d33618fca28d407a" exitCode=0 Nov 03 13:32:35.165951 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:35.158179 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb" event={"ID":"cfd3eded-c3de-4d6b-9a1e-c284549c6c18","Type":"ContainerDied","Data":"405a543b62ebb99ccd78f604fd577bd100d66d8b417afb35d33618fca28d407a"} Nov 03 13:32:35.165951 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:35.158221 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb" event={"ID":"cfd3eded-c3de-4d6b-9a1e-c284549c6c18","Type":"ContainerDied","Data":"9012e4c0b0cbbc9534d9feb16bf34319f9cbcfb4ba9fa0f5c34a1f554378336d"} Nov 03 13:32:35.165951 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:35.158298 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb" Nov 03 13:32:35.170080 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:35.169706 5979 generic.go:356] "Generic (PLEG): container finished" podID="42d9f732-f0cc-4e09-8ef5-cce6fc157dad" containerID="ab52e4bdbd49931b70d29485333cc0e955891ea8c6700ec49000ecb8f3208308" exitCode=0 Nov 03 13:32:35.170080 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:35.169837 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7" Nov 03 13:32:35.170080 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:35.169869 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7" event={"ID":"42d9f732-f0cc-4e09-8ef5-cce6fc157dad","Type":"ContainerDied","Data":"ab52e4bdbd49931b70d29485333cc0e955891ea8c6700ec49000ecb8f3208308"} Nov 03 13:32:35.170080 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:35.169915 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7" event={"ID":"42d9f732-f0cc-4e09-8ef5-cce6fc157dad","Type":"ContainerDied","Data":"ce8594714d02491d87a85bdef53bb653f3b1e2fe0a4a90d457da4797a744c0f0"} Nov 03 13:32:35.175092 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:35.175062 5979 generic.go:356] "Generic (PLEG): container finished" podID="1f4a018c-c28f-4a81-b727-e276ae0156d3" containerID="9edcf518b5926b6444322e10b9d784c097113d24911329ae56638bd93d64f5e2" exitCode=0 Nov 03 13:32:35.175230 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:35.175116 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7" Nov 03 13:32:35.175230 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:35.175187 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7" event={"ID":"1f4a018c-c28f-4a81-b727-e276ae0156d3","Type":"ContainerDied","Data":"9edcf518b5926b6444322e10b9d784c097113d24911329ae56638bd93d64f5e2"} Nov 03 13:32:35.175466 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:35.175227 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7" event={"ID":"1f4a018c-c28f-4a81-b727-e276ae0156d3","Type":"ContainerDied","Data":"ebf5c2674be0f886fa7804b6e6024414c3251984d8938a5038fbefddcade86fa"} Nov 03 13:32:35.184132 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:35.184096 5979 scope.go:122] "RemoveContainer" containerID="bef503c247cd6c60e93a5a66dc1ce2f2cf2aee3c879196084c9f5277d53556f1" Nov 03 13:32:35.185177 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:32:35.184788 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bef503c247cd6c60e93a5a66dc1ce2f2cf2aee3c879196084c9f5277d53556f1\": container with ID starting with bef503c247cd6c60e93a5a66dc1ce2f2cf2aee3c879196084c9f5277d53556f1 not found: ID does not exist" containerID="bef503c247cd6c60e93a5a66dc1ce2f2cf2aee3c879196084c9f5277d53556f1" Nov 03 13:32:35.185177 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:35.184827 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bef503c247cd6c60e93a5a66dc1ce2f2cf2aee3c879196084c9f5277d53556f1"} err="failed to get container status \"bef503c247cd6c60e93a5a66dc1ce2f2cf2aee3c879196084c9f5277d53556f1\": rpc error: code = NotFound desc = could not find container \"bef503c247cd6c60e93a5a66dc1ce2f2cf2aee3c879196084c9f5277d53556f1\": container with ID starting with bef503c247cd6c60e93a5a66dc1ce2f2cf2aee3c879196084c9f5277d53556f1 not found: ID does not exist" Nov 03 13:32:35.185177 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:35.184844 5979 scope.go:122] "RemoveContainer" containerID="405a543b62ebb99ccd78f604fd577bd100d66d8b417afb35d33618fca28d407a" Nov 03 13:32:35.223400 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:35.223380 5979 scope.go:122] "RemoveContainer" containerID="405a543b62ebb99ccd78f604fd577bd100d66d8b417afb35d33618fca28d407a" Nov 03 13:32:35.223780 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:32:35.223757 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"405a543b62ebb99ccd78f604fd577bd100d66d8b417afb35d33618fca28d407a\": container with ID starting with 405a543b62ebb99ccd78f604fd577bd100d66d8b417afb35d33618fca28d407a not found: ID does not exist" containerID="405a543b62ebb99ccd78f604fd577bd100d66d8b417afb35d33618fca28d407a" Nov 03 13:32:35.223929 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:35.223790 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"405a543b62ebb99ccd78f604fd577bd100d66d8b417afb35d33618fca28d407a"} err="failed to get container status \"405a543b62ebb99ccd78f604fd577bd100d66d8b417afb35d33618fca28d407a\": rpc error: code = NotFound desc = could not find container \"405a543b62ebb99ccd78f604fd577bd100d66d8b417afb35d33618fca28d407a\": container with ID starting with 405a543b62ebb99ccd78f604fd577bd100d66d8b417afb35d33618fca28d407a not found: ID does not exist" Nov 03 13:32:35.223929 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:35.223814 5979 scope.go:122] "RemoveContainer" containerID="ab52e4bdbd49931b70d29485333cc0e955891ea8c6700ec49000ecb8f3208308" Nov 03 13:32:35.235435 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:35.235417 5979 scope.go:122] "RemoveContainer" containerID="ab52e4bdbd49931b70d29485333cc0e955891ea8c6700ec49000ecb8f3208308" Nov 03 13:32:35.235766 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:32:35.235742 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab52e4bdbd49931b70d29485333cc0e955891ea8c6700ec49000ecb8f3208308\": container with ID starting with ab52e4bdbd49931b70d29485333cc0e955891ea8c6700ec49000ecb8f3208308 not found: ID does not exist" containerID="ab52e4bdbd49931b70d29485333cc0e955891ea8c6700ec49000ecb8f3208308" Nov 03 13:32:35.235811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:35.235775 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab52e4bdbd49931b70d29485333cc0e955891ea8c6700ec49000ecb8f3208308"} err="failed to get container status \"ab52e4bdbd49931b70d29485333cc0e955891ea8c6700ec49000ecb8f3208308\": rpc error: code = NotFound desc = could not find container \"ab52e4bdbd49931b70d29485333cc0e955891ea8c6700ec49000ecb8f3208308\": container with ID starting with ab52e4bdbd49931b70d29485333cc0e955891ea8c6700ec49000ecb8f3208308 not found: ID does not exist" Nov 03 13:32:35.235811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:35.235793 5979 scope.go:122] "RemoveContainer" containerID="9edcf518b5926b6444322e10b9d784c097113d24911329ae56638bd93d64f5e2" Nov 03 13:32:35.245296 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:35.245273 5979 scope.go:122] "RemoveContainer" containerID="9edcf518b5926b6444322e10b9d784c097113d24911329ae56638bd93d64f5e2" Nov 03 13:32:35.245620 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:32:35.245587 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9edcf518b5926b6444322e10b9d784c097113d24911329ae56638bd93d64f5e2\": container with ID starting with 9edcf518b5926b6444322e10b9d784c097113d24911329ae56638bd93d64f5e2 not found: ID does not exist" containerID="9edcf518b5926b6444322e10b9d784c097113d24911329ae56638bd93d64f5e2" Nov 03 13:32:35.245735 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:35.245625 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9edcf518b5926b6444322e10b9d784c097113d24911329ae56638bd93d64f5e2"} err="failed to get container status \"9edcf518b5926b6444322e10b9d784c097113d24911329ae56638bd93d64f5e2\": rpc error: code = NotFound desc = could not find container \"9edcf518b5926b6444322e10b9d784c097113d24911329ae56638bd93d64f5e2\": container with ID starting with 9edcf518b5926b6444322e10b9d784c097113d24911329ae56638bd93d64f5e2 not found: ID does not exist" Nov 03 13:32:35.319095 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:35.319061 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb"] Nov 03 13:32:35.365303 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:35.365275 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-s9bqb"] Nov 03 13:32:35.509887 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:35.509805 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-28d87c-f8886f745-ggq7c"] Nov 03 13:32:35.547360 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:35.547329 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn"] Nov 03 13:32:35.554137 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:35.554105 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-k8kzn"] Nov 03 13:32:35.610301 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:35.610269 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-28d87c-6c79c6f6dc-pvhkm"] Nov 03 13:32:35.655013 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:35.654982 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7"] Nov 03 13:32:35.659816 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:35.659789 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-5psw7"] Nov 03 13:32:35.873270 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:35.873188 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7"] Nov 03 13:32:35.880492 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:35.880451 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-pg2f7"] Nov 03 13:32:36.128251 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:36.128172 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f4a018c-c28f-4a81-b727-e276ae0156d3" path="/var/lib/kubelet/pods/1f4a018c-c28f-4a81-b727-e276ae0156d3/volumes" Nov 03 13:32:36.128696 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:36.128681 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42d9f732-f0cc-4e09-8ef5-cce6fc157dad" path="/var/lib/kubelet/pods/42d9f732-f0cc-4e09-8ef5-cce6fc157dad/volumes" Nov 03 13:32:36.129129 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:36.129116 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfd3eded-c3de-4d6b-9a1e-c284549c6c18" path="/var/lib/kubelet/pods/cfd3eded-c3de-4d6b-9a1e-c284549c6c18/volumes" Nov 03 13:32:36.129881 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:36.129863 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5" path="/var/lib/kubelet/pods/e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5/volumes" Nov 03 13:32:36.130298 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:36.130286 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f156e9bb-8f04-4805-b8ff-ca1561fb598c" path="/var/lib/kubelet/pods/f156e9bb-8f04-4805-b8ff-ca1561fb598c/volumes" Nov 03 13:32:37.275740 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:32:37.275694 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:32:38.209243 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:38.209208 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:32:40.280391 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:40.280346 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm"] Nov 03 13:32:40.281369 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:40.281341 5979 cpu_manager.go:414] "containerMap: removing container" podUID="42d9f732-f0cc-4e09-8ef5-cce6fc157dad" containerName="istio-proxy" Nov 03 13:32:40.281369 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:40.281368 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42d9f732-f0cc-4e09-8ef5-cce6fc157dad" containerName="istio-proxy" Nov 03 13:32:40.281643 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:40.281401 5979 cpu_manager.go:414] "containerMap: removing container" podUID="f156e9bb-8f04-4805-b8ff-ca1561fb598c" containerName="istio-proxy" Nov 03 13:32:40.281643 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:40.281409 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f156e9bb-8f04-4805-b8ff-ca1561fb598c" containerName="istio-proxy" Nov 03 13:32:40.281643 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:40.281433 5979 cpu_manager.go:414] "containerMap: removing container" podUID="1f4a018c-c28f-4a81-b727-e276ae0156d3" containerName="istio-proxy" Nov 03 13:32:40.281643 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:40.281441 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f4a018c-c28f-4a81-b727-e276ae0156d3" containerName="istio-proxy" Nov 03 13:32:40.281643 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:40.281462 5979 cpu_manager.go:414] "containerMap: removing container" podUID="e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5" containerName="istio-proxy" Nov 03 13:32:40.281643 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:40.281485 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5" containerName="istio-proxy" Nov 03 13:32:40.281643 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:40.281508 5979 cpu_manager.go:414] "containerMap: removing container" podUID="e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5" containerName="istio-proxy" Nov 03 13:32:40.281643 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:40.281514 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5" containerName="istio-proxy" Nov 03 13:32:40.281643 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:40.281542 5979 cpu_manager.go:414] "containerMap: removing container" podUID="cfd3eded-c3de-4d6b-9a1e-c284549c6c18" containerName="istio-proxy" Nov 03 13:32:40.281643 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:40.281549 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfd3eded-c3de-4d6b-9a1e-c284549c6c18" containerName="istio-proxy" Nov 03 13:32:40.282205 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:40.281692 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="f156e9bb-8f04-4805-b8ff-ca1561fb598c" containerName="istio-proxy" Nov 03 13:32:40.282205 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:40.281707 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f4a018c-c28f-4a81-b727-e276ae0156d3" containerName="istio-proxy" Nov 03 13:32:40.282205 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:40.281717 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5" containerName="istio-proxy" Nov 03 13:32:40.282205 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:40.281729 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0ad1760-927c-4eaf-9ae9-0a2e85f4c0d5" containerName="istio-proxy" Nov 03 13:32:40.282205 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:40.281741 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfd3eded-c3de-4d6b-9a1e-c284549c6c18" containerName="istio-proxy" Nov 03 13:32:40.282205 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:40.281753 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="42d9f732-f0cc-4e09-8ef5-cce6fc157dad" containerName="istio-proxy" Nov 03 13:32:40.299601 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:40.299572 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm" Nov 03 13:32:40.359745 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:40.359714 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm"] Nov 03 13:32:40.440936 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:40.417941 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-istiod-ca-cert\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm\" (UID: \"5fd8d2bb-4812-41bc-88a9-0d8249c25c1c\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm" Nov 03 13:32:40.441275 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:40.441240 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-istio-token\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm\" (UID: \"5fd8d2bb-4812-41bc-88a9-0d8249c25c1c\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm" Nov 03 13:32:40.441456 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:40.441438 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-istio-podinfo\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm\" (UID: \"5fd8d2bb-4812-41bc-88a9-0d8249c25c1c\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm" Nov 03 13:32:40.441690 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:40.441670 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-workload-socket\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm\" (UID: \"5fd8d2bb-4812-41bc-88a9-0d8249c25c1c\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm" Nov 03 13:32:40.441819 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:40.441804 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-credential-socket\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm\" (UID: \"5fd8d2bb-4812-41bc-88a9-0d8249c25c1c\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm" Nov 03 13:32:40.442031 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:40.442003 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-istio-envoy\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm\" (UID: \"5fd8d2bb-4812-41bc-88a9-0d8249c25c1c\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm" Nov 03 13:32:40.442165 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:40.442108 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8gs7\" (UniqueName: \"kubernetes.io/projected/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-kube-api-access-t8gs7\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm\" (UID: \"5fd8d2bb-4812-41bc-88a9-0d8249c25c1c\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm" Nov 03 13:32:40.442258 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:40.442202 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-workload-certs\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm\" (UID: \"5fd8d2bb-4812-41bc-88a9-0d8249c25c1c\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm" Nov 03 13:32:40.442355 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:40.442334 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-istio-data\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm\" (UID: \"5fd8d2bb-4812-41bc-88a9-0d8249c25c1c\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm" Nov 03 13:32:40.511017 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:40.510960 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-28d87c-f8886f745-ggq7c" podUID="15e26ebd-f572-4f06-b7cc-55f415f492ad" containerName="payload-processing" containerID="cri-o://8587e7ed08e92ba11caddb03e2794062c76520f060966cf5454777b593245dd4" gracePeriod=25 Nov 03 13:32:40.544071 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:40.543994 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-istio-podinfo\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm\" (UID: \"5fd8d2bb-4812-41bc-88a9-0d8249c25c1c\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm" Nov 03 13:32:40.544187 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:40.544094 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-workload-socket\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm\" (UID: \"5fd8d2bb-4812-41bc-88a9-0d8249c25c1c\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm" Nov 03 13:32:40.544187 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:40.544126 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-credential-socket\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm\" (UID: \"5fd8d2bb-4812-41bc-88a9-0d8249c25c1c\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm" Nov 03 13:32:40.544187 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:40.544169 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-istio-envoy\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm\" (UID: \"5fd8d2bb-4812-41bc-88a9-0d8249c25c1c\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm" Nov 03 13:32:40.544360 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:40.544228 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8gs7\" (UniqueName: \"kubernetes.io/projected/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-kube-api-access-t8gs7\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm\" (UID: \"5fd8d2bb-4812-41bc-88a9-0d8249c25c1c\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm" Nov 03 13:32:40.544360 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:40.544264 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-workload-certs\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm\" (UID: \"5fd8d2bb-4812-41bc-88a9-0d8249c25c1c\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm" Nov 03 13:32:40.544360 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:40.544342 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-istio-data\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm\" (UID: \"5fd8d2bb-4812-41bc-88a9-0d8249c25c1c\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm" Nov 03 13:32:40.544534 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:40.544459 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-istiod-ca-cert\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm\" (UID: \"5fd8d2bb-4812-41bc-88a9-0d8249c25c1c\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm" Nov 03 13:32:40.544534 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:40.544509 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-istio-token\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm\" (UID: \"5fd8d2bb-4812-41bc-88a9-0d8249c25c1c\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm" Nov 03 13:32:40.544639 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:40.544542 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-workload-socket\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm\" (UID: \"5fd8d2bb-4812-41bc-88a9-0d8249c25c1c\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm" Nov 03 13:32:40.545178 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:40.545010 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-credential-socket\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm\" (UID: \"5fd8d2bb-4812-41bc-88a9-0d8249c25c1c\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm" Nov 03 13:32:40.545178 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:40.545143 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-istio-data\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm\" (UID: \"5fd8d2bb-4812-41bc-88a9-0d8249c25c1c\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm" Nov 03 13:32:40.545367 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:40.545317 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-istiod-ca-cert\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm\" (UID: \"5fd8d2bb-4812-41bc-88a9-0d8249c25c1c\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm" Nov 03 13:32:40.545639 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:40.545620 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-workload-certs\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm\" (UID: \"5fd8d2bb-4812-41bc-88a9-0d8249c25c1c\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm" Nov 03 13:32:40.546812 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:40.546777 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-istio-envoy\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm\" (UID: \"5fd8d2bb-4812-41bc-88a9-0d8249c25c1c\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm" Nov 03 13:32:40.546936 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:40.546814 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-istio-podinfo\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm\" (UID: \"5fd8d2bb-4812-41bc-88a9-0d8249c25c1c\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm" Nov 03 13:32:40.598935 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:40.598906 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-istio-token\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm\" (UID: \"5fd8d2bb-4812-41bc-88a9-0d8249c25c1c\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm" Nov 03 13:32:40.599072 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:40.599052 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8gs7\" (UniqueName: \"kubernetes.io/projected/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-kube-api-access-t8gs7\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm\" (UID: \"5fd8d2bb-4812-41bc-88a9-0d8249c25c1c\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm" Nov 03 13:32:40.610884 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:40.610855 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-28d87c-6c79c6f6dc-pvhkm" podUID="332311c4-5b9c-4cb2-b48d-ec4308a75c41" containerName="payload-pre-processing" containerID="cri-o://8e673f1f77c22e6ed530bf7fe1c9d8edf7c5bb1bfb164e56fd695ec6b5df5865" gracePeriod=25 Nov 03 13:32:40.615372 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:40.615348 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm" Nov 03 13:32:40.833527 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:40.833494 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-28d87c-f8886f745-ggq7c" Nov 03 13:32:40.885933 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:40.885907 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-28d87c-6c79c6f6dc-pvhkm" Nov 03 13:32:40.950690 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:40.950643 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15e26ebd-f572-4f06-b7cc-55f415f492ad-kube-api-access-57pj8\" (UniqueName: \"kubernetes.io/projected/15e26ebd-f572-4f06-b7cc-55f415f492ad-kube-api-access-57pj8\") pod \"15e26ebd-f572-4f06-b7cc-55f415f492ad\" (UID: \"15e26ebd-f572-4f06-b7cc-55f415f492ad\") " Nov 03 13:32:40.950888 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:40.950842 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15e26ebd-f572-4f06-b7cc-55f415f492ad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15e26ebd-f572-4f06-b7cc-55f415f492ad-plugins-config-volume\") pod \"15e26ebd-f572-4f06-b7cc-55f415f492ad\" (UID: \"15e26ebd-f572-4f06-b7cc-55f415f492ad\") " Nov 03 13:32:40.951176 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:40.951154 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15e26ebd-f572-4f06-b7cc-55f415f492ad-plugins-config-volume" pod "15e26ebd-f572-4f06-b7cc-55f415f492ad" (UID: "15e26ebd-f572-4f06-b7cc-55f415f492ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:32:40.951353 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:40.951335 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15e26ebd-f572-4f06-b7cc-55f415f492ad-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:40.952822 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:40.952794 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15e26ebd-f572-4f06-b7cc-55f415f492ad-kube-api-access-57pj8" pod "15e26ebd-f572-4f06-b7cc-55f415f492ad" (UID: "15e26ebd-f572-4f06-b7cc-55f415f492ad"). InnerVolumeSpecName "kube-api-access-57pj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:32:41.052006 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:41.051968 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/332311c4-5b9c-4cb2-b48d-ec4308a75c41-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/332311c4-5b9c-4cb2-b48d-ec4308a75c41-plugins-config-volume\") pod \"332311c4-5b9c-4cb2-b48d-ec4308a75c41\" (UID: \"332311c4-5b9c-4cb2-b48d-ec4308a75c41\") " Nov 03 13:32:41.052205 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:41.052138 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/332311c4-5b9c-4cb2-b48d-ec4308a75c41-kube-api-access-5d52v\" (UniqueName: \"kubernetes.io/projected/332311c4-5b9c-4cb2-b48d-ec4308a75c41-kube-api-access-5d52v\") pod \"332311c4-5b9c-4cb2-b48d-ec4308a75c41\" (UID: \"332311c4-5b9c-4cb2-b48d-ec4308a75c41\") " Nov 03 13:32:41.052440 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:41.052384 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/332311c4-5b9c-4cb2-b48d-ec4308a75c41-plugins-config-volume" pod "332311c4-5b9c-4cb2-b48d-ec4308a75c41" (UID: "332311c4-5b9c-4cb2-b48d-ec4308a75c41"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:32:41.052726 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:41.052704 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57pj8\" (UniqueName: \"kubernetes.io/projected/15e26ebd-f572-4f06-b7cc-55f415f492ad-kube-api-access-57pj8\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:41.052923 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:41.052730 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/332311c4-5b9c-4cb2-b48d-ec4308a75c41-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:41.054781 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:41.054757 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/332311c4-5b9c-4cb2-b48d-ec4308a75c41-kube-api-access-5d52v" pod "332311c4-5b9c-4cb2-b48d-ec4308a75c41" (UID: "332311c4-5b9c-4cb2-b48d-ec4308a75c41"). InnerVolumeSpecName "kube-api-access-5d52v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:32:41.110459 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:41.110382 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm"] Nov 03 13:32:41.116041 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:32:41.116017 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fd8d2bb_4812_41bc_88a9_0d8249c25c1c.slice/crio-872b5552b2289e4897198dfa4d1516c460f2d4685636041f3faf41134db25c0c WatchSource:0}: Error finding container 872b5552b2289e4897198dfa4d1516c460f2d4685636041f3faf41134db25c0c: Status 404 returned error can't find the container with id 872b5552b2289e4897198dfa4d1516c460f2d4685636041f3faf41134db25c0c Nov 03 13:32:41.118565 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:41.118537 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:32:41.118629 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:41.118599 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:32:41.118668 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:41.118634 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:32:41.154615 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:41.154593 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d52v\" (UniqueName: \"kubernetes.io/projected/332311c4-5b9c-4cb2-b48d-ec4308a75c41-kube-api-access-5d52v\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:32:41.306792 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:41.306742 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm" event={"ID":"5fd8d2bb-4812-41bc-88a9-0d8249c25c1c","Type":"ContainerStarted","Data":"1f7a2ff3e89e525a4021063055e7c71af56a94f5c449df20286ac3336aff5e1c"} Nov 03 13:32:41.307237 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:41.306802 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm" event={"ID":"5fd8d2bb-4812-41bc-88a9-0d8249c25c1c","Type":"ContainerStarted","Data":"872b5552b2289e4897198dfa4d1516c460f2d4685636041f3faf41134db25c0c"} Nov 03 13:32:41.308353 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:41.308316 5979 generic.go:356] "Generic (PLEG): container finished" podID="15e26ebd-f572-4f06-b7cc-55f415f492ad" containerID="8587e7ed08e92ba11caddb03e2794062c76520f060966cf5454777b593245dd4" exitCode=0 Nov 03 13:32:41.308508 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:41.308346 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-28d87c-f8886f745-ggq7c" event={"ID":"15e26ebd-f572-4f06-b7cc-55f415f492ad","Type":"ContainerDied","Data":"8587e7ed08e92ba11caddb03e2794062c76520f060966cf5454777b593245dd4"} Nov 03 13:32:41.308508 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:41.308394 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-28d87c-f8886f745-ggq7c" Nov 03 13:32:41.308508 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:41.308413 5979 scope.go:122] "RemoveContainer" containerID="8587e7ed08e92ba11caddb03e2794062c76520f060966cf5454777b593245dd4" Nov 03 13:32:41.308508 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:41.308401 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-28d87c-f8886f745-ggq7c" event={"ID":"15e26ebd-f572-4f06-b7cc-55f415f492ad","Type":"ContainerDied","Data":"2774e048c543c9815de9f29c5eb21ca6e4d6c3807d64c2252d12a51c665d2f8d"} Nov 03 13:32:41.310011 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:41.309988 5979 generic.go:356] "Generic (PLEG): container finished" podID="332311c4-5b9c-4cb2-b48d-ec4308a75c41" containerID="8e673f1f77c22e6ed530bf7fe1c9d8edf7c5bb1bfb164e56fd695ec6b5df5865" exitCode=0 Nov 03 13:32:41.310117 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:41.310044 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-28d87c-6c79c6f6dc-pvhkm" Nov 03 13:32:41.310117 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:41.310075 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-28d87c-6c79c6f6dc-pvhkm" event={"ID":"332311c4-5b9c-4cb2-b48d-ec4308a75c41","Type":"ContainerDied","Data":"8e673f1f77c22e6ed530bf7fe1c9d8edf7c5bb1bfb164e56fd695ec6b5df5865"} Nov 03 13:32:41.310215 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:41.310134 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-28d87c-6c79c6f6dc-pvhkm" event={"ID":"332311c4-5b9c-4cb2-b48d-ec4308a75c41","Type":"ContainerDied","Data":"dd85e3d1c537712bb1fd9534c79508f37949a6dab5ec09a9c9e3c844177c21a9"} Nov 03 13:32:41.320051 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:41.320032 5979 scope.go:122] "RemoveContainer" containerID="8587e7ed08e92ba11caddb03e2794062c76520f060966cf5454777b593245dd4" Nov 03 13:32:41.320316 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:32:41.320298 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8587e7ed08e92ba11caddb03e2794062c76520f060966cf5454777b593245dd4\": container with ID starting with 8587e7ed08e92ba11caddb03e2794062c76520f060966cf5454777b593245dd4 not found: ID does not exist" containerID="8587e7ed08e92ba11caddb03e2794062c76520f060966cf5454777b593245dd4" Nov 03 13:32:41.320409 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:41.320325 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8587e7ed08e92ba11caddb03e2794062c76520f060966cf5454777b593245dd4"} err="failed to get container status \"8587e7ed08e92ba11caddb03e2794062c76520f060966cf5454777b593245dd4\": rpc error: code = NotFound desc = could not find container \"8587e7ed08e92ba11caddb03e2794062c76520f060966cf5454777b593245dd4\": container with ID starting with 8587e7ed08e92ba11caddb03e2794062c76520f060966cf5454777b593245dd4 not found: ID does not exist" Nov 03 13:32:41.320409 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:41.320349 5979 scope.go:122] "RemoveContainer" containerID="8e673f1f77c22e6ed530bf7fe1c9d8edf7c5bb1bfb164e56fd695ec6b5df5865" Nov 03 13:32:41.330226 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:41.330202 5979 scope.go:122] "RemoveContainer" containerID="8e673f1f77c22e6ed530bf7fe1c9d8edf7c5bb1bfb164e56fd695ec6b5df5865" Nov 03 13:32:41.330498 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:32:41.330452 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e673f1f77c22e6ed530bf7fe1c9d8edf7c5bb1bfb164e56fd695ec6b5df5865\": container with ID starting with 8e673f1f77c22e6ed530bf7fe1c9d8edf7c5bb1bfb164e56fd695ec6b5df5865 not found: ID does not exist" containerID="8e673f1f77c22e6ed530bf7fe1c9d8edf7c5bb1bfb164e56fd695ec6b5df5865" Nov 03 13:32:41.330565 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:41.330505 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e673f1f77c22e6ed530bf7fe1c9d8edf7c5bb1bfb164e56fd695ec6b5df5865"} err="failed to get container status \"8e673f1f77c22e6ed530bf7fe1c9d8edf7c5bb1bfb164e56fd695ec6b5df5865\": rpc error: code = NotFound desc = could not find container \"8e673f1f77c22e6ed530bf7fe1c9d8edf7c5bb1bfb164e56fd695ec6b5df5865\": container with ID starting with 8e673f1f77c22e6ed530bf7fe1c9d8edf7c5bb1bfb164e56fd695ec6b5df5865 not found: ID does not exist" Nov 03 13:32:41.428393 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:41.428335 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm" podStartSLOduration=1.428315166 podStartE2EDuration="1.428315166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:32:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:32:41.427383053 +0000 UTC m=+1643.438071324" watchObservedRunningTime="2026-11-03 13:32:41.428315166 +0000 UTC m=+1643.439003409" Nov 03 13:32:41.553257 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:41.553228 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-28d87c-f8886f745-ggq7c"] Nov 03 13:32:41.564999 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:41.564971 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-28d87c-f8886f745-ggq7c"] Nov 03 13:32:41.616402 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:41.616358 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm" Nov 03 13:32:41.683244 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:41.683147 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-28d87c-6c79c6f6dc-pvhkm"] Nov 03 13:32:41.690050 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:41.690022 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-28d87c-6c79c6f6dc-pvhkm"] Nov 03 13:32:42.117826 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:42.117749 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15e26ebd-f572-4f06-b7cc-55f415f492ad" path="/var/lib/kubelet/pods/15e26ebd-f572-4f06-b7cc-55f415f492ad/volumes" Nov 03 13:32:42.118239 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:42.118224 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="332311c4-5b9c-4cb2-b48d-ec4308a75c41" path="/var/lib/kubelet/pods/332311c4-5b9c-4cb2-b48d-ec4308a75c41/volumes" Nov 03 13:32:42.617804 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:42.617770 5979 patch_prober.go:29] interesting pod/e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Nov 03 13:32:42.618261 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:42.617830 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm" podUID="5fd8d2bb-4812-41bc-88a9-0d8249c25c1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Nov 03 13:32:43.620742 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:43.620707 5979 patch_prober.go:29] interesting pod/e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Nov 03 13:32:43.626662 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:43.626608 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm" podUID="5fd8d2bb-4812-41bc-88a9-0d8249c25c1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Nov 03 13:32:44.616390 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:44.616354 5979 patch_prober.go:29] interesting pod/e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Nov 03 13:32:44.616564 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:44.616411 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm" podUID="5fd8d2bb-4812-41bc-88a9-0d8249c25c1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Nov 03 13:32:45.431964 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:45.431928 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa2568-d5dd645b7-pvp9f"] Nov 03 13:32:45.432658 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:45.432638 5979 cpu_manager.go:414] "containerMap: removing container" podUID="332311c4-5b9c-4cb2-b48d-ec4308a75c41" containerName="payload-pre-processing" Nov 03 13:32:45.432752 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:45.432661 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="332311c4-5b9c-4cb2-b48d-ec4308a75c41" containerName="payload-pre-processing" Nov 03 13:32:45.432752 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:45.432687 5979 cpu_manager.go:414] "containerMap: removing container" podUID="15e26ebd-f572-4f06-b7cc-55f415f492ad" containerName="payload-processing" Nov 03 13:32:45.432752 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:45.432695 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15e26ebd-f572-4f06-b7cc-55f415f492ad" containerName="payload-processing" Nov 03 13:32:45.432861 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:45.432853 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="332311c4-5b9c-4cb2-b48d-ec4308a75c41" containerName="payload-pre-processing" Nov 03 13:32:45.432896 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:45.432863 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="15e26ebd-f572-4f06-b7cc-55f415f492ad" containerName="payload-processing" Nov 03 13:32:45.436255 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:45.436235 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa2568-d5dd645b7-pvp9f" Nov 03 13:32:45.488068 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:45.488033 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-fa2568-serving-cert\"" Nov 03 13:32:45.528612 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:45.528563 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwjcl\" (UniqueName: \"kubernetes.io/projected/e6506f7e-7151-4cf3-9f2d-fefbba0a3783-kube-api-access-nwjcl\") pod \"maas-api-e2e-ipp-a-fa2568-d5dd645b7-pvp9f\" (UID: \"e6506f7e-7151-4cf3-9f2d-fefbba0a3783\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa2568-d5dd645b7-pvp9f" Nov 03 13:32:45.528785 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:45.528627 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6506f7e-7151-4cf3-9f2d-fefbba0a3783-maas-api-tls\") pod \"maas-api-e2e-ipp-a-fa2568-d5dd645b7-pvp9f\" (UID: \"e6506f7e-7151-4cf3-9f2d-fefbba0a3783\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa2568-d5dd645b7-pvp9f" Nov 03 13:32:45.528785 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:45.528750 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6506f7e-7151-4cf3-9f2d-fefbba0a3783-metrics-certs\") pod \"maas-api-e2e-ipp-a-fa2568-d5dd645b7-pvp9f\" (UID: \"e6506f7e-7151-4cf3-9f2d-fefbba0a3783\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa2568-d5dd645b7-pvp9f" Nov 03 13:32:45.528943 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:45.528852 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6506f7e-7151-4cf3-9f2d-fefbba0a3783-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-fa2568-d5dd645b7-pvp9f\" (UID: \"e6506f7e-7151-4cf3-9f2d-fefbba0a3783\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa2568-d5dd645b7-pvp9f" Nov 03 13:32:45.534390 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:45.534356 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-dgx9m\"" Nov 03 13:32:45.534550 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:45.534411 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 13:32:45.534615 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:45.534560 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 13:32:45.534615 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:45.534589 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 13:32:45.538233 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:45.538213 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa2568-d5dd645b7-pvp9f"] Nov 03 13:32:45.616619 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:45.616582 5979 patch_prober.go:29] interesting pod/e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Nov 03 13:32:45.616786 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:45.616640 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm" podUID="5fd8d2bb-4812-41bc-88a9-0d8249c25c1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Nov 03 13:32:45.630268 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:45.630228 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwjcl\" (UniqueName: \"kubernetes.io/projected/e6506f7e-7151-4cf3-9f2d-fefbba0a3783-kube-api-access-nwjcl\") pod \"maas-api-e2e-ipp-a-fa2568-d5dd645b7-pvp9f\" (UID: \"e6506f7e-7151-4cf3-9f2d-fefbba0a3783\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa2568-d5dd645b7-pvp9f" Nov 03 13:32:45.630394 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:45.630305 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6506f7e-7151-4cf3-9f2d-fefbba0a3783-maas-api-tls\") pod \"maas-api-e2e-ipp-a-fa2568-d5dd645b7-pvp9f\" (UID: \"e6506f7e-7151-4cf3-9f2d-fefbba0a3783\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa2568-d5dd645b7-pvp9f" Nov 03 13:32:45.630394 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:45.630356 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6506f7e-7151-4cf3-9f2d-fefbba0a3783-metrics-certs\") pod \"maas-api-e2e-ipp-a-fa2568-d5dd645b7-pvp9f\" (UID: \"e6506f7e-7151-4cf3-9f2d-fefbba0a3783\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa2568-d5dd645b7-pvp9f" Nov 03 13:32:45.630514 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:45.630423 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6506f7e-7151-4cf3-9f2d-fefbba0a3783-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-fa2568-d5dd645b7-pvp9f\" (UID: \"e6506f7e-7151-4cf3-9f2d-fefbba0a3783\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa2568-d5dd645b7-pvp9f" Nov 03 13:32:45.630514 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:32:45.630438 5979 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa2568-serving-cert: secret "maas-api-e2e-ipp-a-fa2568-serving-cert" not found Nov 03 13:32:45.630628 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:32:45.630536 5979 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6506f7e-7151-4cf3-9f2d-fefbba0a3783-maas-api-tls podName:e6506f7e-7151-4cf3-9f2d-fefbba0a3783 nodeName:}" failed. No retries permitted until 2026-11-03 13:32:46.130517922 +0000 UTC m=+1648.141206141 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e6506f7e-7151-4cf3-9f2d-fefbba0a3783-maas-api-tls") pod "maas-api-e2e-ipp-a-fa2568-d5dd645b7-pvp9f" (UID: "e6506f7e-7151-4cf3-9f2d-fefbba0a3783") : secret "maas-api-e2e-ipp-a-fa2568-serving-cert" not found Nov 03 13:32:45.631153 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:45.631126 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6506f7e-7151-4cf3-9f2d-fefbba0a3783-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-fa2568-d5dd645b7-pvp9f\" (UID: \"e6506f7e-7151-4cf3-9f2d-fefbba0a3783\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa2568-d5dd645b7-pvp9f" Nov 03 13:32:45.632803 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:45.632785 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6506f7e-7151-4cf3-9f2d-fefbba0a3783-metrics-certs\") pod \"maas-api-e2e-ipp-a-fa2568-d5dd645b7-pvp9f\" (UID: \"e6506f7e-7151-4cf3-9f2d-fefbba0a3783\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa2568-d5dd645b7-pvp9f" Nov 03 13:32:45.726682 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:45.726572 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwjcl\" (UniqueName: \"kubernetes.io/projected/e6506f7e-7151-4cf3-9f2d-fefbba0a3783-kube-api-access-nwjcl\") pod \"maas-api-e2e-ipp-a-fa2568-d5dd645b7-pvp9f\" (UID: \"e6506f7e-7151-4cf3-9f2d-fefbba0a3783\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa2568-d5dd645b7-pvp9f" Nov 03 13:32:46.138873 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:46.138840 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6506f7e-7151-4cf3-9f2d-fefbba0a3783-maas-api-tls\") pod \"maas-api-e2e-ipp-a-fa2568-d5dd645b7-pvp9f\" (UID: \"e6506f7e-7151-4cf3-9f2d-fefbba0a3783\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa2568-d5dd645b7-pvp9f" Nov 03 13:32:46.141285 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:46.141259 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6506f7e-7151-4cf3-9f2d-fefbba0a3783-maas-api-tls\") pod \"maas-api-e2e-ipp-a-fa2568-d5dd645b7-pvp9f\" (UID: \"e6506f7e-7151-4cf3-9f2d-fefbba0a3783\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa2568-d5dd645b7-pvp9f" Nov 03 13:32:46.366684 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:46.365724 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa2568-d5dd645b7-pvp9f" Nov 03 13:32:46.615713 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:46.615679 5979 patch_prober.go:29] interesting pod/e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Nov 03 13:32:46.616132 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:46.615734 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm" podUID="5fd8d2bb-4812-41bc-88a9-0d8249c25c1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Nov 03 13:32:46.740924 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:46.740853 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa2568-d5dd645b7-pvp9f"] Nov 03 13:32:46.773668 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:32:46.773627 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6506f7e_7151_4cf3_9f2d_fefbba0a3783.slice/crio-9634b582ce02c399b4383432dc6fe3f1b25c3d8724adaf97e1b81482c80f2dd4 WatchSource:0}: Error finding container 9634b582ce02c399b4383432dc6fe3f1b25c3d8724adaf97e1b81482c80f2dd4: Status 404 returned error can't find the container with id 9634b582ce02c399b4383432dc6fe3f1b25c3d8724adaf97e1b81482c80f2dd4 Nov 03 13:32:46.775346 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:46.775324 5979 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 13:32:47.435453 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:47.435411 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa2568-d5dd645b7-pvp9f" event={"ID":"e6506f7e-7151-4cf3-9f2d-fefbba0a3783","Type":"ContainerStarted","Data":"9634b582ce02c399b4383432dc6fe3f1b25c3d8724adaf97e1b81482c80f2dd4"} Nov 03 13:32:47.670231 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:47.670194 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm" Nov 03 13:32:47.672076 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:47.672052 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm" Nov 03 13:32:47.685086 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:47.685054 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm" Nov 03 13:32:49.510832 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:49.503932 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa2568-d5dd645b7-pvp9f" event={"ID":"e6506f7e-7151-4cf3-9f2d-fefbba0a3783","Type":"ContainerStarted","Data":"5bf168b7972cf903e6f1931a05395880327d5e83f2420bf885f22f194e1942e3"} Nov 03 13:32:49.510832 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:49.504030 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa2568-d5dd645b7-pvp9f" Nov 03 13:32:49.618346 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:49.618285 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa2568-d5dd645b7-pvp9f" podStartSLOduration=2.348169358 podStartE2EDuration="4.618266221s" totalImagesPullingTime="2.270096863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:32:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:32:49.616051389 +0000 UTC m=+1651.626739631" watchObservedRunningTime="2026-11-03 13:32:49.618266221 +0000 UTC m=+1651.628954464" Nov 03 13:32:55.376559 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:55.376518 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74"] Nov 03 13:32:55.380466 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:55.380446 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74" Nov 03 13:32:55.481731 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:55.481671 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74"] Nov 03 13:32:55.539533 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:55.539499 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-istio-envoy\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74\" (UID: \"22fad5ad-56d4-4d01-944f-a5db77a4a3ee\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74" Nov 03 13:32:55.539727 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:55.539642 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-workload-socket\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74\" (UID: \"22fad5ad-56d4-4d01-944f-a5db77a4a3ee\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74" Nov 03 13:32:55.539777 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:55.539751 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-istio-podinfo\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74\" (UID: \"22fad5ad-56d4-4d01-944f-a5db77a4a3ee\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74" Nov 03 13:32:55.539861 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:55.539845 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-istiod-ca-cert\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74\" (UID: \"22fad5ad-56d4-4d01-944f-a5db77a4a3ee\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74" Nov 03 13:32:55.539921 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:55.539908 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-istio-token\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74\" (UID: \"22fad5ad-56d4-4d01-944f-a5db77a4a3ee\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74" Nov 03 13:32:55.539961 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:55.539942 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-workload-certs\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74\" (UID: \"22fad5ad-56d4-4d01-944f-a5db77a4a3ee\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74" Nov 03 13:32:55.540010 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:55.539981 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-istio-data\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74\" (UID: \"22fad5ad-56d4-4d01-944f-a5db77a4a3ee\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74" Nov 03 13:32:55.540049 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:55.540025 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-credential-socket\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74\" (UID: \"22fad5ad-56d4-4d01-944f-a5db77a4a3ee\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74" Nov 03 13:32:55.540092 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:55.540074 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cz2w\" (UniqueName: \"kubernetes.io/projected/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-kube-api-access-8cz2w\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74\" (UID: \"22fad5ad-56d4-4d01-944f-a5db77a4a3ee\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74" Nov 03 13:32:55.641819 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:55.641782 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-istio-data\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74\" (UID: \"22fad5ad-56d4-4d01-944f-a5db77a4a3ee\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74" Nov 03 13:32:55.641819 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:55.641821 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-credential-socket\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74\" (UID: \"22fad5ad-56d4-4d01-944f-a5db77a4a3ee\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74" Nov 03 13:32:55.642043 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:55.641838 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cz2w\" (UniqueName: \"kubernetes.io/projected/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-kube-api-access-8cz2w\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74\" (UID: \"22fad5ad-56d4-4d01-944f-a5db77a4a3ee\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74" Nov 03 13:32:55.642043 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:55.641931 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-istio-envoy\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74\" (UID: \"22fad5ad-56d4-4d01-944f-a5db77a4a3ee\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74" Nov 03 13:32:55.642043 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:55.641975 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-workload-socket\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74\" (UID: \"22fad5ad-56d4-4d01-944f-a5db77a4a3ee\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74" Nov 03 13:32:55.642043 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:55.642004 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-istio-podinfo\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74\" (UID: \"22fad5ad-56d4-4d01-944f-a5db77a4a3ee\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74" Nov 03 13:32:55.642259 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:55.642090 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-istiod-ca-cert\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74\" (UID: \"22fad5ad-56d4-4d01-944f-a5db77a4a3ee\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74" Nov 03 13:32:55.642259 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:55.642115 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-istio-token\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74\" (UID: \"22fad5ad-56d4-4d01-944f-a5db77a4a3ee\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74" Nov 03 13:32:55.642259 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:55.642157 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-workload-certs\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74\" (UID: \"22fad5ad-56d4-4d01-944f-a5db77a4a3ee\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74" Nov 03 13:32:55.642259 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:55.642234 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-credential-socket\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74\" (UID: \"22fad5ad-56d4-4d01-944f-a5db77a4a3ee\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74" Nov 03 13:32:55.642580 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:55.642410 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-workload-socket\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74\" (UID: \"22fad5ad-56d4-4d01-944f-a5db77a4a3ee\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74" Nov 03 13:32:55.642580 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:55.642437 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-istio-data\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74\" (UID: \"22fad5ad-56d4-4d01-944f-a5db77a4a3ee\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74" Nov 03 13:32:55.642675 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:55.642663 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-workload-certs\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74\" (UID: \"22fad5ad-56d4-4d01-944f-a5db77a4a3ee\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74" Nov 03 13:32:55.642781 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:55.642761 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-istiod-ca-cert\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74\" (UID: \"22fad5ad-56d4-4d01-944f-a5db77a4a3ee\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74" Nov 03 13:32:55.644245 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:55.644217 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-istio-envoy\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74\" (UID: \"22fad5ad-56d4-4d01-944f-a5db77a4a3ee\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74" Nov 03 13:32:55.644600 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:55.644579 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-istio-podinfo\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74\" (UID: \"22fad5ad-56d4-4d01-944f-a5db77a4a3ee\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74" Nov 03 13:32:55.697101 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:55.697075 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-istio-token\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74\" (UID: \"22fad5ad-56d4-4d01-944f-a5db77a4a3ee\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74" Nov 03 13:32:55.697777 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:55.697759 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cz2w\" (UniqueName: \"kubernetes.io/projected/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-kube-api-access-8cz2w\") pod \"e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74\" (UID: \"22fad5ad-56d4-4d01-944f-a5db77a4a3ee\") " pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74" Nov 03 13:32:55.994925 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:55.994845 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74" Nov 03 13:32:56.423915 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:56.423876 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74"] Nov 03 13:32:56.430125 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:32:56.430097 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22fad5ad_56d4_4d01_944f_a5db77a4a3ee.slice/crio-05e3218cea31daae4e10e4a40184f85210e2658b5775cc69ae58f8a06bd7f2a6 WatchSource:0}: Error finding container 05e3218cea31daae4e10e4a40184f85210e2658b5775cc69ae58f8a06bd7f2a6: Status 404 returned error can't find the container with id 05e3218cea31daae4e10e4a40184f85210e2658b5775cc69ae58f8a06bd7f2a6 Nov 03 13:32:56.432824 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:56.432794 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:32:56.432897 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:56.432859 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:32:56.432897 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:56.432889 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:32:56.522416 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:56.522389 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa2568-d5dd645b7-pvp9f" Nov 03 13:32:56.553130 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:56.553094 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74" event={"ID":"22fad5ad-56d4-4d01-944f-a5db77a4a3ee","Type":"ContainerStarted","Data":"0c0701613949d61961abc7b953bbaa3f49b57d39e2aa21425167e5e5880f6828"} Nov 03 13:32:56.553310 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:56.553154 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74" event={"ID":"22fad5ad-56d4-4d01-944f-a5db77a4a3ee","Type":"ContainerStarted","Data":"05e3218cea31daae4e10e4a40184f85210e2658b5775cc69ae58f8a06bd7f2a6"} Nov 03 13:32:56.746515 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:56.746390 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74" podStartSLOduration=1.746376718 podStartE2EDuration="1.746376718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:32:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:32:56.74414874 +0000 UTC m=+1658.754836993" watchObservedRunningTime="2026-11-03 13:32:56.746376718 +0000 UTC m=+1658.757064960" Nov 03 13:32:56.995287 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:56.995244 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74" Nov 03 13:32:57.997350 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:57.997315 5979 patch_prober.go:29] interesting pod/e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Nov 03 13:32:57.997869 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:57.997377 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74" podUID="22fad5ad-56d4-4d01-944f-a5db77a4a3ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Nov 03 13:32:58.995868 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:58.995825 5979 patch_prober.go:29] interesting pod/e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Nov 03 13:32:58.996058 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:58.995912 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74" podUID="22fad5ad-56d4-4d01-944f-a5db77a4a3ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Nov 03 13:32:59.995885 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:59.995832 5979 patch_prober.go:29] interesting pod/e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Nov 03 13:32:59.996323 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:32:59.995955 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74" podUID="22fad5ad-56d4-4d01-944f-a5db77a4a3ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Nov 03 13:33:00.997194 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:00.997154 5979 patch_prober.go:29] interesting pod/e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Nov 03 13:33:00.997834 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:00.997801 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74" podUID="22fad5ad-56d4-4d01-944f-a5db77a4a3ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Nov 03 13:33:01.995645 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:01.995607 5979 patch_prober.go:29] interesting pod/e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Nov 03 13:33:01.995861 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:01.995685 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74" podUID="22fad5ad-56d4-4d01-944f-a5db77a4a3ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Nov 03 13:33:02.069832 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:02.069791 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cfe580d-p9kpx"] Nov 03 13:33:02.105855 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:02.105820 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cfe580d-p9kpx"] Nov 03 13:33:02.116740 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:02.116712 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d24c04d4-388d-4eea-8aa0-2d14c245b017" path="/var/lib/kubelet/pods/d24c04d4-388d-4eea-8aa0-2d14c245b017/volumes" Nov 03 13:33:03.401071 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:03.401036 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74" Nov 03 13:33:03.401512 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:03.401449 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74" Nov 03 13:33:03.402236 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:03.402217 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74" Nov 03 13:33:07.235796 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:33:07.235741 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:33:09.115044 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:09.115013 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7603-76227f"] Nov 03 13:33:09.119174 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:09.119059 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7603-76227f" Nov 03 13:33:09.132398 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:09.132371 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lczmh\" (UniqueName: \"kubernetes.io/projected/014b8894-0c63-40c5-a6f3-eff36e867fbf-kube-api-access-lczmh\") pod \"test-curl-7603-76227f\" (UID: \"014b8894-0c63-40c5-a6f3-eff36e867fbf\") " pod="opendatahub/test-curl-7603-76227f" Nov 03 13:33:09.203523 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:09.192595 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7603-76227f"] Nov 03 13:33:09.221237 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:09.221208 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-kmbtk\"" Nov 03 13:33:09.233792 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:09.233762 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lczmh\" (UniqueName: \"kubernetes.io/projected/014b8894-0c63-40c5-a6f3-eff36e867fbf-kube-api-access-lczmh\") pod \"test-curl-7603-76227f\" (UID: \"014b8894-0c63-40c5-a6f3-eff36e867fbf\") " pod="opendatahub/test-curl-7603-76227f" Nov 03 13:33:09.293591 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:09.293463 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lczmh\" (UniqueName: \"kubernetes.io/projected/014b8894-0c63-40c5-a6f3-eff36e867fbf-kube-api-access-lczmh\") pod \"test-curl-7603-76227f\" (UID: \"014b8894-0c63-40c5-a6f3-eff36e867fbf\") " pod="opendatahub/test-curl-7603-76227f" Nov 03 13:33:09.432770 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:09.432741 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7603-76227f" Nov 03 13:33:09.917431 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:09.917389 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7603-76227f"] Nov 03 13:33:09.924151 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:33:09.924116 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod014b8894_0c63_40c5_a6f3_eff36e867fbf.slice/crio-691e9e9a38d39796910f34455f4ef0cfbfec45d26e18c2dc5eec1c3df9008c5e WatchSource:0}: Error finding container 691e9e9a38d39796910f34455f4ef0cfbfec45d26e18c2dc5eec1c3df9008c5e: Status 404 returned error can't find the container with id 691e9e9a38d39796910f34455f4ef0cfbfec45d26e18c2dc5eec1c3df9008c5e Nov 03 13:33:10.821110 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:10.821055 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7603-76227f" event={"ID":"014b8894-0c63-40c5-a6f3-eff36e867fbf","Type":"ContainerStarted","Data":"691e9e9a38d39796910f34455f4ef0cfbfec45d26e18c2dc5eec1c3df9008c5e"} Nov 03 13:33:13.857228 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:13.857142 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7603-76227f" event={"ID":"014b8894-0c63-40c5-a6f3-eff36e867fbf","Type":"ContainerStarted","Data":"5b14b39d49e5b05bfa477d2a06d74c11d5e09290f92bdcda7a77ec81fc088a5e"} Nov 03 13:33:13.963410 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:13.963352 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7603-76227f" podStartSLOduration=1.41693307 podStartE2EDuration="4.963334153s" totalImagesPullingTime="3.546401083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:33:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:33:13.962218527 +0000 UTC m=+1675.972906769" watchObservedRunningTime="2026-11-03 13:33:13.963334153 +0000 UTC m=+1675.974022396" Nov 03 13:33:15.514574 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:15.514530 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7603-76227f"] Nov 03 13:33:15.517787 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:15.517756 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7603-76227f"] Nov 03 13:33:15.872061 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:15.871971 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7603-76227f" podUID="014b8894-0c63-40c5-a6f3-eff36e867fbf" containerName="test-curl-7603-76227f" containerID="cri-o://5b14b39d49e5b05bfa477d2a06d74c11d5e09290f92bdcda7a77ec81fc088a5e" gracePeriod=2 Nov 03 13:33:15.920174 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:33:15.920125 5979 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7603-76227f\" is forbidden: User \"system:node:ip-10-0-130-198.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-198.us-west-2.compute.internal' and this object" podUID="014b8894-0c63-40c5-a6f3-eff36e867fbf" pod="opendatahub/test-curl-7603-76227f" Nov 03 13:33:16.358001 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:16.357963 5979 patch_prober.go:29] interesting pod/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": read tcp 10.134.0.2:50870->10.134.0.145:15021: read: connection reset by peer" start-of-body= Nov 03 13:33:16.358201 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:16.358045 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg" podUID="ec489f9b-9c50-45d9-aefb-1da511837bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": read tcp 10.134.0.2:50870->10.134.0.145:15021: read: connection reset by peer" Nov 03 13:33:16.880185 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:16.880151 5979 generic.go:356] "Generic (PLEG): container finished" podID="80fa8184-94c4-4ce1-8354-92681a69a6bf" containerID="4d964944b8213e907ff08f64d1e48efce13f570076c250aa322688d326ad21c3" exitCode=0 Nov 03 13:33:16.880185 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:16.880224 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s" event={"ID":"80fa8184-94c4-4ce1-8354-92681a69a6bf","Type":"ContainerDied","Data":"4d964944b8213e907ff08f64d1e48efce13f570076c250aa322688d326ad21c3"} Nov 03 13:33:16.880780 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:16.880728 5979 scope.go:122] "RemoveContainer" containerID="4d964944b8213e907ff08f64d1e48efce13f570076c250aa322688d326ad21c3" Nov 03 13:33:16.882143 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:16.882120 5979 generic.go:356] "Generic (PLEG): container finished" podID="ec489f9b-9c50-45d9-aefb-1da511837bae" containerID="899494b34430f7ca67b7b29888b402f22dd9dcf680dbf73212bb456dd2c2b313" exitCode=0 Nov 03 13:33:16.882281 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:16.882176 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg" event={"ID":"ec489f9b-9c50-45d9-aefb-1da511837bae","Type":"ContainerDied","Data":"899494b34430f7ca67b7b29888b402f22dd9dcf680dbf73212bb456dd2c2b313"} Nov 03 13:33:16.882359 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:16.882304 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:33:16.882428 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:16.882368 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:33:16.882428 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:16.882401 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:33:16.882627 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:16.882610 5979 scope.go:122] "RemoveContainer" containerID="899494b34430f7ca67b7b29888b402f22dd9dcf680dbf73212bb456dd2c2b313" Nov 03 13:33:16.884116 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:16.884090 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:33:16.884196 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:16.884162 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:33:16.884239 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:16.884206 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:33:17.893879 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:17.893852 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s" event={"ID":"80fa8184-94c4-4ce1-8354-92681a69a6bf","Type":"ContainerStarted","Data":"03374afcced2e1fa408e68b3ec85774ff85263d64eef1a300e9ff4c12ecc9ea9"} Nov 03 13:33:17.895875 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:17.895854 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg" event={"ID":"ec489f9b-9c50-45d9-aefb-1da511837bae","Type":"ContainerStarted","Data":"91997c5037e74bd7a546bbfe607c64651bf1160f9c7e914de4625a68f86045a6"} Nov 03 13:33:18.192537 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:18.192509 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg" Nov 03 13:33:18.192837 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:18.192810 5979 patch_prober.go:29] interesting pod/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 03 13:33:18.192895 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:18.192874 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg" podUID="ec489f9b-9c50-45d9-aefb-1da511837bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 03 13:33:18.214009 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:18.213982 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7603-76227f" Nov 03 13:33:18.259622 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:18.259585 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s" Nov 03 13:33:18.261743 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:33:18.261711 5979 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7603-76227f\" is forbidden: User \"system:node:ip-10-0-130-198.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-198.us-west-2.compute.internal' and this object" podUID="014b8894-0c63-40c5-a6f3-eff36e867fbf" pod="opendatahub/test-curl-7603-76227f" Nov 03 13:33:18.385002 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:18.383527 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/014b8894-0c63-40c5-a6f3-eff36e867fbf-kube-api-access-lczmh\" (UniqueName: \"kubernetes.io/projected/014b8894-0c63-40c5-a6f3-eff36e867fbf-kube-api-access-lczmh\") pod \"014b8894-0c63-40c5-a6f3-eff36e867fbf\" (UID: \"014b8894-0c63-40c5-a6f3-eff36e867fbf\") " Nov 03 13:33:18.388581 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:18.387626 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/014b8894-0c63-40c5-a6f3-eff36e867fbf-kube-api-access-lczmh" pod "014b8894-0c63-40c5-a6f3-eff36e867fbf" (UID: "014b8894-0c63-40c5-a6f3-eff36e867fbf"). InnerVolumeSpecName "kube-api-access-lczmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:33:18.489549 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:18.489248 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lczmh\" (UniqueName: \"kubernetes.io/projected/014b8894-0c63-40c5-a6f3-eff36e867fbf-kube-api-access-lczmh\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:18.830358 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:18.830283 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s" Nov 03 13:33:18.831710 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:18.831685 5979 patch_prober.go:29] interesting pod/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Nov 03 13:33:18.831818 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:18.831738 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s" podUID="80fa8184-94c4-4ce1-8354-92681a69a6bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Nov 03 13:33:18.905637 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:18.905595 5979 generic.go:356] "Generic (PLEG): container finished" podID="014b8894-0c63-40c5-a6f3-eff36e867fbf" containerID="5b14b39d49e5b05bfa477d2a06d74c11d5e09290f92bdcda7a77ec81fc088a5e" exitCode=137 Nov 03 13:33:18.906028 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:18.905923 5979 scope.go:122] "RemoveContainer" containerID="5b14b39d49e5b05bfa477d2a06d74c11d5e09290f92bdcda7a77ec81fc088a5e" Nov 03 13:33:18.906076 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:18.906052 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7603-76227f" Nov 03 13:33:18.917015 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:18.916997 5979 scope.go:122] "RemoveContainer" containerID="5b14b39d49e5b05bfa477d2a06d74c11d5e09290f92bdcda7a77ec81fc088a5e" Nov 03 13:33:18.917311 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:33:18.917288 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b14b39d49e5b05bfa477d2a06d74c11d5e09290f92bdcda7a77ec81fc088a5e\": container with ID starting with 5b14b39d49e5b05bfa477d2a06d74c11d5e09290f92bdcda7a77ec81fc088a5e not found: ID does not exist" containerID="5b14b39d49e5b05bfa477d2a06d74c11d5e09290f92bdcda7a77ec81fc088a5e" Nov 03 13:33:18.917360 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:18.917320 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b14b39d49e5b05bfa477d2a06d74c11d5e09290f92bdcda7a77ec81fc088a5e"} err="failed to get container status \"5b14b39d49e5b05bfa477d2a06d74c11d5e09290f92bdcda7a77ec81fc088a5e\": rpc error: code = NotFound desc = could not find container \"5b14b39d49e5b05bfa477d2a06d74c11d5e09290f92bdcda7a77ec81fc088a5e\": container with ID starting with 5b14b39d49e5b05bfa477d2a06d74c11d5e09290f92bdcda7a77ec81fc088a5e not found: ID does not exist" Nov 03 13:33:18.968497 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:33:18.959745 5979 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7603-76227f\" is forbidden: User \"system:node:ip-10-0-130-198.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-198.us-west-2.compute.internal' and this object" podUID="014b8894-0c63-40c5-a6f3-eff36e867fbf" pod="opendatahub/test-curl-7603-76227f" Nov 03 13:33:19.007609 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:33:19.007564 5979 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7603-76227f\" is forbidden: User \"system:node:ip-10-0-130-198.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-198.us-west-2.compute.internal' and this object" podUID="014b8894-0c63-40c5-a6f3-eff36e867fbf" pod="opendatahub/test-curl-7603-76227f" Nov 03 13:33:19.047495 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:19.044411 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37ae8021-5bwfl"] Nov 03 13:33:19.075493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:19.075152 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37ae8021-5bwfl"] Nov 03 13:33:19.092050 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:19.089488 5979 scope.go:122] "RemoveContainer" containerID="c2b839e72b5321a77bd94d966ff62660f84ef96187e9e109091b253d0dcf4beb" Nov 03 13:33:19.197497 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:19.196924 5979 patch_prober.go:29] interesting pod/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 03 13:33:19.197497 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:19.196982 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg" podUID="ec489f9b-9c50-45d9-aefb-1da511837bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 03 13:33:19.829750 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:19.829714 5979 patch_prober.go:29] interesting pod/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Nov 03 13:33:19.829954 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:19.829776 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s" podUID="80fa8184-94c4-4ce1-8354-92681a69a6bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Nov 03 13:33:20.118718 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:20.118646 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="014b8894-0c63-40c5-a6f3-eff36e867fbf" path="/var/lib/kubelet/pods/014b8894-0c63-40c5-a6f3-eff36e867fbf/volumes" Nov 03 13:33:20.119126 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:20.119092 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adbe9b58-f764-4b66-bf64-fcc44f186a4d" path="/var/lib/kubelet/pods/adbe9b58-f764-4b66-bf64-fcc44f186a4d/volumes" Nov 03 13:33:20.193916 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:20.193877 5979 patch_prober.go:29] interesting pod/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 03 13:33:20.194194 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:20.194164 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg" podUID="ec489f9b-9c50-45d9-aefb-1da511837bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 03 13:33:20.830262 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:20.830215 5979 patch_prober.go:29] interesting pod/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Nov 03 13:33:20.830457 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:20.830270 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s" podUID="80fa8184-94c4-4ce1-8354-92681a69a6bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Nov 03 13:33:21.192947 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:21.192908 5979 patch_prober.go:29] interesting pod/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 03 13:33:21.193560 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:21.193531 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg" podUID="ec489f9b-9c50-45d9-aefb-1da511837bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 03 13:33:21.834268 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:21.834228 5979 patch_prober.go:29] interesting pod/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Nov 03 13:33:21.834584 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:21.834542 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s" podUID="80fa8184-94c4-4ce1-8354-92681a69a6bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Nov 03 13:33:22.081866 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:22.081823 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:33:22.192975 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:22.192931 5979 patch_prober.go:29] interesting pod/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 03 13:33:22.193607 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:22.193572 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg" podUID="ec489f9b-9c50-45d9-aefb-1da511837bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 03 13:33:22.829950 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:22.829914 5979 patch_prober.go:29] interesting pod/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Nov 03 13:33:22.830125 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:22.829974 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s" podUID="80fa8184-94c4-4ce1-8354-92681a69a6bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Nov 03 13:33:23.194922 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:23.194877 5979 patch_prober.go:29] interesting pod/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Nov 03 13:33:23.214414 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:23.214373 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg" podUID="ec489f9b-9c50-45d9-aefb-1da511837bae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Nov 03 13:33:23.829828 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:23.829792 5979 patch_prober.go:29] interesting pod/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body= Nov 03 13:33:23.830023 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:23.829846 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s" podUID="80fa8184-94c4-4ce1-8354-92681a69a6bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" Nov 03 13:33:24.198140 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:24.198108 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg" Nov 03 13:33:24.198570 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:24.198502 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg" Nov 03 13:33:24.199238 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:24.199219 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg" Nov 03 13:33:24.834452 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:24.834417 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s" Nov 03 13:33:24.835629 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:24.835610 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s" Nov 03 13:33:33.074797 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:33.074704 5979 generic.go:356] "Generic (PLEG): container finished" podID="80f447c3-327a-4726-b976-8ce7adcd92fa" containerID="0d6b04afa5b09b783bbb268c1ce484933b3ea0b2010040be34269f66ef17ef95" exitCode=0 Nov 03 13:33:33.074797 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:33.074742 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s" event={"ID":"80f447c3-327a-4726-b976-8ce7adcd92fa","Type":"ContainerDied","Data":"0d6b04afa5b09b783bbb268c1ce484933b3ea0b2010040be34269f66ef17ef95"} Nov 03 13:33:33.075239 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:33.075115 5979 scope.go:122] "RemoveContainer" containerID="0d6b04afa5b09b783bbb268c1ce484933b3ea0b2010040be34269f66ef17ef95" Nov 03 13:33:33.076683 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:33.076631 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:33:33.076813 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:33.076723 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:33:33.076813 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:33.076797 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:33:34.047642 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:34.047611 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bb55afd-ch2rj"] Nov 03 13:33:34.083503 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:34.083459 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s" event={"ID":"80f447c3-327a-4726-b976-8ce7adcd92fa","Type":"ContainerStarted","Data":"c7db051d1389b916fd8cefea8d7b19b247ad25bcbba96a1755555a3a1888375e"} Nov 03 13:33:34.093745 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:34.093717 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1bb55afd-ch2rj"] Nov 03 13:33:34.116759 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:34.116727 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23cdc8aa-735c-49b4-8952-3750dddfab79" path="/var/lib/kubelet/pods/23cdc8aa-735c-49b4-8952-3750dddfab79/volumes" Nov 03 13:33:34.409299 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:34.409258 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s" Nov 03 13:33:34.409558 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:34.409509 5979 patch_prober.go:29] interesting pod/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 03 13:33:34.409651 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:34.409586 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s" podUID="80f447c3-327a-4726-b976-8ce7adcd92fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 03 13:33:35.408897 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:35.408861 5979 patch_prober.go:29] interesting pod/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 03 13:33:35.409321 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:35.408928 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s" podUID="80f447c3-327a-4726-b976-8ce7adcd92fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 03 13:33:36.408834 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:36.408798 5979 patch_prober.go:29] interesting pod/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 03 13:33:36.409052 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:36.408860 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s" podUID="80f447c3-327a-4726-b976-8ce7adcd92fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 03 13:33:37.234420 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:33:37.234380 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:33:37.408811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:37.408775 5979 patch_prober.go:29] interesting pod/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 03 13:33:37.408990 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:37.408829 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s" podUID="80f447c3-327a-4726-b976-8ce7adcd92fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 03 13:33:38.414032 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:38.413987 5979 patch_prober.go:29] interesting pod/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 03 13:33:38.425623 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:38.425574 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s" podUID="80f447c3-327a-4726-b976-8ce7adcd92fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 03 13:33:39.409678 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:39.409644 5979 patch_prober.go:29] interesting pod/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Nov 03 13:33:39.409978 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:39.409952 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s" podUID="80f447c3-327a-4726-b976-8ce7adcd92fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Nov 03 13:33:40.414098 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:40.414069 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s" Nov 03 13:33:40.414521 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:40.414500 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s" Nov 03 13:33:40.415129 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:40.415112 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s" Nov 03 13:33:43.260534 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:43.260496 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7"] Nov 03 13:33:43.260931 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:43.260707 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7" podUID="dd4c2ad4-8ff1-40ba-91e5-5e4556213933" containerName="istio-proxy" containerID="cri-o://d3068b8ae120b3170cc8fcc8321f4b752fd67bf29eb9021e5e34bced70e469da" gracePeriod=30 Nov 03 13:33:43.275780 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:43.275751 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd"] Nov 03 13:33:43.275972 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:43.275937 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd" podUID="abf1926b-6e99-4b17-846d-8c2e9c924d9f" containerName="istio-proxy" containerID="cri-o://e3e0a46dbb0df4c84f8db0b7b74830383b859441ebfbf7778e1a938b65cabcd4" gracePeriod=30 Nov 03 13:33:45.856787 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:45.856740 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s"] Nov 03 13:33:45.857400 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:45.857383 5979 cpu_manager.go:414] "containerMap: removing container" podUID="014b8894-0c63-40c5-a6f3-eff36e867fbf" containerName="test-curl-7603-76227f" Nov 03 13:33:45.857443 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:45.857402 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="014b8894-0c63-40c5-a6f3-eff36e867fbf" containerName="test-curl-7603-76227f" Nov 03 13:33:45.857558 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:45.857548 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="014b8894-0c63-40c5-a6f3-eff36e867fbf" containerName="test-curl-7603-76227f" Nov 03 13:33:45.860291 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:45.860273 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s" Nov 03 13:33:45.919010 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:45.918974 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-f6f9ed-openshift-default-dockercfg-7zlmb\"" Nov 03 13:33:45.960397 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:45.960357 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s"] Nov 03 13:33:45.993654 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:45.993622 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09fccbb2-307c-4752-a4d7-f9986eb5f61c-workload-socket\") pod \"e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s\" (UID: \"09fccbb2-307c-4752-a4d7-f9986eb5f61c\") " pod="openshift-ingress/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s" Nov 03 13:33:45.993822 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:45.993735 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09fccbb2-307c-4752-a4d7-f9986eb5f61c-istio-token\") pod \"e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s\" (UID: \"09fccbb2-307c-4752-a4d7-f9986eb5f61c\") " pod="openshift-ingress/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s" Nov 03 13:33:45.993822 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:45.993783 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r72tr\" (UniqueName: \"kubernetes.io/projected/09fccbb2-307c-4752-a4d7-f9986eb5f61c-kube-api-access-r72tr\") pod \"e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s\" (UID: \"09fccbb2-307c-4752-a4d7-f9986eb5f61c\") " pod="openshift-ingress/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s" Nov 03 13:33:45.993822 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:45.993804 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09fccbb2-307c-4752-a4d7-f9986eb5f61c-workload-certs\") pod \"e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s\" (UID: \"09fccbb2-307c-4752-a4d7-f9986eb5f61c\") " pod="openshift-ingress/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s" Nov 03 13:33:45.993956 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:45.993842 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09fccbb2-307c-4752-a4d7-f9986eb5f61c-credential-socket\") pod \"e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s\" (UID: \"09fccbb2-307c-4752-a4d7-f9986eb5f61c\") " pod="openshift-ingress/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s" Nov 03 13:33:45.993956 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:45.993870 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09fccbb2-307c-4752-a4d7-f9986eb5f61c-istiod-ca-cert\") pod \"e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s\" (UID: \"09fccbb2-307c-4752-a4d7-f9986eb5f61c\") " pod="openshift-ingress/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s" Nov 03 13:33:45.994074 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:45.994051 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09fccbb2-307c-4752-a4d7-f9986eb5f61c-istio-data\") pod \"e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s\" (UID: \"09fccbb2-307c-4752-a4d7-f9986eb5f61c\") " pod="openshift-ingress/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s" Nov 03 13:33:45.994137 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:45.994124 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09fccbb2-307c-4752-a4d7-f9986eb5f61c-istio-envoy\") pod \"e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s\" (UID: \"09fccbb2-307c-4752-a4d7-f9986eb5f61c\") " pod="openshift-ingress/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s" Nov 03 13:33:45.994180 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:45.994149 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09fccbb2-307c-4752-a4d7-f9986eb5f61c-istio-podinfo\") pod \"e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s\" (UID: \"09fccbb2-307c-4752-a4d7-f9986eb5f61c\") " pod="openshift-ingress/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s" Nov 03 13:33:46.095460 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:46.095422 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09fccbb2-307c-4752-a4d7-f9986eb5f61c-istio-data\") pod \"e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s\" (UID: \"09fccbb2-307c-4752-a4d7-f9986eb5f61c\") " pod="openshift-ingress/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s" Nov 03 13:33:46.095697 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:46.095542 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09fccbb2-307c-4752-a4d7-f9986eb5f61c-istio-envoy\") pod \"e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s\" (UID: \"09fccbb2-307c-4752-a4d7-f9986eb5f61c\") " pod="openshift-ingress/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s" Nov 03 13:33:46.095697 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:46.095565 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09fccbb2-307c-4752-a4d7-f9986eb5f61c-istio-podinfo\") pod \"e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s\" (UID: \"09fccbb2-307c-4752-a4d7-f9986eb5f61c\") " pod="openshift-ingress/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s" Nov 03 13:33:46.095697 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:46.095648 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09fccbb2-307c-4752-a4d7-f9986eb5f61c-workload-socket\") pod \"e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s\" (UID: \"09fccbb2-307c-4752-a4d7-f9986eb5f61c\") " pod="openshift-ingress/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s" Nov 03 13:33:46.095884 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:46.095724 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09fccbb2-307c-4752-a4d7-f9986eb5f61c-istio-token\") pod \"e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s\" (UID: \"09fccbb2-307c-4752-a4d7-f9986eb5f61c\") " pod="openshift-ingress/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s" Nov 03 13:33:46.095955 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:46.095869 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r72tr\" (UniqueName: \"kubernetes.io/projected/09fccbb2-307c-4752-a4d7-f9986eb5f61c-kube-api-access-r72tr\") pod \"e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s\" (UID: \"09fccbb2-307c-4752-a4d7-f9986eb5f61c\") " pod="openshift-ingress/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s" Nov 03 13:33:46.095955 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:46.095908 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09fccbb2-307c-4752-a4d7-f9986eb5f61c-istio-data\") pod \"e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s\" (UID: \"09fccbb2-307c-4752-a4d7-f9986eb5f61c\") " pod="openshift-ingress/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s" Nov 03 13:33:46.095955 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:46.095927 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09fccbb2-307c-4752-a4d7-f9986eb5f61c-workload-certs\") pod \"e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s\" (UID: \"09fccbb2-307c-4752-a4d7-f9986eb5f61c\") " pod="openshift-ingress/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s" Nov 03 13:33:46.096127 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:46.095998 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09fccbb2-307c-4752-a4d7-f9986eb5f61c-credential-socket\") pod \"e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s\" (UID: \"09fccbb2-307c-4752-a4d7-f9986eb5f61c\") " pod="openshift-ingress/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s" Nov 03 13:33:46.096127 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:46.096028 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09fccbb2-307c-4752-a4d7-f9986eb5f61c-istiod-ca-cert\") pod \"e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s\" (UID: \"09fccbb2-307c-4752-a4d7-f9986eb5f61c\") " pod="openshift-ingress/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s" Nov 03 13:33:46.096127 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:46.096064 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09fccbb2-307c-4752-a4d7-f9986eb5f61c-workload-socket\") pod \"e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s\" (UID: \"09fccbb2-307c-4752-a4d7-f9986eb5f61c\") " pod="openshift-ingress/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s" Nov 03 13:33:46.096404 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:46.096233 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09fccbb2-307c-4752-a4d7-f9986eb5f61c-workload-certs\") pod \"e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s\" (UID: \"09fccbb2-307c-4752-a4d7-f9986eb5f61c\") " pod="openshift-ingress/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s" Nov 03 13:33:46.096404 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:46.096314 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09fccbb2-307c-4752-a4d7-f9986eb5f61c-credential-socket\") pod \"e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s\" (UID: \"09fccbb2-307c-4752-a4d7-f9986eb5f61c\") " pod="openshift-ingress/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s" Nov 03 13:33:46.096738 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:46.096717 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09fccbb2-307c-4752-a4d7-f9986eb5f61c-istiod-ca-cert\") pod \"e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s\" (UID: \"09fccbb2-307c-4752-a4d7-f9986eb5f61c\") " pod="openshift-ingress/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s" Nov 03 13:33:46.097958 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:46.097937 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09fccbb2-307c-4752-a4d7-f9986eb5f61c-istio-envoy\") pod \"e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s\" (UID: \"09fccbb2-307c-4752-a4d7-f9986eb5f61c\") " pod="openshift-ingress/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s" Nov 03 13:33:46.098348 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:46.098325 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09fccbb2-307c-4752-a4d7-f9986eb5f61c-istio-podinfo\") pod \"e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s\" (UID: \"09fccbb2-307c-4752-a4d7-f9986eb5f61c\") " pod="openshift-ingress/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s" Nov 03 13:33:46.171493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:46.170557 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r72tr\" (UniqueName: \"kubernetes.io/projected/09fccbb2-307c-4752-a4d7-f9986eb5f61c-kube-api-access-r72tr\") pod \"e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s\" (UID: \"09fccbb2-307c-4752-a4d7-f9986eb5f61c\") " pod="openshift-ingress/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s" Nov 03 13:33:46.179493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:46.172012 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09fccbb2-307c-4752-a4d7-f9986eb5f61c-istio-token\") pod \"e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s\" (UID: \"09fccbb2-307c-4752-a4d7-f9986eb5f61c\") " pod="openshift-ingress/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s" Nov 03 13:33:46.179493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:46.179367 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s" Nov 03 13:33:46.627970 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:46.627118 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s"] Nov 03 13:33:46.633405 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:33:46.633376 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09fccbb2_307c_4752_a4d7_f9986eb5f61c.slice/crio-d061e9ab10270fb74cbf3d8a074bc5530062ae4e2a89642de1b27d7eb93ee166 WatchSource:0}: Error finding container d061e9ab10270fb74cbf3d8a074bc5530062ae4e2a89642de1b27d7eb93ee166: Status 404 returned error can't find the container with id d061e9ab10270fb74cbf3d8a074bc5530062ae4e2a89642de1b27d7eb93ee166 Nov 03 13:33:46.636160 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:46.636123 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:33:46.636272 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:46.636192 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:33:46.636272 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:46.636221 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:33:47.212025 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:47.211984 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s" event={"ID":"09fccbb2-307c-4752-a4d7-f9986eb5f61c","Type":"ContainerStarted","Data":"5188e424135679f01285add632aa930ddcc42d7f7793c56b89a020827a80ff5c"} Nov 03 13:33:47.212539 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:47.212045 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s" event={"ID":"09fccbb2-307c-4752-a4d7-f9986eb5f61c","Type":"ContainerStarted","Data":"d061e9ab10270fb74cbf3d8a074bc5530062ae4e2a89642de1b27d7eb93ee166"} Nov 03 13:33:47.327892 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:47.327842 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s" podStartSLOduration=2.327827582 podStartE2EDuration="2.327827582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:33:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:33:47.325930632 +0000 UTC m=+1709.336618887" watchObservedRunningTime="2026-11-03 13:33:47.327827582 +0000 UTC m=+1709.338515824" Nov 03 13:33:48.179997 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:48.179963 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s" Nov 03 13:33:48.185104 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:48.185080 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s" Nov 03 13:33:48.218523 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:48.218463 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s" Nov 03 13:33:48.219437 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:48.219415 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s" Nov 03 13:33:48.600131 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:48.600102 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7" Nov 03 13:33:48.614687 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:48.614660 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd" Nov 03 13:33:48.736245 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:48.736143 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abf1926b-6e99-4b17-846d-8c2e9c924d9f-kube-api-access-4shkz\" (UniqueName: \"kubernetes.io/projected/abf1926b-6e99-4b17-846d-8c2e9c924d9f-kube-api-access-4shkz\") pod \"abf1926b-6e99-4b17-846d-8c2e9c924d9f\" (UID: \"abf1926b-6e99-4b17-846d-8c2e9c924d9f\") " Nov 03 13:33:48.736245 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:48.736194 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-istio-envoy\") pod \"dd4c2ad4-8ff1-40ba-91e5-5e4556213933\" (UID: \"dd4c2ad4-8ff1-40ba-91e5-5e4556213933\") " Nov 03 13:33:48.736512 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:48.736328 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abf1926b-6e99-4b17-846d-8c2e9c924d9f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abf1926b-6e99-4b17-846d-8c2e9c924d9f-workload-certs\") pod \"abf1926b-6e99-4b17-846d-8c2e9c924d9f\" (UID: \"abf1926b-6e99-4b17-846d-8c2e9c924d9f\") " Nov 03 13:33:48.736512 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:48.736411 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-istio-data\") pod \"dd4c2ad4-8ff1-40ba-91e5-5e4556213933\" (UID: \"dd4c2ad4-8ff1-40ba-91e5-5e4556213933\") " Nov 03 13:33:48.736649 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:48.736566 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-credential-socket\") pod \"dd4c2ad4-8ff1-40ba-91e5-5e4556213933\" (UID: \"dd4c2ad4-8ff1-40ba-91e5-5e4556213933\") " Nov 03 13:33:48.736649 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:48.736627 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abf1926b-6e99-4b17-846d-8c2e9c924d9f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf1926b-6e99-4b17-846d-8c2e9c924d9f-credential-socket\") pod \"abf1926b-6e99-4b17-846d-8c2e9c924d9f\" (UID: \"abf1926b-6e99-4b17-846d-8c2e9c924d9f\") " Nov 03 13:33:48.736761 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:48.736659 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-istiod-ca-cert\") pod \"dd4c2ad4-8ff1-40ba-91e5-5e4556213933\" (UID: \"dd4c2ad4-8ff1-40ba-91e5-5e4556213933\") " Nov 03 13:33:48.736761 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:48.736740 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-kube-api-access-kg29v\" (UniqueName: \"kubernetes.io/projected/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-kube-api-access-kg29v\") pod \"dd4c2ad4-8ff1-40ba-91e5-5e4556213933\" (UID: \"dd4c2ad4-8ff1-40ba-91e5-5e4556213933\") " Nov 03 13:33:48.736874 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:48.736761 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abf1926b-6e99-4b17-846d-8c2e9c924d9f-workload-certs" pod "abf1926b-6e99-4b17-846d-8c2e9c924d9f" (UID: "abf1926b-6e99-4b17-846d-8c2e9c924d9f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:33:48.736874 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:48.736772 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abf1926b-6e99-4b17-846d-8c2e9c924d9f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abf1926b-6e99-4b17-846d-8c2e9c924d9f-istio-data\") pod \"abf1926b-6e99-4b17-846d-8c2e9c924d9f\" (UID: \"abf1926b-6e99-4b17-846d-8c2e9c924d9f\") " Nov 03 13:33:48.736874 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:48.736867 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-istio-podinfo\") pod \"dd4c2ad4-8ff1-40ba-91e5-5e4556213933\" (UID: \"dd4c2ad4-8ff1-40ba-91e5-5e4556213933\") " Nov 03 13:33:48.737818 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:48.736908 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abf1926b-6e99-4b17-846d-8c2e9c924d9f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abf1926b-6e99-4b17-846d-8c2e9c924d9f-istiod-ca-cert\") pod \"abf1926b-6e99-4b17-846d-8c2e9c924d9f\" (UID: \"abf1926b-6e99-4b17-846d-8c2e9c924d9f\") " Nov 03 13:33:48.737818 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:48.736948 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-workload-socket\") pod \"dd4c2ad4-8ff1-40ba-91e5-5e4556213933\" (UID: \"dd4c2ad4-8ff1-40ba-91e5-5e4556213933\") " Nov 03 13:33:48.737818 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:48.736983 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abf1926b-6e99-4b17-846d-8c2e9c924d9f-istio-token\" (UniqueName: \"kubernetes.io/projected/abf1926b-6e99-4b17-846d-8c2e9c924d9f-istio-token\") pod \"abf1926b-6e99-4b17-846d-8c2e9c924d9f\" (UID: \"abf1926b-6e99-4b17-846d-8c2e9c924d9f\") " Nov 03 13:33:48.737818 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:48.737026 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-istio-token\" (UniqueName: \"kubernetes.io/projected/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-istio-token\") pod \"dd4c2ad4-8ff1-40ba-91e5-5e4556213933\" (UID: \"dd4c2ad4-8ff1-40ba-91e5-5e4556213933\") " Nov 03 13:33:48.737818 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:48.737086 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abf1926b-6e99-4b17-846d-8c2e9c924d9f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abf1926b-6e99-4b17-846d-8c2e9c924d9f-istio-envoy\") pod \"abf1926b-6e99-4b17-846d-8c2e9c924d9f\" (UID: \"abf1926b-6e99-4b17-846d-8c2e9c924d9f\") " Nov 03 13:33:48.737818 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:48.737143 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/abf1926b-6e99-4b17-846d-8c2e9c924d9f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abf1926b-6e99-4b17-846d-8c2e9c924d9f-istio-podinfo\") pod \"abf1926b-6e99-4b17-846d-8c2e9c924d9f\" (UID: \"abf1926b-6e99-4b17-846d-8c2e9c924d9f\") " Nov 03 13:33:48.737818 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:48.737149 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abf1926b-6e99-4b17-846d-8c2e9c924d9f-istio-data" pod "abf1926b-6e99-4b17-846d-8c2e9c924d9f" (UID: "abf1926b-6e99-4b17-846d-8c2e9c924d9f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:33:48.737818 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:48.737210 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abf1926b-6e99-4b17-846d-8c2e9c924d9f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf1926b-6e99-4b17-846d-8c2e9c924d9f-workload-socket\") pod \"abf1926b-6e99-4b17-846d-8c2e9c924d9f\" (UID: \"abf1926b-6e99-4b17-846d-8c2e9c924d9f\") " Nov 03 13:33:48.737818 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:48.737240 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-workload-certs\") pod \"dd4c2ad4-8ff1-40ba-91e5-5e4556213933\" (UID: \"dd4c2ad4-8ff1-40ba-91e5-5e4556213933\") " Nov 03 13:33:48.737818 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:48.737358 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-istio-data" pod "dd4c2ad4-8ff1-40ba-91e5-5e4556213933" (UID: "dd4c2ad4-8ff1-40ba-91e5-5e4556213933"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:33:48.737818 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:48.737687 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-credential-socket" pod "dd4c2ad4-8ff1-40ba-91e5-5e4556213933" (UID: "dd4c2ad4-8ff1-40ba-91e5-5e4556213933"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:33:48.738427 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:48.737863 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abf1926b-6e99-4b17-846d-8c2e9c924d9f-credential-socket" pod "abf1926b-6e99-4b17-846d-8c2e9c924d9f" (UID: "abf1926b-6e99-4b17-846d-8c2e9c924d9f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:33:48.738427 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:48.738060 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abf1926b-6e99-4b17-846d-8c2e9c924d9f-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:48.738427 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:48.738084 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abf1926b-6e99-4b17-846d-8c2e9c924d9f-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:48.738427 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:48.738100 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:48.738427 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:48.738117 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:48.738427 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:48.738132 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf1926b-6e99-4b17-846d-8c2e9c924d9f-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:48.738427 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:48.738136 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-istiod-ca-cert" pod "dd4c2ad4-8ff1-40ba-91e5-5e4556213933" (UID: "dd4c2ad4-8ff1-40ba-91e5-5e4556213933"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:33:48.739245 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:48.739216 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abf1926b-6e99-4b17-846d-8c2e9c924d9f-workload-socket" pod "abf1926b-6e99-4b17-846d-8c2e9c924d9f" (UID: "abf1926b-6e99-4b17-846d-8c2e9c924d9f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:33:48.740190 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:48.739642 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-workload-socket" pod "dd4c2ad4-8ff1-40ba-91e5-5e4556213933" (UID: "dd4c2ad4-8ff1-40ba-91e5-5e4556213933"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:33:48.740190 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:48.739937 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abf1926b-6e99-4b17-846d-8c2e9c924d9f-istiod-ca-cert" pod "abf1926b-6e99-4b17-846d-8c2e9c924d9f" (UID: "abf1926b-6e99-4b17-846d-8c2e9c924d9f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:33:48.740832 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:48.740794 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-workload-certs" pod "dd4c2ad4-8ff1-40ba-91e5-5e4556213933" (UID: "dd4c2ad4-8ff1-40ba-91e5-5e4556213933"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:33:48.740968 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:48.740923 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abf1926b-6e99-4b17-846d-8c2e9c924d9f-kube-api-access-4shkz" pod "abf1926b-6e99-4b17-846d-8c2e9c924d9f" (UID: "abf1926b-6e99-4b17-846d-8c2e9c924d9f"). InnerVolumeSpecName "kube-api-access-4shkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:33:48.741194 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:48.741166 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-istio-envoy" pod "dd4c2ad4-8ff1-40ba-91e5-5e4556213933" (UID: "dd4c2ad4-8ff1-40ba-91e5-5e4556213933"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:33:48.741550 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:48.741361 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-istio-podinfo" pod "dd4c2ad4-8ff1-40ba-91e5-5e4556213933" (UID: "dd4c2ad4-8ff1-40ba-91e5-5e4556213933"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:33:48.741550 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:48.741510 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abf1926b-6e99-4b17-846d-8c2e9c924d9f-istio-token" pod "abf1926b-6e99-4b17-846d-8c2e9c924d9f" (UID: "abf1926b-6e99-4b17-846d-8c2e9c924d9f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:33:48.741550 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:48.741516 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/abf1926b-6e99-4b17-846d-8c2e9c924d9f-istio-podinfo" pod "abf1926b-6e99-4b17-846d-8c2e9c924d9f" (UID: "abf1926b-6e99-4b17-846d-8c2e9c924d9f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:33:48.741550 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:48.741538 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-kube-api-access-kg29v" pod "dd4c2ad4-8ff1-40ba-91e5-5e4556213933" (UID: "dd4c2ad4-8ff1-40ba-91e5-5e4556213933"). InnerVolumeSpecName "kube-api-access-kg29v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:33:48.741993 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:48.741975 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abf1926b-6e99-4b17-846d-8c2e9c924d9f-istio-envoy" pod "abf1926b-6e99-4b17-846d-8c2e9c924d9f" (UID: "abf1926b-6e99-4b17-846d-8c2e9c924d9f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:33:48.741993 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:48.741985 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-istio-token" pod "dd4c2ad4-8ff1-40ba-91e5-5e4556213933" (UID: "dd4c2ad4-8ff1-40ba-91e5-5e4556213933"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:33:48.839787 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:48.839750 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:48.839787 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:48.839782 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kg29v\" (UniqueName: \"kubernetes.io/projected/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-kube-api-access-kg29v\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:48.839787 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:48.839794 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:48.840102 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:48.839802 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abf1926b-6e99-4b17-846d-8c2e9c924d9f-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:48.840102 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:48.839811 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:48.840102 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:48.839820 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abf1926b-6e99-4b17-846d-8c2e9c924d9f-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:48.840102 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:48.839828 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:48.840102 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:48.839840 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abf1926b-6e99-4b17-846d-8c2e9c924d9f-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:48.840102 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:48.839853 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abf1926b-6e99-4b17-846d-8c2e9c924d9f-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:48.840102 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:48.839864 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf1926b-6e99-4b17-846d-8c2e9c924d9f-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:48.840102 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:48.839872 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:48.840102 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:48.839881 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4shkz\" (UniqueName: \"kubernetes.io/projected/abf1926b-6e99-4b17-846d-8c2e9c924d9f-kube-api-access-4shkz\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:48.840102 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:48.839889 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd4c2ad4-8ff1-40ba-91e5-5e4556213933-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:33:49.226070 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:49.226033 5979 generic.go:356] "Generic (PLEG): container finished" podID="abf1926b-6e99-4b17-846d-8c2e9c924d9f" containerID="e3e0a46dbb0df4c84f8db0b7b74830383b859441ebfbf7778e1a938b65cabcd4" exitCode=0 Nov 03 13:33:49.226600 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:49.226104 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd" Nov 03 13:33:49.226600 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:49.226118 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd" event={"ID":"abf1926b-6e99-4b17-846d-8c2e9c924d9f","Type":"ContainerDied","Data":"e3e0a46dbb0df4c84f8db0b7b74830383b859441ebfbf7778e1a938b65cabcd4"} Nov 03 13:33:49.226600 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:49.226197 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd" event={"ID":"abf1926b-6e99-4b17-846d-8c2e9c924d9f","Type":"ContainerDied","Data":"18b8b44197dff8ee96a79035599651f8948c92d7047e7cb7906b0b2633e74299"} Nov 03 13:33:49.226600 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:49.226229 5979 scope.go:122] "RemoveContainer" containerID="e3e0a46dbb0df4c84f8db0b7b74830383b859441ebfbf7778e1a938b65cabcd4" Nov 03 13:33:49.228023 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:49.227999 5979 generic.go:356] "Generic (PLEG): container finished" podID="dd4c2ad4-8ff1-40ba-91e5-5e4556213933" containerID="d3068b8ae120b3170cc8fcc8321f4b752fd67bf29eb9021e5e34bced70e469da" exitCode=0 Nov 03 13:33:49.228154 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:49.228070 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7" Nov 03 13:33:49.228154 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:49.228073 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7" event={"ID":"dd4c2ad4-8ff1-40ba-91e5-5e4556213933","Type":"ContainerDied","Data":"d3068b8ae120b3170cc8fcc8321f4b752fd67bf29eb9021e5e34bced70e469da"} Nov 03 13:33:49.228154 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:49.228114 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7" event={"ID":"dd4c2ad4-8ff1-40ba-91e5-5e4556213933","Type":"ContainerDied","Data":"e106c4d7857b264c76ea934db802a9a1c370bc731e7884ae6fbb2dd46dae97e6"} Nov 03 13:33:49.238633 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:49.238543 5979 scope.go:122] "RemoveContainer" containerID="e3e0a46dbb0df4c84f8db0b7b74830383b859441ebfbf7778e1a938b65cabcd4" Nov 03 13:33:49.238980 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:33:49.238852 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3e0a46dbb0df4c84f8db0b7b74830383b859441ebfbf7778e1a938b65cabcd4\": container with ID starting with e3e0a46dbb0df4c84f8db0b7b74830383b859441ebfbf7778e1a938b65cabcd4 not found: ID does not exist" containerID="e3e0a46dbb0df4c84f8db0b7b74830383b859441ebfbf7778e1a938b65cabcd4" Nov 03 13:33:49.238980 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:49.238891 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3e0a46dbb0df4c84f8db0b7b74830383b859441ebfbf7778e1a938b65cabcd4"} err="failed to get container status \"e3e0a46dbb0df4c84f8db0b7b74830383b859441ebfbf7778e1a938b65cabcd4\": rpc error: code = NotFound desc = could not find container \"e3e0a46dbb0df4c84f8db0b7b74830383b859441ebfbf7778e1a938b65cabcd4\": container with ID starting with e3e0a46dbb0df4c84f8db0b7b74830383b859441ebfbf7778e1a938b65cabcd4 not found: ID does not exist" Nov 03 13:33:49.238980 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:49.238914 5979 scope.go:122] "RemoveContainer" containerID="d3068b8ae120b3170cc8fcc8321f4b752fd67bf29eb9021e5e34bced70e469da" Nov 03 13:33:49.249398 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:49.249382 5979 scope.go:122] "RemoveContainer" containerID="7da7678ef1c981b9e5f62d2acaf668e2134b7db5116739fde82b81dffa45b13d" Nov 03 13:33:49.260948 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:49.260916 5979 scope.go:122] "RemoveContainer" containerID="d3068b8ae120b3170cc8fcc8321f4b752fd67bf29eb9021e5e34bced70e469da" Nov 03 13:33:49.261259 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:33:49.261236 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3068b8ae120b3170cc8fcc8321f4b752fd67bf29eb9021e5e34bced70e469da\": container with ID starting with d3068b8ae120b3170cc8fcc8321f4b752fd67bf29eb9021e5e34bced70e469da not found: ID does not exist" containerID="d3068b8ae120b3170cc8fcc8321f4b752fd67bf29eb9021e5e34bced70e469da" Nov 03 13:33:49.261346 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:49.261267 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3068b8ae120b3170cc8fcc8321f4b752fd67bf29eb9021e5e34bced70e469da"} err="failed to get container status \"d3068b8ae120b3170cc8fcc8321f4b752fd67bf29eb9021e5e34bced70e469da\": rpc error: code = NotFound desc = could not find container \"d3068b8ae120b3170cc8fcc8321f4b752fd67bf29eb9021e5e34bced70e469da\": container with ID starting with d3068b8ae120b3170cc8fcc8321f4b752fd67bf29eb9021e5e34bced70e469da not found: ID does not exist" Nov 03 13:33:49.261346 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:49.261285 5979 scope.go:122] "RemoveContainer" containerID="7da7678ef1c981b9e5f62d2acaf668e2134b7db5116739fde82b81dffa45b13d" Nov 03 13:33:49.261594 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:33:49.261575 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7da7678ef1c981b9e5f62d2acaf668e2134b7db5116739fde82b81dffa45b13d\": container with ID starting with 7da7678ef1c981b9e5f62d2acaf668e2134b7db5116739fde82b81dffa45b13d not found: ID does not exist" containerID="7da7678ef1c981b9e5f62d2acaf668e2134b7db5116739fde82b81dffa45b13d" Nov 03 13:33:49.261639 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:49.261601 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7da7678ef1c981b9e5f62d2acaf668e2134b7db5116739fde82b81dffa45b13d"} err="failed to get container status \"7da7678ef1c981b9e5f62d2acaf668e2134b7db5116739fde82b81dffa45b13d\": rpc error: code = NotFound desc = could not find container \"7da7678ef1c981b9e5f62d2acaf668e2134b7db5116739fde82b81dffa45b13d\": container with ID starting with 7da7678ef1c981b9e5f62d2acaf668e2134b7db5116739fde82b81dffa45b13d not found: ID does not exist" Nov 03 13:33:49.381629 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:49.381591 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7"] Nov 03 13:33:49.427942 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:49.427895 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-cdnw7"] Nov 03 13:33:49.490600 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:49.490509 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd"] Nov 03 13:33:49.495402 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:49.495374 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-28d87c-openshift-default-59978677dc-ts2fd"] Nov 03 13:33:50.117130 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:50.117099 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abf1926b-6e99-4b17-846d-8c2e9c924d9f" path="/var/lib/kubelet/pods/abf1926b-6e99-4b17-846d-8c2e9c924d9f/volumes" Nov 03 13:33:50.117623 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:50.117609 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd4c2ad4-8ff1-40ba-91e5-5e4556213933" path="/var/lib/kubelet/pods/dd4c2ad4-8ff1-40ba-91e5-5e4556213933/volumes" Nov 03 13:33:54.585114 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:54.585081 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr"] Nov 03 13:33:54.586028 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:54.586004 5979 cpu_manager.go:414] "containerMap: removing container" podUID="abf1926b-6e99-4b17-846d-8c2e9c924d9f" containerName="istio-proxy" Nov 03 13:33:54.586028 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:54.586028 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abf1926b-6e99-4b17-846d-8c2e9c924d9f" containerName="istio-proxy" Nov 03 13:33:54.586255 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:54.586042 5979 cpu_manager.go:414] "containerMap: removing container" podUID="dd4c2ad4-8ff1-40ba-91e5-5e4556213933" containerName="istio-proxy" Nov 03 13:33:54.586255 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:54.586050 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd4c2ad4-8ff1-40ba-91e5-5e4556213933" containerName="istio-proxy" Nov 03 13:33:54.586255 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:54.586097 5979 cpu_manager.go:414] "containerMap: removing container" podUID="dd4c2ad4-8ff1-40ba-91e5-5e4556213933" containerName="istio-proxy" Nov 03 13:33:54.586255 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:54.586107 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd4c2ad4-8ff1-40ba-91e5-5e4556213933" containerName="istio-proxy" Nov 03 13:33:54.586578 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:54.586282 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="abf1926b-6e99-4b17-846d-8c2e9c924d9f" containerName="istio-proxy" Nov 03 13:33:54.586578 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:54.586295 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd4c2ad4-8ff1-40ba-91e5-5e4556213933" containerName="istio-proxy" Nov 03 13:33:54.586578 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:54.586312 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd4c2ad4-8ff1-40ba-91e5-5e4556213933" containerName="istio-proxy" Nov 03 13:33:54.590564 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:54.590545 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr" Nov 03 13:33:54.641244 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:54.641217 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-5c8192-openshift-default-dockercfg-mv27x\"" Nov 03 13:33:54.693188 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:54.693155 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr"] Nov 03 13:33:54.763893 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:54.763856 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc8p2\" (UniqueName: \"kubernetes.io/projected/7fd6dca0-648c-4476-a1a5-8a7407bf598f-kube-api-access-zc8p2\") pod \"e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr\" (UID: \"7fd6dca0-648c-4476-a1a5-8a7407bf598f\") " pod="openshift-ingress/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr" Nov 03 13:33:54.764087 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:54.763915 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fd6dca0-648c-4476-a1a5-8a7407bf598f-credential-socket\") pod \"e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr\" (UID: \"7fd6dca0-648c-4476-a1a5-8a7407bf598f\") " pod="openshift-ingress/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr" Nov 03 13:33:54.764087 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:54.764038 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fd6dca0-648c-4476-a1a5-8a7407bf598f-istio-token\") pod \"e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr\" (UID: \"7fd6dca0-648c-4476-a1a5-8a7407bf598f\") " pod="openshift-ingress/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr" Nov 03 13:33:54.764205 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:54.764108 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fd6dca0-648c-4476-a1a5-8a7407bf598f-istio-data\") pod \"e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr\" (UID: \"7fd6dca0-648c-4476-a1a5-8a7407bf598f\") " pod="openshift-ingress/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr" Nov 03 13:33:54.764205 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:54.764158 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fd6dca0-648c-4476-a1a5-8a7407bf598f-istio-envoy\") pod \"e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr\" (UID: \"7fd6dca0-648c-4476-a1a5-8a7407bf598f\") " pod="openshift-ingress/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr" Nov 03 13:33:54.764291 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:54.764264 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fd6dca0-648c-4476-a1a5-8a7407bf598f-workload-certs\") pod \"e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr\" (UID: \"7fd6dca0-648c-4476-a1a5-8a7407bf598f\") " pod="openshift-ingress/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr" Nov 03 13:33:54.764291 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:54.764283 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fd6dca0-648c-4476-a1a5-8a7407bf598f-workload-socket\") pod \"e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr\" (UID: \"7fd6dca0-648c-4476-a1a5-8a7407bf598f\") " pod="openshift-ingress/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr" Nov 03 13:33:54.764366 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:54.764314 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fd6dca0-648c-4476-a1a5-8a7407bf598f-istiod-ca-cert\") pod \"e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr\" (UID: \"7fd6dca0-648c-4476-a1a5-8a7407bf598f\") " pod="openshift-ingress/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr" Nov 03 13:33:54.764429 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:54.764406 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fd6dca0-648c-4476-a1a5-8a7407bf598f-istio-podinfo\") pod \"e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr\" (UID: \"7fd6dca0-648c-4476-a1a5-8a7407bf598f\") " pod="openshift-ingress/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr" Nov 03 13:33:54.881420 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:54.877551 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fd6dca0-648c-4476-a1a5-8a7407bf598f-istio-podinfo\") pod \"e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr\" (UID: \"7fd6dca0-648c-4476-a1a5-8a7407bf598f\") " pod="openshift-ingress/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr" Nov 03 13:33:54.881789 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:54.881753 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc8p2\" (UniqueName: \"kubernetes.io/projected/7fd6dca0-648c-4476-a1a5-8a7407bf598f-kube-api-access-zc8p2\") pod \"e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr\" (UID: \"7fd6dca0-648c-4476-a1a5-8a7407bf598f\") " pod="openshift-ingress/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr" Nov 03 13:33:54.887503 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:54.881964 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fd6dca0-648c-4476-a1a5-8a7407bf598f-credential-socket\") pod \"e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr\" (UID: \"7fd6dca0-648c-4476-a1a5-8a7407bf598f\") " pod="openshift-ingress/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr" Nov 03 13:33:54.887827 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:54.887791 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fd6dca0-648c-4476-a1a5-8a7407bf598f-istio-token\") pod \"e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr\" (UID: \"7fd6dca0-648c-4476-a1a5-8a7407bf598f\") " pod="openshift-ingress/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr" Nov 03 13:33:54.888119 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:54.888098 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fd6dca0-648c-4476-a1a5-8a7407bf598f-istio-data\") pod \"e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr\" (UID: \"7fd6dca0-648c-4476-a1a5-8a7407bf598f\") " pod="openshift-ingress/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr" Nov 03 13:33:54.888300 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:54.888283 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fd6dca0-648c-4476-a1a5-8a7407bf598f-istio-envoy\") pod \"e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr\" (UID: \"7fd6dca0-648c-4476-a1a5-8a7407bf598f\") " pod="openshift-ingress/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr" Nov 03 13:33:54.888546 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:54.888521 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fd6dca0-648c-4476-a1a5-8a7407bf598f-workload-certs\") pod \"e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr\" (UID: \"7fd6dca0-648c-4476-a1a5-8a7407bf598f\") " pod="openshift-ingress/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr" Nov 03 13:33:54.888713 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:54.888686 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fd6dca0-648c-4476-a1a5-8a7407bf598f-workload-socket\") pod \"e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr\" (UID: \"7fd6dca0-648c-4476-a1a5-8a7407bf598f\") " pod="openshift-ingress/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr" Nov 03 13:33:54.888896 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:54.888877 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fd6dca0-648c-4476-a1a5-8a7407bf598f-istiod-ca-cert\") pod \"e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr\" (UID: \"7fd6dca0-648c-4476-a1a5-8a7407bf598f\") " pod="openshift-ingress/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr" Nov 03 13:33:54.894672 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:54.890591 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fd6dca0-648c-4476-a1a5-8a7407bf598f-istiod-ca-cert\") pod \"e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr\" (UID: \"7fd6dca0-648c-4476-a1a5-8a7407bf598f\") " pod="openshift-ingress/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr" Nov 03 13:33:54.894865 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:54.890835 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fd6dca0-648c-4476-a1a5-8a7407bf598f-workload-certs\") pod \"e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr\" (UID: \"7fd6dca0-648c-4476-a1a5-8a7407bf598f\") " pod="openshift-ingress/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr" Nov 03 13:33:54.894985 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:54.891052 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fd6dca0-648c-4476-a1a5-8a7407bf598f-workload-socket\") pod \"e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr\" (UID: \"7fd6dca0-648c-4476-a1a5-8a7407bf598f\") " pod="openshift-ingress/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr" Nov 03 13:33:54.895093 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:54.891613 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fd6dca0-648c-4476-a1a5-8a7407bf598f-istio-data\") pod \"e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr\" (UID: \"7fd6dca0-648c-4476-a1a5-8a7407bf598f\") " pod="openshift-ingress/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr" Nov 03 13:33:54.895179 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:54.894378 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fd6dca0-648c-4476-a1a5-8a7407bf598f-credential-socket\") pod \"e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr\" (UID: \"7fd6dca0-648c-4476-a1a5-8a7407bf598f\") " pod="openshift-ingress/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr" Nov 03 13:33:54.931283 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:54.931250 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fd6dca0-648c-4476-a1a5-8a7407bf598f-istio-podinfo\") pod \"e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr\" (UID: \"7fd6dca0-648c-4476-a1a5-8a7407bf598f\") " pod="openshift-ingress/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr" Nov 03 13:33:54.940217 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:54.937086 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fd6dca0-648c-4476-a1a5-8a7407bf598f-istio-envoy\") pod \"e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr\" (UID: \"7fd6dca0-648c-4476-a1a5-8a7407bf598f\") " pod="openshift-ingress/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr" Nov 03 13:33:54.948439 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:54.946453 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc8p2\" (UniqueName: \"kubernetes.io/projected/7fd6dca0-648c-4476-a1a5-8a7407bf598f-kube-api-access-zc8p2\") pod \"e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr\" (UID: \"7fd6dca0-648c-4476-a1a5-8a7407bf598f\") " pod="openshift-ingress/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr" Nov 03 13:33:54.955407 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:54.955345 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:33:54.956277 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:54.956238 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fd6dca0-648c-4476-a1a5-8a7407bf598f-istio-token\") pod \"e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr\" (UID: \"7fd6dca0-648c-4476-a1a5-8a7407bf598f\") " pod="openshift-ingress/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr" Nov 03 13:33:55.204390 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:55.204359 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr" Nov 03 13:33:55.634535 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:55.634507 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr"] Nov 03 13:33:55.640725 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:33:55.640697 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fd6dca0_648c_4476_a1a5_8a7407bf598f.slice/crio-f7803d2e0d6c26f3bcbb042204d6d3c571d6e7f4b852eb7c4770c0828b17e072 WatchSource:0}: Error finding container f7803d2e0d6c26f3bcbb042204d6d3c571d6e7f4b852eb7c4770c0828b17e072: Status 404 returned error can't find the container with id f7803d2e0d6c26f3bcbb042204d6d3c571d6e7f4b852eb7c4770c0828b17e072 Nov 03 13:33:55.643255 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:55.643225 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:33:55.643344 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:55.643291 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:33:55.643344 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:55.643318 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:33:56.290345 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:56.290313 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr" event={"ID":"7fd6dca0-648c-4476-a1a5-8a7407bf598f","Type":"ContainerStarted","Data":"c0696ee584a93c5517435d6824775f80ed90c756889e69f557c8b761b8295daf"} Nov 03 13:33:56.290522 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:56.290364 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr" event={"ID":"7fd6dca0-648c-4476-a1a5-8a7407bf598f","Type":"ContainerStarted","Data":"f7803d2e0d6c26f3bcbb042204d6d3c571d6e7f4b852eb7c4770c0828b17e072"} Nov 03 13:33:56.532494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:56.525758 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr" podStartSLOduration=2.52569051 podStartE2EDuration="2.52569051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:33:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:33:56.502591558 +0000 UTC m=+1718.513279802" watchObservedRunningTime="2026-11-03 13:33:56.52569051 +0000 UTC m=+1718.536378763" Nov 03 13:33:56.711493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:56.710560 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f6f9ed-667f6c778c-cq5t6"] Nov 03 13:33:56.724505 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:56.724352 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6f9ed-667f6c778c-cq5t6" Nov 03 13:33:56.773066 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:56.773031 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f6f9ed-667f6c778c-cq5t6"] Nov 03 13:33:56.774439 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:56.774411 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f6f9ed-dockercfg-5wf9p\"" Nov 03 13:33:56.774626 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:56.774483 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f6f9ed\"" Nov 03 13:33:56.828509 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:56.825175 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whrl4\" (UniqueName: \"kubernetes.io/projected/fbfb58ac-c12a-4312-a8f3-96c5bfeac157-kube-api-access-whrl4\") pod \"payload-pre-processing-e2e-ipp-a-f6f9ed-667f6c778c-cq5t6\" (UID: \"fbfb58ac-c12a-4312-a8f3-96c5bfeac157\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6f9ed-667f6c778c-cq5t6" Nov 03 13:33:56.828509 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:56.825426 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbfb58ac-c12a-4312-a8f3-96c5bfeac157-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f6f9ed-667f6c778c-cq5t6\" (UID: \"fbfb58ac-c12a-4312-a8f3-96c5bfeac157\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6f9ed-667f6c778c-cq5t6" Nov 03 13:33:56.953425 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:56.941839 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbfb58ac-c12a-4312-a8f3-96c5bfeac157-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f6f9ed-667f6c778c-cq5t6\" (UID: \"fbfb58ac-c12a-4312-a8f3-96c5bfeac157\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6f9ed-667f6c778c-cq5t6" Nov 03 13:33:56.953880 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:56.953847 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whrl4\" (UniqueName: \"kubernetes.io/projected/fbfb58ac-c12a-4312-a8f3-96c5bfeac157-kube-api-access-whrl4\") pod \"payload-pre-processing-e2e-ipp-a-f6f9ed-667f6c778c-cq5t6\" (UID: \"fbfb58ac-c12a-4312-a8f3-96c5bfeac157\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6f9ed-667f6c778c-cq5t6" Nov 03 13:33:56.955426 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:56.955397 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbfb58ac-c12a-4312-a8f3-96c5bfeac157-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f6f9ed-667f6c778c-cq5t6\" (UID: \"fbfb58ac-c12a-4312-a8f3-96c5bfeac157\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6f9ed-667f6c778c-cq5t6" Nov 03 13:33:56.963911 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:56.963813 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f6f9ed-554b464fd5-kzpbv"] Nov 03 13:33:56.989126 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:56.988566 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f6f9ed-554b464fd5-kzpbv"] Nov 03 13:33:56.989126 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:56.988672 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6f9ed-554b464fd5-kzpbv" Nov 03 13:33:57.008446 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:57.008407 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whrl4\" (UniqueName: \"kubernetes.io/projected/fbfb58ac-c12a-4312-a8f3-96c5bfeac157-kube-api-access-whrl4\") pod \"payload-pre-processing-e2e-ipp-a-f6f9ed-667f6c778c-cq5t6\" (UID: \"fbfb58ac-c12a-4312-a8f3-96c5bfeac157\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6f9ed-667f6c778c-cq5t6" Nov 03 13:33:57.041850 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:57.041816 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6f9ed-667f6c778c-cq5t6" Nov 03 13:33:57.165536 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:57.163742 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f57d7f68-a80f-4fdf-84b8-a76509ff413d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f6f9ed-554b464fd5-kzpbv\" (UID: \"f57d7f68-a80f-4fdf-84b8-a76509ff413d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f6f9ed-554b464fd5-kzpbv" Nov 03 13:33:57.165536 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:57.163896 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmtct\" (UniqueName: \"kubernetes.io/projected/f57d7f68-a80f-4fdf-84b8-a76509ff413d-kube-api-access-tmtct\") pod \"payload-processing-e2e-ipp-a-f6f9ed-554b464fd5-kzpbv\" (UID: \"f57d7f68-a80f-4fdf-84b8-a76509ff413d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f6f9ed-554b464fd5-kzpbv" Nov 03 13:33:57.212202 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:57.207860 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr" Nov 03 13:33:57.228618 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:57.228534 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr" Nov 03 13:33:57.281198 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:57.281121 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f57d7f68-a80f-4fdf-84b8-a76509ff413d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f6f9ed-554b464fd5-kzpbv\" (UID: \"f57d7f68-a80f-4fdf-84b8-a76509ff413d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f6f9ed-554b464fd5-kzpbv" Nov 03 13:33:57.281386 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:57.281284 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmtct\" (UniqueName: \"kubernetes.io/projected/f57d7f68-a80f-4fdf-84b8-a76509ff413d-kube-api-access-tmtct\") pod \"payload-processing-e2e-ipp-a-f6f9ed-554b464fd5-kzpbv\" (UID: \"f57d7f68-a80f-4fdf-84b8-a76509ff413d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f6f9ed-554b464fd5-kzpbv" Nov 03 13:33:57.281834 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:57.281814 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f57d7f68-a80f-4fdf-84b8-a76509ff413d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f6f9ed-554b464fd5-kzpbv\" (UID: \"f57d7f68-a80f-4fdf-84b8-a76509ff413d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f6f9ed-554b464fd5-kzpbv" Nov 03 13:33:57.297294 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:57.297266 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr" Nov 03 13:33:57.298333 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:57.298310 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr" Nov 03 13:33:57.337719 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:57.337688 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmtct\" (UniqueName: \"kubernetes.io/projected/f57d7f68-a80f-4fdf-84b8-a76509ff413d-kube-api-access-tmtct\") pod \"payload-processing-e2e-ipp-a-f6f9ed-554b464fd5-kzpbv\" (UID: \"f57d7f68-a80f-4fdf-84b8-a76509ff413d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f6f9ed-554b464fd5-kzpbv" Nov 03 13:33:57.415558 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:57.415521 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f6f9ed-667f6c778c-cq5t6"] Nov 03 13:33:57.421482 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:33:57.421437 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbfb58ac_c12a_4312_a8f3_96c5bfeac157.slice/crio-327e4c90b9863d1ed6eac19d09e38aa1c31813a220edcad72f0ae8cc8d930013 WatchSource:0}: Error finding container 327e4c90b9863d1ed6eac19d09e38aa1c31813a220edcad72f0ae8cc8d930013: Status 404 returned error can't find the container with id 327e4c90b9863d1ed6eac19d09e38aa1c31813a220edcad72f0ae8cc8d930013 Nov 03 13:33:57.442553 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:57.442526 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-461b2d-6466bf6b8f-48fhn"] Nov 03 13:33:57.549351 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:57.549299 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-461b2d-7d984ffbf9-262hb"] Nov 03 13:33:57.604970 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:57.604921 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6f9ed-554b464fd5-kzpbv" Nov 03 13:33:58.021269 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:58.020565 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f6f9ed-554b464fd5-kzpbv"] Nov 03 13:33:58.067093 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:33:58.067056 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf57d7f68_a80f_4fdf_84b8_a76509ff413d.slice/crio-f808b5427632bc7e267c76294b15649334c6581c51fc1532941ba4ff105697e3 WatchSource:0}: Error finding container f808b5427632bc7e267c76294b15649334c6581c51fc1532941ba4ff105697e3: Status 404 returned error can't find the container with id f808b5427632bc7e267c76294b15649334c6581c51fc1532941ba4ff105697e3 Nov 03 13:33:58.306886 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:58.306782 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6f9ed-667f6c778c-cq5t6" event={"ID":"fbfb58ac-c12a-4312-a8f3-96c5bfeac157","Type":"ContainerStarted","Data":"def8cc76a0aea8c0d663d5128cce35a9d29bda964e1393bf09470c1b4900a54b"} Nov 03 13:33:58.306886 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:58.306840 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6f9ed-667f6c778c-cq5t6" event={"ID":"fbfb58ac-c12a-4312-a8f3-96c5bfeac157","Type":"ContainerStarted","Data":"327e4c90b9863d1ed6eac19d09e38aa1c31813a220edcad72f0ae8cc8d930013"} Nov 03 13:33:58.306886 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:58.306865 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6f9ed-667f6c778c-cq5t6" Nov 03 13:33:58.308605 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:58.308579 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6f9ed-554b464fd5-kzpbv" event={"ID":"f57d7f68-a80f-4fdf-84b8-a76509ff413d","Type":"ContainerStarted","Data":"0328831c02bbf4d67aa0673d4f16ab9669b528009721c1698fd7ae56e44ac377"} Nov 03 13:33:58.308717 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:58.308622 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6f9ed-554b464fd5-kzpbv" event={"ID":"f57d7f68-a80f-4fdf-84b8-a76509ff413d","Type":"ContainerStarted","Data":"f808b5427632bc7e267c76294b15649334c6581c51fc1532941ba4ff105697e3"} Nov 03 13:33:58.308821 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:58.308810 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6f9ed-554b464fd5-kzpbv" Nov 03 13:33:58.418398 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:58.418347 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6f9ed-667f6c778c-cq5t6" podStartSLOduration=2.418333331 podStartE2EDuration="2.418333331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:33:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:33:58.415040434 +0000 UTC m=+1720.425728676" watchObservedRunningTime="2026-11-03 13:33:58.418333331 +0000 UTC m=+1720.429021580" Nov 03 13:33:58.540533 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:33:58.540354 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6f9ed-554b464fd5-kzpbv" podStartSLOduration=2.540335192 podStartE2EDuration="2.540335192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:33:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:33:58.537813089 +0000 UTC m=+1720.548501334" watchObservedRunningTime="2026-11-03 13:33:58.540335192 +0000 UTC m=+1720.551023435" Nov 03 13:34:00.061689 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:00.061606 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5295ab29-79c7656b49-gxvjq"] Nov 03 13:34:00.169127 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:00.169094 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5295ab29-6d58657867-ls4rq"] Nov 03 13:34:02.443209 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:02.443163 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-461b2d-6466bf6b8f-48fhn" podUID="bb292b8a-a577-42de-a66b-646fdf0e22b8" containerName="payload-processing" containerID="cri-o://015ca4a909b0339e1d848ea381ea24f38a1ec1f377cd2e719fdf9a303b27c8b1" gracePeriod=25 Nov 03 13:34:02.550283 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:02.550220 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-461b2d-7d984ffbf9-262hb" podUID="be6c35e9-cf38-4a4b-a393-932c02403030" containerName="payload-pre-processing" containerID="cri-o://172910e9b60ba468df1baf2ff58e9b868aa048154002fe3dab07e6a1279a9584" gracePeriod=25 Nov 03 13:34:02.552457 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:02.552427 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fa2568-794d67c965-wnxdb"] Nov 03 13:34:02.677527 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:02.673278 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fa2568-6cb8658547-kqzwp"] Nov 03 13:34:02.756745 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:02.756715 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-461b2d-6466bf6b8f-48fhn" Nov 03 13:34:02.887568 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:02.887542 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-461b2d-7d984ffbf9-262hb" Nov 03 13:34:02.902962 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:02.902928 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb292b8a-a577-42de-a66b-646fdf0e22b8-kube-api-access-v27mc\" (UniqueName: \"kubernetes.io/projected/bb292b8a-a577-42de-a66b-646fdf0e22b8-kube-api-access-v27mc\") pod \"bb292b8a-a577-42de-a66b-646fdf0e22b8\" (UID: \"bb292b8a-a577-42de-a66b-646fdf0e22b8\") " Nov 03 13:34:02.903147 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:02.903061 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb292b8a-a577-42de-a66b-646fdf0e22b8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb292b8a-a577-42de-a66b-646fdf0e22b8-plugins-config-volume\") pod \"bb292b8a-a577-42de-a66b-646fdf0e22b8\" (UID: \"bb292b8a-a577-42de-a66b-646fdf0e22b8\") " Nov 03 13:34:02.903428 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:02.903404 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb292b8a-a577-42de-a66b-646fdf0e22b8-plugins-config-volume" pod "bb292b8a-a577-42de-a66b-646fdf0e22b8" (UID: "bb292b8a-a577-42de-a66b-646fdf0e22b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:34:02.903672 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:02.903652 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb292b8a-a577-42de-a66b-646fdf0e22b8-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:34:02.905084 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:02.905064 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb292b8a-a577-42de-a66b-646fdf0e22b8-kube-api-access-v27mc" pod "bb292b8a-a577-42de-a66b-646fdf0e22b8" (UID: "bb292b8a-a577-42de-a66b-646fdf0e22b8"). InnerVolumeSpecName "kube-api-access-v27mc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:34:03.006354 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:03.006269 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be6c35e9-cf38-4a4b-a393-932c02403030-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be6c35e9-cf38-4a4b-a393-932c02403030-plugins-config-volume\") pod \"be6c35e9-cf38-4a4b-a393-932c02403030\" (UID: \"be6c35e9-cf38-4a4b-a393-932c02403030\") " Nov 03 13:34:03.006584 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:03.006418 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be6c35e9-cf38-4a4b-a393-932c02403030-kube-api-access-28v2f\" (UniqueName: \"kubernetes.io/projected/be6c35e9-cf38-4a4b-a393-932c02403030-kube-api-access-28v2f\") pod \"be6c35e9-cf38-4a4b-a393-932c02403030\" (UID: \"be6c35e9-cf38-4a4b-a393-932c02403030\") " Nov 03 13:34:03.007442 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:03.007075 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v27mc\" (UniqueName: \"kubernetes.io/projected/bb292b8a-a577-42de-a66b-646fdf0e22b8-kube-api-access-v27mc\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:34:03.008101 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:03.008070 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be6c35e9-cf38-4a4b-a393-932c02403030-plugins-config-volume" pod "be6c35e9-cf38-4a4b-a393-932c02403030" (UID: "be6c35e9-cf38-4a4b-a393-932c02403030"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:34:03.069437 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:03.065389 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be6c35e9-cf38-4a4b-a393-932c02403030-kube-api-access-28v2f" pod "be6c35e9-cf38-4a4b-a393-932c02403030" (UID: "be6c35e9-cf38-4a4b-a393-932c02403030"). InnerVolumeSpecName "kube-api-access-28v2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:34:03.108337 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:03.108297 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be6c35e9-cf38-4a4b-a393-932c02403030-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:34:03.108337 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:03.108333 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28v2f\" (UniqueName: \"kubernetes.io/projected/be6c35e9-cf38-4a4b-a393-932c02403030-kube-api-access-28v2f\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:34:03.390571 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:03.390533 5979 generic.go:356] "Generic (PLEG): container finished" podID="be6c35e9-cf38-4a4b-a393-932c02403030" containerID="172910e9b60ba468df1baf2ff58e9b868aa048154002fe3dab07e6a1279a9584" exitCode=0 Nov 03 13:34:03.390571 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:03.390571 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-461b2d-7d984ffbf9-262hb" Nov 03 13:34:03.390817 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:03.390616 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-461b2d-7d984ffbf9-262hb" event={"ID":"be6c35e9-cf38-4a4b-a393-932c02403030","Type":"ContainerDied","Data":"172910e9b60ba468df1baf2ff58e9b868aa048154002fe3dab07e6a1279a9584"} Nov 03 13:34:03.390817 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:03.390697 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-461b2d-7d984ffbf9-262hb" event={"ID":"be6c35e9-cf38-4a4b-a393-932c02403030","Type":"ContainerDied","Data":"1bd1f72c1dad8be1937839385c8fe7b4104c0d9ca08346e12117a8c8a39fb1b6"} Nov 03 13:34:03.390817 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:03.390723 5979 scope.go:122] "RemoveContainer" containerID="172910e9b60ba468df1baf2ff58e9b868aa048154002fe3dab07e6a1279a9584" Nov 03 13:34:03.392106 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:03.392055 5979 generic.go:356] "Generic (PLEG): container finished" podID="bb292b8a-a577-42de-a66b-646fdf0e22b8" containerID="015ca4a909b0339e1d848ea381ea24f38a1ec1f377cd2e719fdf9a303b27c8b1" exitCode=0 Nov 03 13:34:03.392106 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:03.392093 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-461b2d-6466bf6b8f-48fhn" event={"ID":"bb292b8a-a577-42de-a66b-646fdf0e22b8","Type":"ContainerDied","Data":"015ca4a909b0339e1d848ea381ea24f38a1ec1f377cd2e719fdf9a303b27c8b1"} Nov 03 13:34:03.392269 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:03.392110 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-461b2d-6466bf6b8f-48fhn" Nov 03 13:34:03.392269 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:03.392144 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-461b2d-6466bf6b8f-48fhn" event={"ID":"bb292b8a-a577-42de-a66b-646fdf0e22b8","Type":"ContainerDied","Data":"e9596a430131586bab7417487ecc65a889fa54d196ba38bfe5b84648c7a25437"} Nov 03 13:34:03.402546 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:03.402522 5979 scope.go:122] "RemoveContainer" containerID="172910e9b60ba468df1baf2ff58e9b868aa048154002fe3dab07e6a1279a9584" Nov 03 13:34:03.402895 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:34:03.402863 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"172910e9b60ba468df1baf2ff58e9b868aa048154002fe3dab07e6a1279a9584\": container with ID starting with 172910e9b60ba468df1baf2ff58e9b868aa048154002fe3dab07e6a1279a9584 not found: ID does not exist" containerID="172910e9b60ba468df1baf2ff58e9b868aa048154002fe3dab07e6a1279a9584" Nov 03 13:34:03.403007 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:03.402903 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"172910e9b60ba468df1baf2ff58e9b868aa048154002fe3dab07e6a1279a9584"} err="failed to get container status \"172910e9b60ba468df1baf2ff58e9b868aa048154002fe3dab07e6a1279a9584\": rpc error: code = NotFound desc = could not find container \"172910e9b60ba468df1baf2ff58e9b868aa048154002fe3dab07e6a1279a9584\": container with ID starting with 172910e9b60ba468df1baf2ff58e9b868aa048154002fe3dab07e6a1279a9584 not found: ID does not exist" Nov 03 13:34:03.403007 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:03.402928 5979 scope.go:122] "RemoveContainer" containerID="015ca4a909b0339e1d848ea381ea24f38a1ec1f377cd2e719fdf9a303b27c8b1" Nov 03 13:34:03.412696 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:03.412679 5979 scope.go:122] "RemoveContainer" containerID="015ca4a909b0339e1d848ea381ea24f38a1ec1f377cd2e719fdf9a303b27c8b1" Nov 03 13:34:03.412956 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:34:03.412936 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"015ca4a909b0339e1d848ea381ea24f38a1ec1f377cd2e719fdf9a303b27c8b1\": container with ID starting with 015ca4a909b0339e1d848ea381ea24f38a1ec1f377cd2e719fdf9a303b27c8b1 not found: ID does not exist" containerID="015ca4a909b0339e1d848ea381ea24f38a1ec1f377cd2e719fdf9a303b27c8b1" Nov 03 13:34:03.413010 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:03.412963 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"015ca4a909b0339e1d848ea381ea24f38a1ec1f377cd2e719fdf9a303b27c8b1"} err="failed to get container status \"015ca4a909b0339e1d848ea381ea24f38a1ec1f377cd2e719fdf9a303b27c8b1\": rpc error: code = NotFound desc = could not find container \"015ca4a909b0339e1d848ea381ea24f38a1ec1f377cd2e719fdf9a303b27c8b1\": container with ID starting with 015ca4a909b0339e1d848ea381ea24f38a1ec1f377cd2e719fdf9a303b27c8b1 not found: ID does not exist" Nov 03 13:34:03.547890 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:03.547856 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-461b2d-7d984ffbf9-262hb"] Nov 03 13:34:03.553790 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:03.553764 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-461b2d-7d984ffbf9-262hb"] Nov 03 13:34:03.655443 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:03.655355 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-461b2d-6466bf6b8f-48fhn"] Nov 03 13:34:03.659879 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:03.659849 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-461b2d-6466bf6b8f-48fhn"] Nov 03 13:34:04.175525 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:04.172931 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb292b8a-a577-42de-a66b-646fdf0e22b8" path="/var/lib/kubelet/pods/bb292b8a-a577-42de-a66b-646fdf0e22b8/volumes" Nov 03 13:34:04.175525 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:04.173847 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be6c35e9-cf38-4a4b-a393-932c02403030" path="/var/lib/kubelet/pods/be6c35e9-cf38-4a4b-a393-932c02403030/volumes" Nov 03 13:34:05.062251 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:05.062205 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-5295ab29-79c7656b49-gxvjq" podUID="96afb289-d271-43d2-aeb8-3deaa841af80" containerName="payload-processing" containerID="cri-o://27846f2111f3bc6e6d58007b420245f9bc9f09118d311e667cb2e99893b49bf7" gracePeriod=25 Nov 03 13:34:05.170020 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:05.169924 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-5295ab29-6d58657867-ls4rq" podUID="e421da61-cb60-47e5-8da3-936ec53af260" containerName="payload-pre-processing" containerID="cri-o://2b6c90cf03d523392a7d8de0ad54a095b9eaeb08e33d728fe5ca2557a1378487" gracePeriod=25 Nov 03 13:34:05.339502 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:05.339452 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5295ab29-79c7656b49-gxvjq" Nov 03 13:34:05.438362 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:05.438337 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5295ab29-6d58657867-ls4rq" Nov 03 13:34:05.444847 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:05.444821 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96afb289-d271-43d2-aeb8-3deaa841af80-kube-api-access-qvhff\" (UniqueName: \"kubernetes.io/projected/96afb289-d271-43d2-aeb8-3deaa841af80-kube-api-access-qvhff\") pod \"96afb289-d271-43d2-aeb8-3deaa841af80\" (UID: \"96afb289-d271-43d2-aeb8-3deaa841af80\") " Nov 03 13:34:05.445046 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:05.445028 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96afb289-d271-43d2-aeb8-3deaa841af80-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96afb289-d271-43d2-aeb8-3deaa841af80-plugins-config-volume\") pod \"96afb289-d271-43d2-aeb8-3deaa841af80\" (UID: \"96afb289-d271-43d2-aeb8-3deaa841af80\") " Nov 03 13:34:05.445457 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:05.445429 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96afb289-d271-43d2-aeb8-3deaa841af80-plugins-config-volume" pod "96afb289-d271-43d2-aeb8-3deaa841af80" (UID: "96afb289-d271-43d2-aeb8-3deaa841af80"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:34:05.445687 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:05.445668 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96afb289-d271-43d2-aeb8-3deaa841af80-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:34:05.450849 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:05.450822 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96afb289-d271-43d2-aeb8-3deaa841af80-kube-api-access-qvhff" pod "96afb289-d271-43d2-aeb8-3deaa841af80" (UID: "96afb289-d271-43d2-aeb8-3deaa841af80"). InnerVolumeSpecName "kube-api-access-qvhff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:34:05.471733 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:05.471697 5979 generic.go:356] "Generic (PLEG): container finished" podID="e421da61-cb60-47e5-8da3-936ec53af260" containerID="2b6c90cf03d523392a7d8de0ad54a095b9eaeb08e33d728fe5ca2557a1378487" exitCode=0 Nov 03 13:34:05.471891 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:05.471743 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5295ab29-6d58657867-ls4rq" Nov 03 13:34:05.471891 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:05.471760 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5295ab29-6d58657867-ls4rq" event={"ID":"e421da61-cb60-47e5-8da3-936ec53af260","Type":"ContainerDied","Data":"2b6c90cf03d523392a7d8de0ad54a095b9eaeb08e33d728fe5ca2557a1378487"} Nov 03 13:34:05.471891 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:05.471855 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5295ab29-6d58657867-ls4rq" event={"ID":"e421da61-cb60-47e5-8da3-936ec53af260","Type":"ContainerDied","Data":"6d49123dbc365325d405f7d3dca7fb829a9df731535e0db1bb6a58d466de2cb0"} Nov 03 13:34:05.471891 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:05.471889 5979 scope.go:122] "RemoveContainer" containerID="2b6c90cf03d523392a7d8de0ad54a095b9eaeb08e33d728fe5ca2557a1378487" Nov 03 13:34:05.473834 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:05.473809 5979 generic.go:356] "Generic (PLEG): container finished" podID="96afb289-d271-43d2-aeb8-3deaa841af80" containerID="27846f2111f3bc6e6d58007b420245f9bc9f09118d311e667cb2e99893b49bf7" exitCode=0 Nov 03 13:34:05.473968 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:05.473873 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5295ab29-79c7656b49-gxvjq" Nov 03 13:34:05.473968 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:05.473893 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5295ab29-79c7656b49-gxvjq" event={"ID":"96afb289-d271-43d2-aeb8-3deaa841af80","Type":"ContainerDied","Data":"27846f2111f3bc6e6d58007b420245f9bc9f09118d311e667cb2e99893b49bf7"} Nov 03 13:34:05.473968 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:05.473951 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5295ab29-79c7656b49-gxvjq" event={"ID":"96afb289-d271-43d2-aeb8-3deaa841af80","Type":"ContainerDied","Data":"f6976eedb8d63a6627b14e8831b73349b69783dbfe97335578e0a6fe3120a6f6"} Nov 03 13:34:05.484099 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:05.484078 5979 scope.go:122] "RemoveContainer" containerID="2b6c90cf03d523392a7d8de0ad54a095b9eaeb08e33d728fe5ca2557a1378487" Nov 03 13:34:05.484525 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:34:05.484369 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b6c90cf03d523392a7d8de0ad54a095b9eaeb08e33d728fe5ca2557a1378487\": container with ID starting with 2b6c90cf03d523392a7d8de0ad54a095b9eaeb08e33d728fe5ca2557a1378487 not found: ID does not exist" containerID="2b6c90cf03d523392a7d8de0ad54a095b9eaeb08e33d728fe5ca2557a1378487" Nov 03 13:34:05.484525 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:05.484404 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b6c90cf03d523392a7d8de0ad54a095b9eaeb08e33d728fe5ca2557a1378487"} err="failed to get container status \"2b6c90cf03d523392a7d8de0ad54a095b9eaeb08e33d728fe5ca2557a1378487\": rpc error: code = NotFound desc = could not find container \"2b6c90cf03d523392a7d8de0ad54a095b9eaeb08e33d728fe5ca2557a1378487\": container with ID starting with 2b6c90cf03d523392a7d8de0ad54a095b9eaeb08e33d728fe5ca2557a1378487 not found: ID does not exist" Nov 03 13:34:05.484525 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:05.484427 5979 scope.go:122] "RemoveContainer" containerID="27846f2111f3bc6e6d58007b420245f9bc9f09118d311e667cb2e99893b49bf7" Nov 03 13:34:05.500044 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:05.500011 5979 scope.go:122] "RemoveContainer" containerID="27846f2111f3bc6e6d58007b420245f9bc9f09118d311e667cb2e99893b49bf7" Nov 03 13:34:05.500318 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:34:05.500291 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27846f2111f3bc6e6d58007b420245f9bc9f09118d311e667cb2e99893b49bf7\": container with ID starting with 27846f2111f3bc6e6d58007b420245f9bc9f09118d311e667cb2e99893b49bf7 not found: ID does not exist" containerID="27846f2111f3bc6e6d58007b420245f9bc9f09118d311e667cb2e99893b49bf7" Nov 03 13:34:05.500374 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:05.500319 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27846f2111f3bc6e6d58007b420245f9bc9f09118d311e667cb2e99893b49bf7"} err="failed to get container status \"27846f2111f3bc6e6d58007b420245f9bc9f09118d311e667cb2e99893b49bf7\": rpc error: code = NotFound desc = could not find container \"27846f2111f3bc6e6d58007b420245f9bc9f09118d311e667cb2e99893b49bf7\": container with ID starting with 27846f2111f3bc6e6d58007b420245f9bc9f09118d311e667cb2e99893b49bf7 not found: ID does not exist" Nov 03 13:34:05.546247 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:05.546223 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e421da61-cb60-47e5-8da3-936ec53af260-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e421da61-cb60-47e5-8da3-936ec53af260-plugins-config-volume\") pod \"e421da61-cb60-47e5-8da3-936ec53af260\" (UID: \"e421da61-cb60-47e5-8da3-936ec53af260\") " Nov 03 13:34:05.546524 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:05.546458 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e421da61-cb60-47e5-8da3-936ec53af260-kube-api-access-jdn2k\" (UniqueName: \"kubernetes.io/projected/e421da61-cb60-47e5-8da3-936ec53af260-kube-api-access-jdn2k\") pod \"e421da61-cb60-47e5-8da3-936ec53af260\" (UID: \"e421da61-cb60-47e5-8da3-936ec53af260\") " Nov 03 13:34:05.546616 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:05.546591 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e421da61-cb60-47e5-8da3-936ec53af260-plugins-config-volume" pod "e421da61-cb60-47e5-8da3-936ec53af260" (UID: "e421da61-cb60-47e5-8da3-936ec53af260"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:34:05.547021 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:05.546995 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvhff\" (UniqueName: \"kubernetes.io/projected/96afb289-d271-43d2-aeb8-3deaa841af80-kube-api-access-qvhff\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:34:05.547021 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:05.547021 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e421da61-cb60-47e5-8da3-936ec53af260-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:34:05.548646 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:05.548620 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e421da61-cb60-47e5-8da3-936ec53af260-kube-api-access-jdn2k" pod "e421da61-cb60-47e5-8da3-936ec53af260" (UID: "e421da61-cb60-47e5-8da3-936ec53af260"). InnerVolumeSpecName "kube-api-access-jdn2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:34:05.648382 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:05.648353 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdn2k\" (UniqueName: \"kubernetes.io/projected/e421da61-cb60-47e5-8da3-936ec53af260-kube-api-access-jdn2k\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:34:05.715291 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:05.715211 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5295ab29-79c7656b49-gxvjq"] Nov 03 13:34:05.732393 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:05.732362 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5295ab29-79c7656b49-gxvjq"] Nov 03 13:34:05.895568 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:05.895529 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5295ab29-6d58657867-ls4rq"] Nov 03 13:34:05.898589 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:05.898557 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5295ab29-6d58657867-ls4rq"] Nov 03 13:34:06.117114 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:06.117078 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96afb289-d271-43d2-aeb8-3deaa841af80" path="/var/lib/kubelet/pods/96afb289-d271-43d2-aeb8-3deaa841af80/volumes" Nov 03 13:34:06.117717 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:06.117697 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e421da61-cb60-47e5-8da3-936ec53af260" path="/var/lib/kubelet/pods/e421da61-cb60-47e5-8da3-936ec53af260/volumes" Nov 03 13:34:07.233778 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:34:07.233745 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:34:07.554093 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:07.554005 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa2568-794d67c965-wnxdb" podUID="2a6a387d-1db5-42e3-8d68-11fec4436ea0" containerName="payload-processing" containerID="cri-o://0aea09ff01ebe71b2ee93515fad85c75f614ffe80f74f4ad58c7afd6bd4a873d" gracePeriod=25 Nov 03 13:34:07.673857 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:07.673823 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa2568-6cb8658547-kqzwp" podUID="ecd0ef06-5ba4-496f-9470-43a606f8c687" containerName="payload-pre-processing" containerID="cri-o://07c1dd2164c21afe3b068853634f64698faffee285d0a0d8694fcbc760a1c23b" gracePeriod=25 Nov 03 13:34:07.918312 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:07.918275 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa2568-794d67c965-wnxdb" Nov 03 13:34:08.101383 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:08.101022 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a6a387d-1db5-42e3-8d68-11fec4436ea0-kube-api-access-qmlc2\" (UniqueName: \"kubernetes.io/projected/2a6a387d-1db5-42e3-8d68-11fec4436ea0-kube-api-access-qmlc2\") pod \"2a6a387d-1db5-42e3-8d68-11fec4436ea0\" (UID: \"2a6a387d-1db5-42e3-8d68-11fec4436ea0\") " Nov 03 13:34:08.101383 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:08.101237 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a6a387d-1db5-42e3-8d68-11fec4436ea0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a6a387d-1db5-42e3-8d68-11fec4436ea0-plugins-config-volume\") pod \"2a6a387d-1db5-42e3-8d68-11fec4436ea0\" (UID: \"2a6a387d-1db5-42e3-8d68-11fec4436ea0\") " Nov 03 13:34:08.102164 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:08.102131 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a6a387d-1db5-42e3-8d68-11fec4436ea0-plugins-config-volume" pod "2a6a387d-1db5-42e3-8d68-11fec4436ea0" (UID: "2a6a387d-1db5-42e3-8d68-11fec4436ea0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:34:08.105529 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:08.105460 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a6a387d-1db5-42e3-8d68-11fec4436ea0-kube-api-access-qmlc2" pod "2a6a387d-1db5-42e3-8d68-11fec4436ea0" (UID: "2a6a387d-1db5-42e3-8d68-11fec4436ea0"). InnerVolumeSpecName "kube-api-access-qmlc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:34:08.119243 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:08.118175 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa2568-6cb8658547-kqzwp" Nov 03 13:34:08.202965 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:08.202926 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a6a387d-1db5-42e3-8d68-11fec4436ea0-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:34:08.202965 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:08.202962 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmlc2\" (UniqueName: \"kubernetes.io/projected/2a6a387d-1db5-42e3-8d68-11fec4436ea0-kube-api-access-qmlc2\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:34:08.303790 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:08.303750 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecd0ef06-5ba4-496f-9470-43a606f8c687-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecd0ef06-5ba4-496f-9470-43a606f8c687-plugins-config-volume\") pod \"ecd0ef06-5ba4-496f-9470-43a606f8c687\" (UID: \"ecd0ef06-5ba4-496f-9470-43a606f8c687\") " Nov 03 13:34:08.304290 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:08.303806 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecd0ef06-5ba4-496f-9470-43a606f8c687-kube-api-access-c5qgm\" (UniqueName: \"kubernetes.io/projected/ecd0ef06-5ba4-496f-9470-43a606f8c687-kube-api-access-c5qgm\") pod \"ecd0ef06-5ba4-496f-9470-43a606f8c687\" (UID: \"ecd0ef06-5ba4-496f-9470-43a606f8c687\") " Nov 03 13:34:08.304290 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:08.304199 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecd0ef06-5ba4-496f-9470-43a606f8c687-plugins-config-volume" pod "ecd0ef06-5ba4-496f-9470-43a606f8c687" (UID: "ecd0ef06-5ba4-496f-9470-43a606f8c687"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:34:08.304708 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:08.304686 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecd0ef06-5ba4-496f-9470-43a606f8c687-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:34:08.306126 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:08.306105 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecd0ef06-5ba4-496f-9470-43a606f8c687-kube-api-access-c5qgm" pod "ecd0ef06-5ba4-496f-9470-43a606f8c687" (UID: "ecd0ef06-5ba4-496f-9470-43a606f8c687"). InnerVolumeSpecName "kube-api-access-c5qgm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:34:08.405727 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:08.405697 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5qgm\" (UniqueName: \"kubernetes.io/projected/ecd0ef06-5ba4-496f-9470-43a606f8c687-kube-api-access-c5qgm\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:34:08.507647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:08.507607 5979 generic.go:356] "Generic (PLEG): container finished" podID="ecd0ef06-5ba4-496f-9470-43a606f8c687" containerID="07c1dd2164c21afe3b068853634f64698faffee285d0a0d8694fcbc760a1c23b" exitCode=0 Nov 03 13:34:08.507857 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:08.507665 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa2568-6cb8658547-kqzwp" Nov 03 13:34:08.507857 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:08.507694 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa2568-6cb8658547-kqzwp" event={"ID":"ecd0ef06-5ba4-496f-9470-43a606f8c687","Type":"ContainerDied","Data":"07c1dd2164c21afe3b068853634f64698faffee285d0a0d8694fcbc760a1c23b"} Nov 03 13:34:08.507857 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:08.507780 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa2568-6cb8658547-kqzwp" event={"ID":"ecd0ef06-5ba4-496f-9470-43a606f8c687","Type":"ContainerDied","Data":"b88b70fce06227f7c9d00b63aeaf02045c1509663cdc73a2e3678d61059ee83d"} Nov 03 13:34:08.507857 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:08.507811 5979 scope.go:122] "RemoveContainer" containerID="07c1dd2164c21afe3b068853634f64698faffee285d0a0d8694fcbc760a1c23b" Nov 03 13:34:08.509112 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:08.509087 5979 generic.go:356] "Generic (PLEG): container finished" podID="2a6a387d-1db5-42e3-8d68-11fec4436ea0" containerID="0aea09ff01ebe71b2ee93515fad85c75f614ffe80f74f4ad58c7afd6bd4a873d" exitCode=0 Nov 03 13:34:08.509232 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:08.509117 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa2568-794d67c965-wnxdb" event={"ID":"2a6a387d-1db5-42e3-8d68-11fec4436ea0","Type":"ContainerDied","Data":"0aea09ff01ebe71b2ee93515fad85c75f614ffe80f74f4ad58c7afd6bd4a873d"} Nov 03 13:34:08.509232 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:08.509146 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa2568-794d67c965-wnxdb" Nov 03 13:34:08.509232 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:08.509159 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa2568-794d67c965-wnxdb" event={"ID":"2a6a387d-1db5-42e3-8d68-11fec4436ea0","Type":"ContainerDied","Data":"289f14d04ecd33d6d87b5ef867afdb954873546afcaa473342a2475e288dcca6"} Nov 03 13:34:08.518829 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:08.518810 5979 scope.go:122] "RemoveContainer" containerID="07c1dd2164c21afe3b068853634f64698faffee285d0a0d8694fcbc760a1c23b" Nov 03 13:34:08.519080 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:34:08.519060 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07c1dd2164c21afe3b068853634f64698faffee285d0a0d8694fcbc760a1c23b\": container with ID starting with 07c1dd2164c21afe3b068853634f64698faffee285d0a0d8694fcbc760a1c23b not found: ID does not exist" containerID="07c1dd2164c21afe3b068853634f64698faffee285d0a0d8694fcbc760a1c23b" Nov 03 13:34:08.519125 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:08.519088 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07c1dd2164c21afe3b068853634f64698faffee285d0a0d8694fcbc760a1c23b"} err="failed to get container status \"07c1dd2164c21afe3b068853634f64698faffee285d0a0d8694fcbc760a1c23b\": rpc error: code = NotFound desc = could not find container \"07c1dd2164c21afe3b068853634f64698faffee285d0a0d8694fcbc760a1c23b\": container with ID starting with 07c1dd2164c21afe3b068853634f64698faffee285d0a0d8694fcbc760a1c23b not found: ID does not exist" Nov 03 13:34:08.519125 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:08.519110 5979 scope.go:122] "RemoveContainer" containerID="0aea09ff01ebe71b2ee93515fad85c75f614ffe80f74f4ad58c7afd6bd4a873d" Nov 03 13:34:08.528259 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:08.528243 5979 scope.go:122] "RemoveContainer" containerID="0aea09ff01ebe71b2ee93515fad85c75f614ffe80f74f4ad58c7afd6bd4a873d" Nov 03 13:34:08.528519 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:34:08.528500 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0aea09ff01ebe71b2ee93515fad85c75f614ffe80f74f4ad58c7afd6bd4a873d\": container with ID starting with 0aea09ff01ebe71b2ee93515fad85c75f614ffe80f74f4ad58c7afd6bd4a873d not found: ID does not exist" containerID="0aea09ff01ebe71b2ee93515fad85c75f614ffe80f74f4ad58c7afd6bd4a873d" Nov 03 13:34:08.528587 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:08.528525 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0aea09ff01ebe71b2ee93515fad85c75f614ffe80f74f4ad58c7afd6bd4a873d"} err="failed to get container status \"0aea09ff01ebe71b2ee93515fad85c75f614ffe80f74f4ad58c7afd6bd4a873d\": rpc error: code = NotFound desc = could not find container \"0aea09ff01ebe71b2ee93515fad85c75f614ffe80f74f4ad58c7afd6bd4a873d\": container with ID starting with 0aea09ff01ebe71b2ee93515fad85c75f614ffe80f74f4ad58c7afd6bd4a873d not found: ID does not exist" Nov 03 13:34:08.664809 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:08.664729 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fa2568-794d67c965-wnxdb"] Nov 03 13:34:08.711169 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:08.711133 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fa2568-794d67c965-wnxdb"] Nov 03 13:34:08.768705 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:08.768668 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fa2568-6cb8658547-kqzwp"] Nov 03 13:34:08.771972 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:08.771941 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fa2568-6cb8658547-kqzwp"] Nov 03 13:34:09.366078 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:09.366053 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6f9ed-554b464fd5-kzpbv" Nov 03 13:34:09.374114 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:09.374093 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6f9ed-667f6c778c-cq5t6" Nov 03 13:34:09.379457 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:09.379437 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5c8192-7c4d4b65cf-wm2vg"] Nov 03 13:34:09.380281 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:09.380259 5979 cpu_manager.go:414] "containerMap: removing container" podUID="bb292b8a-a577-42de-a66b-646fdf0e22b8" containerName="payload-processing" Nov 03 13:34:09.380281 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:09.380281 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb292b8a-a577-42de-a66b-646fdf0e22b8" containerName="payload-processing" Nov 03 13:34:09.380394 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:09.380324 5979 cpu_manager.go:414] "containerMap: removing container" podUID="2a6a387d-1db5-42e3-8d68-11fec4436ea0" containerName="payload-processing" Nov 03 13:34:09.380394 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:09.380330 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a6a387d-1db5-42e3-8d68-11fec4436ea0" containerName="payload-processing" Nov 03 13:34:09.380394 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:09.380348 5979 cpu_manager.go:414] "containerMap: removing container" podUID="be6c35e9-cf38-4a4b-a393-932c02403030" containerName="payload-pre-processing" Nov 03 13:34:09.380394 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:09.380353 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be6c35e9-cf38-4a4b-a393-932c02403030" containerName="payload-pre-processing" Nov 03 13:34:09.380394 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:09.380365 5979 cpu_manager.go:414] "containerMap: removing container" podUID="e421da61-cb60-47e5-8da3-936ec53af260" containerName="payload-pre-processing" Nov 03 13:34:09.380394 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:09.380370 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e421da61-cb60-47e5-8da3-936ec53af260" containerName="payload-pre-processing" Nov 03 13:34:09.380394 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:09.380379 5979 cpu_manager.go:414] "containerMap: removing container" podUID="96afb289-d271-43d2-aeb8-3deaa841af80" containerName="payload-processing" Nov 03 13:34:09.380394 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:09.380383 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96afb289-d271-43d2-aeb8-3deaa841af80" containerName="payload-processing" Nov 03 13:34:09.380394 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:09.380390 5979 cpu_manager.go:414] "containerMap: removing container" podUID="ecd0ef06-5ba4-496f-9470-43a606f8c687" containerName="payload-pre-processing" Nov 03 13:34:09.380394 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:09.380394 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecd0ef06-5ba4-496f-9470-43a606f8c687" containerName="payload-pre-processing" Nov 03 13:34:09.380719 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:09.380506 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a6a387d-1db5-42e3-8d68-11fec4436ea0" containerName="payload-processing" Nov 03 13:34:09.380719 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:09.380518 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="be6c35e9-cf38-4a4b-a393-932c02403030" containerName="payload-pre-processing" Nov 03 13:34:09.380719 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:09.380525 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="e421da61-cb60-47e5-8da3-936ec53af260" containerName="payload-pre-processing" Nov 03 13:34:09.380719 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:09.380532 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="96afb289-d271-43d2-aeb8-3deaa841af80" containerName="payload-processing" Nov 03 13:34:09.380719 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:09.380537 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecd0ef06-5ba4-496f-9470-43a606f8c687" containerName="payload-pre-processing" Nov 03 13:34:09.380719 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:09.380545 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb292b8a-a577-42de-a66b-646fdf0e22b8" containerName="payload-processing" Nov 03 13:34:09.383079 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:09.383061 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c8192-7c4d4b65cf-wm2vg" Nov 03 13:34:09.433269 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:09.433240 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-5c8192-dockercfg-pbxqb\"" Nov 03 13:34:09.433269 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:09.433240 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-5c8192\"" Nov 03 13:34:09.440076 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:09.440051 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5c8192-7c4d4b65cf-wm2vg"] Nov 03 13:34:09.443084 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:09.443037 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6jqw\" (UniqueName: \"kubernetes.io/projected/10c68fea-ba03-4f0c-85a9-01f113c0f65d-kube-api-access-x6jqw\") pod \"payload-pre-processing-e2e-ipp-b-5c8192-7c4d4b65cf-wm2vg\" (UID: \"10c68fea-ba03-4f0c-85a9-01f113c0f65d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c8192-7c4d4b65cf-wm2vg" Nov 03 13:34:09.443446 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:09.443426 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10c68fea-ba03-4f0c-85a9-01f113c0f65d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5c8192-7c4d4b65cf-wm2vg\" (UID: \"10c68fea-ba03-4f0c-85a9-01f113c0f65d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c8192-7c4d4b65cf-wm2vg" Nov 03 13:34:09.581494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:09.576293 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6jqw\" (UniqueName: \"kubernetes.io/projected/10c68fea-ba03-4f0c-85a9-01f113c0f65d-kube-api-access-x6jqw\") pod \"payload-pre-processing-e2e-ipp-b-5c8192-7c4d4b65cf-wm2vg\" (UID: \"10c68fea-ba03-4f0c-85a9-01f113c0f65d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c8192-7c4d4b65cf-wm2vg" Nov 03 13:34:09.581494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:09.576436 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10c68fea-ba03-4f0c-85a9-01f113c0f65d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5c8192-7c4d4b65cf-wm2vg\" (UID: \"10c68fea-ba03-4f0c-85a9-01f113c0f65d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c8192-7c4d4b65cf-wm2vg" Nov 03 13:34:09.581494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:09.577308 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10c68fea-ba03-4f0c-85a9-01f113c0f65d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5c8192-7c4d4b65cf-wm2vg\" (UID: \"10c68fea-ba03-4f0c-85a9-01f113c0f65d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c8192-7c4d4b65cf-wm2vg" Nov 03 13:34:09.694726 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:09.694675 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5c8192-54f4dd4bf5-qjv8l"] Nov 03 13:34:09.695336 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:09.695301 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6jqw\" (UniqueName: \"kubernetes.io/projected/10c68fea-ba03-4f0c-85a9-01f113c0f65d-kube-api-access-x6jqw\") pod \"payload-pre-processing-e2e-ipp-b-5c8192-7c4d4b65cf-wm2vg\" (UID: \"10c68fea-ba03-4f0c-85a9-01f113c0f65d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c8192-7c4d4b65cf-wm2vg" Nov 03 13:34:09.697529 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:09.697505 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c8192-7c4d4b65cf-wm2vg" Nov 03 13:34:09.718707 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:09.718671 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5c8192-54f4dd4bf5-qjv8l"] Nov 03 13:34:09.718912 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:09.718794 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c8192-54f4dd4bf5-qjv8l" Nov 03 13:34:09.882310 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:09.882262 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca5d6e86-e31a-4a7f-b71b-54216e92e668-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5c8192-54f4dd4bf5-qjv8l\" (UID: \"ca5d6e86-e31a-4a7f-b71b-54216e92e668\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5c8192-54f4dd4bf5-qjv8l" Nov 03 13:34:09.882505 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:09.882416 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lgk5\" (UniqueName: \"kubernetes.io/projected/ca5d6e86-e31a-4a7f-b71b-54216e92e668-kube-api-access-4lgk5\") pod \"payload-processing-e2e-ipp-b-5c8192-54f4dd4bf5-qjv8l\" (UID: \"ca5d6e86-e31a-4a7f-b71b-54216e92e668\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5c8192-54f4dd4bf5-qjv8l" Nov 03 13:34:09.984381 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:09.984298 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lgk5\" (UniqueName: \"kubernetes.io/projected/ca5d6e86-e31a-4a7f-b71b-54216e92e668-kube-api-access-4lgk5\") pod \"payload-processing-e2e-ipp-b-5c8192-54f4dd4bf5-qjv8l\" (UID: \"ca5d6e86-e31a-4a7f-b71b-54216e92e668\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5c8192-54f4dd4bf5-qjv8l" Nov 03 13:34:09.984540 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:09.984508 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca5d6e86-e31a-4a7f-b71b-54216e92e668-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5c8192-54f4dd4bf5-qjv8l\" (UID: \"ca5d6e86-e31a-4a7f-b71b-54216e92e668\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5c8192-54f4dd4bf5-qjv8l" Nov 03 13:34:09.985142 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:09.985124 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca5d6e86-e31a-4a7f-b71b-54216e92e668-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5c8192-54f4dd4bf5-qjv8l\" (UID: \"ca5d6e86-e31a-4a7f-b71b-54216e92e668\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5c8192-54f4dd4bf5-qjv8l" Nov 03 13:34:10.039011 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:10.038974 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lgk5\" (UniqueName: \"kubernetes.io/projected/ca5d6e86-e31a-4a7f-b71b-54216e92e668-kube-api-access-4lgk5\") pod \"payload-processing-e2e-ipp-b-5c8192-54f4dd4bf5-qjv8l\" (UID: \"ca5d6e86-e31a-4a7f-b71b-54216e92e668\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5c8192-54f4dd4bf5-qjv8l" Nov 03 13:34:10.065630 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:10.065597 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c8192-54f4dd4bf5-qjv8l" Nov 03 13:34:10.088973 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:10.088944 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5c8192-7c4d4b65cf-wm2vg"] Nov 03 13:34:10.094634 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:34:10.094603 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10c68fea_ba03_4f0c_85a9_01f113c0f65d.slice/crio-980eed2205fefcf8c123f2fcfd37cef3118c70a805bb067c9230f9ed284de80d WatchSource:0}: Error finding container 980eed2205fefcf8c123f2fcfd37cef3118c70a805bb067c9230f9ed284de80d: Status 404 returned error can't find the container with id 980eed2205fefcf8c123f2fcfd37cef3118c70a805bb067c9230f9ed284de80d Nov 03 13:34:10.119159 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:10.119122 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a6a387d-1db5-42e3-8d68-11fec4436ea0" path="/var/lib/kubelet/pods/2a6a387d-1db5-42e3-8d68-11fec4436ea0/volumes" Nov 03 13:34:10.119729 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:10.119706 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecd0ef06-5ba4-496f-9470-43a606f8c687" path="/var/lib/kubelet/pods/ecd0ef06-5ba4-496f-9470-43a606f8c687/volumes" Nov 03 13:34:10.450563 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:10.450533 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5c8192-54f4dd4bf5-qjv8l"] Nov 03 13:34:10.455976 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:34:10.455950 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca5d6e86_e31a_4a7f_b71b_54216e92e668.slice/crio-d223dd0a1deb8bc9f4674c835ef12d7907a9eace75c562c6a7db4b0ba6ec05a3 WatchSource:0}: Error finding container d223dd0a1deb8bc9f4674c835ef12d7907a9eace75c562c6a7db4b0ba6ec05a3: Status 404 returned error can't find the container with id d223dd0a1deb8bc9f4674c835ef12d7907a9eace75c562c6a7db4b0ba6ec05a3 Nov 03 13:34:10.711905 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:10.711791 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c8192-54f4dd4bf5-qjv8l" event={"ID":"ca5d6e86-e31a-4a7f-b71b-54216e92e668","Type":"ContainerStarted","Data":"3c9aa2a28c385d0f8f2998af5f6f518f738d5c076e7db1d946b477661023fb25"} Nov 03 13:34:10.711905 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:10.711859 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c8192-54f4dd4bf5-qjv8l" event={"ID":"ca5d6e86-e31a-4a7f-b71b-54216e92e668","Type":"ContainerStarted","Data":"d223dd0a1deb8bc9f4674c835ef12d7907a9eace75c562c6a7db4b0ba6ec05a3"} Nov 03 13:34:10.712115 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:10.711886 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c8192-54f4dd4bf5-qjv8l" Nov 03 13:34:10.713770 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:10.713732 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c8192-7c4d4b65cf-wm2vg" event={"ID":"10c68fea-ba03-4f0c-85a9-01f113c0f65d","Type":"ContainerStarted","Data":"1e51c064970925eae34be195cef6ffb20b5577084f437361a2ef3596fd9149e3"} Nov 03 13:34:10.713903 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:10.713782 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c8192-7c4d4b65cf-wm2vg" event={"ID":"10c68fea-ba03-4f0c-85a9-01f113c0f65d","Type":"ContainerStarted","Data":"980eed2205fefcf8c123f2fcfd37cef3118c70a805bb067c9230f9ed284de80d"} Nov 03 13:34:10.713903 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:10.713845 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c8192-7c4d4b65cf-wm2vg" Nov 03 13:34:10.821344 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:10.821288 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c8192-54f4dd4bf5-qjv8l" podStartSLOduration=1.821273592 podStartE2EDuration="1.821273592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:34:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:34:10.818215207 +0000 UTC m=+1732.828903446" watchObservedRunningTime="2026-11-03 13:34:10.821273592 +0000 UTC m=+1732.831961834" Nov 03 13:34:10.931066 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:10.931006 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c8192-7c4d4b65cf-wm2vg" podStartSLOduration=1.9309897390000001 podStartE2EDuration="1.930989739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:34:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:34:10.928508946 +0000 UTC m=+1732.939197190" watchObservedRunningTime="2026-11-03 13:34:10.930989739 +0000 UTC m=+1732.941677981" Nov 03 13:34:13.154306 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:13.154273 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l"] Nov 03 13:34:13.158171 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:13.158150 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l" Nov 03 13:34:13.229370 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:13.225928 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-d29e07-openshift-default-dockercfg-stf84\"" Nov 03 13:34:13.453353 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:13.453268 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l"] Nov 03 13:34:13.454922 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:13.454874 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-istiod-ca-cert\") pod \"e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l\" (UID: \"ddd1f97b-814c-43b5-a81b-201bd1ef2c9b\") " pod="openshift-ingress/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l" Nov 03 13:34:13.455236 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:13.455207 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht9mb\" (UniqueName: \"kubernetes.io/projected/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-kube-api-access-ht9mb\") pod \"e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l\" (UID: \"ddd1f97b-814c-43b5-a81b-201bd1ef2c9b\") " pod="openshift-ingress/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l" Nov 03 13:34:13.455421 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:13.455398 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-istio-data\") pod \"e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l\" (UID: \"ddd1f97b-814c-43b5-a81b-201bd1ef2c9b\") " pod="openshift-ingress/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l" Nov 03 13:34:13.456025 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:13.455986 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-istio-podinfo\") pod \"e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l\" (UID: \"ddd1f97b-814c-43b5-a81b-201bd1ef2c9b\") " pod="openshift-ingress/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l" Nov 03 13:34:13.456226 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:13.456208 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-workload-certs\") pod \"e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l\" (UID: \"ddd1f97b-814c-43b5-a81b-201bd1ef2c9b\") " pod="openshift-ingress/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l" Nov 03 13:34:13.456365 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:13.456346 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-istio-token\") pod \"e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l\" (UID: \"ddd1f97b-814c-43b5-a81b-201bd1ef2c9b\") " pod="openshift-ingress/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l" Nov 03 13:34:13.456547 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:13.456514 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-credential-socket\") pod \"e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l\" (UID: \"ddd1f97b-814c-43b5-a81b-201bd1ef2c9b\") " pod="openshift-ingress/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l" Nov 03 13:34:13.456849 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:13.456827 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-workload-socket\") pod \"e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l\" (UID: \"ddd1f97b-814c-43b5-a81b-201bd1ef2c9b\") " pod="openshift-ingress/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l" Nov 03 13:34:13.457017 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:13.457000 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-istio-envoy\") pod \"e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l\" (UID: \"ddd1f97b-814c-43b5-a81b-201bd1ef2c9b\") " pod="openshift-ingress/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l" Nov 03 13:34:13.566510 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:13.566016 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht9mb\" (UniqueName: \"kubernetes.io/projected/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-kube-api-access-ht9mb\") pod \"e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l\" (UID: \"ddd1f97b-814c-43b5-a81b-201bd1ef2c9b\") " pod="openshift-ingress/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l" Nov 03 13:34:13.566510 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:13.566098 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-istio-data\") pod \"e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l\" (UID: \"ddd1f97b-814c-43b5-a81b-201bd1ef2c9b\") " pod="openshift-ingress/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l" Nov 03 13:34:13.566510 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:13.566182 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-istio-podinfo\") pod \"e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l\" (UID: \"ddd1f97b-814c-43b5-a81b-201bd1ef2c9b\") " pod="openshift-ingress/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l" Nov 03 13:34:13.566510 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:13.566209 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-workload-certs\") pod \"e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l\" (UID: \"ddd1f97b-814c-43b5-a81b-201bd1ef2c9b\") " pod="openshift-ingress/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l" Nov 03 13:34:13.566510 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:13.566232 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-istio-token\") pod \"e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l\" (UID: \"ddd1f97b-814c-43b5-a81b-201bd1ef2c9b\") " pod="openshift-ingress/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l" Nov 03 13:34:13.566510 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:13.566267 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-credential-socket\") pod \"e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l\" (UID: \"ddd1f97b-814c-43b5-a81b-201bd1ef2c9b\") " pod="openshift-ingress/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l" Nov 03 13:34:13.566510 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:13.566337 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-workload-socket\") pod \"e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l\" (UID: \"ddd1f97b-814c-43b5-a81b-201bd1ef2c9b\") " pod="openshift-ingress/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l" Nov 03 13:34:13.566510 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:13.566373 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-istio-envoy\") pod \"e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l\" (UID: \"ddd1f97b-814c-43b5-a81b-201bd1ef2c9b\") " pod="openshift-ingress/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l" Nov 03 13:34:13.566510 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:13.566444 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-istiod-ca-cert\") pod \"e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l\" (UID: \"ddd1f97b-814c-43b5-a81b-201bd1ef2c9b\") " pod="openshift-ingress/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l" Nov 03 13:34:13.566909 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:13.566707 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-workload-certs\") pod \"e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l\" (UID: \"ddd1f97b-814c-43b5-a81b-201bd1ef2c9b\") " pod="openshift-ingress/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l" Nov 03 13:34:13.566968 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:13.566948 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-istio-data\") pod \"e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l\" (UID: \"ddd1f97b-814c-43b5-a81b-201bd1ef2c9b\") " pod="openshift-ingress/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l" Nov 03 13:34:13.567218 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:13.567198 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-istiod-ca-cert\") pod \"e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l\" (UID: \"ddd1f97b-814c-43b5-a81b-201bd1ef2c9b\") " pod="openshift-ingress/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l" Nov 03 13:34:13.567753 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:13.567728 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-workload-socket\") pod \"e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l\" (UID: \"ddd1f97b-814c-43b5-a81b-201bd1ef2c9b\") " pod="openshift-ingress/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l" Nov 03 13:34:13.567959 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:13.567934 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-credential-socket\") pod \"e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l\" (UID: \"ddd1f97b-814c-43b5-a81b-201bd1ef2c9b\") " pod="openshift-ingress/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l" Nov 03 13:34:13.570188 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:13.570160 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-istio-podinfo\") pod \"e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l\" (UID: \"ddd1f97b-814c-43b5-a81b-201bd1ef2c9b\") " pod="openshift-ingress/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l" Nov 03 13:34:13.573846 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:13.573827 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-istio-envoy\") pod \"e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l\" (UID: \"ddd1f97b-814c-43b5-a81b-201bd1ef2c9b\") " pod="openshift-ingress/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l" Nov 03 13:34:13.620886 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:13.620854 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-istio-token\") pod \"e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l\" (UID: \"ddd1f97b-814c-43b5-a81b-201bd1ef2c9b\") " pod="openshift-ingress/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l" Nov 03 13:34:13.620886 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:13.620873 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht9mb\" (UniqueName: \"kubernetes.io/projected/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-kube-api-access-ht9mb\") pod \"e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l\" (UID: \"ddd1f97b-814c-43b5-a81b-201bd1ef2c9b\") " pod="openshift-ingress/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l" Nov 03 13:34:13.771505 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:13.771401 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l" Nov 03 13:34:14.207329 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:14.207290 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l"] Nov 03 13:34:14.219822 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:14.219788 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:34:14.219927 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:14.219861 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:34:14.219927 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:14.219889 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:34:14.747410 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:14.747369 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l" event={"ID":"ddd1f97b-814c-43b5-a81b-201bd1ef2c9b","Type":"ContainerStarted","Data":"01ff368eaf37eeb14f397fd8731124fe04ee652a6271303d9a55e7228f7b00fa"} Nov 03 13:34:14.747623 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:14.747436 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l" event={"ID":"ddd1f97b-814c-43b5-a81b-201bd1ef2c9b","Type":"ContainerStarted","Data":"c2d057d9669a2a56c70df97d8ad5305714f27774e81ce71d462c432561892262"} Nov 03 13:34:14.772186 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:14.772158 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l" Nov 03 13:34:14.862439 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:14.862385 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l" podStartSLOduration=1.862372711 podStartE2EDuration="1.862372711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:34:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:34:14.859878479 +0000 UTC m=+1736.870566721" watchObservedRunningTime="2026-11-03 13:34:14.862372711 +0000 UTC m=+1736.873060953" Nov 03 13:34:14.914777 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:14.914741 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-be8d0a-5cd4fcdcbf-2kt7h"] Nov 03 13:34:15.026178 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:15.026044 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-be8d0a-689bdb4cf5-l4wtk"] Nov 03 13:34:15.778138 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:15.778111 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l" Nov 03 13:34:16.784494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:16.767770 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l" Nov 03 13:34:16.784494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:16.769633 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l" Nov 03 13:34:19.219440 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:19.219405 5979 scope.go:122] "RemoveContainer" containerID="2eeb7233bdc9b02b3b4d2f968f604665b63e4fa70af980c86f7a2d78a7d02e18" Nov 03 13:34:19.236751 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:19.236725 5979 scope.go:122] "RemoveContainer" containerID="5b8a3322f250b20945fe1dd93cd7f5b55b40a5b49a618c3d562dea5fd89e9f5b" Nov 03 13:34:19.915886 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:19.915822 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-be8d0a-5cd4fcdcbf-2kt7h" podUID="43bab3d9-bb26-412c-b00d-3d980b572eeb" containerName="payload-processing" containerID="cri-o://db51c05f272c49422799cef406b5b23c1dc45f7f81ddb56e0ae46e115ffd3b6a" gracePeriod=25 Nov 03 13:34:20.026828 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:20.026761 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be8d0a-689bdb4cf5-l4wtk" podUID="5e533eaa-a35c-475e-a88e-6dbf38f3f37e" containerName="payload-pre-processing" containerID="cri-o://deb208ba8a8e33c6e94679d073c5c15deeefdb7693dfc8b1e0b9364ed5990db1" gracePeriod=25 Nov 03 13:34:20.185780 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:20.185750 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-be8d0a-5cd4fcdcbf-2kt7h" Nov 03 13:34:20.326159 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:20.326123 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43bab3d9-bb26-412c-b00d-3d980b572eeb-kube-api-access-wnm46\" (UniqueName: \"kubernetes.io/projected/43bab3d9-bb26-412c-b00d-3d980b572eeb-kube-api-access-wnm46\") pod \"43bab3d9-bb26-412c-b00d-3d980b572eeb\" (UID: \"43bab3d9-bb26-412c-b00d-3d980b572eeb\") " Nov 03 13:34:20.326673 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:20.326288 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43bab3d9-bb26-412c-b00d-3d980b572eeb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43bab3d9-bb26-412c-b00d-3d980b572eeb-plugins-config-volume\") pod \"43bab3d9-bb26-412c-b00d-3d980b572eeb\" (UID: \"43bab3d9-bb26-412c-b00d-3d980b572eeb\") " Nov 03 13:34:20.327415 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:20.327387 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43bab3d9-bb26-412c-b00d-3d980b572eeb-plugins-config-volume" pod "43bab3d9-bb26-412c-b00d-3d980b572eeb" (UID: "43bab3d9-bb26-412c-b00d-3d980b572eeb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:34:20.360426 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:20.360392 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43bab3d9-bb26-412c-b00d-3d980b572eeb-kube-api-access-wnm46" pod "43bab3d9-bb26-412c-b00d-3d980b572eeb" (UID: "43bab3d9-bb26-412c-b00d-3d980b572eeb"). InnerVolumeSpecName "kube-api-access-wnm46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:34:20.366352 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:20.366037 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be8d0a-689bdb4cf5-l4wtk" Nov 03 13:34:20.427895 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:20.427815 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43bab3d9-bb26-412c-b00d-3d980b572eeb-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:34:20.427895 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:20.427842 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnm46\" (UniqueName: \"kubernetes.io/projected/43bab3d9-bb26-412c-b00d-3d980b572eeb-kube-api-access-wnm46\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:34:20.529160 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:20.529131 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e533eaa-a35c-475e-a88e-6dbf38f3f37e-kube-api-access-l5wb4\" (UniqueName: \"kubernetes.io/projected/5e533eaa-a35c-475e-a88e-6dbf38f3f37e-kube-api-access-l5wb4\") pod \"5e533eaa-a35c-475e-a88e-6dbf38f3f37e\" (UID: \"5e533eaa-a35c-475e-a88e-6dbf38f3f37e\") " Nov 03 13:34:20.529333 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:20.529228 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e533eaa-a35c-475e-a88e-6dbf38f3f37e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e533eaa-a35c-475e-a88e-6dbf38f3f37e-plugins-config-volume\") pod \"5e533eaa-a35c-475e-a88e-6dbf38f3f37e\" (UID: \"5e533eaa-a35c-475e-a88e-6dbf38f3f37e\") " Nov 03 13:34:20.529709 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:20.529670 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e533eaa-a35c-475e-a88e-6dbf38f3f37e-plugins-config-volume" pod "5e533eaa-a35c-475e-a88e-6dbf38f3f37e" (UID: "5e533eaa-a35c-475e-a88e-6dbf38f3f37e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:34:20.531241 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:20.531216 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e533eaa-a35c-475e-a88e-6dbf38f3f37e-kube-api-access-l5wb4" pod "5e533eaa-a35c-475e-a88e-6dbf38f3f37e" (UID: "5e533eaa-a35c-475e-a88e-6dbf38f3f37e"). InnerVolumeSpecName "kube-api-access-l5wb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:34:20.630781 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:20.630750 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5wb4\" (UniqueName: \"kubernetes.io/projected/5e533eaa-a35c-475e-a88e-6dbf38f3f37e-kube-api-access-l5wb4\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:34:20.630781 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:20.630776 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e533eaa-a35c-475e-a88e-6dbf38f3f37e-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:34:20.796043 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:20.795946 5979 generic.go:356] "Generic (PLEG): container finished" podID="5e533eaa-a35c-475e-a88e-6dbf38f3f37e" containerID="deb208ba8a8e33c6e94679d073c5c15deeefdb7693dfc8b1e0b9364ed5990db1" exitCode=0 Nov 03 13:34:20.796043 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:20.796004 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be8d0a-689bdb4cf5-l4wtk" Nov 03 13:34:20.796268 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:20.796038 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be8d0a-689bdb4cf5-l4wtk" event={"ID":"5e533eaa-a35c-475e-a88e-6dbf38f3f37e","Type":"ContainerDied","Data":"deb208ba8a8e33c6e94679d073c5c15deeefdb7693dfc8b1e0b9364ed5990db1"} Nov 03 13:34:20.796268 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:20.796097 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be8d0a-689bdb4cf5-l4wtk" event={"ID":"5e533eaa-a35c-475e-a88e-6dbf38f3f37e","Type":"ContainerDied","Data":"1e8be6e5ff77acad1af3625b64416e977a956147e0337e1d7f729dbcaa395d93"} Nov 03 13:34:20.796268 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:20.796122 5979 scope.go:122] "RemoveContainer" containerID="deb208ba8a8e33c6e94679d073c5c15deeefdb7693dfc8b1e0b9364ed5990db1" Nov 03 13:34:20.797749 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:20.797724 5979 generic.go:356] "Generic (PLEG): container finished" podID="43bab3d9-bb26-412c-b00d-3d980b572eeb" containerID="db51c05f272c49422799cef406b5b23c1dc45f7f81ddb56e0ae46e115ffd3b6a" exitCode=0 Nov 03 13:34:20.797895 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:20.797753 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-be8d0a-5cd4fcdcbf-2kt7h" event={"ID":"43bab3d9-bb26-412c-b00d-3d980b572eeb","Type":"ContainerDied","Data":"db51c05f272c49422799cef406b5b23c1dc45f7f81ddb56e0ae46e115ffd3b6a"} Nov 03 13:34:20.797895 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:20.797778 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-be8d0a-5cd4fcdcbf-2kt7h" Nov 03 13:34:20.797895 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:20.797799 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-be8d0a-5cd4fcdcbf-2kt7h" event={"ID":"43bab3d9-bb26-412c-b00d-3d980b572eeb","Type":"ContainerDied","Data":"31fc49290e0096bb73dd5cfc83df269af8ecbc421a00c24e2181e514a6936169"} Nov 03 13:34:20.806446 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:20.806428 5979 scope.go:122] "RemoveContainer" containerID="deb208ba8a8e33c6e94679d073c5c15deeefdb7693dfc8b1e0b9364ed5990db1" Nov 03 13:34:20.806780 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:34:20.806750 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"deb208ba8a8e33c6e94679d073c5c15deeefdb7693dfc8b1e0b9364ed5990db1\": container with ID starting with deb208ba8a8e33c6e94679d073c5c15deeefdb7693dfc8b1e0b9364ed5990db1 not found: ID does not exist" containerID="deb208ba8a8e33c6e94679d073c5c15deeefdb7693dfc8b1e0b9364ed5990db1" Nov 03 13:34:20.806851 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:20.806790 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"deb208ba8a8e33c6e94679d073c5c15deeefdb7693dfc8b1e0b9364ed5990db1"} err="failed to get container status \"deb208ba8a8e33c6e94679d073c5c15deeefdb7693dfc8b1e0b9364ed5990db1\": rpc error: code = NotFound desc = could not find container \"deb208ba8a8e33c6e94679d073c5c15deeefdb7693dfc8b1e0b9364ed5990db1\": container with ID starting with deb208ba8a8e33c6e94679d073c5c15deeefdb7693dfc8b1e0b9364ed5990db1 not found: ID does not exist" Nov 03 13:34:20.806851 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:20.806815 5979 scope.go:122] "RemoveContainer" containerID="db51c05f272c49422799cef406b5b23c1dc45f7f81ddb56e0ae46e115ffd3b6a" Nov 03 13:34:20.816845 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:20.816827 5979 scope.go:122] "RemoveContainer" containerID="db51c05f272c49422799cef406b5b23c1dc45f7f81ddb56e0ae46e115ffd3b6a" Nov 03 13:34:20.817116 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:34:20.817094 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db51c05f272c49422799cef406b5b23c1dc45f7f81ddb56e0ae46e115ffd3b6a\": container with ID starting with db51c05f272c49422799cef406b5b23c1dc45f7f81ddb56e0ae46e115ffd3b6a not found: ID does not exist" containerID="db51c05f272c49422799cef406b5b23c1dc45f7f81ddb56e0ae46e115ffd3b6a" Nov 03 13:34:20.817177 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:20.817125 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db51c05f272c49422799cef406b5b23c1dc45f7f81ddb56e0ae46e115ffd3b6a"} err="failed to get container status \"db51c05f272c49422799cef406b5b23c1dc45f7f81ddb56e0ae46e115ffd3b6a\": rpc error: code = NotFound desc = could not find container \"db51c05f272c49422799cef406b5b23c1dc45f7f81ddb56e0ae46e115ffd3b6a\": container with ID starting with db51c05f272c49422799cef406b5b23c1dc45f7f81ddb56e0ae46e115ffd3b6a not found: ID does not exist" Nov 03 13:34:20.950016 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:20.949984 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-be8d0a-689bdb4cf5-l4wtk"] Nov 03 13:34:20.996767 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:20.996736 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-be8d0a-689bdb4cf5-l4wtk"] Nov 03 13:34:21.056563 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:21.056480 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-be8d0a-5cd4fcdcbf-2kt7h"] Nov 03 13:34:21.063176 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:21.063145 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-be8d0a-5cd4fcdcbf-2kt7h"] Nov 03 13:34:21.729092 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:21.729063 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c8192-7c4d4b65cf-wm2vg" Nov 03 13:34:21.729625 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:21.729143 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c8192-54f4dd4bf5-qjv8l" Nov 03 13:34:22.168493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:22.166188 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43bab3d9-bb26-412c-b00d-3d980b572eeb" path="/var/lib/kubelet/pods/43bab3d9-bb26-412c-b00d-3d980b572eeb/volumes" Nov 03 13:34:22.183699 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:22.183656 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e533eaa-a35c-475e-a88e-6dbf38f3f37e" path="/var/lib/kubelet/pods/5e533eaa-a35c-475e-a88e-6dbf38f3f37e/volumes" Nov 03 13:34:22.377496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:22.374831 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d29e07-58df5c765-x69w6"] Nov 03 13:34:22.393494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:22.392531 5979 cpu_manager.go:414] "containerMap: removing container" podUID="43bab3d9-bb26-412c-b00d-3d980b572eeb" containerName="payload-processing" Nov 03 13:34:22.393494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:22.392564 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43bab3d9-bb26-412c-b00d-3d980b572eeb" containerName="payload-processing" Nov 03 13:34:22.393494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:22.392586 5979 cpu_manager.go:414] "containerMap: removing container" podUID="5e533eaa-a35c-475e-a88e-6dbf38f3f37e" containerName="payload-pre-processing" Nov 03 13:34:22.393494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:22.392596 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e533eaa-a35c-475e-a88e-6dbf38f3f37e" containerName="payload-pre-processing" Nov 03 13:34:22.393494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:22.392771 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="43bab3d9-bb26-412c-b00d-3d980b572eeb" containerName="payload-processing" Nov 03 13:34:22.393494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:22.392786 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e533eaa-a35c-475e-a88e-6dbf38f3f37e" containerName="payload-pre-processing" Nov 03 13:34:22.415493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:22.407800 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d29e07-58df5c765-x69w6" Nov 03 13:34:22.440513 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:22.436070 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d29e07-58df5c765-x69w6"] Nov 03 13:34:22.462495 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:22.461538 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-d29e07\"" Nov 03 13:34:22.478494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:22.476279 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-d29e07-dockercfg-pczkm\"" Nov 03 13:34:22.572726 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:22.571742 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z858b\" (UniqueName: \"kubernetes.io/projected/bd3ebbea-ef03-4183-8e8e-cd89fb9cae36-kube-api-access-z858b\") pod \"payload-processing-e2e-ipp-b-d29e07-58df5c765-x69w6\" (UID: \"bd3ebbea-ef03-4183-8e8e-cd89fb9cae36\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d29e07-58df5c765-x69w6" Nov 03 13:34:22.572726 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:22.571795 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd3ebbea-ef03-4183-8e8e-cd89fb9cae36-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d29e07-58df5c765-x69w6\" (UID: \"bd3ebbea-ef03-4183-8e8e-cd89fb9cae36\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d29e07-58df5c765-x69w6" Nov 03 13:34:22.581112 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:22.581074 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d29e07-56b6bbfbcf-pjntv"] Nov 03 13:34:22.587526 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:22.585450 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d29e07-56b6bbfbcf-pjntv" Nov 03 13:34:22.631072 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:22.631039 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d29e07-56b6bbfbcf-pjntv"] Nov 03 13:34:22.672844 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:22.672811 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z858b\" (UniqueName: \"kubernetes.io/projected/bd3ebbea-ef03-4183-8e8e-cd89fb9cae36-kube-api-access-z858b\") pod \"payload-processing-e2e-ipp-b-d29e07-58df5c765-x69w6\" (UID: \"bd3ebbea-ef03-4183-8e8e-cd89fb9cae36\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d29e07-58df5c765-x69w6" Nov 03 13:34:22.672844 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:22.672852 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd3ebbea-ef03-4183-8e8e-cd89fb9cae36-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d29e07-58df5c765-x69w6\" (UID: \"bd3ebbea-ef03-4183-8e8e-cd89fb9cae36\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d29e07-58df5c765-x69w6" Nov 03 13:34:22.673079 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:22.672879 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c72f8e91-a138-46d6-8440-94cac7793866-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d29e07-56b6bbfbcf-pjntv\" (UID: \"c72f8e91-a138-46d6-8440-94cac7793866\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d29e07-56b6bbfbcf-pjntv" Nov 03 13:34:22.673079 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:22.672922 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhrvm\" (UniqueName: \"kubernetes.io/projected/c72f8e91-a138-46d6-8440-94cac7793866-kube-api-access-lhrvm\") pod \"payload-pre-processing-e2e-ipp-b-d29e07-56b6bbfbcf-pjntv\" (UID: \"c72f8e91-a138-46d6-8440-94cac7793866\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d29e07-56b6bbfbcf-pjntv" Nov 03 13:34:22.673547 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:22.673528 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd3ebbea-ef03-4183-8e8e-cd89fb9cae36-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d29e07-58df5c765-x69w6\" (UID: \"bd3ebbea-ef03-4183-8e8e-cd89fb9cae36\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d29e07-58df5c765-x69w6" Nov 03 13:34:22.727613 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:22.727540 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z858b\" (UniqueName: \"kubernetes.io/projected/bd3ebbea-ef03-4183-8e8e-cd89fb9cae36-kube-api-access-z858b\") pod \"payload-processing-e2e-ipp-b-d29e07-58df5c765-x69w6\" (UID: \"bd3ebbea-ef03-4183-8e8e-cd89fb9cae36\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d29e07-58df5c765-x69w6" Nov 03 13:34:22.754895 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:22.754863 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d29e07-58df5c765-x69w6" Nov 03 13:34:22.774112 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:22.774087 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c72f8e91-a138-46d6-8440-94cac7793866-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d29e07-56b6bbfbcf-pjntv\" (UID: \"c72f8e91-a138-46d6-8440-94cac7793866\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d29e07-56b6bbfbcf-pjntv" Nov 03 13:34:22.774217 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:22.774133 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhrvm\" (UniqueName: \"kubernetes.io/projected/c72f8e91-a138-46d6-8440-94cac7793866-kube-api-access-lhrvm\") pod \"payload-pre-processing-e2e-ipp-b-d29e07-56b6bbfbcf-pjntv\" (UID: \"c72f8e91-a138-46d6-8440-94cac7793866\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d29e07-56b6bbfbcf-pjntv" Nov 03 13:34:22.775017 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:22.774991 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c72f8e91-a138-46d6-8440-94cac7793866-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d29e07-56b6bbfbcf-pjntv\" (UID: \"c72f8e91-a138-46d6-8440-94cac7793866\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d29e07-56b6bbfbcf-pjntv" Nov 03 13:34:22.828935 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:22.828909 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhrvm\" (UniqueName: \"kubernetes.io/projected/c72f8e91-a138-46d6-8440-94cac7793866-kube-api-access-lhrvm\") pod \"payload-pre-processing-e2e-ipp-b-d29e07-56b6bbfbcf-pjntv\" (UID: \"c72f8e91-a138-46d6-8440-94cac7793866\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d29e07-56b6bbfbcf-pjntv" Nov 03 13:34:22.907805 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:22.907771 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d29e07-56b6bbfbcf-pjntv" Nov 03 13:34:23.137595 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:23.137564 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d29e07-58df5c765-x69w6"] Nov 03 13:34:23.143758 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:34:23.143728 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd3ebbea_ef03_4183_8e8e_cd89fb9cae36.slice/crio-7ffb43c506a8325e2802c53d4d6f7202fdbe1268f693b204a396a333083ab477 WatchSource:0}: Error finding container 7ffb43c506a8325e2802c53d4d6f7202fdbe1268f693b204a396a333083ab477: Status 404 returned error can't find the container with id 7ffb43c506a8325e2802c53d4d6f7202fdbe1268f693b204a396a333083ab477 Nov 03 13:34:23.281667 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:23.281634 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d29e07-56b6bbfbcf-pjntv"] Nov 03 13:34:23.288224 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:34:23.288196 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc72f8e91_a138_46d6_8440_94cac7793866.slice/crio-27324ed7f30d60e8fedd5b01de88e86d41e394ffaf45d6a5fddd4dd198997a99 WatchSource:0}: Error finding container 27324ed7f30d60e8fedd5b01de88e86d41e394ffaf45d6a5fddd4dd198997a99: Status 404 returned error can't find the container with id 27324ed7f30d60e8fedd5b01de88e86d41e394ffaf45d6a5fddd4dd198997a99 Nov 03 13:34:23.825445 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:23.825406 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d29e07-56b6bbfbcf-pjntv" event={"ID":"c72f8e91-a138-46d6-8440-94cac7793866","Type":"ContainerStarted","Data":"72e1dd56071e5017d6fa1b4962d8354354f6d0ea3f9d5215606fca1f601f10df"} Nov 03 13:34:23.825989 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:23.825497 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d29e07-56b6bbfbcf-pjntv" event={"ID":"c72f8e91-a138-46d6-8440-94cac7793866","Type":"ContainerStarted","Data":"27324ed7f30d60e8fedd5b01de88e86d41e394ffaf45d6a5fddd4dd198997a99"} Nov 03 13:34:23.825989 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:23.825633 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d29e07-56b6bbfbcf-pjntv" Nov 03 13:34:23.827251 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:23.827221 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d29e07-58df5c765-x69w6" event={"ID":"bd3ebbea-ef03-4183-8e8e-cd89fb9cae36","Type":"ContainerStarted","Data":"43106f1f127806d43a3a50abc2cdffa5ee8b39776f9ea343fe592ec081d1d0ad"} Nov 03 13:34:23.827391 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:23.827268 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d29e07-58df5c765-x69w6" event={"ID":"bd3ebbea-ef03-4183-8e8e-cd89fb9cae36","Type":"ContainerStarted","Data":"7ffb43c506a8325e2802c53d4d6f7202fdbe1268f693b204a396a333083ab477"} Nov 03 13:34:23.827391 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:23.827292 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d29e07-58df5c765-x69w6" Nov 03 13:34:23.936201 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:23.936143 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d29e07-56b6bbfbcf-pjntv" podStartSLOduration=1.936130565 podStartE2EDuration="1.936130565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:34:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:34:23.934052132 +0000 UTC m=+1745.944740375" watchObservedRunningTime="2026-11-03 13:34:23.936130565 +0000 UTC m=+1745.946818807" Nov 03 13:34:24.044434 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:24.044376 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-d29e07-58df5c765-x69w6" podStartSLOduration=2.044362322 podStartE2EDuration="2.044362322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:34:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:34:24.04137933 +0000 UTC m=+1746.052067566" watchObservedRunningTime="2026-11-03 13:34:24.044362322 +0000 UTC m=+1746.055050563" Nov 03 13:34:34.838275 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:34.838239 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d29e07-56b6bbfbcf-pjntv" Nov 03 13:34:34.838836 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:34.838640 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d29e07-58df5c765-x69w6" Nov 03 13:34:37.235367 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:34:37.235330 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:34:43.359689 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:43.359655 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw"] Nov 03 13:34:43.360098 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:43.359859 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw" podUID="e5478d2d-1afc-4d18-9e91-e1965f3f51c0" containerName="istio-proxy" containerID="cri-o://be2293debe68f293b71ce940c4694059c4861b51eabf8ea268630439a72c0a2a" gracePeriod=30 Nov 03 13:34:46.758336 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:46.758300 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5295ab29-pcxj8"] Nov 03 13:34:46.763196 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:46.763173 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5295ab29-pcxj8" Nov 03 13:34:46.816267 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:46.816237 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5295ab29-pcxj8"] Nov 03 13:34:46.858507 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:46.858461 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:34:46.858507 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:46.858506 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:34:46.874328 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:46.874294 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4c6bbfe-7d2c-498e-9a5d-f2194f4d7d9c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-5295ab29-pcxj8\" (UID: \"d4c6bbfe-7d2c-498e-9a5d-f2194f4d7d9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5295ab29-pcxj8" Nov 03 13:34:46.874465 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:46.874436 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s62js\" (UniqueName: \"kubernetes.io/projected/d4c6bbfe-7d2c-498e-9a5d-f2194f4d7d9c-kube-api-access-s62js\") pod \"maas-api-revoke-keys-e2e-mt-5295ab29-pcxj8\" (UID: \"d4c6bbfe-7d2c-498e-9a5d-f2194f4d7d9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5295ab29-pcxj8" Nov 03 13:34:46.874644 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:46.874625 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4c6bbfe-7d2c-498e-9a5d-f2194f4d7d9c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-5295ab29-pcxj8\" (UID: \"d4c6bbfe-7d2c-498e-9a5d-f2194f4d7d9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5295ab29-pcxj8" Nov 03 13:34:46.976277 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:46.976238 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4c6bbfe-7d2c-498e-9a5d-f2194f4d7d9c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-5295ab29-pcxj8\" (UID: \"d4c6bbfe-7d2c-498e-9a5d-f2194f4d7d9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5295ab29-pcxj8" Nov 03 13:34:46.976449 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:46.976337 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s62js\" (UniqueName: \"kubernetes.io/projected/d4c6bbfe-7d2c-498e-9a5d-f2194f4d7d9c-kube-api-access-s62js\") pod \"maas-api-revoke-keys-e2e-mt-5295ab29-pcxj8\" (UID: \"d4c6bbfe-7d2c-498e-9a5d-f2194f4d7d9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5295ab29-pcxj8" Nov 03 13:34:46.976449 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:46.976439 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4c6bbfe-7d2c-498e-9a5d-f2194f4d7d9c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-5295ab29-pcxj8\" (UID: \"d4c6bbfe-7d2c-498e-9a5d-f2194f4d7d9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5295ab29-pcxj8" Nov 03 13:34:46.976971 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:46.976941 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4c6bbfe-7d2c-498e-9a5d-f2194f4d7d9c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-5295ab29-pcxj8\" (UID: \"d4c6bbfe-7d2c-498e-9a5d-f2194f4d7d9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5295ab29-pcxj8" Nov 03 13:34:46.978785 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:46.978762 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4c6bbfe-7d2c-498e-9a5d-f2194f4d7d9c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-5295ab29-pcxj8\" (UID: \"d4c6bbfe-7d2c-498e-9a5d-f2194f4d7d9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5295ab29-pcxj8" Nov 03 13:34:47.032779 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:47.032711 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s62js\" (UniqueName: \"kubernetes.io/projected/d4c6bbfe-7d2c-498e-9a5d-f2194f4d7d9c-kube-api-access-s62js\") pod \"maas-api-revoke-keys-e2e-mt-5295ab29-pcxj8\" (UID: \"d4c6bbfe-7d2c-498e-9a5d-f2194f4d7d9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5295ab29-pcxj8" Nov 03 13:34:47.078301 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:47.078261 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5295ab29-pcxj8" Nov 03 13:34:47.454482 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:47.454443 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5295ab29-pcxj8"] Nov 03 13:34:47.461496 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:34:47.461434 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4c6bbfe_7d2c_498e_9a5d_f2194f4d7d9c.slice/crio-5914ca42af3a0d84b85b499b48c083d8c50c98f3303f320bd6bb2741332ff2c9 WatchSource:0}: Error finding container 5914ca42af3a0d84b85b499b48c083d8c50c98f3303f320bd6bb2741332ff2c9: Status 404 returned error can't find the container with id 5914ca42af3a0d84b85b499b48c083d8c50c98f3303f320bd6bb2741332ff2c9 Nov 03 13:34:48.072451 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:48.065874 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5295ab29-pcxj8" event={"ID":"d4c6bbfe-7d2c-498e-9a5d-f2194f4d7d9c","Type":"ContainerStarted","Data":"eefcfb9dc482406a86b0d8dbf0a1e90c62d8a2d9ed10e71af9c1bf1db3c11862"} Nov 03 13:34:48.072451 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:48.065940 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5295ab29-pcxj8" event={"ID":"d4c6bbfe-7d2c-498e-9a5d-f2194f4d7d9c","Type":"ContainerStarted","Data":"5914ca42af3a0d84b85b499b48c083d8c50c98f3303f320bd6bb2741332ff2c9"} Nov 03 13:34:48.176386 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:48.176307 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5295ab29-pcxj8" podStartSLOduration=2.176289603 podStartE2EDuration="2.176289603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:34:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:34:48.175420999 +0000 UTC m=+1770.186109241" watchObservedRunningTime="2026-11-03 13:34:48.176289603 +0000 UTC m=+1770.186977846" Nov 03 13:34:48.678493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:48.672881 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw" Nov 03 13:34:48.837602 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:48.835950 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-credential-socket\") pod \"e5478d2d-1afc-4d18-9e91-e1965f3f51c0\" (UID: \"e5478d2d-1afc-4d18-9e91-e1965f3f51c0\") " Nov 03 13:34:48.837602 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:48.836029 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-workload-socket\") pod \"e5478d2d-1afc-4d18-9e91-e1965f3f51c0\" (UID: \"e5478d2d-1afc-4d18-9e91-e1965f3f51c0\") " Nov 03 13:34:48.837602 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:48.836075 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-workload-certs\") pod \"e5478d2d-1afc-4d18-9e91-e1965f3f51c0\" (UID: \"e5478d2d-1afc-4d18-9e91-e1965f3f51c0\") " Nov 03 13:34:48.837602 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:48.836151 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-istiod-ca-cert\") pod \"e5478d2d-1afc-4d18-9e91-e1965f3f51c0\" (UID: \"e5478d2d-1afc-4d18-9e91-e1965f3f51c0\") " Nov 03 13:34:48.837602 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:48.836228 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-istio-data\") pod \"e5478d2d-1afc-4d18-9e91-e1965f3f51c0\" (UID: \"e5478d2d-1afc-4d18-9e91-e1965f3f51c0\") " Nov 03 13:34:48.837602 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:48.836293 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-istio-podinfo\") pod \"e5478d2d-1afc-4d18-9e91-e1965f3f51c0\" (UID: \"e5478d2d-1afc-4d18-9e91-e1965f3f51c0\") " Nov 03 13:34:48.837602 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:48.836328 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-istio-token\" (UniqueName: \"kubernetes.io/projected/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-istio-token\") pod \"e5478d2d-1afc-4d18-9e91-e1965f3f51c0\" (UID: \"e5478d2d-1afc-4d18-9e91-e1965f3f51c0\") " Nov 03 13:34:48.837602 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:48.836447 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-kube-api-access-4zd4d\" (UniqueName: \"kubernetes.io/projected/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-kube-api-access-4zd4d\") pod \"e5478d2d-1afc-4d18-9e91-e1965f3f51c0\" (UID: \"e5478d2d-1afc-4d18-9e91-e1965f3f51c0\") " Nov 03 13:34:48.837602 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:48.836557 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-istio-envoy\") pod \"e5478d2d-1afc-4d18-9e91-e1965f3f51c0\" (UID: \"e5478d2d-1afc-4d18-9e91-e1965f3f51c0\") " Nov 03 13:34:48.840589 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:48.840554 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-workload-socket" pod "e5478d2d-1afc-4d18-9e91-e1965f3f51c0" (UID: "e5478d2d-1afc-4d18-9e91-e1965f3f51c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:34:48.840998 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:48.840975 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-credential-socket" pod "e5478d2d-1afc-4d18-9e91-e1965f3f51c0" (UID: "e5478d2d-1afc-4d18-9e91-e1965f3f51c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:34:48.843968 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:48.843943 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-istiod-ca-cert" pod "e5478d2d-1afc-4d18-9e91-e1965f3f51c0" (UID: "e5478d2d-1afc-4d18-9e91-e1965f3f51c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:34:48.846331 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:48.846295 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-workload-certs" pod "e5478d2d-1afc-4d18-9e91-e1965f3f51c0" (UID: "e5478d2d-1afc-4d18-9e91-e1965f3f51c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:34:48.846536 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:48.844133 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-istio-data" pod "e5478d2d-1afc-4d18-9e91-e1965f3f51c0" (UID: "e5478d2d-1afc-4d18-9e91-e1965f3f51c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:34:48.851016 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:48.850968 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-istio-envoy" pod "e5478d2d-1afc-4d18-9e91-e1965f3f51c0" (UID: "e5478d2d-1afc-4d18-9e91-e1965f3f51c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:34:48.851692 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:48.851663 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-istio-podinfo" pod "e5478d2d-1afc-4d18-9e91-e1965f3f51c0" (UID: "e5478d2d-1afc-4d18-9e91-e1965f3f51c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:34:48.857497 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:48.853955 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-kube-api-access-4zd4d" pod "e5478d2d-1afc-4d18-9e91-e1965f3f51c0" (UID: "e5478d2d-1afc-4d18-9e91-e1965f3f51c0"). InnerVolumeSpecName "kube-api-access-4zd4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:34:48.867529 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:48.865166 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-istio-token" pod "e5478d2d-1afc-4d18-9e91-e1965f3f51c0" (UID: "e5478d2d-1afc-4d18-9e91-e1965f3f51c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:34:48.938560 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:48.938384 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zd4d\" (UniqueName: \"kubernetes.io/projected/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-kube-api-access-4zd4d\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:34:48.938560 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:48.938414 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:34:48.938560 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:48.938425 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:34:48.938560 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:48.938433 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:34:48.938560 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:48.938443 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:34:48.938560 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:48.938451 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:34:48.938560 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:48.938460 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:34:48.938560 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:48.938482 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:34:48.938560 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:48.938492 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5478d2d-1afc-4d18-9e91-e1965f3f51c0-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:34:49.125684 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:49.125649 5979 generic.go:356] "Generic (PLEG): container finished" podID="d4c6bbfe-7d2c-498e-9a5d-f2194f4d7d9c" containerID="eefcfb9dc482406a86b0d8dbf0a1e90c62d8a2d9ed10e71af9c1bf1db3c11862" exitCode=0 Nov 03 13:34:49.126437 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:49.126294 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5295ab29-pcxj8" event={"ID":"d4c6bbfe-7d2c-498e-9a5d-f2194f4d7d9c","Type":"ContainerDied","Data":"eefcfb9dc482406a86b0d8dbf0a1e90c62d8a2d9ed10e71af9c1bf1db3c11862"} Nov 03 13:34:49.131508 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:49.129521 5979 generic.go:356] "Generic (PLEG): container finished" podID="e5478d2d-1afc-4d18-9e91-e1965f3f51c0" containerID="be2293debe68f293b71ce940c4694059c4861b51eabf8ea268630439a72c0a2a" exitCode=0 Nov 03 13:34:49.131508 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:49.129572 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw" event={"ID":"e5478d2d-1afc-4d18-9e91-e1965f3f51c0","Type":"ContainerDied","Data":"be2293debe68f293b71ce940c4694059c4861b51eabf8ea268630439a72c0a2a"} Nov 03 13:34:49.131508 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:49.129613 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw" event={"ID":"e5478d2d-1afc-4d18-9e91-e1965f3f51c0","Type":"ContainerDied","Data":"cb53c77d2317236788eff372f383a4c359a5639fd85431e03de9c0e6fef19619"} Nov 03 13:34:49.131508 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:49.129641 5979 scope.go:122] "RemoveContainer" containerID="be2293debe68f293b71ce940c4694059c4861b51eabf8ea268630439a72c0a2a" Nov 03 13:34:49.131508 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:49.129788 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw" Nov 03 13:34:49.160714 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:49.160066 5979 scope.go:122] "RemoveContainer" containerID="4c37c527438e7778be040ed075fd4a776f6721feb812f6539f6939850020aba2" Nov 03 13:34:49.176936 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:49.176875 5979 scope.go:122] "RemoveContainer" containerID="be2293debe68f293b71ce940c4694059c4861b51eabf8ea268630439a72c0a2a" Nov 03 13:34:49.177367 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:34:49.177342 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be2293debe68f293b71ce940c4694059c4861b51eabf8ea268630439a72c0a2a\": container with ID starting with be2293debe68f293b71ce940c4694059c4861b51eabf8ea268630439a72c0a2a not found: ID does not exist" containerID="be2293debe68f293b71ce940c4694059c4861b51eabf8ea268630439a72c0a2a" Nov 03 13:34:49.177518 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:49.177376 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be2293debe68f293b71ce940c4694059c4861b51eabf8ea268630439a72c0a2a"} err="failed to get container status \"be2293debe68f293b71ce940c4694059c4861b51eabf8ea268630439a72c0a2a\": rpc error: code = NotFound desc = could not find container \"be2293debe68f293b71ce940c4694059c4861b51eabf8ea268630439a72c0a2a\": container with ID starting with be2293debe68f293b71ce940c4694059c4861b51eabf8ea268630439a72c0a2a not found: ID does not exist" Nov 03 13:34:49.177518 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:49.177406 5979 scope.go:122] "RemoveContainer" containerID="4c37c527438e7778be040ed075fd4a776f6721feb812f6539f6939850020aba2" Nov 03 13:34:49.177794 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:34:49.177774 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c37c527438e7778be040ed075fd4a776f6721feb812f6539f6939850020aba2\": container with ID starting with 4c37c527438e7778be040ed075fd4a776f6721feb812f6539f6939850020aba2 not found: ID does not exist" containerID="4c37c527438e7778be040ed075fd4a776f6721feb812f6539f6939850020aba2" Nov 03 13:34:49.177894 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:49.177800 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c37c527438e7778be040ed075fd4a776f6721feb812f6539f6939850020aba2"} err="failed to get container status \"4c37c527438e7778be040ed075fd4a776f6721feb812f6539f6939850020aba2\": rpc error: code = NotFound desc = could not find container \"4c37c527438e7778be040ed075fd4a776f6721feb812f6539f6939850020aba2\": container with ID starting with 4c37c527438e7778be040ed075fd4a776f6721feb812f6539f6939850020aba2 not found: ID does not exist" Nov 03 13:34:49.291393 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:49.291291 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:34:49.354774 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:49.354732 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw"] Nov 03 13:34:49.359841 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:49.359813 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-461b2d-openshift-default-74889b4fd7-mwqlw"] Nov 03 13:34:49.682670 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:49.682632 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fa2568-67ffc88795-t7jsv"] Nov 03 13:34:49.683249 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:49.683235 5979 cpu_manager.go:414] "containerMap: removing container" podUID="e5478d2d-1afc-4d18-9e91-e1965f3f51c0" containerName="istio-proxy" Nov 03 13:34:49.683303 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:49.683250 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5478d2d-1afc-4d18-9e91-e1965f3f51c0" containerName="istio-proxy" Nov 03 13:34:49.683303 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:49.683266 5979 cpu_manager.go:414] "containerMap: removing container" podUID="e5478d2d-1afc-4d18-9e91-e1965f3f51c0" containerName="istio-proxy" Nov 03 13:34:49.683303 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:49.683272 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5478d2d-1afc-4d18-9e91-e1965f3f51c0" containerName="istio-proxy" Nov 03 13:34:49.683458 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:49.683404 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5478d2d-1afc-4d18-9e91-e1965f3f51c0" containerName="istio-proxy" Nov 03 13:34:49.683458 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:49.683419 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5478d2d-1afc-4d18-9e91-e1965f3f51c0" containerName="istio-proxy" Nov 03 13:34:49.688528 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:49.688507 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa2568-67ffc88795-t7jsv" Nov 03 13:34:49.741758 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:49.741724 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-fa2568-dockercfg-nvm9v\"" Nov 03 13:34:49.742997 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:49.742971 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-fa2568\"" Nov 03 13:34:49.746161 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:49.746132 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fa2568-67ffc88795-t7jsv"] Nov 03 13:34:49.851583 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:49.851546 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lp5p\" (UniqueName: \"kubernetes.io/projected/0f997da5-e10f-4926-92c0-f5559fb13567-kube-api-access-8lp5p\") pod \"payload-pre-processing-e2e-ipp-a-fa2568-67ffc88795-t7jsv\" (UID: \"0f997da5-e10f-4926-92c0-f5559fb13567\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa2568-67ffc88795-t7jsv" Nov 03 13:34:49.851583 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:49.851582 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f997da5-e10f-4926-92c0-f5559fb13567-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fa2568-67ffc88795-t7jsv\" (UID: \"0f997da5-e10f-4926-92c0-f5559fb13567\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa2568-67ffc88795-t7jsv" Nov 03 13:34:49.952859 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:49.952762 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lp5p\" (UniqueName: \"kubernetes.io/projected/0f997da5-e10f-4926-92c0-f5559fb13567-kube-api-access-8lp5p\") pod \"payload-pre-processing-e2e-ipp-a-fa2568-67ffc88795-t7jsv\" (UID: \"0f997da5-e10f-4926-92c0-f5559fb13567\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa2568-67ffc88795-t7jsv" Nov 03 13:34:49.952859 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:49.952807 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f997da5-e10f-4926-92c0-f5559fb13567-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fa2568-67ffc88795-t7jsv\" (UID: \"0f997da5-e10f-4926-92c0-f5559fb13567\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa2568-67ffc88795-t7jsv" Nov 03 13:34:49.953633 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:49.953614 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f997da5-e10f-4926-92c0-f5559fb13567-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fa2568-67ffc88795-t7jsv\" (UID: \"0f997da5-e10f-4926-92c0-f5559fb13567\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa2568-67ffc88795-t7jsv" Nov 03 13:34:50.008796 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:50.008764 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lp5p\" (UniqueName: \"kubernetes.io/projected/0f997da5-e10f-4926-92c0-f5559fb13567-kube-api-access-8lp5p\") pod \"payload-pre-processing-e2e-ipp-a-fa2568-67ffc88795-t7jsv\" (UID: \"0f997da5-e10f-4926-92c0-f5559fb13567\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa2568-67ffc88795-t7jsv" Nov 03 13:34:50.116459 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:50.116422 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5478d2d-1afc-4d18-9e91-e1965f3f51c0" path="/var/lib/kubelet/pods/e5478d2d-1afc-4d18-9e91-e1965f3f51c0/volumes" Nov 03 13:34:50.284628 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:50.284598 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5295ab29-pcxj8" Nov 03 13:34:50.303207 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:50.303184 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa2568-67ffc88795-t7jsv" Nov 03 13:34:50.330798 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:50.330754 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fa2568-6cbd46f946-6fn4v"] Nov 03 13:34:50.331665 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:50.331643 5979 cpu_manager.go:414] "containerMap: removing container" podUID="d4c6bbfe-7d2c-498e-9a5d-f2194f4d7d9c" containerName="revoke-keys" Nov 03 13:34:50.331665 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:50.331664 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4c6bbfe-7d2c-498e-9a5d-f2194f4d7d9c" containerName="revoke-keys" Nov 03 13:34:50.331889 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:50.331875 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4c6bbfe-7d2c-498e-9a5d-f2194f4d7d9c" containerName="revoke-keys" Nov 03 13:34:50.335917 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:50.335887 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa2568-6cbd46f946-6fn4v" Nov 03 13:34:50.391763 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:50.391731 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fa2568-6cbd46f946-6fn4v"] Nov 03 13:34:50.459813 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:50.459786 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4c6bbfe-7d2c-498e-9a5d-f2194f4d7d9c-kube-api-access-s62js\" (UniqueName: \"kubernetes.io/projected/d4c6bbfe-7d2c-498e-9a5d-f2194f4d7d9c-kube-api-access-s62js\") pod \"d4c6bbfe-7d2c-498e-9a5d-f2194f4d7d9c\" (UID: \"d4c6bbfe-7d2c-498e-9a5d-f2194f4d7d9c\") " Nov 03 13:34:50.460007 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:50.459846 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4c6bbfe-7d2c-498e-9a5d-f2194f4d7d9c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4c6bbfe-7d2c-498e-9a5d-f2194f4d7d9c-maas-api-xks-ca\") pod \"d4c6bbfe-7d2c-498e-9a5d-f2194f4d7d9c\" (UID: \"d4c6bbfe-7d2c-498e-9a5d-f2194f4d7d9c\") " Nov 03 13:34:50.460092 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:50.460012 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4c6bbfe-7d2c-498e-9a5d-f2194f4d7d9c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4c6bbfe-7d2c-498e-9a5d-f2194f4d7d9c-maas-api-openshift-ca\") pod \"d4c6bbfe-7d2c-498e-9a5d-f2194f4d7d9c\" (UID: \"d4c6bbfe-7d2c-498e-9a5d-f2194f4d7d9c\") " Nov 03 13:34:50.460299 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:50.460274 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p86n7\" (UniqueName: \"kubernetes.io/projected/92d72487-02a6-4172-a2b7-9d13411a94f7-kube-api-access-p86n7\") pod \"payload-processing-e2e-ipp-a-fa2568-6cbd46f946-6fn4v\" (UID: \"92d72487-02a6-4172-a2b7-9d13411a94f7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fa2568-6cbd46f946-6fn4v" Nov 03 13:34:50.460424 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:50.460336 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4c6bbfe-7d2c-498e-9a5d-f2194f4d7d9c-maas-api-openshift-ca" pod "d4c6bbfe-7d2c-498e-9a5d-f2194f4d7d9c" (UID: "d4c6bbfe-7d2c-498e-9a5d-f2194f4d7d9c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:34:50.460511 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:50.460423 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92d72487-02a6-4172-a2b7-9d13411a94f7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fa2568-6cbd46f946-6fn4v\" (UID: \"92d72487-02a6-4172-a2b7-9d13411a94f7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fa2568-6cbd46f946-6fn4v" Nov 03 13:34:50.460725 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:50.460706 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4c6bbfe-7d2c-498e-9a5d-f2194f4d7d9c-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:34:50.461879 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:50.461855 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4c6bbfe-7d2c-498e-9a5d-f2194f4d7d9c-kube-api-access-s62js" pod "d4c6bbfe-7d2c-498e-9a5d-f2194f4d7d9c" (UID: "d4c6bbfe-7d2c-498e-9a5d-f2194f4d7d9c"). InnerVolumeSpecName "kube-api-access-s62js". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:34:50.461957 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:50.461938 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4c6bbfe-7d2c-498e-9a5d-f2194f4d7d9c-maas-api-xks-ca" pod "d4c6bbfe-7d2c-498e-9a5d-f2194f4d7d9c" (UID: "d4c6bbfe-7d2c-498e-9a5d-f2194f4d7d9c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:34:50.562113 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:50.562031 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92d72487-02a6-4172-a2b7-9d13411a94f7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fa2568-6cbd46f946-6fn4v\" (UID: \"92d72487-02a6-4172-a2b7-9d13411a94f7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fa2568-6cbd46f946-6fn4v" Nov 03 13:34:50.562317 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:50.562279 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p86n7\" (UniqueName: \"kubernetes.io/projected/92d72487-02a6-4172-a2b7-9d13411a94f7-kube-api-access-p86n7\") pod \"payload-processing-e2e-ipp-a-fa2568-6cbd46f946-6fn4v\" (UID: \"92d72487-02a6-4172-a2b7-9d13411a94f7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fa2568-6cbd46f946-6fn4v" Nov 03 13:34:50.562406 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:50.562380 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s62js\" (UniqueName: \"kubernetes.io/projected/d4c6bbfe-7d2c-498e-9a5d-f2194f4d7d9c-kube-api-access-s62js\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:34:50.562406 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:50.562400 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4c6bbfe-7d2c-498e-9a5d-f2194f4d7d9c-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:34:50.562850 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:50.562828 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92d72487-02a6-4172-a2b7-9d13411a94f7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fa2568-6cbd46f946-6fn4v\" (UID: \"92d72487-02a6-4172-a2b7-9d13411a94f7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fa2568-6cbd46f946-6fn4v" Nov 03 13:34:50.625772 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:50.625743 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p86n7\" (UniqueName: \"kubernetes.io/projected/92d72487-02a6-4172-a2b7-9d13411a94f7-kube-api-access-p86n7\") pod \"payload-processing-e2e-ipp-a-fa2568-6cbd46f946-6fn4v\" (UID: \"92d72487-02a6-4172-a2b7-9d13411a94f7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fa2568-6cbd46f946-6fn4v" Nov 03 13:34:50.653362 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:50.653330 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa2568-6cbd46f946-6fn4v" Nov 03 13:34:50.699164 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:50.699131 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fa2568-67ffc88795-t7jsv"] Nov 03 13:34:50.704593 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:34:50.704555 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f997da5_e10f_4926_92c0_f5559fb13567.slice/crio-d587384f98d3997ee2433ff5208082daf5d6650975402e59d5e3de8ae6532210 WatchSource:0}: Error finding container d587384f98d3997ee2433ff5208082daf5d6650975402e59d5e3de8ae6532210: Status 404 returned error can't find the container with id d587384f98d3997ee2433ff5208082daf5d6650975402e59d5e3de8ae6532210 Nov 03 13:34:51.035062 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:51.035032 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fa2568-6cbd46f946-6fn4v"] Nov 03 13:34:51.041080 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:34:51.041047 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92d72487_02a6_4172_a2b7_9d13411a94f7.slice/crio-0d5a8ee220b3962423684259f2f93f25cb183ce0085f79f4be4787448b336dce WatchSource:0}: Error finding container 0d5a8ee220b3962423684259f2f93f25cb183ce0085f79f4be4787448b336dce: Status 404 returned error can't find the container with id 0d5a8ee220b3962423684259f2f93f25cb183ce0085f79f4be4787448b336dce Nov 03 13:34:51.146908 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:51.146876 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa2568-6cbd46f946-6fn4v" event={"ID":"92d72487-02a6-4172-a2b7-9d13411a94f7","Type":"ContainerStarted","Data":"0d5a8ee220b3962423684259f2f93f25cb183ce0085f79f4be4787448b336dce"} Nov 03 13:34:51.148428 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:51.148407 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5295ab29-pcxj8" Nov 03 13:34:51.148428 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:51.148419 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5295ab29-pcxj8" event={"ID":"d4c6bbfe-7d2c-498e-9a5d-f2194f4d7d9c","Type":"ContainerDied","Data":"5914ca42af3a0d84b85b499b48c083d8c50c98f3303f320bd6bb2741332ff2c9"} Nov 03 13:34:51.148614 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:51.148462 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5914ca42af3a0d84b85b499b48c083d8c50c98f3303f320bd6bb2741332ff2c9" Nov 03 13:34:51.149824 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:51.149784 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa2568-67ffc88795-t7jsv" event={"ID":"0f997da5-e10f-4926-92c0-f5559fb13567","Type":"ContainerStarted","Data":"d587384f98d3997ee2433ff5208082daf5d6650975402e59d5e3de8ae6532210"} Nov 03 13:34:51.584089 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:51.584051 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6f9ed-7f68fdb665-jhx2v"] Nov 03 13:34:51.589680 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:51.589660 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6f9ed-7f68fdb665-jhx2v" Nov 03 13:34:51.642486 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:51.642440 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-f6f9ed-serving-cert\"" Nov 03 13:34:51.647114 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:51.647082 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6f9ed-7f68fdb665-jhx2v"] Nov 03 13:34:51.677186 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:51.677155 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4839945c-72ce-4a48-9879-1d94da9c3b48-metrics-certs\") pod \"maas-api-e2e-ipp-a-f6f9ed-7f68fdb665-jhx2v\" (UID: \"4839945c-72ce-4a48-9879-1d94da9c3b48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6f9ed-7f68fdb665-jhx2v" Nov 03 13:34:51.677353 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:51.677234 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnzkz\" (UniqueName: \"kubernetes.io/projected/4839945c-72ce-4a48-9879-1d94da9c3b48-kube-api-access-vnzkz\") pod \"maas-api-e2e-ipp-a-f6f9ed-7f68fdb665-jhx2v\" (UID: \"4839945c-72ce-4a48-9879-1d94da9c3b48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6f9ed-7f68fdb665-jhx2v" Nov 03 13:34:51.677422 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:51.677390 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4839945c-72ce-4a48-9879-1d94da9c3b48-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f6f9ed-7f68fdb665-jhx2v\" (UID: \"4839945c-72ce-4a48-9879-1d94da9c3b48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6f9ed-7f68fdb665-jhx2v" Nov 03 13:34:51.677641 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:51.677617 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4839945c-72ce-4a48-9879-1d94da9c3b48-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f6f9ed-7f68fdb665-jhx2v\" (UID: \"4839945c-72ce-4a48-9879-1d94da9c3b48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6f9ed-7f68fdb665-jhx2v" Nov 03 13:34:51.778819 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:51.778786 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4839945c-72ce-4a48-9879-1d94da9c3b48-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f6f9ed-7f68fdb665-jhx2v\" (UID: \"4839945c-72ce-4a48-9879-1d94da9c3b48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6f9ed-7f68fdb665-jhx2v" Nov 03 13:34:51.778993 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:51.778825 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4839945c-72ce-4a48-9879-1d94da9c3b48-metrics-certs\") pod \"maas-api-e2e-ipp-a-f6f9ed-7f68fdb665-jhx2v\" (UID: \"4839945c-72ce-4a48-9879-1d94da9c3b48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6f9ed-7f68fdb665-jhx2v" Nov 03 13:34:51.778993 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:51.778881 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnzkz\" (UniqueName: \"kubernetes.io/projected/4839945c-72ce-4a48-9879-1d94da9c3b48-kube-api-access-vnzkz\") pod \"maas-api-e2e-ipp-a-f6f9ed-7f68fdb665-jhx2v\" (UID: \"4839945c-72ce-4a48-9879-1d94da9c3b48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6f9ed-7f68fdb665-jhx2v" Nov 03 13:34:51.778993 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:34:51.778925 5979 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6f9ed-serving-cert: secret "maas-api-e2e-ipp-a-f6f9ed-serving-cert" not found Nov 03 13:34:51.778993 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:51.778943 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4839945c-72ce-4a48-9879-1d94da9c3b48-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f6f9ed-7f68fdb665-jhx2v\" (UID: \"4839945c-72ce-4a48-9879-1d94da9c3b48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6f9ed-7f68fdb665-jhx2v" Nov 03 13:34:51.778993 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:34:51.778985 5979 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4839945c-72ce-4a48-9879-1d94da9c3b48-maas-api-tls podName:4839945c-72ce-4a48-9879-1d94da9c3b48 nodeName:}" failed. No retries permitted until 2026-11-03 13:34:52.278967544 +0000 UTC m=+1774.289655765 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4839945c-72ce-4a48-9879-1d94da9c3b48-maas-api-tls") pod "maas-api-e2e-ipp-a-f6f9ed-7f68fdb665-jhx2v" (UID: "4839945c-72ce-4a48-9879-1d94da9c3b48") : secret "maas-api-e2e-ipp-a-f6f9ed-serving-cert" not found Nov 03 13:34:51.779668 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:51.779649 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4839945c-72ce-4a48-9879-1d94da9c3b48-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f6f9ed-7f68fdb665-jhx2v\" (UID: \"4839945c-72ce-4a48-9879-1d94da9c3b48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6f9ed-7f68fdb665-jhx2v" Nov 03 13:34:51.781165 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:51.781147 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4839945c-72ce-4a48-9879-1d94da9c3b48-metrics-certs\") pod \"maas-api-e2e-ipp-a-f6f9ed-7f68fdb665-jhx2v\" (UID: \"4839945c-72ce-4a48-9879-1d94da9c3b48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6f9ed-7f68fdb665-jhx2v" Nov 03 13:34:51.852063 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:51.851912 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnzkz\" (UniqueName: \"kubernetes.io/projected/4839945c-72ce-4a48-9879-1d94da9c3b48-kube-api-access-vnzkz\") pod \"maas-api-e2e-ipp-a-f6f9ed-7f68fdb665-jhx2v\" (UID: \"4839945c-72ce-4a48-9879-1d94da9c3b48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6f9ed-7f68fdb665-jhx2v" Nov 03 13:34:52.291152 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:52.291106 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4839945c-72ce-4a48-9879-1d94da9c3b48-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f6f9ed-7f68fdb665-jhx2v\" (UID: \"4839945c-72ce-4a48-9879-1d94da9c3b48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6f9ed-7f68fdb665-jhx2v" Nov 03 13:34:52.294942 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:52.294913 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4839945c-72ce-4a48-9879-1d94da9c3b48-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f6f9ed-7f68fdb665-jhx2v\" (UID: \"4839945c-72ce-4a48-9879-1d94da9c3b48\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6f9ed-7f68fdb665-jhx2v" Nov 03 13:34:52.504042 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:52.503996 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6f9ed-7f68fdb665-jhx2v" Nov 03 13:34:52.908833 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:52.908781 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6f9ed-7f68fdb665-jhx2v"] Nov 03 13:34:52.913227 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:34:52.913201 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4839945c_72ce_4a48_9879_1d94da9c3b48.slice/crio-710bc84e605e936e6740b28bb5350db9f2b1eaf8d1068a0df22a2612a4b773df WatchSource:0}: Error finding container 710bc84e605e936e6740b28bb5350db9f2b1eaf8d1068a0df22a2612a4b773df: Status 404 returned error can't find the container with id 710bc84e605e936e6740b28bb5350db9f2b1eaf8d1068a0df22a2612a4b773df Nov 03 13:34:53.170653 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:53.170529 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6f9ed-7f68fdb665-jhx2v" event={"ID":"4839945c-72ce-4a48-9879-1d94da9c3b48","Type":"ContainerStarted","Data":"710bc84e605e936e6740b28bb5350db9f2b1eaf8d1068a0df22a2612a4b773df"} Nov 03 13:34:53.897924 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:53.897898 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e7ca74-848576db5f-kgxq7"] Nov 03 13:34:54.023542 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:54.022463 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e7ca74-74db7b7bdc-gz58m"] Nov 03 13:34:54.180889 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:54.180837 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa2568-6cbd46f946-6fn4v" event={"ID":"92d72487-02a6-4172-a2b7-9d13411a94f7","Type":"ContainerStarted","Data":"2630b423d6fe9f67b3684bdb13cf6f0bae5e9b4868c00fc9540e3aaf4dbb3124"} Nov 03 13:34:54.181085 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:54.180908 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa2568-6cbd46f946-6fn4v" Nov 03 13:34:54.182536 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:54.182504 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa2568-67ffc88795-t7jsv" event={"ID":"0f997da5-e10f-4926-92c0-f5559fb13567","Type":"ContainerStarted","Data":"06d3c34da84ccbfb8a95fe4040a7d23503b61b18c2517ca9648dae28f4d13abc"} Nov 03 13:34:54.182669 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:54.182582 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa2568-67ffc88795-t7jsv" Nov 03 13:34:54.184235 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:54.184214 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6f9ed-7f68fdb665-jhx2v" event={"ID":"4839945c-72ce-4a48-9879-1d94da9c3b48","Type":"ContainerStarted","Data":"b74f6155ecedac22889eae636711d091301b3855657ec2c65ed7e2e0a058f824"} Nov 03 13:34:54.184337 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:54.184282 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6f9ed-7f68fdb665-jhx2v" Nov 03 13:34:54.298041 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:54.297936 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa2568-6cbd46f946-6fn4v" podStartSLOduration=1.70424456 podStartE2EDuration="4.297920337s" totalImagesPullingTime="2.593675777s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:34:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:34:54.295452953 +0000 UTC m=+1776.306141195" watchObservedRunningTime="2026-11-03 13:34:54.297920337 +0000 UTC m=+1776.308608579" Nov 03 13:34:54.412289 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:54.412213 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa2568-67ffc88795-t7jsv" podStartSLOduration=2.676547781 podStartE2EDuration="5.412194452s" totalImagesPullingTime="2.735646671s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:34:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:34:54.407193199 +0000 UTC m=+1776.417881438" watchObservedRunningTime="2026-11-03 13:34:54.412194452 +0000 UTC m=+1776.422882701" Nov 03 13:34:54.519630 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:54.519579 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6f9ed-7f68fdb665-jhx2v" podStartSLOduration=2.621032597 podStartE2EDuration="3.519565392s" totalImagesPullingTime="898.532795ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:34:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:34:54.516433187 +0000 UTC m=+1776.527121429" watchObservedRunningTime="2026-11-03 13:34:54.519565392 +0000 UTC m=+1776.530253633" Nov 03 13:34:58.898230 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:58.898171 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-e7ca74-848576db5f-kgxq7" podUID="f5d98b25-2641-4cbb-b288-38142a3869fd" containerName="payload-processing" containerID="cri-o://27bcbe9d87b6e4e661a5d465e456afadcb58f9843d5d0b84ab1cb6825a93df78" gracePeriod=25 Nov 03 13:34:59.023068 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:59.023012 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e7ca74-74db7b7bdc-gz58m" podUID="211eda79-4ce3-4b71-a4e5-73b86f72ac15" containerName="payload-pre-processing" containerID="cri-o://e3fa584e570991631115cdb20fbe5cdf02ffd94ba654ba9f66051d2790adaeb7" gracePeriod=25 Nov 03 13:34:59.183423 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:59.183395 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e7ca74-848576db5f-kgxq7" Nov 03 13:34:59.249499 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:59.249433 5979 generic.go:356] "Generic (PLEG): container finished" podID="211eda79-4ce3-4b71-a4e5-73b86f72ac15" containerID="e3fa584e570991631115cdb20fbe5cdf02ffd94ba654ba9f66051d2790adaeb7" exitCode=0 Nov 03 13:34:59.249695 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:59.249527 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e7ca74-74db7b7bdc-gz58m" event={"ID":"211eda79-4ce3-4b71-a4e5-73b86f72ac15","Type":"ContainerDied","Data":"e3fa584e570991631115cdb20fbe5cdf02ffd94ba654ba9f66051d2790adaeb7"} Nov 03 13:34:59.251314 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:59.251283 5979 generic.go:356] "Generic (PLEG): container finished" podID="f5d98b25-2641-4cbb-b288-38142a3869fd" containerID="27bcbe9d87b6e4e661a5d465e456afadcb58f9843d5d0b84ab1cb6825a93df78" exitCode=0 Nov 03 13:34:59.251491 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:59.251367 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e7ca74-848576db5f-kgxq7" event={"ID":"f5d98b25-2641-4cbb-b288-38142a3869fd","Type":"ContainerDied","Data":"27bcbe9d87b6e4e661a5d465e456afadcb58f9843d5d0b84ab1cb6825a93df78"} Nov 03 13:34:59.251491 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:59.251421 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e7ca74-848576db5f-kgxq7" event={"ID":"f5d98b25-2641-4cbb-b288-38142a3869fd","Type":"ContainerDied","Data":"3a7ff767a6d59ce78aaae29666ac66398e82935cd57ba101a9577b1c70543128"} Nov 03 13:34:59.251491 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:59.251446 5979 scope.go:122] "RemoveContainer" containerID="27bcbe9d87b6e4e661a5d465e456afadcb58f9843d5d0b84ab1cb6825a93df78" Nov 03 13:34:59.251491 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:59.251465 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e7ca74-848576db5f-kgxq7" Nov 03 13:34:59.262813 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:59.262787 5979 scope.go:122] "RemoveContainer" containerID="27bcbe9d87b6e4e661a5d465e456afadcb58f9843d5d0b84ab1cb6825a93df78" Nov 03 13:34:59.263094 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:34:59.263067 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27bcbe9d87b6e4e661a5d465e456afadcb58f9843d5d0b84ab1cb6825a93df78\": container with ID starting with 27bcbe9d87b6e4e661a5d465e456afadcb58f9843d5d0b84ab1cb6825a93df78 not found: ID does not exist" containerID="27bcbe9d87b6e4e661a5d465e456afadcb58f9843d5d0b84ab1cb6825a93df78" Nov 03 13:34:59.263195 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:59.263103 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27bcbe9d87b6e4e661a5d465e456afadcb58f9843d5d0b84ab1cb6825a93df78"} err="failed to get container status \"27bcbe9d87b6e4e661a5d465e456afadcb58f9843d5d0b84ab1cb6825a93df78\": rpc error: code = NotFound desc = could not find container \"27bcbe9d87b6e4e661a5d465e456afadcb58f9843d5d0b84ab1cb6825a93df78\": container with ID starting with 27bcbe9d87b6e4e661a5d465e456afadcb58f9843d5d0b84ab1cb6825a93df78 not found: ID does not exist" Nov 03 13:34:59.278798 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:59.278768 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5d98b25-2641-4cbb-b288-38142a3869fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5d98b25-2641-4cbb-b288-38142a3869fd-plugins-config-volume\") pod \"f5d98b25-2641-4cbb-b288-38142a3869fd\" (UID: \"f5d98b25-2641-4cbb-b288-38142a3869fd\") " Nov 03 13:34:59.278942 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:59.278921 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5d98b25-2641-4cbb-b288-38142a3869fd-kube-api-access-vrxmn\" (UniqueName: \"kubernetes.io/projected/f5d98b25-2641-4cbb-b288-38142a3869fd-kube-api-access-vrxmn\") pod \"f5d98b25-2641-4cbb-b288-38142a3869fd\" (UID: \"f5d98b25-2641-4cbb-b288-38142a3869fd\") " Nov 03 13:34:59.279201 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:59.279172 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5d98b25-2641-4cbb-b288-38142a3869fd-plugins-config-volume" pod "f5d98b25-2641-4cbb-b288-38142a3869fd" (UID: "f5d98b25-2641-4cbb-b288-38142a3869fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:34:59.279620 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:59.279599 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5d98b25-2641-4cbb-b288-38142a3869fd-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:34:59.281129 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:59.281111 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5d98b25-2641-4cbb-b288-38142a3869fd-kube-api-access-vrxmn" pod "f5d98b25-2641-4cbb-b288-38142a3869fd" (UID: "f5d98b25-2641-4cbb-b288-38142a3869fd"). InnerVolumeSpecName "kube-api-access-vrxmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:34:59.290435 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:59.290403 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e7ca74-74db7b7bdc-gz58m" Nov 03 13:34:59.380338 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:59.380304 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/211eda79-4ce3-4b71-a4e5-73b86f72ac15-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/211eda79-4ce3-4b71-a4e5-73b86f72ac15-plugins-config-volume\") pod \"211eda79-4ce3-4b71-a4e5-73b86f72ac15\" (UID: \"211eda79-4ce3-4b71-a4e5-73b86f72ac15\") " Nov 03 13:34:59.380540 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:59.380456 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/211eda79-4ce3-4b71-a4e5-73b86f72ac15-kube-api-access-pmjhb\" (UniqueName: \"kubernetes.io/projected/211eda79-4ce3-4b71-a4e5-73b86f72ac15-kube-api-access-pmjhb\") pod \"211eda79-4ce3-4b71-a4e5-73b86f72ac15\" (UID: \"211eda79-4ce3-4b71-a4e5-73b86f72ac15\") " Nov 03 13:34:59.380693 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:59.380671 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/211eda79-4ce3-4b71-a4e5-73b86f72ac15-plugins-config-volume" pod "211eda79-4ce3-4b71-a4e5-73b86f72ac15" (UID: "211eda79-4ce3-4b71-a4e5-73b86f72ac15"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:34:59.380956 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:59.380929 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrxmn\" (UniqueName: \"kubernetes.io/projected/f5d98b25-2641-4cbb-b288-38142a3869fd-kube-api-access-vrxmn\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:34:59.380956 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:59.380949 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/211eda79-4ce3-4b71-a4e5-73b86f72ac15-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:34:59.382519 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:59.382498 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/211eda79-4ce3-4b71-a4e5-73b86f72ac15-kube-api-access-pmjhb" pod "211eda79-4ce3-4b71-a4e5-73b86f72ac15" (UID: "211eda79-4ce3-4b71-a4e5-73b86f72ac15"). InnerVolumeSpecName "kube-api-access-pmjhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:34:59.482441 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:59.482324 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmjhb\" (UniqueName: \"kubernetes.io/projected/211eda79-4ce3-4b71-a4e5-73b86f72ac15-kube-api-access-pmjhb\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:34:59.666571 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:59.666536 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e7ca74-848576db5f-kgxq7"] Nov 03 13:34:59.671656 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:34:59.671631 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e7ca74-848576db5f-kgxq7"] Nov 03 13:35:00.117718 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:00.117682 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5d98b25-2641-4cbb-b288-38142a3869fd" path="/var/lib/kubelet/pods/f5d98b25-2641-4cbb-b288-38142a3869fd/volumes" Nov 03 13:35:00.197315 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:00.197244 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6f9ed-7f68fdb665-jhx2v" Nov 03 13:35:00.260352 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:00.260317 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e7ca74-74db7b7bdc-gz58m" event={"ID":"211eda79-4ce3-4b71-a4e5-73b86f72ac15","Type":"ContainerDied","Data":"a7ad9c4e56276cef8159f13b2a37a59bba4183583575952e59d646708e46f54d"} Nov 03 13:35:00.260535 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:00.260373 5979 scope.go:122] "RemoveContainer" containerID="e3fa584e570991631115cdb20fbe5cdf02ffd94ba654ba9f66051d2790adaeb7" Nov 03 13:35:00.260535 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:00.260326 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e7ca74-74db7b7bdc-gz58m" Nov 03 13:35:00.415035 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:00.415002 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e7ca74-74db7b7bdc-gz58m"] Nov 03 13:35:00.420953 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:00.420918 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e7ca74-74db7b7bdc-gz58m"] Nov 03 13:35:02.116749 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:02.116716 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="211eda79-4ce3-4b71-a4e5-73b86f72ac15" path="/var/lib/kubelet/pods/211eda79-4ce3-4b71-a4e5-73b86f72ac15/volumes" Nov 03 13:35:03.050253 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:03.050221 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a111273-pdcvs"] Nov 03 13:35:03.096336 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:03.096298 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a111273-pdcvs"] Nov 03 13:35:03.895093 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:03.895050 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c8192-55bbd4cd78-8jfgv"] Nov 03 13:35:03.896524 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:03.896492 5979 cpu_manager.go:414] "containerMap: removing container" podUID="f5d98b25-2641-4cbb-b288-38142a3869fd" containerName="payload-processing" Nov 03 13:35:03.896524 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:03.896520 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5d98b25-2641-4cbb-b288-38142a3869fd" containerName="payload-processing" Nov 03 13:35:03.896723 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:03.896561 5979 cpu_manager.go:414] "containerMap: removing container" podUID="211eda79-4ce3-4b71-a4e5-73b86f72ac15" containerName="payload-pre-processing" Nov 03 13:35:03.896723 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:03.896570 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="211eda79-4ce3-4b71-a4e5-73b86f72ac15" containerName="payload-pre-processing" Nov 03 13:35:03.896979 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:03.896963 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5d98b25-2641-4cbb-b288-38142a3869fd" containerName="payload-processing" Nov 03 13:35:03.897032 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:03.896982 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="211eda79-4ce3-4b71-a4e5-73b86f72ac15" containerName="payload-pre-processing" Nov 03 13:35:03.903427 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:03.903403 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c8192-55bbd4cd78-8jfgv" Nov 03 13:35:03.950538 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:03.950505 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c8192-55bbd4cd78-8jfgv"] Nov 03 13:35:03.993343 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:03.993306 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-5c8192-serving-cert\"" Nov 03 13:35:04.001390 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:04.001362 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp9t5\" (UniqueName: \"kubernetes.io/projected/e13199dc-0be9-4635-92f7-61706f259156-kube-api-access-kp9t5\") pod \"maas-api-e2e-ipp-b-5c8192-55bbd4cd78-8jfgv\" (UID: \"e13199dc-0be9-4635-92f7-61706f259156\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c8192-55bbd4cd78-8jfgv" Nov 03 13:35:04.001528 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:04.001515 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e13199dc-0be9-4635-92f7-61706f259156-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5c8192-55bbd4cd78-8jfgv\" (UID: \"e13199dc-0be9-4635-92f7-61706f259156\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c8192-55bbd4cd78-8jfgv" Nov 03 13:35:04.001591 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:04.001539 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e13199dc-0be9-4635-92f7-61706f259156-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5c8192-55bbd4cd78-8jfgv\" (UID: \"e13199dc-0be9-4635-92f7-61706f259156\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c8192-55bbd4cd78-8jfgv" Nov 03 13:35:04.001690 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:04.001673 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e13199dc-0be9-4635-92f7-61706f259156-metrics-certs\") pod \"maas-api-e2e-ipp-b-5c8192-55bbd4cd78-8jfgv\" (UID: \"e13199dc-0be9-4635-92f7-61706f259156\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c8192-55bbd4cd78-8jfgv" Nov 03 13:35:04.103240 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:04.103197 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e13199dc-0be9-4635-92f7-61706f259156-metrics-certs\") pod \"maas-api-e2e-ipp-b-5c8192-55bbd4cd78-8jfgv\" (UID: \"e13199dc-0be9-4635-92f7-61706f259156\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c8192-55bbd4cd78-8jfgv" Nov 03 13:35:04.103440 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:04.103333 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp9t5\" (UniqueName: \"kubernetes.io/projected/e13199dc-0be9-4635-92f7-61706f259156-kube-api-access-kp9t5\") pod \"maas-api-e2e-ipp-b-5c8192-55bbd4cd78-8jfgv\" (UID: \"e13199dc-0be9-4635-92f7-61706f259156\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c8192-55bbd4cd78-8jfgv" Nov 03 13:35:04.103574 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:04.103518 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e13199dc-0be9-4635-92f7-61706f259156-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5c8192-55bbd4cd78-8jfgv\" (UID: \"e13199dc-0be9-4635-92f7-61706f259156\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c8192-55bbd4cd78-8jfgv" Nov 03 13:35:04.103574 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:04.103556 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e13199dc-0be9-4635-92f7-61706f259156-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5c8192-55bbd4cd78-8jfgv\" (UID: \"e13199dc-0be9-4635-92f7-61706f259156\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c8192-55bbd4cd78-8jfgv" Nov 03 13:35:04.103804 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:35:04.103778 5979 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c8192-serving-cert: secret "maas-api-e2e-ipp-b-5c8192-serving-cert" not found Nov 03 13:35:04.103885 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:35:04.103875 5979 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e13199dc-0be9-4635-92f7-61706f259156-maas-api-tls podName:e13199dc-0be9-4635-92f7-61706f259156 nodeName:}" failed. No retries permitted until 2026-11-03 13:35:04.603858605 +0000 UTC m=+1786.614546840 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e13199dc-0be9-4635-92f7-61706f259156-maas-api-tls") pod "maas-api-e2e-ipp-b-5c8192-55bbd4cd78-8jfgv" (UID: "e13199dc-0be9-4635-92f7-61706f259156") : secret "maas-api-e2e-ipp-b-5c8192-serving-cert" not found Nov 03 13:35:04.104163 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:04.104144 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e13199dc-0be9-4635-92f7-61706f259156-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5c8192-55bbd4cd78-8jfgv\" (UID: \"e13199dc-0be9-4635-92f7-61706f259156\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c8192-55bbd4cd78-8jfgv" Nov 03 13:35:04.105611 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:04.105592 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e13199dc-0be9-4635-92f7-61706f259156-metrics-certs\") pod \"maas-api-e2e-ipp-b-5c8192-55bbd4cd78-8jfgv\" (UID: \"e13199dc-0be9-4635-92f7-61706f259156\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c8192-55bbd4cd78-8jfgv" Nov 03 13:35:04.118453 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:04.118424 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01e5557b-1b62-4d56-843e-1685de2da7f1" path="/var/lib/kubelet/pods/01e5557b-1b62-4d56-843e-1685de2da7f1/volumes" Nov 03 13:35:04.158453 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:04.158374 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp9t5\" (UniqueName: \"kubernetes.io/projected/e13199dc-0be9-4635-92f7-61706f259156-kube-api-access-kp9t5\") pod \"maas-api-e2e-ipp-b-5c8192-55bbd4cd78-8jfgv\" (UID: \"e13199dc-0be9-4635-92f7-61706f259156\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c8192-55bbd4cd78-8jfgv" Nov 03 13:35:04.449272 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:04.449192 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5295ab29-pcxj8"] Nov 03 13:35:04.454015 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:04.453986 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5295ab29-pcxj8"] Nov 03 13:35:04.613014 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:04.612980 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e13199dc-0be9-4635-92f7-61706f259156-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5c8192-55bbd4cd78-8jfgv\" (UID: \"e13199dc-0be9-4635-92f7-61706f259156\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c8192-55bbd4cd78-8jfgv" Nov 03 13:35:04.616508 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:04.616442 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e13199dc-0be9-4635-92f7-61706f259156-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5c8192-55bbd4cd78-8jfgv\" (UID: \"e13199dc-0be9-4635-92f7-61706f259156\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c8192-55bbd4cd78-8jfgv" Nov 03 13:35:04.643194 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:04.643153 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl"] Nov 03 13:35:04.643444 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:04.643418 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl" podUID="fc325c76-815f-4a33-b5b6-71421ad64ded" containerName="istio-proxy" containerID="cri-o://178ac86c8cbdd80437a8bf478eb86f1ad0e1e3816991f8b3ccf0ef1eaf3749f1" gracePeriod=30 Nov 03 13:35:04.646203 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:04.646177 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm"] Nov 03 13:35:04.646447 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:04.646414 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm" podUID="5fd8d2bb-4812-41bc-88a9-0d8249c25c1c" containerName="istio-proxy" containerID="cri-o://1f7a2ff3e89e525a4021063055e7c71af56a94f5c449df20286ac3336aff5e1c" gracePeriod=30 Nov 03 13:35:04.666321 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:04.666281 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z"] Nov 03 13:35:04.666518 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:04.666493 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z" podUID="b109e1f4-308a-4557-ad21-bb8ae0d75553" containerName="istio-proxy" containerID="cri-o://6b1c04d6646625b809d10de2d313efd013416a8692895d9bd18da25323a49696" gracePeriod=30 Nov 03 13:35:04.669965 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:04.669934 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74"] Nov 03 13:35:04.670181 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:04.670155 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74" podUID="22fad5ad-56d4-4d01-944f-a5db77a4a3ee" containerName="istio-proxy" containerID="cri-o://0c0701613949d61961abc7b953bbaa3f49b57d39e2aa21425167e5e5880f6828" gracePeriod=30 Nov 03 13:35:04.820196 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:04.820087 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c8192-55bbd4cd78-8jfgv" Nov 03 13:35:05.193973 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:05.193943 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa2568-6cbd46f946-6fn4v" Nov 03 13:35:05.194394 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:05.194015 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa2568-67ffc88795-t7jsv" Nov 03 13:35:05.245309 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:05.245267 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c8192-55bbd4cd78-8jfgv"] Nov 03 13:35:05.252785 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:35:05.252759 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode13199dc_0be9_4635_92f7_61706f259156.slice/crio-378e1e98a280252dc7a15a36d3a2dc8161a7625fa9f80906619141b798977f48 WatchSource:0}: Error finding container 378e1e98a280252dc7a15a36d3a2dc8161a7625fa9f80906619141b798977f48: Status 404 returned error can't find the container with id 378e1e98a280252dc7a15a36d3a2dc8161a7625fa9f80906619141b798977f48 Nov 03 13:35:05.356964 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:05.356932 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c8192-55bbd4cd78-8jfgv" event={"ID":"e13199dc-0be9-4635-92f7-61706f259156","Type":"ContainerStarted","Data":"378e1e98a280252dc7a15a36d3a2dc8161a7625fa9f80906619141b798977f48"} Nov 03 13:35:06.117453 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:06.117418 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4c6bbfe-7d2c-498e-9a5d-f2194f4d7d9c" path="/var/lib/kubelet/pods/d4c6bbfe-7d2c-498e-9a5d-f2194f4d7d9c/volumes" Nov 03 13:35:06.366628 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:06.366544 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c8192-55bbd4cd78-8jfgv" event={"ID":"e13199dc-0be9-4635-92f7-61706f259156","Type":"ContainerStarted","Data":"ccf8eb24a6436b1056bc39276c0ad1053ef98443753507d21ce816047403b676"} Nov 03 13:35:06.367091 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:06.366680 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c8192-55bbd4cd78-8jfgv" Nov 03 13:35:06.484855 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:06.484805 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c8192-55bbd4cd78-8jfgv" podStartSLOduration=2.65487428 podStartE2EDuration="3.484791569s" totalImagesPullingTime="829.917289ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:35:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:35:06.483400393 +0000 UTC m=+1788.494088630" watchObservedRunningTime="2026-11-03 13:35:06.484791569 +0000 UTC m=+1788.495479812" Nov 03 13:35:07.257962 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:35:07.257928 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:35:09.976434 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:09.975516 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm" Nov 03 13:35:09.981537 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:09.981511 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl" Nov 03 13:35:10.036484 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.036437 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z" Nov 03 13:35:10.041709 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.041683 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc325c76-815f-4a33-b5b6-71421ad64ded-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc325c76-815f-4a33-b5b6-71421ad64ded-istio-data\") pod \"fc325c76-815f-4a33-b5b6-71421ad64ded\" (UID: \"fc325c76-815f-4a33-b5b6-71421ad64ded\") " Nov 03 13:35:10.041850 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.041725 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc325c76-815f-4a33-b5b6-71421ad64ded-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc325c76-815f-4a33-b5b6-71421ad64ded-workload-socket\") pod \"fc325c76-815f-4a33-b5b6-71421ad64ded\" (UID: \"fc325c76-815f-4a33-b5b6-71421ad64ded\") " Nov 03 13:35:10.041850 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.041753 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-istio-token\" (UniqueName: \"kubernetes.io/projected/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-istio-token\") pod \"5fd8d2bb-4812-41bc-88a9-0d8249c25c1c\" (UID: \"5fd8d2bb-4812-41bc-88a9-0d8249c25c1c\") " Nov 03 13:35:10.041850 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.041799 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-workload-certs\") pod \"5fd8d2bb-4812-41bc-88a9-0d8249c25c1c\" (UID: \"5fd8d2bb-4812-41bc-88a9-0d8249c25c1c\") " Nov 03 13:35:10.042022 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.041869 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-istio-envoy\") pod \"5fd8d2bb-4812-41bc-88a9-0d8249c25c1c\" (UID: \"5fd8d2bb-4812-41bc-88a9-0d8249c25c1c\") " Nov 03 13:35:10.042022 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.041943 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc325c76-815f-4a33-b5b6-71421ad64ded-istio-data" pod "fc325c76-815f-4a33-b5b6-71421ad64ded" (UID: "fc325c76-815f-4a33-b5b6-71421ad64ded"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:35:10.042022 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.041970 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-kube-api-access-t8gs7\" (UniqueName: \"kubernetes.io/projected/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-kube-api-access-t8gs7\") pod \"5fd8d2bb-4812-41bc-88a9-0d8249c25c1c\" (UID: \"5fd8d2bb-4812-41bc-88a9-0d8249c25c1c\") " Nov 03 13:35:10.042022 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.042002 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc325c76-815f-4a33-b5b6-71421ad64ded-kube-api-access-jtt6z\" (UniqueName: \"kubernetes.io/projected/fc325c76-815f-4a33-b5b6-71421ad64ded-kube-api-access-jtt6z\") pod \"fc325c76-815f-4a33-b5b6-71421ad64ded\" (UID: \"fc325c76-815f-4a33-b5b6-71421ad64ded\") " Nov 03 13:35:10.042239 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.042059 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc325c76-815f-4a33-b5b6-71421ad64ded-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc325c76-815f-4a33-b5b6-71421ad64ded-istio-envoy\") pod \"fc325c76-815f-4a33-b5b6-71421ad64ded\" (UID: \"fc325c76-815f-4a33-b5b6-71421ad64ded\") " Nov 03 13:35:10.042239 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.042096 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-istiod-ca-cert\") pod \"5fd8d2bb-4812-41bc-88a9-0d8249c25c1c\" (UID: \"5fd8d2bb-4812-41bc-88a9-0d8249c25c1c\") " Nov 03 13:35:10.042239 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.042178 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-istio-podinfo\") pod \"5fd8d2bb-4812-41bc-88a9-0d8249c25c1c\" (UID: \"5fd8d2bb-4812-41bc-88a9-0d8249c25c1c\") " Nov 03 13:35:10.042239 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.042213 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-workload-socket\") pod \"5fd8d2bb-4812-41bc-88a9-0d8249c25c1c\" (UID: \"5fd8d2bb-4812-41bc-88a9-0d8249c25c1c\") " Nov 03 13:35:10.042456 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.042306 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fc325c76-815f-4a33-b5b6-71421ad64ded-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc325c76-815f-4a33-b5b6-71421ad64ded-istio-podinfo\") pod \"fc325c76-815f-4a33-b5b6-71421ad64ded\" (UID: \"fc325c76-815f-4a33-b5b6-71421ad64ded\") " Nov 03 13:35:10.042456 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.042351 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc325c76-815f-4a33-b5b6-71421ad64ded-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc325c76-815f-4a33-b5b6-71421ad64ded-credential-socket\") pod \"fc325c76-815f-4a33-b5b6-71421ad64ded\" (UID: \"fc325c76-815f-4a33-b5b6-71421ad64ded\") " Nov 03 13:35:10.042456 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.042387 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc325c76-815f-4a33-b5b6-71421ad64ded-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc325c76-815f-4a33-b5b6-71421ad64ded-workload-certs\") pod \"fc325c76-815f-4a33-b5b6-71421ad64ded\" (UID: \"fc325c76-815f-4a33-b5b6-71421ad64ded\") " Nov 03 13:35:10.042456 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.042414 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-istio-data\") pod \"5fd8d2bb-4812-41bc-88a9-0d8249c25c1c\" (UID: \"5fd8d2bb-4812-41bc-88a9-0d8249c25c1c\") " Nov 03 13:35:10.042456 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.042456 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc325c76-815f-4a33-b5b6-71421ad64ded-istio-token\" (UniqueName: \"kubernetes.io/projected/fc325c76-815f-4a33-b5b6-71421ad64ded-istio-token\") pod \"fc325c76-815f-4a33-b5b6-71421ad64ded\" (UID: \"fc325c76-815f-4a33-b5b6-71421ad64ded\") " Nov 03 13:35:10.042763 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.042502 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-credential-socket\") pod \"5fd8d2bb-4812-41bc-88a9-0d8249c25c1c\" (UID: \"5fd8d2bb-4812-41bc-88a9-0d8249c25c1c\") " Nov 03 13:35:10.042763 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.042525 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc325c76-815f-4a33-b5b6-71421ad64ded-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc325c76-815f-4a33-b5b6-71421ad64ded-istiod-ca-cert\") pod \"fc325c76-815f-4a33-b5b6-71421ad64ded\" (UID: \"fc325c76-815f-4a33-b5b6-71421ad64ded\") " Nov 03 13:35:10.043001 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.042981 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc325c76-815f-4a33-b5b6-71421ad64ded-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.043308 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.043287 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc325c76-815f-4a33-b5b6-71421ad64ded-istiod-ca-cert" pod "fc325c76-815f-4a33-b5b6-71421ad64ded" (UID: "fc325c76-815f-4a33-b5b6-71421ad64ded"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:35:10.074982 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.074844 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-workload-certs" pod "5fd8d2bb-4812-41bc-88a9-0d8249c25c1c" (UID: "5fd8d2bb-4812-41bc-88a9-0d8249c25c1c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:35:10.075941 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.075915 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-istiod-ca-cert" pod "5fd8d2bb-4812-41bc-88a9-0d8249c25c1c" (UID: "5fd8d2bb-4812-41bc-88a9-0d8249c25c1c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:35:10.076078 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.076036 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-workload-socket" pod "5fd8d2bb-4812-41bc-88a9-0d8249c25c1c" (UID: "5fd8d2bb-4812-41bc-88a9-0d8249c25c1c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:35:10.076144 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.076067 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-istio-data" pod "5fd8d2bb-4812-41bc-88a9-0d8249c25c1c" (UID: "5fd8d2bb-4812-41bc-88a9-0d8249c25c1c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:35:10.076321 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.076299 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc325c76-815f-4a33-b5b6-71421ad64ded-credential-socket" pod "fc325c76-815f-4a33-b5b6-71421ad64ded" (UID: "fc325c76-815f-4a33-b5b6-71421ad64ded"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:35:10.076675 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.076457 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc325c76-815f-4a33-b5b6-71421ad64ded-workload-socket" pod "fc325c76-815f-4a33-b5b6-71421ad64ded" (UID: "fc325c76-815f-4a33-b5b6-71421ad64ded"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:35:10.076799 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.076524 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-credential-socket" pod "5fd8d2bb-4812-41bc-88a9-0d8249c25c1c" (UID: "5fd8d2bb-4812-41bc-88a9-0d8249c25c1c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:35:10.076799 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.076533 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc325c76-815f-4a33-b5b6-71421ad64ded-workload-certs" pod "fc325c76-815f-4a33-b5b6-71421ad64ded" (UID: "fc325c76-815f-4a33-b5b6-71421ad64ded"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:35:10.077673 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.077647 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-kube-api-access-t8gs7" pod "5fd8d2bb-4812-41bc-88a9-0d8249c25c1c" (UID: "5fd8d2bb-4812-41bc-88a9-0d8249c25c1c"). InnerVolumeSpecName "kube-api-access-t8gs7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:35:10.078251 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.078226 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-istio-token" pod "5fd8d2bb-4812-41bc-88a9-0d8249c25c1c" (UID: "5fd8d2bb-4812-41bc-88a9-0d8249c25c1c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:35:10.080057 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.080034 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-istio-podinfo" pod "5fd8d2bb-4812-41bc-88a9-0d8249c25c1c" (UID: "5fd8d2bb-4812-41bc-88a9-0d8249c25c1c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:35:10.083594 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.083548 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-istio-envoy" pod "5fd8d2bb-4812-41bc-88a9-0d8249c25c1c" (UID: "5fd8d2bb-4812-41bc-88a9-0d8249c25c1c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:35:10.083594 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.083554 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc325c76-815f-4a33-b5b6-71421ad64ded-istio-envoy" pod "fc325c76-815f-4a33-b5b6-71421ad64ded" (UID: "fc325c76-815f-4a33-b5b6-71421ad64ded"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:35:10.083594 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.083571 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc325c76-815f-4a33-b5b6-71421ad64ded-istio-podinfo" pod "fc325c76-815f-4a33-b5b6-71421ad64ded" (UID: "fc325c76-815f-4a33-b5b6-71421ad64ded"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:35:10.083980 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.083953 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc325c76-815f-4a33-b5b6-71421ad64ded-kube-api-access-jtt6z" pod "fc325c76-815f-4a33-b5b6-71421ad64ded" (UID: "fc325c76-815f-4a33-b5b6-71421ad64ded"). InnerVolumeSpecName "kube-api-access-jtt6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:35:10.084986 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.084961 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc325c76-815f-4a33-b5b6-71421ad64ded-istio-token" pod "fc325c76-815f-4a33-b5b6-71421ad64ded" (UID: "fc325c76-815f-4a33-b5b6-71421ad64ded"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:35:10.098886 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.098865 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74" Nov 03 13:35:10.167021 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.166975 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b109e1f4-308a-4557-ad21-bb8ae0d75553-kube-api-access-5kxx4\" (UniqueName: \"kubernetes.io/projected/b109e1f4-308a-4557-ad21-bb8ae0d75553-kube-api-access-5kxx4\") pod \"b109e1f4-308a-4557-ad21-bb8ae0d75553\" (UID: \"b109e1f4-308a-4557-ad21-bb8ae0d75553\") " Nov 03 13:35:10.167243 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.167035 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-istio-token\" (UniqueName: \"kubernetes.io/projected/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-istio-token\") pod \"22fad5ad-56d4-4d01-944f-a5db77a4a3ee\" (UID: \"22fad5ad-56d4-4d01-944f-a5db77a4a3ee\") " Nov 03 13:35:10.167243 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.167092 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b109e1f4-308a-4557-ad21-bb8ae0d75553-istio-token\" (UniqueName: \"kubernetes.io/projected/b109e1f4-308a-4557-ad21-bb8ae0d75553-istio-token\") pod \"b109e1f4-308a-4557-ad21-bb8ae0d75553\" (UID: \"b109e1f4-308a-4557-ad21-bb8ae0d75553\") " Nov 03 13:35:10.167243 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.167121 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-workload-certs\") pod \"22fad5ad-56d4-4d01-944f-a5db77a4a3ee\" (UID: \"22fad5ad-56d4-4d01-944f-a5db77a4a3ee\") " Nov 03 13:35:10.167243 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.167152 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b109e1f4-308a-4557-ad21-bb8ae0d75553-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b109e1f4-308a-4557-ad21-bb8ae0d75553-istiod-ca-cert\") pod \"b109e1f4-308a-4557-ad21-bb8ae0d75553\" (UID: \"b109e1f4-308a-4557-ad21-bb8ae0d75553\") " Nov 03 13:35:10.167243 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.167214 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b109e1f4-308a-4557-ad21-bb8ae0d75553-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b109e1f4-308a-4557-ad21-bb8ae0d75553-istio-data\") pod \"b109e1f4-308a-4557-ad21-bb8ae0d75553\" (UID: \"b109e1f4-308a-4557-ad21-bb8ae0d75553\") " Nov 03 13:35:10.167607 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.167262 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b109e1f4-308a-4557-ad21-bb8ae0d75553-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b109e1f4-308a-4557-ad21-bb8ae0d75553-istio-envoy\") pod \"b109e1f4-308a-4557-ad21-bb8ae0d75553\" (UID: \"b109e1f4-308a-4557-ad21-bb8ae0d75553\") " Nov 03 13:35:10.167607 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.167303 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b109e1f4-308a-4557-ad21-bb8ae0d75553-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b109e1f4-308a-4557-ad21-bb8ae0d75553-workload-socket\") pod \"b109e1f4-308a-4557-ad21-bb8ae0d75553\" (UID: \"b109e1f4-308a-4557-ad21-bb8ae0d75553\") " Nov 03 13:35:10.167607 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.167341 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-kube-api-access-8cz2w\" (UniqueName: \"kubernetes.io/projected/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-kube-api-access-8cz2w\") pod \"22fad5ad-56d4-4d01-944f-a5db77a4a3ee\" (UID: \"22fad5ad-56d4-4d01-944f-a5db77a4a3ee\") " Nov 03 13:35:10.167607 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.167405 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b109e1f4-308a-4557-ad21-bb8ae0d75553-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b109e1f4-308a-4557-ad21-bb8ae0d75553-credential-socket\") pod \"b109e1f4-308a-4557-ad21-bb8ae0d75553\" (UID: \"b109e1f4-308a-4557-ad21-bb8ae0d75553\") " Nov 03 13:35:10.167607 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.167518 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-istio-data\") pod \"22fad5ad-56d4-4d01-944f-a5db77a4a3ee\" (UID: \"22fad5ad-56d4-4d01-944f-a5db77a4a3ee\") " Nov 03 13:35:10.167607 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.167585 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-istio-podinfo\") pod \"22fad5ad-56d4-4d01-944f-a5db77a4a3ee\" (UID: \"22fad5ad-56d4-4d01-944f-a5db77a4a3ee\") " Nov 03 13:35:10.167955 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.167620 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-credential-socket\") pod \"22fad5ad-56d4-4d01-944f-a5db77a4a3ee\" (UID: \"22fad5ad-56d4-4d01-944f-a5db77a4a3ee\") " Nov 03 13:35:10.167955 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.167653 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b109e1f4-308a-4557-ad21-bb8ae0d75553-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b109e1f4-308a-4557-ad21-bb8ae0d75553-workload-certs\") pod \"b109e1f4-308a-4557-ad21-bb8ae0d75553\" (UID: \"b109e1f4-308a-4557-ad21-bb8ae0d75553\") " Nov 03 13:35:10.167955 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.167698 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-istiod-ca-cert\") pod \"22fad5ad-56d4-4d01-944f-a5db77a4a3ee\" (UID: \"22fad5ad-56d4-4d01-944f-a5db77a4a3ee\") " Nov 03 13:35:10.167955 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.167754 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b109e1f4-308a-4557-ad21-bb8ae0d75553-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b109e1f4-308a-4557-ad21-bb8ae0d75553-istio-podinfo\") pod \"b109e1f4-308a-4557-ad21-bb8ae0d75553\" (UID: \"b109e1f4-308a-4557-ad21-bb8ae0d75553\") " Nov 03 13:35:10.167955 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.167853 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-workload-socket\") pod \"22fad5ad-56d4-4d01-944f-a5db77a4a3ee\" (UID: \"22fad5ad-56d4-4d01-944f-a5db77a4a3ee\") " Nov 03 13:35:10.167955 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.167942 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-istio-envoy\") pod \"22fad5ad-56d4-4d01-944f-a5db77a4a3ee\" (UID: \"22fad5ad-56d4-4d01-944f-a5db77a4a3ee\") " Nov 03 13:35:10.169037 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.168969 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.169037 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.168999 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.169037 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.169015 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc325c76-815f-4a33-b5b6-71421ad64ded-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.169037 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.169029 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc325c76-815f-4a33-b5b6-71421ad64ded-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.169037 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.169044 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc325c76-815f-4a33-b5b6-71421ad64ded-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.169037 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.169058 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.169037 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.169072 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc325c76-815f-4a33-b5b6-71421ad64ded-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.169724 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.169086 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.169724 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.169103 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc325c76-815f-4a33-b5b6-71421ad64ded-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.169724 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.169184 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc325c76-815f-4a33-b5b6-71421ad64ded-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.169724 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.169201 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.169724 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.169216 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.169724 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.169230 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.169724 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.169245 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8gs7\" (UniqueName: \"kubernetes.io/projected/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-kube-api-access-t8gs7\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.169724 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.169263 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtt6z\" (UniqueName: \"kubernetes.io/projected/fc325c76-815f-4a33-b5b6-71421ad64ded-kube-api-access-jtt6z\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.169724 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.169279 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc325c76-815f-4a33-b5b6-71421ad64ded-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.169724 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.169294 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.178991 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.175358 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b109e1f4-308a-4557-ad21-bb8ae0d75553-credential-socket" pod "b109e1f4-308a-4557-ad21-bb8ae0d75553" (UID: "b109e1f4-308a-4557-ad21-bb8ae0d75553"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:35:10.178991 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.176808 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b109e1f4-308a-4557-ad21-bb8ae0d75553-istiod-ca-cert" pod "b109e1f4-308a-4557-ad21-bb8ae0d75553" (UID: "b109e1f4-308a-4557-ad21-bb8ae0d75553"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:35:10.178991 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.177140 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b109e1f4-308a-4557-ad21-bb8ae0d75553-workload-certs" pod "b109e1f4-308a-4557-ad21-bb8ae0d75553" (UID: "b109e1f4-308a-4557-ad21-bb8ae0d75553"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:35:10.178991 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.177432 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-credential-socket" pod "22fad5ad-56d4-4d01-944f-a5db77a4a3ee" (UID: "22fad5ad-56d4-4d01-944f-a5db77a4a3ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:35:10.178991 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.177888 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-workload-socket" pod "22fad5ad-56d4-4d01-944f-a5db77a4a3ee" (UID: "22fad5ad-56d4-4d01-944f-a5db77a4a3ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:35:10.178991 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.177993 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-istio-data" pod "22fad5ad-56d4-4d01-944f-a5db77a4a3ee" (UID: "22fad5ad-56d4-4d01-944f-a5db77a4a3ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:35:10.178991 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.178344 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-istiod-ca-cert" pod "22fad5ad-56d4-4d01-944f-a5db77a4a3ee" (UID: "22fad5ad-56d4-4d01-944f-a5db77a4a3ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:35:10.180050 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.179604 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b109e1f4-308a-4557-ad21-bb8ae0d75553-kube-api-access-5kxx4" pod "b109e1f4-308a-4557-ad21-bb8ae0d75553" (UID: "b109e1f4-308a-4557-ad21-bb8ae0d75553"). InnerVolumeSpecName "kube-api-access-5kxx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:35:10.180050 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.179755 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-workload-certs" pod "22fad5ad-56d4-4d01-944f-a5db77a4a3ee" (UID: "22fad5ad-56d4-4d01-944f-a5db77a4a3ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:35:10.180537 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.180504 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-istio-podinfo" pod "22fad5ad-56d4-4d01-944f-a5db77a4a3ee" (UID: "22fad5ad-56d4-4d01-944f-a5db77a4a3ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:35:10.180537 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.180524 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-istio-token" pod "22fad5ad-56d4-4d01-944f-a5db77a4a3ee" (UID: "22fad5ad-56d4-4d01-944f-a5db77a4a3ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:35:10.182410 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.182361 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-istio-envoy" pod "22fad5ad-56d4-4d01-944f-a5db77a4a3ee" (UID: "22fad5ad-56d4-4d01-944f-a5db77a4a3ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:35:10.184048 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.184013 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b109e1f4-308a-4557-ad21-bb8ae0d75553-istio-data" pod "b109e1f4-308a-4557-ad21-bb8ae0d75553" (UID: "b109e1f4-308a-4557-ad21-bb8ae0d75553"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:35:10.184248 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.184205 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b109e1f4-308a-4557-ad21-bb8ae0d75553-workload-socket" pod "b109e1f4-308a-4557-ad21-bb8ae0d75553" (UID: "b109e1f4-308a-4557-ad21-bb8ae0d75553"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:35:10.185993 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.185938 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-kube-api-access-8cz2w" pod "22fad5ad-56d4-4d01-944f-a5db77a4a3ee" (UID: "22fad5ad-56d4-4d01-944f-a5db77a4a3ee"). InnerVolumeSpecName "kube-api-access-8cz2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:35:10.185993 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.185946 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b109e1f4-308a-4557-ad21-bb8ae0d75553-istio-podinfo" pod "b109e1f4-308a-4557-ad21-bb8ae0d75553" (UID: "b109e1f4-308a-4557-ad21-bb8ae0d75553"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:35:10.186415 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.186394 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b109e1f4-308a-4557-ad21-bb8ae0d75553-istio-token" pod "b109e1f4-308a-4557-ad21-bb8ae0d75553" (UID: "b109e1f4-308a-4557-ad21-bb8ae0d75553"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:35:10.187218 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.187186 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b109e1f4-308a-4557-ad21-bb8ae0d75553-istio-envoy" pod "b109e1f4-308a-4557-ad21-bb8ae0d75553" (UID: "b109e1f4-308a-4557-ad21-bb8ae0d75553"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:35:10.319549 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.313460 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b109e1f4-308a-4557-ad21-bb8ae0d75553-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.319549 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.313514 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.319549 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.313529 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.319549 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.313542 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kxx4\" (UniqueName: \"kubernetes.io/projected/b109e1f4-308a-4557-ad21-bb8ae0d75553-kube-api-access-5kxx4\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.319549 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.313578 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.319549 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.313590 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b109e1f4-308a-4557-ad21-bb8ae0d75553-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.319549 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.313602 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.319549 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.313614 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b109e1f4-308a-4557-ad21-bb8ae0d75553-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.319549 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.313626 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b109e1f4-308a-4557-ad21-bb8ae0d75553-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.319549 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.313656 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b109e1f4-308a-4557-ad21-bb8ae0d75553-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.319549 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.313673 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b109e1f4-308a-4557-ad21-bb8ae0d75553-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.319549 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.313687 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cz2w\" (UniqueName: \"kubernetes.io/projected/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-kube-api-access-8cz2w\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.319549 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.313700 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b109e1f4-308a-4557-ad21-bb8ae0d75553-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.319549 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.313726 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.319549 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.313739 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.319549 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.313751 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.319549 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.313764 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b109e1f4-308a-4557-ad21-bb8ae0d75553-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.319549 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.313776 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22fad5ad-56d4-4d01-944f-a5db77a4a3ee-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:10.399156 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.399123 5979 generic.go:356] "Generic (PLEG): container finished" podID="22fad5ad-56d4-4d01-944f-a5db77a4a3ee" containerID="0c0701613949d61961abc7b953bbaa3f49b57d39e2aa21425167e5e5880f6828" exitCode=0 Nov 03 13:35:10.399360 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.399179 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74" Nov 03 13:35:10.399360 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.399207 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74" event={"ID":"22fad5ad-56d4-4d01-944f-a5db77a4a3ee","Type":"ContainerDied","Data":"0c0701613949d61961abc7b953bbaa3f49b57d39e2aa21425167e5e5880f6828"} Nov 03 13:35:10.399360 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.399275 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74" event={"ID":"22fad5ad-56d4-4d01-944f-a5db77a4a3ee","Type":"ContainerDied","Data":"05e3218cea31daae4e10e4a40184f85210e2658b5775cc69ae58f8a06bd7f2a6"} Nov 03 13:35:10.399360 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.399305 5979 scope.go:122] "RemoveContainer" containerID="0c0701613949d61961abc7b953bbaa3f49b57d39e2aa21425167e5e5880f6828" Nov 03 13:35:10.400730 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.400684 5979 generic.go:356] "Generic (PLEG): container finished" podID="5fd8d2bb-4812-41bc-88a9-0d8249c25c1c" containerID="1f7a2ff3e89e525a4021063055e7c71af56a94f5c449df20286ac3336aff5e1c" exitCode=0 Nov 03 13:35:10.400730 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.400723 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm" event={"ID":"5fd8d2bb-4812-41bc-88a9-0d8249c25c1c","Type":"ContainerDied","Data":"1f7a2ff3e89e525a4021063055e7c71af56a94f5c449df20286ac3336aff5e1c"} Nov 03 13:35:10.400997 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.400756 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm" Nov 03 13:35:10.400997 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.400763 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm" event={"ID":"5fd8d2bb-4812-41bc-88a9-0d8249c25c1c","Type":"ContainerDied","Data":"872b5552b2289e4897198dfa4d1516c460f2d4685636041f3faf41134db25c0c"} Nov 03 13:35:10.402255 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.402231 5979 generic.go:356] "Generic (PLEG): container finished" podID="fc325c76-815f-4a33-b5b6-71421ad64ded" containerID="178ac86c8cbdd80437a8bf478eb86f1ad0e1e3816991f8b3ccf0ef1eaf3749f1" exitCode=0 Nov 03 13:35:10.402371 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.402281 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl" Nov 03 13:35:10.402371 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.402310 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl" event={"ID":"fc325c76-815f-4a33-b5b6-71421ad64ded","Type":"ContainerDied","Data":"178ac86c8cbdd80437a8bf478eb86f1ad0e1e3816991f8b3ccf0ef1eaf3749f1"} Nov 03 13:35:10.402371 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.402359 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl" event={"ID":"fc325c76-815f-4a33-b5b6-71421ad64ded","Type":"ContainerDied","Data":"c7d0ae7bce8b5873397fec1f92f117add9ed1a1b75d9c89a78306b6c8c0c7844"} Nov 03 13:35:10.404464 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.404444 5979 generic.go:356] "Generic (PLEG): container finished" podID="b109e1f4-308a-4557-ad21-bb8ae0d75553" containerID="6b1c04d6646625b809d10de2d313efd013416a8692895d9bd18da25323a49696" exitCode=0 Nov 03 13:35:10.404606 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.404506 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z" event={"ID":"b109e1f4-308a-4557-ad21-bb8ae0d75553","Type":"ContainerDied","Data":"6b1c04d6646625b809d10de2d313efd013416a8692895d9bd18da25323a49696"} Nov 03 13:35:10.404606 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.404536 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z" event={"ID":"b109e1f4-308a-4557-ad21-bb8ae0d75553","Type":"ContainerDied","Data":"fb70cb800393dcf2d4cc6454fc956812ad91c838297da2e232929cbd6ffd6f29"} Nov 03 13:35:10.404606 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.404511 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z" Nov 03 13:35:10.412368 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.412353 5979 scope.go:122] "RemoveContainer" containerID="0c0701613949d61961abc7b953bbaa3f49b57d39e2aa21425167e5e5880f6828" Nov 03 13:35:10.412652 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:35:10.412630 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c0701613949d61961abc7b953bbaa3f49b57d39e2aa21425167e5e5880f6828\": container with ID starting with 0c0701613949d61961abc7b953bbaa3f49b57d39e2aa21425167e5e5880f6828 not found: ID does not exist" containerID="0c0701613949d61961abc7b953bbaa3f49b57d39e2aa21425167e5e5880f6828" Nov 03 13:35:10.412838 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.412661 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c0701613949d61961abc7b953bbaa3f49b57d39e2aa21425167e5e5880f6828"} err="failed to get container status \"0c0701613949d61961abc7b953bbaa3f49b57d39e2aa21425167e5e5880f6828\": rpc error: code = NotFound desc = could not find container \"0c0701613949d61961abc7b953bbaa3f49b57d39e2aa21425167e5e5880f6828\": container with ID starting with 0c0701613949d61961abc7b953bbaa3f49b57d39e2aa21425167e5e5880f6828 not found: ID does not exist" Nov 03 13:35:10.412838 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.412682 5979 scope.go:122] "RemoveContainer" containerID="1f7a2ff3e89e525a4021063055e7c71af56a94f5c449df20286ac3336aff5e1c" Nov 03 13:35:10.426384 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.426361 5979 scope.go:122] "RemoveContainer" containerID="1f7a2ff3e89e525a4021063055e7c71af56a94f5c449df20286ac3336aff5e1c" Nov 03 13:35:10.426725 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:35:10.426704 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f7a2ff3e89e525a4021063055e7c71af56a94f5c449df20286ac3336aff5e1c\": container with ID starting with 1f7a2ff3e89e525a4021063055e7c71af56a94f5c449df20286ac3336aff5e1c not found: ID does not exist" containerID="1f7a2ff3e89e525a4021063055e7c71af56a94f5c449df20286ac3336aff5e1c" Nov 03 13:35:10.426770 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.426733 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f7a2ff3e89e525a4021063055e7c71af56a94f5c449df20286ac3336aff5e1c"} err="failed to get container status \"1f7a2ff3e89e525a4021063055e7c71af56a94f5c449df20286ac3336aff5e1c\": rpc error: code = NotFound desc = could not find container \"1f7a2ff3e89e525a4021063055e7c71af56a94f5c449df20286ac3336aff5e1c\": container with ID starting with 1f7a2ff3e89e525a4021063055e7c71af56a94f5c449df20286ac3336aff5e1c not found: ID does not exist" Nov 03 13:35:10.426770 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.426750 5979 scope.go:122] "RemoveContainer" containerID="178ac86c8cbdd80437a8bf478eb86f1ad0e1e3816991f8b3ccf0ef1eaf3749f1" Nov 03 13:35:10.474988 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.474938 5979 scope.go:122] "RemoveContainer" containerID="178ac86c8cbdd80437a8bf478eb86f1ad0e1e3816991f8b3ccf0ef1eaf3749f1" Nov 03 13:35:10.480569 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:35:10.480528 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"178ac86c8cbdd80437a8bf478eb86f1ad0e1e3816991f8b3ccf0ef1eaf3749f1\": container with ID starting with 178ac86c8cbdd80437a8bf478eb86f1ad0e1e3816991f8b3ccf0ef1eaf3749f1 not found: ID does not exist" containerID="178ac86c8cbdd80437a8bf478eb86f1ad0e1e3816991f8b3ccf0ef1eaf3749f1" Nov 03 13:35:10.480800 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.480576 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"178ac86c8cbdd80437a8bf478eb86f1ad0e1e3816991f8b3ccf0ef1eaf3749f1"} err="failed to get container status \"178ac86c8cbdd80437a8bf478eb86f1ad0e1e3816991f8b3ccf0ef1eaf3749f1\": rpc error: code = NotFound desc = could not find container \"178ac86c8cbdd80437a8bf478eb86f1ad0e1e3816991f8b3ccf0ef1eaf3749f1\": container with ID starting with 178ac86c8cbdd80437a8bf478eb86f1ad0e1e3816991f8b3ccf0ef1eaf3749f1 not found: ID does not exist" Nov 03 13:35:10.480800 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.480604 5979 scope.go:122] "RemoveContainer" containerID="6b1c04d6646625b809d10de2d313efd013416a8692895d9bd18da25323a49696" Nov 03 13:35:10.510831 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.510789 5979 scope.go:122] "RemoveContainer" containerID="6b1c04d6646625b809d10de2d313efd013416a8692895d9bd18da25323a49696" Nov 03 13:35:10.511297 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:35:10.511248 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b1c04d6646625b809d10de2d313efd013416a8692895d9bd18da25323a49696\": container with ID starting with 6b1c04d6646625b809d10de2d313efd013416a8692895d9bd18da25323a49696 not found: ID does not exist" containerID="6b1c04d6646625b809d10de2d313efd013416a8692895d9bd18da25323a49696" Nov 03 13:35:10.511418 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.511319 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b1c04d6646625b809d10de2d313efd013416a8692895d9bd18da25323a49696"} err="failed to get container status \"6b1c04d6646625b809d10de2d313efd013416a8692895d9bd18da25323a49696\": rpc error: code = NotFound desc = could not find container \"6b1c04d6646625b809d10de2d313efd013416a8692895d9bd18da25323a49696\": container with ID starting with 6b1c04d6646625b809d10de2d313efd013416a8692895d9bd18da25323a49696 not found: ID does not exist" Nov 03 13:35:10.687202 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.687169 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm"] Nov 03 13:35:10.693673 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.693644 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-6g5zm"] Nov 03 13:35:10.796269 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.796234 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74"] Nov 03 13:35:10.800983 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.800958 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-nnd74"] Nov 03 13:35:10.904812 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.904775 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl"] Nov 03 13:35:10.909021 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:10.908995 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-zfnwl"] Nov 03 13:35:11.016752 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:11.016661 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z"] Nov 03 13:35:11.020062 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:11.020036 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5295ab29-openshift-default-7c46b866d5-n779z"] Nov 03 13:35:11.765500 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:11.764821 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba0d2a06-2b9dm"] Nov 03 13:35:11.766497 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:11.765827 5979 cpu_manager.go:414] "containerMap: removing container" podUID="22fad5ad-56d4-4d01-944f-a5db77a4a3ee" containerName="istio-proxy" Nov 03 13:35:11.766497 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:11.765864 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22fad5ad-56d4-4d01-944f-a5db77a4a3ee" containerName="istio-proxy" Nov 03 13:35:11.766497 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:11.765908 5979 cpu_manager.go:414] "containerMap: removing container" podUID="fc325c76-815f-4a33-b5b6-71421ad64ded" containerName="istio-proxy" Nov 03 13:35:11.766497 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:11.765917 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc325c76-815f-4a33-b5b6-71421ad64ded" containerName="istio-proxy" Nov 03 13:35:11.766497 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:11.765969 5979 cpu_manager.go:414] "containerMap: removing container" podUID="5fd8d2bb-4812-41bc-88a9-0d8249c25c1c" containerName="istio-proxy" Nov 03 13:35:11.766497 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:11.765978 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fd8d2bb-4812-41bc-88a9-0d8249c25c1c" containerName="istio-proxy" Nov 03 13:35:11.766497 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:11.765996 5979 cpu_manager.go:414] "containerMap: removing container" podUID="b109e1f4-308a-4557-ad21-bb8ae0d75553" containerName="istio-proxy" Nov 03 13:35:11.766497 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:11.766005 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b109e1f4-308a-4557-ad21-bb8ae0d75553" containerName="istio-proxy" Nov 03 13:35:11.766497 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:11.766158 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="22fad5ad-56d4-4d01-944f-a5db77a4a3ee" containerName="istio-proxy" Nov 03 13:35:11.766497 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:11.766178 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc325c76-815f-4a33-b5b6-71421ad64ded" containerName="istio-proxy" Nov 03 13:35:11.766497 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:11.766195 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fd8d2bb-4812-41bc-88a9-0d8249c25c1c" containerName="istio-proxy" Nov 03 13:35:11.766497 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:11.766206 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="b109e1f4-308a-4557-ad21-bb8ae0d75553" containerName="istio-proxy" Nov 03 13:35:11.773977 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:11.773957 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba0d2a06-2b9dm" Nov 03 13:35:11.806437 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:11.806408 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba0d2a06-2b9dm"] Nov 03 13:35:11.823183 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:11.823155 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:35:11.823333 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:11.823188 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:35:11.934465 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:11.934429 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6c1fb16-1f63-4044-b7cc-0fb72ac53b31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba0d2a06-2b9dm\" (UID: \"b6c1fb16-1f63-4044-b7cc-0fb72ac53b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba0d2a06-2b9dm" Nov 03 13:35:11.934698 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:11.934553 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k54x2\" (UniqueName: \"kubernetes.io/projected/b6c1fb16-1f63-4044-b7cc-0fb72ac53b31-kube-api-access-k54x2\") pod \"maas-api-revoke-sub-ba0d2a06-2b9dm\" (UID: \"b6c1fb16-1f63-4044-b7cc-0fb72ac53b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba0d2a06-2b9dm" Nov 03 13:35:11.934781 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:11.934717 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6c1fb16-1f63-4044-b7cc-0fb72ac53b31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba0d2a06-2b9dm\" (UID: \"b6c1fb16-1f63-4044-b7cc-0fb72ac53b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba0d2a06-2b9dm" Nov 03 13:35:12.036400 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:12.036312 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k54x2\" (UniqueName: \"kubernetes.io/projected/b6c1fb16-1f63-4044-b7cc-0fb72ac53b31-kube-api-access-k54x2\") pod \"maas-api-revoke-sub-ba0d2a06-2b9dm\" (UID: \"b6c1fb16-1f63-4044-b7cc-0fb72ac53b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba0d2a06-2b9dm" Nov 03 13:35:12.036400 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:12.036397 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6c1fb16-1f63-4044-b7cc-0fb72ac53b31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba0d2a06-2b9dm\" (UID: \"b6c1fb16-1f63-4044-b7cc-0fb72ac53b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba0d2a06-2b9dm" Nov 03 13:35:12.036915 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:12.036548 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6c1fb16-1f63-4044-b7cc-0fb72ac53b31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba0d2a06-2b9dm\" (UID: \"b6c1fb16-1f63-4044-b7cc-0fb72ac53b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba0d2a06-2b9dm" Nov 03 13:35:12.037148 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:12.037129 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6c1fb16-1f63-4044-b7cc-0fb72ac53b31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba0d2a06-2b9dm\" (UID: \"b6c1fb16-1f63-4044-b7cc-0fb72ac53b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba0d2a06-2b9dm" Nov 03 13:35:12.038709 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:12.038687 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6c1fb16-1f63-4044-b7cc-0fb72ac53b31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba0d2a06-2b9dm\" (UID: \"b6c1fb16-1f63-4044-b7cc-0fb72ac53b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba0d2a06-2b9dm" Nov 03 13:35:12.090789 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:12.090759 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k54x2\" (UniqueName: \"kubernetes.io/projected/b6c1fb16-1f63-4044-b7cc-0fb72ac53b31-kube-api-access-k54x2\") pod \"maas-api-revoke-sub-ba0d2a06-2b9dm\" (UID: \"b6c1fb16-1f63-4044-b7cc-0fb72ac53b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba0d2a06-2b9dm" Nov 03 13:35:12.117289 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:12.117260 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22fad5ad-56d4-4d01-944f-a5db77a4a3ee" path="/var/lib/kubelet/pods/22fad5ad-56d4-4d01-944f-a5db77a4a3ee/volumes" Nov 03 13:35:12.117805 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:12.117791 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fd8d2bb-4812-41bc-88a9-0d8249c25c1c" path="/var/lib/kubelet/pods/5fd8d2bb-4812-41bc-88a9-0d8249c25c1c/volumes" Nov 03 13:35:12.118240 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:12.118227 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b109e1f4-308a-4557-ad21-bb8ae0d75553" path="/var/lib/kubelet/pods/b109e1f4-308a-4557-ad21-bb8ae0d75553/volumes" Nov 03 13:35:12.118997 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:12.118977 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc325c76-815f-4a33-b5b6-71421ad64ded" path="/var/lib/kubelet/pods/fc325c76-815f-4a33-b5b6-71421ad64ded/volumes" Nov 03 13:35:12.379127 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:12.379053 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c8192-55bbd4cd78-8jfgv" Nov 03 13:35:12.387678 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:12.387658 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba0d2a06-2b9dm" Nov 03 13:35:12.783976 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:12.783945 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba0d2a06-2b9dm"] Nov 03 13:35:12.790600 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:35:12.790572 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6c1fb16_1f63_4044_b7cc_0fb72ac53b31.slice/crio-bff4981672aeef5daffe7de97b2015cc291c97ee2cb494965fd8546eef474693 WatchSource:0}: Error finding container bff4981672aeef5daffe7de97b2015cc291c97ee2cb494965fd8546eef474693: Status 404 returned error can't find the container with id bff4981672aeef5daffe7de97b2015cc291c97ee2cb494965fd8546eef474693 Nov 03 13:35:13.433286 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:13.433195 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba0d2a06-2b9dm" event={"ID":"b6c1fb16-1f63-4044-b7cc-0fb72ac53b31","Type":"ContainerStarted","Data":"7b23dfb888b62aa3a607b8e1e5fe8772593cdefe4360eb3876883a4c8ffab79f"} Nov 03 13:35:13.433286 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:13.433245 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba0d2a06-2b9dm" event={"ID":"b6c1fb16-1f63-4044-b7cc-0fb72ac53b31","Type":"ContainerStarted","Data":"bff4981672aeef5daffe7de97b2015cc291c97ee2cb494965fd8546eef474693"} Nov 03 13:35:13.565723 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:13.565653 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba0d2a06-2b9dm" podStartSLOduration=2.565629266 podStartE2EDuration="2.565629266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:35:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:35:13.542529901 +0000 UTC m=+1795.553218144" watchObservedRunningTime="2026-11-03 13:35:13.565629266 +0000 UTC m=+1795.576317508" Nov 03 13:35:14.443193 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:14.443162 5979 generic.go:356] "Generic (PLEG): container finished" podID="b6c1fb16-1f63-4044-b7cc-0fb72ac53b31" containerID="7b23dfb888b62aa3a607b8e1e5fe8772593cdefe4360eb3876883a4c8ffab79f" exitCode=0 Nov 03 13:35:14.443585 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:14.443219 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba0d2a06-2b9dm" event={"ID":"b6c1fb16-1f63-4044-b7cc-0fb72ac53b31","Type":"ContainerDied","Data":"7b23dfb888b62aa3a607b8e1e5fe8772593cdefe4360eb3876883a4c8ffab79f"} Nov 03 13:35:15.615332 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:15.615305 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba0d2a06-2b9dm" Nov 03 13:35:15.711047 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:15.710957 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6c1fb16-1f63-4044-b7cc-0fb72ac53b31-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6c1fb16-1f63-4044-b7cc-0fb72ac53b31-maas-api-xks-ca\") pod \"b6c1fb16-1f63-4044-b7cc-0fb72ac53b31\" (UID: \"b6c1fb16-1f63-4044-b7cc-0fb72ac53b31\") " Nov 03 13:35:15.711223 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:15.711118 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6c1fb16-1f63-4044-b7cc-0fb72ac53b31-kube-api-access-k54x2\" (UniqueName: \"kubernetes.io/projected/b6c1fb16-1f63-4044-b7cc-0fb72ac53b31-kube-api-access-k54x2\") pod \"b6c1fb16-1f63-4044-b7cc-0fb72ac53b31\" (UID: \"b6c1fb16-1f63-4044-b7cc-0fb72ac53b31\") " Nov 03 13:35:15.711611 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:15.711561 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6c1fb16-1f63-4044-b7cc-0fb72ac53b31-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6c1fb16-1f63-4044-b7cc-0fb72ac53b31-maas-api-openshift-ca\") pod \"b6c1fb16-1f63-4044-b7cc-0fb72ac53b31\" (UID: \"b6c1fb16-1f63-4044-b7cc-0fb72ac53b31\") " Nov 03 13:35:15.712193 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:15.712109 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6c1fb16-1f63-4044-b7cc-0fb72ac53b31-maas-api-openshift-ca" pod "b6c1fb16-1f63-4044-b7cc-0fb72ac53b31" (UID: "b6c1fb16-1f63-4044-b7cc-0fb72ac53b31"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:35:15.712888 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:15.712865 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6c1fb16-1f63-4044-b7cc-0fb72ac53b31-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:15.721505 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:15.713587 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6c1fb16-1f63-4044-b7cc-0fb72ac53b31-maas-api-xks-ca" pod "b6c1fb16-1f63-4044-b7cc-0fb72ac53b31" (UID: "b6c1fb16-1f63-4044-b7cc-0fb72ac53b31"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:35:15.721505 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:15.713883 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6c1fb16-1f63-4044-b7cc-0fb72ac53b31-kube-api-access-k54x2" pod "b6c1fb16-1f63-4044-b7cc-0fb72ac53b31" (UID: "b6c1fb16-1f63-4044-b7cc-0fb72ac53b31"). InnerVolumeSpecName "kube-api-access-k54x2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:35:15.721505 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:15.714792 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-be8d0a-7bf74dc5f-lkxx5"] Nov 03 13:35:15.721505 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:15.715644 5979 cpu_manager.go:414] "containerMap: removing container" podUID="b6c1fb16-1f63-4044-b7cc-0fb72ac53b31" containerName="revoke-keys" Nov 03 13:35:15.721505 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:15.715662 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6c1fb16-1f63-4044-b7cc-0fb72ac53b31" containerName="revoke-keys" Nov 03 13:35:15.721505 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:15.715861 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6c1fb16-1f63-4044-b7cc-0fb72ac53b31" containerName="revoke-keys" Nov 03 13:35:15.721505 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:15.719515 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be8d0a-7bf74dc5f-lkxx5" Nov 03 13:35:15.770748 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:15.770702 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-be8d0a-dockercfg-qtk7l\"" Nov 03 13:35:15.773371 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:15.773335 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-be8d0a\"" Nov 03 13:35:15.787128 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:15.787093 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-be8d0a-7bf74dc5f-lkxx5"] Nov 03 13:35:15.813806 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:15.813767 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pfbj\" (UniqueName: \"kubernetes.io/projected/3d59ed0e-e153-4d82-93ad-dd7adf1a3cd7-kube-api-access-9pfbj\") pod \"payload-pre-processing-e2e-ipp-a-be8d0a-7bf74dc5f-lkxx5\" (UID: \"3d59ed0e-e153-4d82-93ad-dd7adf1a3cd7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be8d0a-7bf74dc5f-lkxx5" Nov 03 13:35:15.813953 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:15.813833 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d59ed0e-e153-4d82-93ad-dd7adf1a3cd7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-be8d0a-7bf74dc5f-lkxx5\" (UID: \"3d59ed0e-e153-4d82-93ad-dd7adf1a3cd7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be8d0a-7bf74dc5f-lkxx5" Nov 03 13:35:15.814013 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:15.813971 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6c1fb16-1f63-4044-b7cc-0fb72ac53b31-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:15.814013 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:15.813993 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k54x2\" (UniqueName: \"kubernetes.io/projected/b6c1fb16-1f63-4044-b7cc-0fb72ac53b31-kube-api-access-k54x2\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:35:15.915298 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:15.915256 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pfbj\" (UniqueName: \"kubernetes.io/projected/3d59ed0e-e153-4d82-93ad-dd7adf1a3cd7-kube-api-access-9pfbj\") pod \"payload-pre-processing-e2e-ipp-a-be8d0a-7bf74dc5f-lkxx5\" (UID: \"3d59ed0e-e153-4d82-93ad-dd7adf1a3cd7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be8d0a-7bf74dc5f-lkxx5" Nov 03 13:35:15.915543 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:15.915336 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d59ed0e-e153-4d82-93ad-dd7adf1a3cd7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-be8d0a-7bf74dc5f-lkxx5\" (UID: \"3d59ed0e-e153-4d82-93ad-dd7adf1a3cd7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be8d0a-7bf74dc5f-lkxx5" Nov 03 13:35:15.916109 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:15.916081 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d59ed0e-e153-4d82-93ad-dd7adf1a3cd7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-be8d0a-7bf74dc5f-lkxx5\" (UID: \"3d59ed0e-e153-4d82-93ad-dd7adf1a3cd7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be8d0a-7bf74dc5f-lkxx5" Nov 03 13:35:15.970613 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:15.970517 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pfbj\" (UniqueName: \"kubernetes.io/projected/3d59ed0e-e153-4d82-93ad-dd7adf1a3cd7-kube-api-access-9pfbj\") pod \"payload-pre-processing-e2e-ipp-a-be8d0a-7bf74dc5f-lkxx5\" (UID: \"3d59ed0e-e153-4d82-93ad-dd7adf1a3cd7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be8d0a-7bf74dc5f-lkxx5" Nov 03 13:35:16.055425 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:16.055366 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be8d0a-7bf74dc5f-lkxx5" Nov 03 13:35:16.441343 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:16.441300 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-be8d0a-7bf74dc5f-lkxx5"] Nov 03 13:35:16.445740 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:16.445715 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-be8d0a-58864d5c94-vcxtd"] Nov 03 13:35:16.447499 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:35:16.447442 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d59ed0e_e153_4d82_93ad_dd7adf1a3cd7.slice/crio-fc4de57eb4824780c8ad1ae4bfa73da081cf76bdedede4ca3502909963e92f18 WatchSource:0}: Error finding container fc4de57eb4824780c8ad1ae4bfa73da081cf76bdedede4ca3502909963e92f18: Status 404 returned error can't find the container with id fc4de57eb4824780c8ad1ae4bfa73da081cf76bdedede4ca3502909963e92f18 Nov 03 13:35:16.451345 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:16.451134 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-be8d0a-58864d5c94-vcxtd" Nov 03 13:35:16.459375 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:16.459350 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba0d2a06-2b9dm" event={"ID":"b6c1fb16-1f63-4044-b7cc-0fb72ac53b31","Type":"ContainerDied","Data":"bff4981672aeef5daffe7de97b2015cc291c97ee2cb494965fd8546eef474693"} Nov 03 13:35:16.459455 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:16.459396 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bff4981672aeef5daffe7de97b2015cc291c97ee2cb494965fd8546eef474693" Nov 03 13:35:16.459511 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:16.459461 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba0d2a06-2b9dm" Nov 03 13:35:16.461253 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:16.461227 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be8d0a-7bf74dc5f-lkxx5" event={"ID":"3d59ed0e-e153-4d82-93ad-dd7adf1a3cd7","Type":"ContainerStarted","Data":"fc4de57eb4824780c8ad1ae4bfa73da081cf76bdedede4ca3502909963e92f18"} Nov 03 13:35:16.506346 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:16.505887 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-be8d0a-58864d5c94-vcxtd"] Nov 03 13:35:16.525896 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:16.525620 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc3ef1e4-0333-411b-a762-9a4f7cc8a0db-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-be8d0a-58864d5c94-vcxtd\" (UID: \"fc3ef1e4-0333-411b-a762-9a4f7cc8a0db\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-be8d0a-58864d5c94-vcxtd" Nov 03 13:35:16.525896 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:16.525757 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhfqk\" (UniqueName: \"kubernetes.io/projected/fc3ef1e4-0333-411b-a762-9a4f7cc8a0db-kube-api-access-vhfqk\") pod \"payload-processing-e2e-ipp-a-be8d0a-58864d5c94-vcxtd\" (UID: \"fc3ef1e4-0333-411b-a762-9a4f7cc8a0db\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-be8d0a-58864d5c94-vcxtd" Nov 03 13:35:16.627211 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:16.627172 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc3ef1e4-0333-411b-a762-9a4f7cc8a0db-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-be8d0a-58864d5c94-vcxtd\" (UID: \"fc3ef1e4-0333-411b-a762-9a4f7cc8a0db\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-be8d0a-58864d5c94-vcxtd" Nov 03 13:35:16.627643 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:16.627243 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhfqk\" (UniqueName: \"kubernetes.io/projected/fc3ef1e4-0333-411b-a762-9a4f7cc8a0db-kube-api-access-vhfqk\") pod \"payload-processing-e2e-ipp-a-be8d0a-58864d5c94-vcxtd\" (UID: \"fc3ef1e4-0333-411b-a762-9a4f7cc8a0db\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-be8d0a-58864d5c94-vcxtd" Nov 03 13:35:16.627931 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:16.627910 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc3ef1e4-0333-411b-a762-9a4f7cc8a0db-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-be8d0a-58864d5c94-vcxtd\" (UID: \"fc3ef1e4-0333-411b-a762-9a4f7cc8a0db\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-be8d0a-58864d5c94-vcxtd" Nov 03 13:35:16.691834 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:16.691763 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhfqk\" (UniqueName: \"kubernetes.io/projected/fc3ef1e4-0333-411b-a762-9a4f7cc8a0db-kube-api-access-vhfqk\") pod \"payload-processing-e2e-ipp-a-be8d0a-58864d5c94-vcxtd\" (UID: \"fc3ef1e4-0333-411b-a762-9a4f7cc8a0db\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-be8d0a-58864d5c94-vcxtd" Nov 03 13:35:16.769000 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:16.768965 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-be8d0a-58864d5c94-vcxtd" Nov 03 13:35:17.157323 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:17.157283 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-be8d0a-58864d5c94-vcxtd"] Nov 03 13:35:17.163444 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:35:17.163413 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc3ef1e4_0333_411b_a762_9a4f7cc8a0db.slice/crio-96a21f004ef9f53bdf0e56b03e30445ffdfb0dd653d32dfe0d9a7446a1679374 WatchSource:0}: Error finding container 96a21f004ef9f53bdf0e56b03e30445ffdfb0dd653d32dfe0d9a7446a1679374: Status 404 returned error can't find the container with id 96a21f004ef9f53bdf0e56b03e30445ffdfb0dd653d32dfe0d9a7446a1679374 Nov 03 13:35:17.473411 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:17.473312 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be8d0a-7bf74dc5f-lkxx5" event={"ID":"3d59ed0e-e153-4d82-93ad-dd7adf1a3cd7","Type":"ContainerStarted","Data":"cd60dc983ef8b94418ba0ab27203fd6de8d36f93b94e3a252be3654a14d19379"} Nov 03 13:35:17.473765 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:17.473743 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be8d0a-7bf74dc5f-lkxx5" Nov 03 13:35:17.475408 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:17.475375 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-be8d0a-58864d5c94-vcxtd" event={"ID":"fc3ef1e4-0333-411b-a762-9a4f7cc8a0db","Type":"ContainerStarted","Data":"363bda6db308dd86f0cb172b97b88b6333337b4133f12e2b962eb9bb7723249f"} Nov 03 13:35:17.475525 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:17.475419 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-be8d0a-58864d5c94-vcxtd" event={"ID":"fc3ef1e4-0333-411b-a762-9a4f7cc8a0db","Type":"ContainerStarted","Data":"96a21f004ef9f53bdf0e56b03e30445ffdfb0dd653d32dfe0d9a7446a1679374"} Nov 03 13:35:17.476047 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:17.476027 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-be8d0a-58864d5c94-vcxtd" Nov 03 13:35:17.591141 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:17.591070 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be8d0a-7bf74dc5f-lkxx5" podStartSLOduration=2.591048699 podStartE2EDuration="2.591048699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:35:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:35:17.58419096 +0000 UTC m=+1799.594879211" watchObservedRunningTime="2026-11-03 13:35:17.591048699 +0000 UTC m=+1799.601736939" Nov 03 13:35:17.693839 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:17.693773 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-be8d0a-58864d5c94-vcxtd" podStartSLOduration=1.6937550049999999 podStartE2EDuration="1.693755005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:35:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:35:17.691987465 +0000 UTC m=+1799.702675707" watchObservedRunningTime="2026-11-03 13:35:17.693755005 +0000 UTC m=+1799.704443247" Nov 03 13:35:18.323774 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:18.323043 5979 patch_prober.go:29] interesting pod/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": read tcp 10.134.0.2:44144->10.134.0.161:15021: read: connection reset by peer" start-of-body= Nov 03 13:35:18.323774 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:18.323108 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s" podUID="09fccbb2-307c-4752-a4d7-f9986eb5f61c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": read tcp 10.134.0.2:44144->10.134.0.161:15021: read: connection reset by peer" Nov 03 13:35:18.483648 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:18.483571 5979 generic.go:356] "Generic (PLEG): container finished" podID="09fccbb2-307c-4752-a4d7-f9986eb5f61c" containerID="5188e424135679f01285add632aa930ddcc42d7f7793c56b89a020827a80ff5c" exitCode=0 Nov 03 13:35:18.483799 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:18.483647 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s" event={"ID":"09fccbb2-307c-4752-a4d7-f9986eb5f61c","Type":"ContainerDied","Data":"5188e424135679f01285add632aa930ddcc42d7f7793c56b89a020827a80ff5c"} Nov 03 13:35:18.484179 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:18.484159 5979 scope.go:122] "RemoveContainer" containerID="5188e424135679f01285add632aa930ddcc42d7f7793c56b89a020827a80ff5c" Nov 03 13:35:18.485484 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:18.485442 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:35:18.485607 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:18.485525 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:35:18.485607 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:18.485559 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:35:19.299485 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:19.299443 5979 scope.go:122] "RemoveContainer" containerID="5351b77dbd0e72fa7aab841085efd2d04e7f1fcf75b2ed45a5ff570f7bfd302b" Nov 03 13:35:19.310729 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:19.310708 5979 scope.go:122] "RemoveContainer" containerID="5471d8e4b233b9a9f7afcb7c04815b86f3d5d1b42ddc7c71b9074c90502cbca7" Nov 03 13:35:19.501125 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:19.501088 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s" event={"ID":"09fccbb2-307c-4752-a4d7-f9986eb5f61c","Type":"ContainerStarted","Data":"05b18fb973820ac2c88f3c93162fc00bccd839f3959871d39e86492c6898af84"} Nov 03 13:35:20.179923 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:20.179881 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s" Nov 03 13:35:20.180102 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:20.180076 5979 patch_prober.go:29] interesting pod/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Nov 03 13:35:20.180155 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:20.180133 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s" podUID="09fccbb2-307c-4752-a4d7-f9986eb5f61c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Nov 03 13:35:21.180980 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:21.180941 5979 patch_prober.go:29] interesting pod/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Nov 03 13:35:21.181379 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:21.181007 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s" podUID="09fccbb2-307c-4752-a4d7-f9986eb5f61c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Nov 03 13:35:22.180915 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:22.180877 5979 patch_prober.go:29] interesting pod/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Nov 03 13:35:22.181100 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:22.180932 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s" podUID="09fccbb2-307c-4752-a4d7-f9986eb5f61c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Nov 03 13:35:23.180992 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:23.180956 5979 patch_prober.go:29] interesting pod/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Nov 03 13:35:23.181174 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:23.181025 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s" podUID="09fccbb2-307c-4752-a4d7-f9986eb5f61c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Nov 03 13:35:24.180080 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:24.180039 5979 patch_prober.go:29] interesting pod/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Nov 03 13:35:24.180299 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:24.180100 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s" podUID="09fccbb2-307c-4752-a4d7-f9986eb5f61c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Nov 03 13:35:25.184215 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:25.184183 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s" Nov 03 13:35:25.184729 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:25.184707 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s" Nov 03 13:35:25.185463 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:25.185446 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s" Nov 03 13:35:28.104356 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:28.104292 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d29e07-7bcf49d5f9-fxr6d"] Nov 03 13:35:28.112555 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:28.112520 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d29e07-7bcf49d5f9-fxr6d" Nov 03 13:35:28.163696 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:28.163668 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d29e07-7bcf49d5f9-fxr6d"] Nov 03 13:35:28.206723 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:28.206687 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-d29e07-serving-cert\"" Nov 03 13:35:28.294607 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:28.294572 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ede9e063-9b8f-4ad9-80a1-2d841732c694-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d29e07-7bcf49d5f9-fxr6d\" (UID: \"ede9e063-9b8f-4ad9-80a1-2d841732c694\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d29e07-7bcf49d5f9-fxr6d" Nov 03 13:35:28.294754 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:28.294627 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ede9e063-9b8f-4ad9-80a1-2d841732c694-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d29e07-7bcf49d5f9-fxr6d\" (UID: \"ede9e063-9b8f-4ad9-80a1-2d841732c694\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d29e07-7bcf49d5f9-fxr6d" Nov 03 13:35:28.294820 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:28.294788 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5nqz\" (UniqueName: \"kubernetes.io/projected/ede9e063-9b8f-4ad9-80a1-2d841732c694-kube-api-access-q5nqz\") pod \"maas-api-e2e-ipp-b-d29e07-7bcf49d5f9-fxr6d\" (UID: \"ede9e063-9b8f-4ad9-80a1-2d841732c694\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d29e07-7bcf49d5f9-fxr6d" Nov 03 13:35:28.295019 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:28.295000 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ede9e063-9b8f-4ad9-80a1-2d841732c694-metrics-certs\") pod \"maas-api-e2e-ipp-b-d29e07-7bcf49d5f9-fxr6d\" (UID: \"ede9e063-9b8f-4ad9-80a1-2d841732c694\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d29e07-7bcf49d5f9-fxr6d" Nov 03 13:35:28.400493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:28.399560 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ede9e063-9b8f-4ad9-80a1-2d841732c694-metrics-certs\") pod \"maas-api-e2e-ipp-b-d29e07-7bcf49d5f9-fxr6d\" (UID: \"ede9e063-9b8f-4ad9-80a1-2d841732c694\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d29e07-7bcf49d5f9-fxr6d" Nov 03 13:35:28.400493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:28.399713 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ede9e063-9b8f-4ad9-80a1-2d841732c694-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d29e07-7bcf49d5f9-fxr6d\" (UID: \"ede9e063-9b8f-4ad9-80a1-2d841732c694\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d29e07-7bcf49d5f9-fxr6d" Nov 03 13:35:28.400493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:28.399773 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ede9e063-9b8f-4ad9-80a1-2d841732c694-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d29e07-7bcf49d5f9-fxr6d\" (UID: \"ede9e063-9b8f-4ad9-80a1-2d841732c694\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d29e07-7bcf49d5f9-fxr6d" Nov 03 13:35:28.400493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:28.399882 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5nqz\" (UniqueName: \"kubernetes.io/projected/ede9e063-9b8f-4ad9-80a1-2d841732c694-kube-api-access-q5nqz\") pod \"maas-api-e2e-ipp-b-d29e07-7bcf49d5f9-fxr6d\" (UID: \"ede9e063-9b8f-4ad9-80a1-2d841732c694\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d29e07-7bcf49d5f9-fxr6d" Nov 03 13:35:28.401026 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:35:28.400870 5979 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-d29e07-serving-cert: secret "maas-api-e2e-ipp-b-d29e07-serving-cert" not found Nov 03 13:35:28.401026 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:35:28.400976 5979 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ede9e063-9b8f-4ad9-80a1-2d841732c694-maas-api-tls podName:ede9e063-9b8f-4ad9-80a1-2d841732c694 nodeName:}" failed. No retries permitted until 2026-11-03 13:35:28.900952354 +0000 UTC m=+1810.911640598 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ede9e063-9b8f-4ad9-80a1-2d841732c694-maas-api-tls") pod "maas-api-e2e-ipp-b-d29e07-7bcf49d5f9-fxr6d" (UID: "ede9e063-9b8f-4ad9-80a1-2d841732c694") : secret "maas-api-e2e-ipp-b-d29e07-serving-cert" not found Nov 03 13:35:28.402712 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:28.401916 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ede9e063-9b8f-4ad9-80a1-2d841732c694-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d29e07-7bcf49d5f9-fxr6d\" (UID: \"ede9e063-9b8f-4ad9-80a1-2d841732c694\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d29e07-7bcf49d5f9-fxr6d" Nov 03 13:35:28.410498 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:28.408029 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ede9e063-9b8f-4ad9-80a1-2d841732c694-metrics-certs\") pod \"maas-api-e2e-ipp-b-d29e07-7bcf49d5f9-fxr6d\" (UID: \"ede9e063-9b8f-4ad9-80a1-2d841732c694\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d29e07-7bcf49d5f9-fxr6d" Nov 03 13:35:28.453964 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:28.453928 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5nqz\" (UniqueName: \"kubernetes.io/projected/ede9e063-9b8f-4ad9-80a1-2d841732c694-kube-api-access-q5nqz\") pod \"maas-api-e2e-ipp-b-d29e07-7bcf49d5f9-fxr6d\" (UID: \"ede9e063-9b8f-4ad9-80a1-2d841732c694\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d29e07-7bcf49d5f9-fxr6d" Nov 03 13:35:28.486233 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:28.486206 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be8d0a-7bf74dc5f-lkxx5" Nov 03 13:35:28.908611 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:28.908580 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ede9e063-9b8f-4ad9-80a1-2d841732c694-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d29e07-7bcf49d5f9-fxr6d\" (UID: \"ede9e063-9b8f-4ad9-80a1-2d841732c694\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d29e07-7bcf49d5f9-fxr6d" Nov 03 13:35:28.910998 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:28.910976 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ede9e063-9b8f-4ad9-80a1-2d841732c694-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d29e07-7bcf49d5f9-fxr6d\" (UID: \"ede9e063-9b8f-4ad9-80a1-2d841732c694\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d29e07-7bcf49d5f9-fxr6d" Nov 03 13:35:29.027945 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:29.027907 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d29e07-7bcf49d5f9-fxr6d" Nov 03 13:35:29.408764 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:29.408728 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d29e07-7bcf49d5f9-fxr6d"] Nov 03 13:35:29.415040 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:35:29.415015 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podede9e063_9b8f_4ad9_80a1_2d841732c694.slice/crio-d04a8a7f88c64a61e7a7ff234c96ee4b44062c0ef44dad1f4d7adaf575ca8d93 WatchSource:0}: Error finding container d04a8a7f88c64a61e7a7ff234c96ee4b44062c0ef44dad1f4d7adaf575ca8d93: Status 404 returned error can't find the container with id d04a8a7f88c64a61e7a7ff234c96ee4b44062c0ef44dad1f4d7adaf575ca8d93 Nov 03 13:35:29.503354 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:29.503326 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-be8d0a-58864d5c94-vcxtd" Nov 03 13:35:29.592656 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:29.592622 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d29e07-7bcf49d5f9-fxr6d" event={"ID":"ede9e063-9b8f-4ad9-80a1-2d841732c694","Type":"ContainerStarted","Data":"d04a8a7f88c64a61e7a7ff234c96ee4b44062c0ef44dad1f4d7adaf575ca8d93"} Nov 03 13:35:30.601816 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:30.601722 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d29e07-7bcf49d5f9-fxr6d" event={"ID":"ede9e063-9b8f-4ad9-80a1-2d841732c694","Type":"ContainerStarted","Data":"f95f512f06fd4218f13dbfb4bba5870dbdff2068e7cf8b45e73e5c7c7df99770"} Nov 03 13:35:30.601816 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:30.601806 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d29e07-7bcf49d5f9-fxr6d" Nov 03 13:35:30.716642 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:30.716592 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d29e07-7bcf49d5f9-fxr6d" podStartSLOduration=1.865696419 podStartE2EDuration="2.716578373s" totalImagesPullingTime="850.881954ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:35:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:35:30.715257334 +0000 UTC m=+1812.725945575" watchObservedRunningTime="2026-11-03 13:35:30.716578373 +0000 UTC m=+1812.727266615" Nov 03 13:35:33.343449 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:33.343406 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a770b9-5fbcd499ff-2xsw7"] Nov 03 13:35:33.349649 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:33.349623 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a770b9-5fbcd499ff-2xsw7" Nov 03 13:35:33.368297 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:33.368267 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxj9f\" (UniqueName: \"kubernetes.io/projected/6be3240b-9b5b-4e17-ada4-676bd02c18ff-kube-api-access-lxj9f\") pod \"payload-pre-processing-e2e-ipp-a-a770b9-5fbcd499ff-2xsw7\" (UID: \"6be3240b-9b5b-4e17-ada4-676bd02c18ff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a770b9-5fbcd499ff-2xsw7" Nov 03 13:35:33.368415 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:33.368298 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6be3240b-9b5b-4e17-ada4-676bd02c18ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a770b9-5fbcd499ff-2xsw7\" (UID: \"6be3240b-9b5b-4e17-ada4-676bd02c18ff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a770b9-5fbcd499ff-2xsw7" Nov 03 13:35:33.451530 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:33.448693 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a770b9\"" Nov 03 13:35:33.465619 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:33.465552 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-a770b9-dockercfg-9bkcr\"" Nov 03 13:35:33.470037 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:33.469987 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxj9f\" (UniqueName: \"kubernetes.io/projected/6be3240b-9b5b-4e17-ada4-676bd02c18ff-kube-api-access-lxj9f\") pod \"payload-pre-processing-e2e-ipp-a-a770b9-5fbcd499ff-2xsw7\" (UID: \"6be3240b-9b5b-4e17-ada4-676bd02c18ff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a770b9-5fbcd499ff-2xsw7" Nov 03 13:35:33.470619 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:33.470581 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6be3240b-9b5b-4e17-ada4-676bd02c18ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a770b9-5fbcd499ff-2xsw7\" (UID: \"6be3240b-9b5b-4e17-ada4-676bd02c18ff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a770b9-5fbcd499ff-2xsw7" Nov 03 13:35:33.472239 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:33.472214 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6be3240b-9b5b-4e17-ada4-676bd02c18ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a770b9-5fbcd499ff-2xsw7\" (UID: \"6be3240b-9b5b-4e17-ada4-676bd02c18ff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a770b9-5fbcd499ff-2xsw7" Nov 03 13:35:33.559794 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:33.554680 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxj9f\" (UniqueName: \"kubernetes.io/projected/6be3240b-9b5b-4e17-ada4-676bd02c18ff-kube-api-access-lxj9f\") pod \"payload-pre-processing-e2e-ipp-a-a770b9-5fbcd499ff-2xsw7\" (UID: \"6be3240b-9b5b-4e17-ada4-676bd02c18ff\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a770b9-5fbcd499ff-2xsw7" Nov 03 13:35:33.586590 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:33.586560 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a770b9-5fbcd499ff-2xsw7"] Nov 03 13:35:33.596136 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:33.595936 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a770b9-65459cd457-28vvf"] Nov 03 13:35:33.601156 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:33.601118 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a770b9-65459cd457-28vvf"] Nov 03 13:35:33.601344 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:33.601252 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a770b9-65459cd457-28vvf" Nov 03 13:35:33.665660 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:33.665627 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a770b9-5fbcd499ff-2xsw7" Nov 03 13:35:33.687146 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:33.687068 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/489c2cad-0beb-4090-8a88-16dce56f5471-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a770b9-65459cd457-28vvf\" (UID: \"489c2cad-0beb-4090-8a88-16dce56f5471\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a770b9-65459cd457-28vvf" Nov 03 13:35:33.687298 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:33.687211 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ds6z\" (UniqueName: \"kubernetes.io/projected/489c2cad-0beb-4090-8a88-16dce56f5471-kube-api-access-9ds6z\") pod \"payload-processing-e2e-ipp-a-a770b9-65459cd457-28vvf\" (UID: \"489c2cad-0beb-4090-8a88-16dce56f5471\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a770b9-65459cd457-28vvf" Nov 03 13:35:33.788368 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:33.788334 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ds6z\" (UniqueName: \"kubernetes.io/projected/489c2cad-0beb-4090-8a88-16dce56f5471-kube-api-access-9ds6z\") pod \"payload-processing-e2e-ipp-a-a770b9-65459cd457-28vvf\" (UID: \"489c2cad-0beb-4090-8a88-16dce56f5471\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a770b9-65459cd457-28vvf" Nov 03 13:35:33.788627 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:33.788608 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/489c2cad-0beb-4090-8a88-16dce56f5471-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a770b9-65459cd457-28vvf\" (UID: \"489c2cad-0beb-4090-8a88-16dce56f5471\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a770b9-65459cd457-28vvf" Nov 03 13:35:33.789297 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:33.789278 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/489c2cad-0beb-4090-8a88-16dce56f5471-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a770b9-65459cd457-28vvf\" (UID: \"489c2cad-0beb-4090-8a88-16dce56f5471\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a770b9-65459cd457-28vvf" Nov 03 13:35:33.843426 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:33.843398 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ds6z\" (UniqueName: \"kubernetes.io/projected/489c2cad-0beb-4090-8a88-16dce56f5471-kube-api-access-9ds6z\") pod \"payload-processing-e2e-ipp-a-a770b9-65459cd457-28vvf\" (UID: \"489c2cad-0beb-4090-8a88-16dce56f5471\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a770b9-65459cd457-28vvf" Nov 03 13:35:33.923604 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:33.923573 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a770b9-65459cd457-28vvf" Nov 03 13:35:34.049721 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:34.049677 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a770b9-5fbcd499ff-2xsw7"] Nov 03 13:35:34.055631 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:35:34.055590 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6be3240b_9b5b_4e17_ada4_676bd02c18ff.slice/crio-88891655528a0703595162ebacd4c128ee86896e72de4fed5fcfb0b9df9e8441 WatchSource:0}: Error finding container 88891655528a0703595162ebacd4c128ee86896e72de4fed5fcfb0b9df9e8441: Status 404 returned error can't find the container with id 88891655528a0703595162ebacd4c128ee86896e72de4fed5fcfb0b9df9e8441 Nov 03 13:35:34.327152 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:34.327050 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a770b9-65459cd457-28vvf"] Nov 03 13:35:34.335575 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:35:34.335459 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod489c2cad_0beb_4090_8a88_16dce56f5471.slice/crio-04d07e886543b55a4b1a403bfad0adccc754343e4213761e564f8c2faad25054 WatchSource:0}: Error finding container 04d07e886543b55a4b1a403bfad0adccc754343e4213761e564f8c2faad25054: Status 404 returned error can't find the container with id 04d07e886543b55a4b1a403bfad0adccc754343e4213761e564f8c2faad25054 Nov 03 13:35:34.640555 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:34.640518 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a770b9-5fbcd499ff-2xsw7" event={"ID":"6be3240b-9b5b-4e17-ada4-676bd02c18ff","Type":"ContainerStarted","Data":"8519cad9e861e3b13f6a4a29d5cd67c0e41fefc95e7c75e71e7203b0c6e84b59"} Nov 03 13:35:34.641007 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:34.640583 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a770b9-5fbcd499ff-2xsw7" event={"ID":"6be3240b-9b5b-4e17-ada4-676bd02c18ff","Type":"ContainerStarted","Data":"88891655528a0703595162ebacd4c128ee86896e72de4fed5fcfb0b9df9e8441"} Nov 03 13:35:34.641007 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:34.640723 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a770b9-5fbcd499ff-2xsw7" Nov 03 13:35:34.642213 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:34.642191 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a770b9-65459cd457-28vvf" event={"ID":"489c2cad-0beb-4090-8a88-16dce56f5471","Type":"ContainerStarted","Data":"218ebf061f3ce3b153493cbfee49fa293ab7dc7be95165833a4bdb29ee43ae98"} Nov 03 13:35:34.642340 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:34.642231 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a770b9-65459cd457-28vvf" event={"ID":"489c2cad-0beb-4090-8a88-16dce56f5471","Type":"ContainerStarted","Data":"04d07e886543b55a4b1a403bfad0adccc754343e4213761e564f8c2faad25054"} Nov 03 13:35:34.642340 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:34.642259 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a770b9-65459cd457-28vvf" Nov 03 13:35:34.750575 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:34.750523 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a770b9-5fbcd499ff-2xsw7" podStartSLOduration=1.750508456 podStartE2EDuration="1.750508456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:35:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:35:34.748518401 +0000 UTC m=+1816.759206644" watchObservedRunningTime="2026-11-03 13:35:34.750508456 +0000 UTC m=+1816.761196694" Nov 03 13:35:34.859820 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:34.859764 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-a770b9-65459cd457-28vvf" podStartSLOduration=1.859748983 podStartE2EDuration="1.859748983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:35:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:35:34.856906424 +0000 UTC m=+1816.867594665" watchObservedRunningTime="2026-11-03 13:35:34.859748983 +0000 UTC m=+1816.870437225" Nov 03 13:35:36.614113 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:36.614084 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d29e07-7bcf49d5f9-fxr6d" Nov 03 13:35:37.134372 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:37.134337 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:35:37.237633 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:35:37.237591 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:35:45.652521 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:45.652486 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a770b9-5fbcd499ff-2xsw7" Nov 03 13:35:45.652983 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:45.652579 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a770b9-65459cd457-28vvf" Nov 03 13:35:48.750054 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:48.750016 5979 generic.go:356] "Generic (PLEG): container finished" podID="7fd6dca0-648c-4476-a1a5-8a7407bf598f" containerID="c0696ee584a93c5517435d6824775f80ed90c756889e69f557c8b761b8295daf" exitCode=0 Nov 03 13:35:48.750939 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:48.750093 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr" event={"ID":"7fd6dca0-648c-4476-a1a5-8a7407bf598f","Type":"ContainerDied","Data":"c0696ee584a93c5517435d6824775f80ed90c756889e69f557c8b761b8295daf"} Nov 03 13:35:48.750939 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:48.750601 5979 scope.go:122] "RemoveContainer" containerID="c0696ee584a93c5517435d6824775f80ed90c756889e69f557c8b761b8295daf" Nov 03 13:35:48.752191 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:48.752167 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:35:48.752253 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:48.752229 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:35:48.752294 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:48.752259 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:35:49.776639 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:49.776598 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr" event={"ID":"7fd6dca0-648c-4476-a1a5-8a7407bf598f","Type":"ContainerStarted","Data":"b3e45af3b10566cc23f0ca3f1a964ad58edef29584104bd0730f57fe418d4c72"} Nov 03 13:35:50.204632 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:50.204599 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr" Nov 03 13:35:50.204834 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:50.204747 5979 patch_prober.go:29] interesting pod/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 03 13:35:50.204834 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:50.204813 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr" podUID="7fd6dca0-648c-4476-a1a5-8a7407bf598f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 03 13:35:51.205643 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:51.205609 5979 patch_prober.go:29] interesting pod/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 03 13:35:51.206062 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:51.205672 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr" podUID="7fd6dca0-648c-4476-a1a5-8a7407bf598f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 03 13:35:52.208153 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:52.206889 5979 patch_prober.go:29] interesting pod/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 03 13:35:52.208153 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:52.206950 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr" podUID="7fd6dca0-648c-4476-a1a5-8a7407bf598f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 03 13:35:53.204740 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:53.204699 5979 patch_prober.go:29] interesting pod/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 03 13:35:53.204942 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:53.204757 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr" podUID="7fd6dca0-648c-4476-a1a5-8a7407bf598f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 03 13:35:54.205753 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:54.205721 5979 patch_prober.go:29] interesting pod/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Nov 03 13:35:54.206160 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:54.205777 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr" podUID="7fd6dca0-648c-4476-a1a5-8a7407bf598f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Nov 03 13:35:55.209565 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:55.209527 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr" Nov 03 13:35:55.210045 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:55.209837 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr" Nov 03 13:35:55.210545 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:55.210527 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr" Nov 03 13:35:59.135527 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:59.135484 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg"] Nov 03 13:35:59.136141 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:35:59.135766 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg" podUID="ec489f9b-9c50-45d9-aefb-1da511837bae" containerName="istio-proxy" containerID="cri-o://91997c5037e74bd7a546bbfe607c64651bf1160f9c7e914de4625a68f86045a6" gracePeriod=30 Nov 03 13:36:03.054266 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:03.054230 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18f090c4-z927b"] Nov 03 13:36:03.100769 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:03.100731 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18f090c4-z927b"] Nov 03 13:36:04.116735 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:04.116696 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76db9c1e-ad76-412e-b373-13cce12b2f0c" path="/var/lib/kubelet/pods/76db9c1e-ad76-412e-b373-13cce12b2f0c/volumes" Nov 03 13:36:04.700683 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:04.700654 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg" Nov 03 13:36:04.804381 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:04.804287 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec489f9b-9c50-45d9-aefb-1da511837bae-kube-api-access-zppmj\" (UniqueName: \"kubernetes.io/projected/ec489f9b-9c50-45d9-aefb-1da511837bae-kube-api-access-zppmj\") pod \"ec489f9b-9c50-45d9-aefb-1da511837bae\" (UID: \"ec489f9b-9c50-45d9-aefb-1da511837bae\") " Nov 03 13:36:04.804381 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:04.804358 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec489f9b-9c50-45d9-aefb-1da511837bae-istio-token\" (UniqueName: \"kubernetes.io/projected/ec489f9b-9c50-45d9-aefb-1da511837bae-istio-token\") pod \"ec489f9b-9c50-45d9-aefb-1da511837bae\" (UID: \"ec489f9b-9c50-45d9-aefb-1da511837bae\") " Nov 03 13:36:04.804655 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:04.804399 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec489f9b-9c50-45d9-aefb-1da511837bae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec489f9b-9c50-45d9-aefb-1da511837bae-istio-envoy\") pod \"ec489f9b-9c50-45d9-aefb-1da511837bae\" (UID: \"ec489f9b-9c50-45d9-aefb-1da511837bae\") " Nov 03 13:36:04.804655 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:04.804458 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec489f9b-9c50-45d9-aefb-1da511837bae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec489f9b-9c50-45d9-aefb-1da511837bae-istio-data\") pod \"ec489f9b-9c50-45d9-aefb-1da511837bae\" (UID: \"ec489f9b-9c50-45d9-aefb-1da511837bae\") " Nov 03 13:36:04.804655 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:04.804538 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec489f9b-9c50-45d9-aefb-1da511837bae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec489f9b-9c50-45d9-aefb-1da511837bae-istiod-ca-cert\") pod \"ec489f9b-9c50-45d9-aefb-1da511837bae\" (UID: \"ec489f9b-9c50-45d9-aefb-1da511837bae\") " Nov 03 13:36:04.804655 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:04.804558 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec489f9b-9c50-45d9-aefb-1da511837bae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec489f9b-9c50-45d9-aefb-1da511837bae-workload-socket\") pod \"ec489f9b-9c50-45d9-aefb-1da511837bae\" (UID: \"ec489f9b-9c50-45d9-aefb-1da511837bae\") " Nov 03 13:36:04.804655 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:04.804589 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec489f9b-9c50-45d9-aefb-1da511837bae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec489f9b-9c50-45d9-aefb-1da511837bae-workload-certs\") pod \"ec489f9b-9c50-45d9-aefb-1da511837bae\" (UID: \"ec489f9b-9c50-45d9-aefb-1da511837bae\") " Nov 03 13:36:04.804655 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:04.804651 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec489f9b-9c50-45d9-aefb-1da511837bae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec489f9b-9c50-45d9-aefb-1da511837bae-credential-socket\") pod \"ec489f9b-9c50-45d9-aefb-1da511837bae\" (UID: \"ec489f9b-9c50-45d9-aefb-1da511837bae\") " Nov 03 13:36:04.804905 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:04.804692 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ec489f9b-9c50-45d9-aefb-1da511837bae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec489f9b-9c50-45d9-aefb-1da511837bae-istio-podinfo\") pod \"ec489f9b-9c50-45d9-aefb-1da511837bae\" (UID: \"ec489f9b-9c50-45d9-aefb-1da511837bae\") " Nov 03 13:36:04.804905 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:04.804840 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec489f9b-9c50-45d9-aefb-1da511837bae-workload-socket" pod "ec489f9b-9c50-45d9-aefb-1da511837bae" (UID: "ec489f9b-9c50-45d9-aefb-1da511837bae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:36:04.805014 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:04.804900 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec489f9b-9c50-45d9-aefb-1da511837bae-istiod-ca-cert" pod "ec489f9b-9c50-45d9-aefb-1da511837bae" (UID: "ec489f9b-9c50-45d9-aefb-1da511837bae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:36:04.805014 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:04.804948 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec489f9b-9c50-45d9-aefb-1da511837bae-workload-certs" pod "ec489f9b-9c50-45d9-aefb-1da511837bae" (UID: "ec489f9b-9c50-45d9-aefb-1da511837bae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:36:04.805272 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:04.805182 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec489f9b-9c50-45d9-aefb-1da511837bae-credential-socket" pod "ec489f9b-9c50-45d9-aefb-1da511837bae" (UID: "ec489f9b-9c50-45d9-aefb-1da511837bae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:36:04.805272 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:04.805227 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec489f9b-9c50-45d9-aefb-1da511837bae-istio-data" pod "ec489f9b-9c50-45d9-aefb-1da511837bae" (UID: "ec489f9b-9c50-45d9-aefb-1da511837bae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:36:04.805493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:04.805407 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec489f9b-9c50-45d9-aefb-1da511837bae-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:04.805493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:04.805429 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec489f9b-9c50-45d9-aefb-1da511837bae-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:04.805493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:04.805444 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec489f9b-9c50-45d9-aefb-1da511837bae-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:04.805493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:04.805457 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec489f9b-9c50-45d9-aefb-1da511837bae-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:04.805493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:04.805490 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec489f9b-9c50-45d9-aefb-1da511837bae-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:04.806835 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:04.806810 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec489f9b-9c50-45d9-aefb-1da511837bae-istio-envoy" pod "ec489f9b-9c50-45d9-aefb-1da511837bae" (UID: "ec489f9b-9c50-45d9-aefb-1da511837bae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:36:04.806945 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:04.806864 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec489f9b-9c50-45d9-aefb-1da511837bae-kube-api-access-zppmj" pod "ec489f9b-9c50-45d9-aefb-1da511837bae" (UID: "ec489f9b-9c50-45d9-aefb-1da511837bae"). InnerVolumeSpecName "kube-api-access-zppmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:36:04.806945 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:04.806929 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec489f9b-9c50-45d9-aefb-1da511837bae-istio-token" pod "ec489f9b-9c50-45d9-aefb-1da511837bae" (UID: "ec489f9b-9c50-45d9-aefb-1da511837bae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:36:04.807066 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:04.807051 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec489f9b-9c50-45d9-aefb-1da511837bae-istio-podinfo" pod "ec489f9b-9c50-45d9-aefb-1da511837bae" (UID: "ec489f9b-9c50-45d9-aefb-1da511837bae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:36:04.907257 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:04.907218 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec489f9b-9c50-45d9-aefb-1da511837bae-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:04.907257 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:04.907250 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zppmj\" (UniqueName: \"kubernetes.io/projected/ec489f9b-9c50-45d9-aefb-1da511837bae-kube-api-access-zppmj\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:04.907257 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:04.907263 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec489f9b-9c50-45d9-aefb-1da511837bae-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:04.907630 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:04.907271 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec489f9b-9c50-45d9-aefb-1da511837bae-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:04.979196 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:04.979165 5979 generic.go:356] "Generic (PLEG): container finished" podID="ec489f9b-9c50-45d9-aefb-1da511837bae" containerID="91997c5037e74bd7a546bbfe607c64651bf1160f9c7e914de4625a68f86045a6" exitCode=0 Nov 03 13:36:04.979357 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:04.979222 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg" Nov 03 13:36:04.979357 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:04.979254 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg" event={"ID":"ec489f9b-9c50-45d9-aefb-1da511837bae","Type":"ContainerDied","Data":"91997c5037e74bd7a546bbfe607c64651bf1160f9c7e914de4625a68f86045a6"} Nov 03 13:36:04.979357 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:04.979349 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg" event={"ID":"ec489f9b-9c50-45d9-aefb-1da511837bae","Type":"ContainerDied","Data":"6b3875fbab3e145130f022452fc33059fc64f0b503b26768eea960cc3460732e"} Nov 03 13:36:04.979464 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:04.979378 5979 scope.go:122] "RemoveContainer" containerID="91997c5037e74bd7a546bbfe607c64651bf1160f9c7e914de4625a68f86045a6" Nov 03 13:36:04.990406 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:04.990385 5979 scope.go:122] "RemoveContainer" containerID="899494b34430f7ca67b7b29888b402f22dd9dcf680dbf73212bb456dd2c2b313" Nov 03 13:36:05.000460 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:05.000438 5979 scope.go:122] "RemoveContainer" containerID="91997c5037e74bd7a546bbfe607c64651bf1160f9c7e914de4625a68f86045a6" Nov 03 13:36:05.000757 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:36:05.000737 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91997c5037e74bd7a546bbfe607c64651bf1160f9c7e914de4625a68f86045a6\": container with ID starting with 91997c5037e74bd7a546bbfe607c64651bf1160f9c7e914de4625a68f86045a6 not found: ID does not exist" containerID="91997c5037e74bd7a546bbfe607c64651bf1160f9c7e914de4625a68f86045a6" Nov 03 13:36:05.000803 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:05.000765 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91997c5037e74bd7a546bbfe607c64651bf1160f9c7e914de4625a68f86045a6"} err="failed to get container status \"91997c5037e74bd7a546bbfe607c64651bf1160f9c7e914de4625a68f86045a6\": rpc error: code = NotFound desc = could not find container \"91997c5037e74bd7a546bbfe607c64651bf1160f9c7e914de4625a68f86045a6\": container with ID starting with 91997c5037e74bd7a546bbfe607c64651bf1160f9c7e914de4625a68f86045a6 not found: ID does not exist" Nov 03 13:36:05.000803 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:05.000787 5979 scope.go:122] "RemoveContainer" containerID="899494b34430f7ca67b7b29888b402f22dd9dcf680dbf73212bb456dd2c2b313" Nov 03 13:36:05.001029 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:36:05.001011 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"899494b34430f7ca67b7b29888b402f22dd9dcf680dbf73212bb456dd2c2b313\": container with ID starting with 899494b34430f7ca67b7b29888b402f22dd9dcf680dbf73212bb456dd2c2b313 not found: ID does not exist" containerID="899494b34430f7ca67b7b29888b402f22dd9dcf680dbf73212bb456dd2c2b313" Nov 03 13:36:05.001095 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:05.001038 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"899494b34430f7ca67b7b29888b402f22dd9dcf680dbf73212bb456dd2c2b313"} err="failed to get container status \"899494b34430f7ca67b7b29888b402f22dd9dcf680dbf73212bb456dd2c2b313\": rpc error: code = NotFound desc = could not find container \"899494b34430f7ca67b7b29888b402f22dd9dcf680dbf73212bb456dd2c2b313\": container with ID starting with 899494b34430f7ca67b7b29888b402f22dd9dcf680dbf73212bb456dd2c2b313 not found: ID does not exist" Nov 03 13:36:05.134902 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:05.134869 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg"] Nov 03 13:36:05.138501 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:05.138452 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-e7ca74-openshift-default-789cd67b5c-hcrqg"] Nov 03 13:36:06.116488 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:06.116449 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec489f9b-9c50-45d9-aefb-1da511837bae" path="/var/lib/kubelet/pods/ec489f9b-9c50-45d9-aefb-1da511837bae/volumes" Nov 03 13:36:07.234429 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:36:07.234391 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:36:09.863696 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:09.863615 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf"] Nov 03 13:36:09.864289 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:09.864274 5979 cpu_manager.go:414] "containerMap: removing container" podUID="ec489f9b-9c50-45d9-aefb-1da511837bae" containerName="istio-proxy" Nov 03 13:36:09.864350 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:09.864290 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec489f9b-9c50-45d9-aefb-1da511837bae" containerName="istio-proxy" Nov 03 13:36:09.864350 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:09.864309 5979 cpu_manager.go:414] "containerMap: removing container" podUID="ec489f9b-9c50-45d9-aefb-1da511837bae" containerName="istio-proxy" Nov 03 13:36:09.864350 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:09.864314 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec489f9b-9c50-45d9-aefb-1da511837bae" containerName="istio-proxy" Nov 03 13:36:09.864465 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:09.864454 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec489f9b-9c50-45d9-aefb-1da511837bae" containerName="istio-proxy" Nov 03 13:36:09.864554 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:09.864484 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec489f9b-9c50-45d9-aefb-1da511837bae" containerName="istio-proxy" Nov 03 13:36:09.869939 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:09.869915 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf" Nov 03 13:36:09.925285 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:09.925241 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-5474b7-openshift-default-dockercfg-m6mnc\"" Nov 03 13:36:10.001380 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:10.001346 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf"] Nov 03 13:36:10.096756 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:10.061320 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-istio-podinfo\") pod \"e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf\" (UID: \"efd2a382-b6ea-419f-b33a-9ebfdf0bd61b\") " pod="openshift-ingress/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf" Nov 03 13:36:10.097126 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:10.097093 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-workload-certs\") pod \"e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf\" (UID: \"efd2a382-b6ea-419f-b33a-9ebfdf0bd61b\") " pod="openshift-ingress/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf" Nov 03 13:36:10.097376 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:10.097360 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-istiod-ca-cert\") pod \"e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf\" (UID: \"efd2a382-b6ea-419f-b33a-9ebfdf0bd61b\") " pod="openshift-ingress/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf" Nov 03 13:36:10.097539 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:10.097522 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-istio-envoy\") pod \"e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf\" (UID: \"efd2a382-b6ea-419f-b33a-9ebfdf0bd61b\") " pod="openshift-ingress/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf" Nov 03 13:36:10.097807 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:10.097775 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-istio-data\") pod \"e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf\" (UID: \"efd2a382-b6ea-419f-b33a-9ebfdf0bd61b\") " pod="openshift-ingress/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf" Nov 03 13:36:10.097989 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:10.097961 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-credential-socket\") pod \"e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf\" (UID: \"efd2a382-b6ea-419f-b33a-9ebfdf0bd61b\") " pod="openshift-ingress/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf" Nov 03 13:36:10.098354 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:10.098337 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-workload-socket\") pod \"e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf\" (UID: \"efd2a382-b6ea-419f-b33a-9ebfdf0bd61b\") " pod="openshift-ingress/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf" Nov 03 13:36:10.101008 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:10.100982 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-istio-token\") pod \"e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf\" (UID: \"efd2a382-b6ea-419f-b33a-9ebfdf0bd61b\") " pod="openshift-ingress/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf" Nov 03 13:36:10.101207 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:10.101189 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gcmp\" (UniqueName: \"kubernetes.io/projected/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-kube-api-access-8gcmp\") pod \"e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf\" (UID: \"efd2a382-b6ea-419f-b33a-9ebfdf0bd61b\") " pod="openshift-ingress/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf" Nov 03 13:36:10.203130 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:10.203088 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-istio-podinfo\") pod \"e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf\" (UID: \"efd2a382-b6ea-419f-b33a-9ebfdf0bd61b\") " pod="openshift-ingress/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf" Nov 03 13:36:10.203315 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:10.203143 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-workload-certs\") pod \"e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf\" (UID: \"efd2a382-b6ea-419f-b33a-9ebfdf0bd61b\") " pod="openshift-ingress/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf" Nov 03 13:36:10.203369 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:10.203330 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-istiod-ca-cert\") pod \"e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf\" (UID: \"efd2a382-b6ea-419f-b33a-9ebfdf0bd61b\") " pod="openshift-ingress/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf" Nov 03 13:36:10.203422 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:10.203374 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-istio-envoy\") pod \"e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf\" (UID: \"efd2a382-b6ea-419f-b33a-9ebfdf0bd61b\") " pod="openshift-ingress/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf" Nov 03 13:36:10.203551 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:10.203531 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-istio-data\") pod \"e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf\" (UID: \"efd2a382-b6ea-419f-b33a-9ebfdf0bd61b\") " pod="openshift-ingress/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf" Nov 03 13:36:10.203655 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:10.203582 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-credential-socket\") pod \"e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf\" (UID: \"efd2a382-b6ea-419f-b33a-9ebfdf0bd61b\") " pod="openshift-ingress/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf" Nov 03 13:36:10.203655 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:10.203590 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-workload-certs\") pod \"e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf\" (UID: \"efd2a382-b6ea-419f-b33a-9ebfdf0bd61b\") " pod="openshift-ingress/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf" Nov 03 13:36:10.203655 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:10.203617 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-workload-socket\") pod \"e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf\" (UID: \"efd2a382-b6ea-419f-b33a-9ebfdf0bd61b\") " pod="openshift-ingress/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf" Nov 03 13:36:10.203839 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:10.203693 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-istio-token\") pod \"e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf\" (UID: \"efd2a382-b6ea-419f-b33a-9ebfdf0bd61b\") " pod="openshift-ingress/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf" Nov 03 13:36:10.203839 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:10.203730 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gcmp\" (UniqueName: \"kubernetes.io/projected/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-kube-api-access-8gcmp\") pod \"e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf\" (UID: \"efd2a382-b6ea-419f-b33a-9ebfdf0bd61b\") " pod="openshift-ingress/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf" Nov 03 13:36:10.203962 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:10.203839 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-istio-data\") pod \"e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf\" (UID: \"efd2a382-b6ea-419f-b33a-9ebfdf0bd61b\") " pod="openshift-ingress/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf" Nov 03 13:36:10.204034 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:10.204008 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-istiod-ca-cert\") pod \"e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf\" (UID: \"efd2a382-b6ea-419f-b33a-9ebfdf0bd61b\") " pod="openshift-ingress/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf" Nov 03 13:36:10.204112 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:10.204080 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-credential-socket\") pod \"e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf\" (UID: \"efd2a382-b6ea-419f-b33a-9ebfdf0bd61b\") " pod="openshift-ingress/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf" Nov 03 13:36:10.204178 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:10.204146 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-workload-socket\") pod \"e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf\" (UID: \"efd2a382-b6ea-419f-b33a-9ebfdf0bd61b\") " pod="openshift-ingress/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf" Nov 03 13:36:10.238493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:10.234100 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-istio-podinfo\") pod \"e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf\" (UID: \"efd2a382-b6ea-419f-b33a-9ebfdf0bd61b\") " pod="openshift-ingress/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf" Nov 03 13:36:10.251050 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:10.247699 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-istio-envoy\") pod \"e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf\" (UID: \"efd2a382-b6ea-419f-b33a-9ebfdf0bd61b\") " pod="openshift-ingress/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf" Nov 03 13:36:10.268582 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:10.268449 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gcmp\" (UniqueName: \"kubernetes.io/projected/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-kube-api-access-8gcmp\") pod \"e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf\" (UID: \"efd2a382-b6ea-419f-b33a-9ebfdf0bd61b\") " pod="openshift-ingress/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf" Nov 03 13:36:10.269860 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:10.269807 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-istio-token\") pod \"e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf\" (UID: \"efd2a382-b6ea-419f-b33a-9ebfdf0bd61b\") " pod="openshift-ingress/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf" Nov 03 13:36:10.484733 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:10.484646 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf" Nov 03 13:36:10.998549 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:10.997941 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-02f3a0-66bc949bbc-dk2q5"] Nov 03 13:36:11.003285 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:11.003259 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02f3a0-66bc949bbc-dk2q5" Nov 03 13:36:11.017112 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:11.017073 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf"] Nov 03 13:36:11.022020 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:36:11.021993 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefd2a382_b6ea_419f_b33a_9ebfdf0bd61b.slice/crio-94b0e2cc756552a6364b0ab46f6e154f4ff635fffd5ee13bb81d9aa71bf66a0d WatchSource:0}: Error finding container 94b0e2cc756552a6364b0ab46f6e154f4ff635fffd5ee13bb81d9aa71bf66a0d: Status 404 returned error can't find the container with id 94b0e2cc756552a6364b0ab46f6e154f4ff635fffd5ee13bb81d9aa71bf66a0d Nov 03 13:36:11.025255 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:11.025214 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:36:11.025340 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:11.025304 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:36:11.025383 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:11.025347 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:36:11.073724 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:11.073657 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-02f3a0\"" Nov 03 13:36:11.083489 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:11.082277 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-02f3a0-66bc949bbc-dk2q5"] Nov 03 13:36:11.089009 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:11.087861 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-02f3a0-dockercfg-fwd2p\"" Nov 03 13:36:11.199517 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:11.192101 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-02f3a0-57f7b565b5-xg9v9"] Nov 03 13:36:11.199715 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:11.199512 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh6wz\" (UniqueName: \"kubernetes.io/projected/23c0c7c1-7712-49e1-9dfe-f7eb396bdaa0-kube-api-access-xh6wz\") pod \"payload-pre-processing-e2e-ipp-a-02f3a0-66bc949bbc-dk2q5\" (UID: \"23c0c7c1-7712-49e1-9dfe-f7eb396bdaa0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02f3a0-66bc949bbc-dk2q5" Nov 03 13:36:11.199715 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:11.199593 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23c0c7c1-7712-49e1-9dfe-f7eb396bdaa0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-02f3a0-66bc949bbc-dk2q5\" (UID: \"23c0c7c1-7712-49e1-9dfe-f7eb396bdaa0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02f3a0-66bc949bbc-dk2q5" Nov 03 13:36:11.235932 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:11.235890 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-02f3a0-57f7b565b5-xg9v9"] Nov 03 13:36:11.236099 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:11.236078 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-02f3a0-57f7b565b5-xg9v9" Nov 03 13:36:11.238849 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:11.238820 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf" event={"ID":"efd2a382-b6ea-419f-b33a-9ebfdf0bd61b","Type":"ContainerStarted","Data":"94b0e2cc756552a6364b0ab46f6e154f4ff635fffd5ee13bb81d9aa71bf66a0d"} Nov 03 13:36:11.323491 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:11.322926 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh6wz\" (UniqueName: \"kubernetes.io/projected/23c0c7c1-7712-49e1-9dfe-f7eb396bdaa0-kube-api-access-xh6wz\") pod \"payload-pre-processing-e2e-ipp-a-02f3a0-66bc949bbc-dk2q5\" (UID: \"23c0c7c1-7712-49e1-9dfe-f7eb396bdaa0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02f3a0-66bc949bbc-dk2q5" Nov 03 13:36:11.323491 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:11.323009 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23c0c7c1-7712-49e1-9dfe-f7eb396bdaa0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-02f3a0-66bc949bbc-dk2q5\" (UID: \"23c0c7c1-7712-49e1-9dfe-f7eb396bdaa0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02f3a0-66bc949bbc-dk2q5" Nov 03 13:36:11.335512 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:11.331531 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23c0c7c1-7712-49e1-9dfe-f7eb396bdaa0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-02f3a0-66bc949bbc-dk2q5\" (UID: \"23c0c7c1-7712-49e1-9dfe-f7eb396bdaa0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02f3a0-66bc949bbc-dk2q5" Nov 03 13:36:11.387004 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:11.386963 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh6wz\" (UniqueName: \"kubernetes.io/projected/23c0c7c1-7712-49e1-9dfe-f7eb396bdaa0-kube-api-access-xh6wz\") pod \"payload-pre-processing-e2e-ipp-a-02f3a0-66bc949bbc-dk2q5\" (UID: \"23c0c7c1-7712-49e1-9dfe-f7eb396bdaa0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02f3a0-66bc949bbc-dk2q5" Nov 03 13:36:11.424835 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:11.424797 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daa7244d-0a66-4e91-acb5-3a604fb83aa7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-02f3a0-57f7b565b5-xg9v9\" (UID: \"daa7244d-0a66-4e91-acb5-3a604fb83aa7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-02f3a0-57f7b565b5-xg9v9" Nov 03 13:36:11.425007 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:11.424940 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqkfb\" (UniqueName: \"kubernetes.io/projected/daa7244d-0a66-4e91-acb5-3a604fb83aa7-kube-api-access-nqkfb\") pod \"payload-processing-e2e-ipp-a-02f3a0-57f7b565b5-xg9v9\" (UID: \"daa7244d-0a66-4e91-acb5-3a604fb83aa7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-02f3a0-57f7b565b5-xg9v9" Nov 03 13:36:11.526942 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:11.526851 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daa7244d-0a66-4e91-acb5-3a604fb83aa7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-02f3a0-57f7b565b5-xg9v9\" (UID: \"daa7244d-0a66-4e91-acb5-3a604fb83aa7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-02f3a0-57f7b565b5-xg9v9" Nov 03 13:36:11.527112 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:11.526968 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqkfb\" (UniqueName: \"kubernetes.io/projected/daa7244d-0a66-4e91-acb5-3a604fb83aa7-kube-api-access-nqkfb\") pod \"payload-processing-e2e-ipp-a-02f3a0-57f7b565b5-xg9v9\" (UID: \"daa7244d-0a66-4e91-acb5-3a604fb83aa7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-02f3a0-57f7b565b5-xg9v9" Nov 03 13:36:11.527719 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:11.527700 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daa7244d-0a66-4e91-acb5-3a604fb83aa7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-02f3a0-57f7b565b5-xg9v9\" (UID: \"daa7244d-0a66-4e91-acb5-3a604fb83aa7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-02f3a0-57f7b565b5-xg9v9" Nov 03 13:36:11.580758 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:11.580722 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqkfb\" (UniqueName: \"kubernetes.io/projected/daa7244d-0a66-4e91-acb5-3a604fb83aa7-kube-api-access-nqkfb\") pod \"payload-processing-e2e-ipp-a-02f3a0-57f7b565b5-xg9v9\" (UID: \"daa7244d-0a66-4e91-acb5-3a604fb83aa7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-02f3a0-57f7b565b5-xg9v9" Nov 03 13:36:11.620060 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:11.620026 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02f3a0-66bc949bbc-dk2q5" Nov 03 13:36:11.810960 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:11.810877 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f6f9ed-554b464fd5-kzpbv"] Nov 03 13:36:11.857029 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:11.856997 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-02f3a0-57f7b565b5-xg9v9" Nov 03 13:36:11.918344 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:11.917378 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f6f9ed-667f6c778c-cq5t6"] Nov 03 13:36:12.020229 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:12.020194 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-02f3a0-66bc949bbc-dk2q5"] Nov 03 13:36:12.034694 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:36:12.034641 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23c0c7c1_7712_49e1_9dfe_f7eb396bdaa0.slice/crio-dfe95619d9c0282be12a320978c4c036618aa7c60df6f760c0498d38cb56ed29 WatchSource:0}: Error finding container dfe95619d9c0282be12a320978c4c036618aa7c60df6f760c0498d38cb56ed29: Status 404 returned error can't find the container with id dfe95619d9c0282be12a320978c4c036618aa7c60df6f760c0498d38cb56ed29 Nov 03 13:36:12.330591 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:12.329877 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-02f3a0-57f7b565b5-xg9v9"] Nov 03 13:36:12.337181 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:12.337149 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf" event={"ID":"efd2a382-b6ea-419f-b33a-9ebfdf0bd61b","Type":"ContainerStarted","Data":"d714840db823ea2dea8bdda3ef53b5e66a20c5bc6204dc3ca8b5e2bb2a4b2125"} Nov 03 13:36:12.353155 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:12.352003 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02f3a0-66bc949bbc-dk2q5" event={"ID":"23c0c7c1-7712-49e1-9dfe-f7eb396bdaa0","Type":"ContainerStarted","Data":"cd31b70e93fa1dec703c23eec631868fb238f87aa71f44c1e979b470a8adadb4"} Nov 03 13:36:12.353463 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:12.353434 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02f3a0-66bc949bbc-dk2q5" event={"ID":"23c0c7c1-7712-49e1-9dfe-f7eb396bdaa0","Type":"ContainerStarted","Data":"dfe95619d9c0282be12a320978c4c036618aa7c60df6f760c0498d38cb56ed29"} Nov 03 13:36:12.354546 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:12.354526 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02f3a0-66bc949bbc-dk2q5" Nov 03 13:36:12.469037 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:12.468972 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf" podStartSLOduration=3.468951297 podStartE2EDuration="3.468951297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:36:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:36:12.465729083 +0000 UTC m=+1854.476417337" watchObservedRunningTime="2026-11-03 13:36:12.468951297 +0000 UTC m=+1854.479639539" Nov 03 13:36:12.485539 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:12.485508 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf" Nov 03 13:36:12.491053 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:12.491028 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf" Nov 03 13:36:12.613572 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:12.595751 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02f3a0-66bc949bbc-dk2q5" podStartSLOduration=2.595730589 podStartE2EDuration="2.595730589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:36:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:36:12.593772793 +0000 UTC m=+1854.604461037" watchObservedRunningTime="2026-11-03 13:36:12.595730589 +0000 UTC m=+1854.606418831" Nov 03 13:36:13.046677 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:13.046638 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51b4988d-24mw5"] Nov 03 13:36:13.049718 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:13.049690 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51b4988d-24mw5"] Nov 03 13:36:13.425275 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:13.425233 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-02f3a0-57f7b565b5-xg9v9" event={"ID":"daa7244d-0a66-4e91-acb5-3a604fb83aa7","Type":"ContainerStarted","Data":"068fee9d9edc4d9dfede971afdbf01c32c27378b48f5973e052ea0caf1dda203"} Nov 03 13:36:13.425503 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:13.425302 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-02f3a0-57f7b565b5-xg9v9" event={"ID":"daa7244d-0a66-4e91-acb5-3a604fb83aa7","Type":"ContainerStarted","Data":"d911f4af23de083317ab00f784237670c747e5a81e7a6e5012228c886f78b579"} Nov 03 13:36:13.426539 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:13.426460 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf" Nov 03 13:36:13.426539 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:13.426527 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-02f3a0-57f7b565b5-xg9v9" Nov 03 13:36:13.437767 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:13.437692 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf" Nov 03 13:36:13.542494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:13.537433 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-02f3a0-57f7b565b5-xg9v9" podStartSLOduration=2.53741575 podStartE2EDuration="2.53741575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:36:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:36:13.534719681 +0000 UTC m=+1855.545407926" watchObservedRunningTime="2026-11-03 13:36:13.53741575 +0000 UTC m=+1855.548103993" Nov 03 13:36:14.116880 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:14.116849 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f24a588-62a8-43fa-aa2a-40c6f1e55097" path="/var/lib/kubelet/pods/3f24a588-62a8-43fa-aa2a-40c6f1e55097/volumes" Nov 03 13:36:14.424187 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:14.424157 5979 generic.go:356] "Generic (PLEG): container finished" podID="ddd1f97b-814c-43b5-a81b-201bd1ef2c9b" containerID="01ff368eaf37eeb14f397fd8731124fe04ee652a6271303d9a55e7228f7b00fa" exitCode=0 Nov 03 13:36:14.424388 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:14.424237 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l" event={"ID":"ddd1f97b-814c-43b5-a81b-201bd1ef2c9b","Type":"ContainerDied","Data":"01ff368eaf37eeb14f397fd8731124fe04ee652a6271303d9a55e7228f7b00fa"} Nov 03 13:36:14.424711 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:14.424690 5979 scope.go:122] "RemoveContainer" containerID="01ff368eaf37eeb14f397fd8731124fe04ee652a6271303d9a55e7228f7b00fa" Nov 03 13:36:14.426110 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:14.426084 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:36:14.426168 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:14.426148 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:36:14.426204 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:14.426180 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:36:15.438606 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:15.438566 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l" event={"ID":"ddd1f97b-814c-43b5-a81b-201bd1ef2c9b","Type":"ContainerStarted","Data":"3948467f6687f08d0693e3c03d844c4a4db3c45208c7a9243eac83fc3b2d6d1e"} Nov 03 13:36:15.772569 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:15.771852 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l" Nov 03 13:36:15.773515 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:15.773466 5979 patch_prober.go:29] interesting pod/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Nov 03 13:36:15.773672 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:15.773543 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l" podUID="ddd1f97b-814c-43b5-a81b-201bd1ef2c9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Nov 03 13:36:16.768598 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:16.768561 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l" Nov 03 13:36:16.771758 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:16.771724 5979 patch_prober.go:29] interesting pod/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Nov 03 13:36:16.771879 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:16.771790 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l" podUID="ddd1f97b-814c-43b5-a81b-201bd1ef2c9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Nov 03 13:36:16.811714 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:16.811677 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6f9ed-554b464fd5-kzpbv" podUID="f57d7f68-a80f-4fdf-84b8-a76509ff413d" containerName="payload-processing" containerID="cri-o://0328831c02bbf4d67aa0673d4f16ab9669b528009721c1698fd7ae56e44ac377" gracePeriod=25 Nov 03 13:36:16.918685 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:16.918604 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6f9ed-667f6c778c-cq5t6" podUID="fbfb58ac-c12a-4312-a8f3-96c5bfeac157" containerName="payload-pre-processing" containerID="cri-o://def8cc76a0aea8c0d663d5128cce35a9d29bda964e1393bf09470c1b4900a54b" gracePeriod=25 Nov 03 13:36:17.126939 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:17.126906 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6f9ed-554b464fd5-kzpbv" Nov 03 13:36:17.200636 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:17.199733 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f57d7f68-a80f-4fdf-84b8-a76509ff413d-kube-api-access-tmtct\" (UniqueName: \"kubernetes.io/projected/f57d7f68-a80f-4fdf-84b8-a76509ff413d-kube-api-access-tmtct\") pod \"f57d7f68-a80f-4fdf-84b8-a76509ff413d\" (UID: \"f57d7f68-a80f-4fdf-84b8-a76509ff413d\") " Nov 03 13:36:17.200636 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:17.199903 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f57d7f68-a80f-4fdf-84b8-a76509ff413d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f57d7f68-a80f-4fdf-84b8-a76509ff413d-plugins-config-volume\") pod \"f57d7f68-a80f-4fdf-84b8-a76509ff413d\" (UID: \"f57d7f68-a80f-4fdf-84b8-a76509ff413d\") " Nov 03 13:36:17.200975 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:17.200940 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f57d7f68-a80f-4fdf-84b8-a76509ff413d-plugins-config-volume" pod "f57d7f68-a80f-4fdf-84b8-a76509ff413d" (UID: "f57d7f68-a80f-4fdf-84b8-a76509ff413d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:36:17.206097 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:17.206066 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f57d7f68-a80f-4fdf-84b8-a76509ff413d-kube-api-access-tmtct" pod "f57d7f68-a80f-4fdf-84b8-a76509ff413d" (UID: "f57d7f68-a80f-4fdf-84b8-a76509ff413d"). InnerVolumeSpecName "kube-api-access-tmtct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:36:17.323550 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:17.301944 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6f9ed-667f6c778c-cq5t6" Nov 03 13:36:17.323550 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:17.303018 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmtct\" (UniqueName: \"kubernetes.io/projected/f57d7f68-a80f-4fdf-84b8-a76509ff413d-kube-api-access-tmtct\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:17.323550 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:17.303052 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f57d7f68-a80f-4fdf-84b8-a76509ff413d-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:17.403676 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:17.403645 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbfb58ac-c12a-4312-a8f3-96c5bfeac157-kube-api-access-whrl4\" (UniqueName: \"kubernetes.io/projected/fbfb58ac-c12a-4312-a8f3-96c5bfeac157-kube-api-access-whrl4\") pod \"fbfb58ac-c12a-4312-a8f3-96c5bfeac157\" (UID: \"fbfb58ac-c12a-4312-a8f3-96c5bfeac157\") " Nov 03 13:36:17.403868 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:17.403765 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbfb58ac-c12a-4312-a8f3-96c5bfeac157-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbfb58ac-c12a-4312-a8f3-96c5bfeac157-plugins-config-volume\") pod \"fbfb58ac-c12a-4312-a8f3-96c5bfeac157\" (UID: \"fbfb58ac-c12a-4312-a8f3-96c5bfeac157\") " Nov 03 13:36:17.404164 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:17.404139 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbfb58ac-c12a-4312-a8f3-96c5bfeac157-plugins-config-volume" pod "fbfb58ac-c12a-4312-a8f3-96c5bfeac157" (UID: "fbfb58ac-c12a-4312-a8f3-96c5bfeac157"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:36:17.405700 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:17.405681 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbfb58ac-c12a-4312-a8f3-96c5bfeac157-kube-api-access-whrl4" pod "fbfb58ac-c12a-4312-a8f3-96c5bfeac157" (UID: "fbfb58ac-c12a-4312-a8f3-96c5bfeac157"). InnerVolumeSpecName "kube-api-access-whrl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:36:17.481541 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:17.481466 5979 generic.go:356] "Generic (PLEG): container finished" podID="fbfb58ac-c12a-4312-a8f3-96c5bfeac157" containerID="def8cc76a0aea8c0d663d5128cce35a9d29bda964e1393bf09470c1b4900a54b" exitCode=0 Nov 03 13:36:17.481752 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:17.481716 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6f9ed-667f6c778c-cq5t6" Nov 03 13:36:17.482226 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:17.482198 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6f9ed-667f6c778c-cq5t6" event={"ID":"fbfb58ac-c12a-4312-a8f3-96c5bfeac157","Type":"ContainerDied","Data":"def8cc76a0aea8c0d663d5128cce35a9d29bda964e1393bf09470c1b4900a54b"} Nov 03 13:36:17.482339 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:17.482276 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6f9ed-667f6c778c-cq5t6" event={"ID":"fbfb58ac-c12a-4312-a8f3-96c5bfeac157","Type":"ContainerDied","Data":"327e4c90b9863d1ed6eac19d09e38aa1c31813a220edcad72f0ae8cc8d930013"} Nov 03 13:36:17.482339 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:17.482312 5979 scope.go:122] "RemoveContainer" containerID="def8cc76a0aea8c0d663d5128cce35a9d29bda964e1393bf09470c1b4900a54b" Nov 03 13:36:17.485798 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:17.485752 5979 generic.go:356] "Generic (PLEG): container finished" podID="f57d7f68-a80f-4fdf-84b8-a76509ff413d" containerID="0328831c02bbf4d67aa0673d4f16ab9669b528009721c1698fd7ae56e44ac377" exitCode=0 Nov 03 13:36:17.486596 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:17.486571 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6f9ed-554b464fd5-kzpbv" Nov 03 13:36:17.486836 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:17.486808 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6f9ed-554b464fd5-kzpbv" event={"ID":"f57d7f68-a80f-4fdf-84b8-a76509ff413d","Type":"ContainerDied","Data":"0328831c02bbf4d67aa0673d4f16ab9669b528009721c1698fd7ae56e44ac377"} Nov 03 13:36:17.486934 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:17.486884 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6f9ed-554b464fd5-kzpbv" event={"ID":"f57d7f68-a80f-4fdf-84b8-a76509ff413d","Type":"ContainerDied","Data":"f808b5427632bc7e267c76294b15649334c6581c51fc1532941ba4ff105697e3"} Nov 03 13:36:17.505686 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:17.505270 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whrl4\" (UniqueName: \"kubernetes.io/projected/fbfb58ac-c12a-4312-a8f3-96c5bfeac157-kube-api-access-whrl4\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:17.505686 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:17.505298 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbfb58ac-c12a-4312-a8f3-96c5bfeac157-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:17.506798 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:17.506775 5979 scope.go:122] "RemoveContainer" containerID="def8cc76a0aea8c0d663d5128cce35a9d29bda964e1393bf09470c1b4900a54b" Nov 03 13:36:17.507228 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:36:17.507205 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"def8cc76a0aea8c0d663d5128cce35a9d29bda964e1393bf09470c1b4900a54b\": container with ID starting with def8cc76a0aea8c0d663d5128cce35a9d29bda964e1393bf09470c1b4900a54b not found: ID does not exist" containerID="def8cc76a0aea8c0d663d5128cce35a9d29bda964e1393bf09470c1b4900a54b" Nov 03 13:36:17.507303 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:17.507238 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"def8cc76a0aea8c0d663d5128cce35a9d29bda964e1393bf09470c1b4900a54b"} err="failed to get container status \"def8cc76a0aea8c0d663d5128cce35a9d29bda964e1393bf09470c1b4900a54b\": rpc error: code = NotFound desc = could not find container \"def8cc76a0aea8c0d663d5128cce35a9d29bda964e1393bf09470c1b4900a54b\": container with ID starting with def8cc76a0aea8c0d663d5128cce35a9d29bda964e1393bf09470c1b4900a54b not found: ID does not exist" Nov 03 13:36:17.507303 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:17.507256 5979 scope.go:122] "RemoveContainer" containerID="0328831c02bbf4d67aa0673d4f16ab9669b528009721c1698fd7ae56e44ac377" Nov 03 13:36:17.517954 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:17.517933 5979 scope.go:122] "RemoveContainer" containerID="0328831c02bbf4d67aa0673d4f16ab9669b528009721c1698fd7ae56e44ac377" Nov 03 13:36:17.518237 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:36:17.518208 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0328831c02bbf4d67aa0673d4f16ab9669b528009721c1698fd7ae56e44ac377\": container with ID starting with 0328831c02bbf4d67aa0673d4f16ab9669b528009721c1698fd7ae56e44ac377 not found: ID does not exist" containerID="0328831c02bbf4d67aa0673d4f16ab9669b528009721c1698fd7ae56e44ac377" Nov 03 13:36:17.518304 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:17.518242 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0328831c02bbf4d67aa0673d4f16ab9669b528009721c1698fd7ae56e44ac377"} err="failed to get container status \"0328831c02bbf4d67aa0673d4f16ab9669b528009721c1698fd7ae56e44ac377\": rpc error: code = NotFound desc = could not find container \"0328831c02bbf4d67aa0673d4f16ab9669b528009721c1698fd7ae56e44ac377\": container with ID starting with 0328831c02bbf4d67aa0673d4f16ab9669b528009721c1698fd7ae56e44ac377 not found: ID does not exist" Nov 03 13:36:17.638030 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:17.637997 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f6f9ed-554b464fd5-kzpbv"] Nov 03 13:36:17.642231 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:17.642205 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f6f9ed-554b464fd5-kzpbv"] Nov 03 13:36:17.743024 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:17.742987 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f6f9ed-667f6c778c-cq5t6"] Nov 03 13:36:17.767495 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:17.766548 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f6f9ed-667f6c778c-cq5t6"] Nov 03 13:36:17.774496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:17.773819 5979 patch_prober.go:29] interesting pod/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Nov 03 13:36:17.774496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:17.773877 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l" podUID="ddd1f97b-814c-43b5-a81b-201bd1ef2c9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Nov 03 13:36:18.116194 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:18.116113 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f57d7f68-a80f-4fdf-84b8-a76509ff413d" path="/var/lib/kubelet/pods/f57d7f68-a80f-4fdf-84b8-a76509ff413d/volumes" Nov 03 13:36:18.116543 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:18.116525 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbfb58ac-c12a-4312-a8f3-96c5bfeac157" path="/var/lib/kubelet/pods/fbfb58ac-c12a-4312-a8f3-96c5bfeac157/volumes" Nov 03 13:36:18.772806 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:18.772765 5979 patch_prober.go:29] interesting pod/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Nov 03 13:36:18.773006 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:18.772835 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l" podUID="ddd1f97b-814c-43b5-a81b-201bd1ef2c9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Nov 03 13:36:19.408321 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:19.408278 5979 scope.go:122] "RemoveContainer" containerID="acf9669ed1f269729e033615a7ab21135c6edf455ab6cabf9044b1604440175f" Nov 03 13:36:19.772061 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:19.771975 5979 patch_prober.go:29] interesting pod/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Nov 03 13:36:19.772061 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:19.772037 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l" podUID="ddd1f97b-814c-43b5-a81b-201bd1ef2c9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Nov 03 13:36:20.797787 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:20.797756 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l" Nov 03 13:36:20.801523 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:20.801464 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l" Nov 03 13:36:21.010018 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:21.009978 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5474b7-ccbcc59f-hlkd5"] Nov 03 13:36:21.010751 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:21.010728 5979 cpu_manager.go:414] "containerMap: removing container" podUID="f57d7f68-a80f-4fdf-84b8-a76509ff413d" containerName="payload-processing" Nov 03 13:36:21.010751 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:21.010750 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f57d7f68-a80f-4fdf-84b8-a76509ff413d" containerName="payload-processing" Nov 03 13:36:21.010958 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:21.010780 5979 cpu_manager.go:414] "containerMap: removing container" podUID="fbfb58ac-c12a-4312-a8f3-96c5bfeac157" containerName="payload-pre-processing" Nov 03 13:36:21.010958 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:21.010790 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbfb58ac-c12a-4312-a8f3-96c5bfeac157" containerName="payload-pre-processing" Nov 03 13:36:21.010958 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:21.010933 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="f57d7f68-a80f-4fdf-84b8-a76509ff413d" containerName="payload-processing" Nov 03 13:36:21.010958 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:21.010949 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbfb58ac-c12a-4312-a8f3-96c5bfeac157" containerName="payload-pre-processing" Nov 03 13:36:21.016638 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:21.016615 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5474b7-ccbcc59f-hlkd5" Nov 03 13:36:21.069565 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:21.069464 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-5474b7\"" Nov 03 13:36:21.070708 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:21.070425 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-5474b7-dockercfg-86vg7\"" Nov 03 13:36:21.076931 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:21.076552 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5474b7-ccbcc59f-hlkd5"] Nov 03 13:36:21.112494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:21.103237 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f27c783a-fa71-4db1-a3bb-4ed989e72d08-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5474b7-ccbcc59f-hlkd5\" (UID: \"f27c783a-fa71-4db1-a3bb-4ed989e72d08\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5474b7-ccbcc59f-hlkd5" Nov 03 13:36:21.112494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:21.103348 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s56nj\" (UniqueName: \"kubernetes.io/projected/f27c783a-fa71-4db1-a3bb-4ed989e72d08-kube-api-access-s56nj\") pod \"payload-pre-processing-e2e-ipp-b-5474b7-ccbcc59f-hlkd5\" (UID: \"f27c783a-fa71-4db1-a3bb-4ed989e72d08\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5474b7-ccbcc59f-hlkd5" Nov 03 13:36:21.133496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:21.131921 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5474b7-7496845f45-6ncjj"] Nov 03 13:36:21.214501 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:21.210222 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f27c783a-fa71-4db1-a3bb-4ed989e72d08-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5474b7-ccbcc59f-hlkd5\" (UID: \"f27c783a-fa71-4db1-a3bb-4ed989e72d08\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5474b7-ccbcc59f-hlkd5" Nov 03 13:36:21.214501 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:21.210314 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s56nj\" (UniqueName: \"kubernetes.io/projected/f27c783a-fa71-4db1-a3bb-4ed989e72d08-kube-api-access-s56nj\") pod \"payload-pre-processing-e2e-ipp-b-5474b7-ccbcc59f-hlkd5\" (UID: \"f27c783a-fa71-4db1-a3bb-4ed989e72d08\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5474b7-ccbcc59f-hlkd5" Nov 03 13:36:21.214501 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:21.211261 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f27c783a-fa71-4db1-a3bb-4ed989e72d08-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5474b7-ccbcc59f-hlkd5\" (UID: \"f27c783a-fa71-4db1-a3bb-4ed989e72d08\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5474b7-ccbcc59f-hlkd5" Nov 03 13:36:21.224964 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:21.224424 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5474b7-7496845f45-6ncjj"] Nov 03 13:36:21.224964 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:21.224575 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5474b7-7496845f45-6ncjj" Nov 03 13:36:21.264903 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:21.264861 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s56nj\" (UniqueName: \"kubernetes.io/projected/f27c783a-fa71-4db1-a3bb-4ed989e72d08-kube-api-access-s56nj\") pod \"payload-pre-processing-e2e-ipp-b-5474b7-ccbcc59f-hlkd5\" (UID: \"f27c783a-fa71-4db1-a3bb-4ed989e72d08\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5474b7-ccbcc59f-hlkd5" Nov 03 13:36:21.330439 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:21.330354 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5474b7-ccbcc59f-hlkd5" Nov 03 13:36:21.412809 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:21.412618 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e22aa23e-9a2a-4499-bba1-4da8208e7386-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5474b7-7496845f45-6ncjj\" (UID: \"e22aa23e-9a2a-4499-bba1-4da8208e7386\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5474b7-7496845f45-6ncjj" Nov 03 13:36:21.412809 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:21.412729 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpwq2\" (UniqueName: \"kubernetes.io/projected/e22aa23e-9a2a-4499-bba1-4da8208e7386-kube-api-access-kpwq2\") pod \"payload-processing-e2e-ipp-b-5474b7-7496845f45-6ncjj\" (UID: \"e22aa23e-9a2a-4499-bba1-4da8208e7386\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5474b7-7496845f45-6ncjj" Nov 03 13:36:21.514224 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:21.514186 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpwq2\" (UniqueName: \"kubernetes.io/projected/e22aa23e-9a2a-4499-bba1-4da8208e7386-kube-api-access-kpwq2\") pod \"payload-processing-e2e-ipp-b-5474b7-7496845f45-6ncjj\" (UID: \"e22aa23e-9a2a-4499-bba1-4da8208e7386\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5474b7-7496845f45-6ncjj" Nov 03 13:36:21.514570 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:21.514544 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e22aa23e-9a2a-4499-bba1-4da8208e7386-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5474b7-7496845f45-6ncjj\" (UID: \"e22aa23e-9a2a-4499-bba1-4da8208e7386\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5474b7-7496845f45-6ncjj" Nov 03 13:36:21.515195 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:21.515172 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e22aa23e-9a2a-4499-bba1-4da8208e7386-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5474b7-7496845f45-6ncjj\" (UID: \"e22aa23e-9a2a-4499-bba1-4da8208e7386\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5474b7-7496845f45-6ncjj" Nov 03 13:36:21.567998 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:21.567965 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpwq2\" (UniqueName: \"kubernetes.io/projected/e22aa23e-9a2a-4499-bba1-4da8208e7386-kube-api-access-kpwq2\") pod \"payload-processing-e2e-ipp-b-5474b7-7496845f45-6ncjj\" (UID: \"e22aa23e-9a2a-4499-bba1-4da8208e7386\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5474b7-7496845f45-6ncjj" Nov 03 13:36:21.705807 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:21.705775 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5474b7-ccbcc59f-hlkd5"] Nov 03 13:36:21.713355 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:36:21.713328 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf27c783a_fa71_4db1_a3bb_4ed989e72d08.slice/crio-9275ff900b86f213d8a18e1148813c46f4fa7a72b2de4256ab048e8c0c9dbcb1 WatchSource:0}: Error finding container 9275ff900b86f213d8a18e1148813c46f4fa7a72b2de4256ab048e8c0c9dbcb1: Status 404 returned error can't find the container with id 9275ff900b86f213d8a18e1148813c46f4fa7a72b2de4256ab048e8c0c9dbcb1 Nov 03 13:36:21.843433 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:21.843401 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5474b7-7496845f45-6ncjj" Nov 03 13:36:22.216070 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:22.216029 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5474b7-7496845f45-6ncjj"] Nov 03 13:36:22.222343 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:36:22.222314 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode22aa23e_9a2a_4499_bba1_4da8208e7386.slice/crio-f539d3e77508f72358be5df322617857a5bd29e1a8dfe18635868331a5181d83 WatchSource:0}: Error finding container f539d3e77508f72358be5df322617857a5bd29e1a8dfe18635868331a5181d83: Status 404 returned error can't find the container with id f539d3e77508f72358be5df322617857a5bd29e1a8dfe18635868331a5181d83 Nov 03 13:36:22.527837 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:22.527737 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5474b7-ccbcc59f-hlkd5" event={"ID":"f27c783a-fa71-4db1-a3bb-4ed989e72d08","Type":"ContainerStarted","Data":"1f0b697479115d5629d5b5871550d76e288907c9d11bc661ea6f4662247fb2bb"} Nov 03 13:36:22.527837 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:22.527804 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5474b7-ccbcc59f-hlkd5" Nov 03 13:36:22.527837 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:22.527827 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5474b7-ccbcc59f-hlkd5" event={"ID":"f27c783a-fa71-4db1-a3bb-4ed989e72d08","Type":"ContainerStarted","Data":"9275ff900b86f213d8a18e1148813c46f4fa7a72b2de4256ab048e8c0c9dbcb1"} Nov 03 13:36:22.535752 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:22.535710 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5474b7-7496845f45-6ncjj" event={"ID":"e22aa23e-9a2a-4499-bba1-4da8208e7386","Type":"ContainerStarted","Data":"250546879e63765f67a74a640f85d7484aa1591d7693464e3f614f2a3a7c8a58"} Nov 03 13:36:22.535932 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:22.535782 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5474b7-7496845f45-6ncjj" Nov 03 13:36:22.535932 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:22.535810 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5474b7-7496845f45-6ncjj" event={"ID":"e22aa23e-9a2a-4499-bba1-4da8208e7386","Type":"ContainerStarted","Data":"f539d3e77508f72358be5df322617857a5bd29e1a8dfe18635868331a5181d83"} Nov 03 13:36:22.639462 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:22.639399 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5474b7-ccbcc59f-hlkd5" podStartSLOduration=2.639386049 podStartE2EDuration="2.639386049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:36:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:36:22.636102387 +0000 UTC m=+1864.646790629" watchObservedRunningTime="2026-11-03 13:36:22.639386049 +0000 UTC m=+1864.650074290" Nov 03 13:36:22.747567 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:22.747504 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-5474b7-7496845f45-6ncjj" podStartSLOduration=1.747490322 podStartE2EDuration="1.747490322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:36:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:36:22.743244076 +0000 UTC m=+1864.753932317" watchObservedRunningTime="2026-11-03 13:36:22.747490322 +0000 UTC m=+1864.758178563" Nov 03 13:36:24.429240 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:24.429204 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02f3a0-66bc949bbc-dk2q5" Nov 03 13:36:25.441843 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:25.441814 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-02f3a0-57f7b565b5-xg9v9" Nov 03 13:36:26.444214 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:26.444180 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5c8192-54f4dd4bf5-qjv8l"] Nov 03 13:36:26.566535 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:26.566493 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5c8192-7c4d4b65cf-wm2vg"] Nov 03 13:36:28.010530 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:27.997535 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:36:31.445084 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:31.445022 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c8192-54f4dd4bf5-qjv8l" podUID="ca5d6e86-e31a-4a7f-b71b-54216e92e668" containerName="payload-processing" containerID="cri-o://3c9aa2a28c385d0f8f2998af5f6f518f738d5c076e7db1d946b477661023fb25" gracePeriod=25 Nov 03 13:36:31.576620 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:31.569032 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c8192-7c4d4b65cf-wm2vg" podUID="10c68fea-ba03-4f0c-85a9-01f113c0f65d" containerName="payload-pre-processing" containerID="cri-o://1e51c064970925eae34be195cef6ffb20b5577084f437361a2ef3596fd9149e3" gracePeriod=25 Nov 03 13:36:31.608262 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:31.608224 5979 generic.go:356] "Generic (PLEG): container finished" podID="ca5d6e86-e31a-4a7f-b71b-54216e92e668" containerID="3c9aa2a28c385d0f8f2998af5f6f518f738d5c076e7db1d946b477661023fb25" exitCode=0 Nov 03 13:36:31.608427 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:31.608270 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c8192-54f4dd4bf5-qjv8l" event={"ID":"ca5d6e86-e31a-4a7f-b71b-54216e92e668","Type":"ContainerDied","Data":"3c9aa2a28c385d0f8f2998af5f6f518f738d5c076e7db1d946b477661023fb25"} Nov 03 13:36:31.729035 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:36:31.729002 5979 logging.go:55] [core] [Channel #2689 SubChannel #2690]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.165:50052", ServerName: "10.134.0.165:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.165:50052: connect: connection refused" Nov 03 13:36:31.744561 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:31.744534 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c8192-54f4dd4bf5-qjv8l" Nov 03 13:36:31.836883 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:31.836852 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c8192-7c4d4b65cf-wm2vg" Nov 03 13:36:31.892517 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:31.892456 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca5d6e86-e31a-4a7f-b71b-54216e92e668-kube-api-access-4lgk5\" (UniqueName: \"kubernetes.io/projected/ca5d6e86-e31a-4a7f-b71b-54216e92e668-kube-api-access-4lgk5\") pod \"ca5d6e86-e31a-4a7f-b71b-54216e92e668\" (UID: \"ca5d6e86-e31a-4a7f-b71b-54216e92e668\") " Nov 03 13:36:31.892697 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:31.892657 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca5d6e86-e31a-4a7f-b71b-54216e92e668-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca5d6e86-e31a-4a7f-b71b-54216e92e668-plugins-config-volume\") pod \"ca5d6e86-e31a-4a7f-b71b-54216e92e668\" (UID: \"ca5d6e86-e31a-4a7f-b71b-54216e92e668\") " Nov 03 13:36:31.893110 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:31.893077 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca5d6e86-e31a-4a7f-b71b-54216e92e668-plugins-config-volume" pod "ca5d6e86-e31a-4a7f-b71b-54216e92e668" (UID: "ca5d6e86-e31a-4a7f-b71b-54216e92e668"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:36:31.894713 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:31.894685 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca5d6e86-e31a-4a7f-b71b-54216e92e668-kube-api-access-4lgk5" pod "ca5d6e86-e31a-4a7f-b71b-54216e92e668" (UID: "ca5d6e86-e31a-4a7f-b71b-54216e92e668"). InnerVolumeSpecName "kube-api-access-4lgk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:36:31.994235 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:31.994151 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10c68fea-ba03-4f0c-85a9-01f113c0f65d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10c68fea-ba03-4f0c-85a9-01f113c0f65d-plugins-config-volume\") pod \"10c68fea-ba03-4f0c-85a9-01f113c0f65d\" (UID: \"10c68fea-ba03-4f0c-85a9-01f113c0f65d\") " Nov 03 13:36:31.994412 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:31.994361 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10c68fea-ba03-4f0c-85a9-01f113c0f65d-kube-api-access-x6jqw\" (UniqueName: \"kubernetes.io/projected/10c68fea-ba03-4f0c-85a9-01f113c0f65d-kube-api-access-x6jqw\") pod \"10c68fea-ba03-4f0c-85a9-01f113c0f65d\" (UID: \"10c68fea-ba03-4f0c-85a9-01f113c0f65d\") " Nov 03 13:36:31.994576 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:31.994553 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10c68fea-ba03-4f0c-85a9-01f113c0f65d-plugins-config-volume" pod "10c68fea-ba03-4f0c-85a9-01f113c0f65d" (UID: "10c68fea-ba03-4f0c-85a9-01f113c0f65d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:36:31.994862 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:31.994843 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lgk5\" (UniqueName: \"kubernetes.io/projected/ca5d6e86-e31a-4a7f-b71b-54216e92e668-kube-api-access-4lgk5\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:31.994942 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:31.994866 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10c68fea-ba03-4f0c-85a9-01f113c0f65d-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:31.994942 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:31.994880 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca5d6e86-e31a-4a7f-b71b-54216e92e668-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:31.996530 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:31.996502 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10c68fea-ba03-4f0c-85a9-01f113c0f65d-kube-api-access-x6jqw" pod "10c68fea-ba03-4f0c-85a9-01f113c0f65d" (UID: "10c68fea-ba03-4f0c-85a9-01f113c0f65d"). InnerVolumeSpecName "kube-api-access-x6jqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:36:32.096652 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:32.096612 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6jqw\" (UniqueName: \"kubernetes.io/projected/10c68fea-ba03-4f0c-85a9-01f113c0f65d-kube-api-access-x6jqw\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:32.615847 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:32.615808 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c8192-54f4dd4bf5-qjv8l" Nov 03 13:36:32.615847 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:32.615825 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c8192-54f4dd4bf5-qjv8l" event={"ID":"ca5d6e86-e31a-4a7f-b71b-54216e92e668","Type":"ContainerDied","Data":"d223dd0a1deb8bc9f4674c835ef12d7907a9eace75c562c6a7db4b0ba6ec05a3"} Nov 03 13:36:32.616425 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:32.615892 5979 scope.go:122] "RemoveContainer" containerID="3c9aa2a28c385d0f8f2998af5f6f518f738d5c076e7db1d946b477661023fb25" Nov 03 13:36:32.617392 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:32.617347 5979 generic.go:356] "Generic (PLEG): container finished" podID="10c68fea-ba03-4f0c-85a9-01f113c0f65d" containerID="1e51c064970925eae34be195cef6ffb20b5577084f437361a2ef3596fd9149e3" exitCode=0 Nov 03 13:36:32.617580 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:32.617407 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c8192-7c4d4b65cf-wm2vg" Nov 03 13:36:32.617580 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:32.617422 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c8192-7c4d4b65cf-wm2vg" event={"ID":"10c68fea-ba03-4f0c-85a9-01f113c0f65d","Type":"ContainerDied","Data":"1e51c064970925eae34be195cef6ffb20b5577084f437361a2ef3596fd9149e3"} Nov 03 13:36:32.617580 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:32.617537 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c8192-7c4d4b65cf-wm2vg" event={"ID":"10c68fea-ba03-4f0c-85a9-01f113c0f65d","Type":"ContainerDied","Data":"980eed2205fefcf8c123f2fcfd37cef3118c70a805bb067c9230f9ed284de80d"} Nov 03 13:36:32.627254 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:32.627223 5979 scope.go:122] "RemoveContainer" containerID="1e51c064970925eae34be195cef6ffb20b5577084f437361a2ef3596fd9149e3" Nov 03 13:36:32.635977 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:32.635958 5979 scope.go:122] "RemoveContainer" containerID="1e51c064970925eae34be195cef6ffb20b5577084f437361a2ef3596fd9149e3" Nov 03 13:36:32.636228 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:36:32.636210 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e51c064970925eae34be195cef6ffb20b5577084f437361a2ef3596fd9149e3\": container with ID starting with 1e51c064970925eae34be195cef6ffb20b5577084f437361a2ef3596fd9149e3 not found: ID does not exist" containerID="1e51c064970925eae34be195cef6ffb20b5577084f437361a2ef3596fd9149e3" Nov 03 13:36:32.636276 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:32.636236 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e51c064970925eae34be195cef6ffb20b5577084f437361a2ef3596fd9149e3"} err="failed to get container status \"1e51c064970925eae34be195cef6ffb20b5577084f437361a2ef3596fd9149e3\": rpc error: code = NotFound desc = could not find container \"1e51c064970925eae34be195cef6ffb20b5577084f437361a2ef3596fd9149e3\": container with ID starting with 1e51c064970925eae34be195cef6ffb20b5577084f437361a2ef3596fd9149e3 not found: ID does not exist" Nov 03 13:36:32.726674 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:32.726585 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c8192-54f4dd4bf5-qjv8l" podUID="ca5d6e86-e31a-4a7f-b71b-54216e92e668" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.166:50052\" within 1s: context deadline exceeded" Nov 03 13:36:32.726674 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:36:32.726637 5979 logging.go:55] [core] [Channel #2687 SubChannel #2688]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.166:50052", ServerName: "10.134.0.166:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.166:50052: operation was canceled" Nov 03 13:36:32.728656 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:32.728634 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5c8192-7c4d4b65cf-wm2vg" podUID="10c68fea-ba03-4f0c-85a9-01f113c0f65d" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.165:50052\" within 1s: context deadline exceeded" Nov 03 13:36:32.790658 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:32.790623 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5c8192-7c4d4b65cf-wm2vg"] Nov 03 13:36:32.797827 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:32.797800 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5c8192-7c4d4b65cf-wm2vg"] Nov 03 13:36:32.914732 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:32.914698 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5c8192-54f4dd4bf5-qjv8l"] Nov 03 13:36:32.920031 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:32.920005 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5c8192-54f4dd4bf5-qjv8l"] Nov 03 13:36:33.548435 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:33.548403 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5474b7-7496845f45-6ncjj" Nov 03 13:36:33.548676 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:33.548509 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5474b7-ccbcc59f-hlkd5" Nov 03 13:36:34.115957 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:34.115925 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10c68fea-ba03-4f0c-85a9-01f113c0f65d" path="/var/lib/kubelet/pods/10c68fea-ba03-4f0c-85a9-01f113c0f65d/volumes" Nov 03 13:36:34.116364 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:34.116355 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca5d6e86-e31a-4a7f-b71b-54216e92e668" path="/var/lib/kubelet/pods/ca5d6e86-e31a-4a7f-b71b-54216e92e668/volumes" Nov 03 13:36:37.234833 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:36:37.234795 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:36:42.055799 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:42.055765 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8a2e426-dxzqm"] Nov 03 13:36:42.101712 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:42.101678 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b8a2e426-dxzqm"] Nov 03 13:36:42.116087 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:42.116058 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f5db4ce-3fef-4033-8d83-71495a0b6373" path="/var/lib/kubelet/pods/9f5db4ce-3fef-4033-8d83-71495a0b6373/volumes" Nov 03 13:36:51.448750 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:51.448712 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d29e07-58df5c765-x69w6"] Nov 03 13:36:51.571368 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:51.571318 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d29e07-56b6bbfbcf-pjntv"] Nov 03 13:36:56.449967 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:56.449919 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-d29e07-58df5c765-x69w6" podUID="bd3ebbea-ef03-4183-8e8e-cd89fb9cae36" containerName="payload-processing" containerID="cri-o://43106f1f127806d43a3a50abc2cdffa5ee8b39776f9ea343fe592ec081d1d0ad" gracePeriod=25 Nov 03 13:36:56.572882 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:56.572825 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d29e07-56b6bbfbcf-pjntv" podUID="c72f8e91-a138-46d6-8440-94cac7793866" containerName="payload-pre-processing" containerID="cri-o://72e1dd56071e5017d6fa1b4962d8354354f6d0ea3f9d5215606fca1f601f10df" gracePeriod=25 Nov 03 13:36:56.755862 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:56.755831 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d29e07-58df5c765-x69w6" Nov 03 13:36:56.773567 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:56.773530 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd3ebbea-ef03-4183-8e8e-cd89fb9cae36-kube-api-access-z858b\" (UniqueName: \"kubernetes.io/projected/bd3ebbea-ef03-4183-8e8e-cd89fb9cae36-kube-api-access-z858b\") pod \"bd3ebbea-ef03-4183-8e8e-cd89fb9cae36\" (UID: \"bd3ebbea-ef03-4183-8e8e-cd89fb9cae36\") " Nov 03 13:36:56.773750 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:56.773615 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd3ebbea-ef03-4183-8e8e-cd89fb9cae36-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd3ebbea-ef03-4183-8e8e-cd89fb9cae36-plugins-config-volume\") pod \"bd3ebbea-ef03-4183-8e8e-cd89fb9cae36\" (UID: \"bd3ebbea-ef03-4183-8e8e-cd89fb9cae36\") " Nov 03 13:36:56.774339 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:56.774310 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd3ebbea-ef03-4183-8e8e-cd89fb9cae36-plugins-config-volume" pod "bd3ebbea-ef03-4183-8e8e-cd89fb9cae36" (UID: "bd3ebbea-ef03-4183-8e8e-cd89fb9cae36"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:36:56.779248 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:56.779219 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd3ebbea-ef03-4183-8e8e-cd89fb9cae36-kube-api-access-z858b" pod "bd3ebbea-ef03-4183-8e8e-cd89fb9cae36" (UID: "bd3ebbea-ef03-4183-8e8e-cd89fb9cae36"). InnerVolumeSpecName "kube-api-access-z858b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:36:56.836729 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:56.834330 5979 generic.go:356] "Generic (PLEG): container finished" podID="c72f8e91-a138-46d6-8440-94cac7793866" containerID="72e1dd56071e5017d6fa1b4962d8354354f6d0ea3f9d5215606fca1f601f10df" exitCode=0 Nov 03 13:36:56.836729 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:56.834379 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d29e07-56b6bbfbcf-pjntv" event={"ID":"c72f8e91-a138-46d6-8440-94cac7793866","Type":"ContainerDied","Data":"72e1dd56071e5017d6fa1b4962d8354354f6d0ea3f9d5215606fca1f601f10df"} Nov 03 13:36:56.836729 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:56.836170 5979 generic.go:356] "Generic (PLEG): container finished" podID="bd3ebbea-ef03-4183-8e8e-cd89fb9cae36" containerID="43106f1f127806d43a3a50abc2cdffa5ee8b39776f9ea343fe592ec081d1d0ad" exitCode=0 Nov 03 13:36:56.836729 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:56.836234 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d29e07-58df5c765-x69w6" Nov 03 13:36:56.836729 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:56.836260 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d29e07-58df5c765-x69w6" event={"ID":"bd3ebbea-ef03-4183-8e8e-cd89fb9cae36","Type":"ContainerDied","Data":"43106f1f127806d43a3a50abc2cdffa5ee8b39776f9ea343fe592ec081d1d0ad"} Nov 03 13:36:56.836729 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:56.836302 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d29e07-58df5c765-x69w6" event={"ID":"bd3ebbea-ef03-4183-8e8e-cd89fb9cae36","Type":"ContainerDied","Data":"7ffb43c506a8325e2802c53d4d6f7202fdbe1268f693b204a396a333083ab477"} Nov 03 13:36:56.836729 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:56.836331 5979 scope.go:122] "RemoveContainer" containerID="43106f1f127806d43a3a50abc2cdffa5ee8b39776f9ea343fe592ec081d1d0ad" Nov 03 13:36:56.842521 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:56.841105 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d29e07-56b6bbfbcf-pjntv" Nov 03 13:36:56.849864 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:56.849843 5979 scope.go:122] "RemoveContainer" containerID="43106f1f127806d43a3a50abc2cdffa5ee8b39776f9ea343fe592ec081d1d0ad" Nov 03 13:36:56.850219 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:36:56.850198 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43106f1f127806d43a3a50abc2cdffa5ee8b39776f9ea343fe592ec081d1d0ad\": container with ID starting with 43106f1f127806d43a3a50abc2cdffa5ee8b39776f9ea343fe592ec081d1d0ad not found: ID does not exist" containerID="43106f1f127806d43a3a50abc2cdffa5ee8b39776f9ea343fe592ec081d1d0ad" Nov 03 13:36:56.850285 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:56.850227 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43106f1f127806d43a3a50abc2cdffa5ee8b39776f9ea343fe592ec081d1d0ad"} err="failed to get container status \"43106f1f127806d43a3a50abc2cdffa5ee8b39776f9ea343fe592ec081d1d0ad\": rpc error: code = NotFound desc = could not find container \"43106f1f127806d43a3a50abc2cdffa5ee8b39776f9ea343fe592ec081d1d0ad\": container with ID starting with 43106f1f127806d43a3a50abc2cdffa5ee8b39776f9ea343fe592ec081d1d0ad not found: ID does not exist" Nov 03 13:36:56.875108 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:56.875074 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c72f8e91-a138-46d6-8440-94cac7793866-kube-api-access-lhrvm\" (UniqueName: \"kubernetes.io/projected/c72f8e91-a138-46d6-8440-94cac7793866-kube-api-access-lhrvm\") pod \"c72f8e91-a138-46d6-8440-94cac7793866\" (UID: \"c72f8e91-a138-46d6-8440-94cac7793866\") " Nov 03 13:36:56.875292 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:56.875234 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c72f8e91-a138-46d6-8440-94cac7793866-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c72f8e91-a138-46d6-8440-94cac7793866-plugins-config-volume\") pod \"c72f8e91-a138-46d6-8440-94cac7793866\" (UID: \"c72f8e91-a138-46d6-8440-94cac7793866\") " Nov 03 13:36:56.875617 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:56.875591 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c72f8e91-a138-46d6-8440-94cac7793866-plugins-config-volume" pod "c72f8e91-a138-46d6-8440-94cac7793866" (UID: "c72f8e91-a138-46d6-8440-94cac7793866"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:36:56.875747 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:56.875728 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z858b\" (UniqueName: \"kubernetes.io/projected/bd3ebbea-ef03-4183-8e8e-cd89fb9cae36-kube-api-access-z858b\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:56.875814 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:56.875752 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd3ebbea-ef03-4183-8e8e-cd89fb9cae36-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:56.875814 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:56.875767 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c72f8e91-a138-46d6-8440-94cac7793866-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:56.877276 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:56.877249 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c72f8e91-a138-46d6-8440-94cac7793866-kube-api-access-lhrvm" pod "c72f8e91-a138-46d6-8440-94cac7793866" (UID: "c72f8e91-a138-46d6-8440-94cac7793866"). InnerVolumeSpecName "kube-api-access-lhrvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:36:56.977896 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:56.976967 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhrvm\" (UniqueName: \"kubernetes.io/projected/c72f8e91-a138-46d6-8440-94cac7793866-kube-api-access-lhrvm\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:36:57.079276 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:57.079243 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d29e07-58df5c765-x69w6"] Nov 03 13:36:57.085277 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:57.085250 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d29e07-58df5c765-x69w6"] Nov 03 13:36:57.844525 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:57.844407 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d29e07-56b6bbfbcf-pjntv" Nov 03 13:36:57.844525 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:57.844415 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d29e07-56b6bbfbcf-pjntv" event={"ID":"c72f8e91-a138-46d6-8440-94cac7793866","Type":"ContainerDied","Data":"27324ed7f30d60e8fedd5b01de88e86d41e394ffaf45d6a5fddd4dd198997a99"} Nov 03 13:36:57.845071 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:57.844548 5979 scope.go:122] "RemoveContainer" containerID="72e1dd56071e5017d6fa1b4962d8354354f6d0ea3f9d5215606fca1f601f10df" Nov 03 13:36:57.999711 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:57.999679 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d29e07-56b6bbfbcf-pjntv"] Nov 03 13:36:58.045971 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:58.045939 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d29e07-56b6bbfbcf-pjntv"] Nov 03 13:36:58.116030 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:58.115952 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd3ebbea-ef03-4183-8e8e-cd89fb9cae36" path="/var/lib/kubelet/pods/bd3ebbea-ef03-4183-8e8e-cd89fb9cae36/volumes" Nov 03 13:36:58.116403 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:58.116388 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c72f8e91-a138-46d6-8440-94cac7793866" path="/var/lib/kubelet/pods/c72f8e91-a138-46d6-8440-94cac7793866/volumes" Nov 03 13:36:58.868538 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:36:58.868435 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:37:01.889569 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:01.889532 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa2568-d5dd645b7-pvp9f"] Nov 03 13:37:01.889986 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:01.889779 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa2568-d5dd645b7-pvp9f" podUID="e6506f7e-7151-4cf3-9f2d-fefbba0a3783" containerName="maas-api" containerID="cri-o://5bf168b7972cf903e6f1931a05395880327d5e83f2420bf885f22f194e1942e3" gracePeriod=30 Nov 03 13:37:02.210883 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:02.210856 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa2568-d5dd645b7-pvp9f" Nov 03 13:37:02.272979 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:02.272947 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6506f7e-7151-4cf3-9f2d-fefbba0a3783-kube-api-access-nwjcl\" (UniqueName: \"kubernetes.io/projected/e6506f7e-7151-4cf3-9f2d-fefbba0a3783-kube-api-access-nwjcl\") pod \"e6506f7e-7151-4cf3-9f2d-fefbba0a3783\" (UID: \"e6506f7e-7151-4cf3-9f2d-fefbba0a3783\") " Nov 03 13:37:02.273158 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:02.273036 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6506f7e-7151-4cf3-9f2d-fefbba0a3783-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6506f7e-7151-4cf3-9f2d-fefbba0a3783-maas-api-tls\") pod \"e6506f7e-7151-4cf3-9f2d-fefbba0a3783\" (UID: \"e6506f7e-7151-4cf3-9f2d-fefbba0a3783\") " Nov 03 13:37:02.273158 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:02.273123 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6506f7e-7151-4cf3-9f2d-fefbba0a3783-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6506f7e-7151-4cf3-9f2d-fefbba0a3783-metrics-certs\") pod \"e6506f7e-7151-4cf3-9f2d-fefbba0a3783\" (UID: \"e6506f7e-7151-4cf3-9f2d-fefbba0a3783\") " Nov 03 13:37:02.273237 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:02.273216 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6506f7e-7151-4cf3-9f2d-fefbba0a3783-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6506f7e-7151-4cf3-9f2d-fefbba0a3783-openshift-service-ca\") pod \"e6506f7e-7151-4cf3-9f2d-fefbba0a3783\" (UID: \"e6506f7e-7151-4cf3-9f2d-fefbba0a3783\") " Nov 03 13:37:02.273661 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:02.273635 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6506f7e-7151-4cf3-9f2d-fefbba0a3783-openshift-service-ca" pod "e6506f7e-7151-4cf3-9f2d-fefbba0a3783" (UID: "e6506f7e-7151-4cf3-9f2d-fefbba0a3783"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:37:02.273982 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:02.273960 5979 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6506f7e-7151-4cf3-9f2d-fefbba0a3783-openshift-service-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:02.275228 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:02.275203 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6506f7e-7151-4cf3-9f2d-fefbba0a3783-metrics-certs" pod "e6506f7e-7151-4cf3-9f2d-fefbba0a3783" (UID: "e6506f7e-7151-4cf3-9f2d-fefbba0a3783"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:37:02.275328 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:02.275248 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6506f7e-7151-4cf3-9f2d-fefbba0a3783-maas-api-tls" pod "e6506f7e-7151-4cf3-9f2d-fefbba0a3783" (UID: "e6506f7e-7151-4cf3-9f2d-fefbba0a3783"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:37:02.275328 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:02.275267 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6506f7e-7151-4cf3-9f2d-fefbba0a3783-kube-api-access-nwjcl" pod "e6506f7e-7151-4cf3-9f2d-fefbba0a3783" (UID: "e6506f7e-7151-4cf3-9f2d-fefbba0a3783"). InnerVolumeSpecName "kube-api-access-nwjcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:37:02.382397 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:02.382341 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fa2568-6cbd46f946-6fn4v"] Nov 03 13:37:02.382648 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:02.382617 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa2568-6cbd46f946-6fn4v" podUID="92d72487-02a6-4172-a2b7-9d13411a94f7" containerName="payload-processing" containerID="cri-o://2630b423d6fe9f67b3684bdb13cf6f0bae5e9b4868c00fc9540e3aaf4dbb3124" gracePeriod=30 Nov 03 13:37:02.392515 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:02.392134 5979 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6506f7e-7151-4cf3-9f2d-fefbba0a3783-metrics-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:02.392515 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:02.392166 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwjcl\" (UniqueName: \"kubernetes.io/projected/e6506f7e-7151-4cf3-9f2d-fefbba0a3783-kube-api-access-nwjcl\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:02.392515 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:02.392182 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6506f7e-7151-4cf3-9f2d-fefbba0a3783-maas-api-tls\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:02.428200 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:02.428164 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fa2568-67ffc88795-t7jsv"] Nov 03 13:37:02.428453 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:02.428405 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa2568-67ffc88795-t7jsv" podUID="0f997da5-e10f-4926-92c0-f5559fb13567" containerName="payload-pre-processing" containerID="cri-o://06d3c34da84ccbfb8a95fe4040a7d23503b61b18c2517ca9648dae28f4d13abc" gracePeriod=30 Nov 03 13:37:02.705929 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:02.705849 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa2568-6cbd46f946-6fn4v" Nov 03 13:37:02.735853 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:02.735826 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa2568-67ffc88795-t7jsv" Nov 03 13:37:02.798064 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:02.798025 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92d72487-02a6-4172-a2b7-9d13411a94f7-kube-api-access-p86n7\" (UniqueName: \"kubernetes.io/projected/92d72487-02a6-4172-a2b7-9d13411a94f7-kube-api-access-p86n7\") pod \"92d72487-02a6-4172-a2b7-9d13411a94f7\" (UID: \"92d72487-02a6-4172-a2b7-9d13411a94f7\") " Nov 03 13:37:02.798239 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:02.798152 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f997da5-e10f-4926-92c0-f5559fb13567-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f997da5-e10f-4926-92c0-f5559fb13567-plugins-config-volume\") pod \"0f997da5-e10f-4926-92c0-f5559fb13567\" (UID: \"0f997da5-e10f-4926-92c0-f5559fb13567\") " Nov 03 13:37:02.798239 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:02.798180 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92d72487-02a6-4172-a2b7-9d13411a94f7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92d72487-02a6-4172-a2b7-9d13411a94f7-plugins-config-volume\") pod \"92d72487-02a6-4172-a2b7-9d13411a94f7\" (UID: \"92d72487-02a6-4172-a2b7-9d13411a94f7\") " Nov 03 13:37:02.798344 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:02.798297 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f997da5-e10f-4926-92c0-f5559fb13567-kube-api-access-8lp5p\" (UniqueName: \"kubernetes.io/projected/0f997da5-e10f-4926-92c0-f5559fb13567-kube-api-access-8lp5p\") pod \"0f997da5-e10f-4926-92c0-f5559fb13567\" (UID: \"0f997da5-e10f-4926-92c0-f5559fb13567\") " Nov 03 13:37:02.798607 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:02.798574 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f997da5-e10f-4926-92c0-f5559fb13567-plugins-config-volume" pod "0f997da5-e10f-4926-92c0-f5559fb13567" (UID: "0f997da5-e10f-4926-92c0-f5559fb13567"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:37:02.798607 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:02.798585 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92d72487-02a6-4172-a2b7-9d13411a94f7-plugins-config-volume" pod "92d72487-02a6-4172-a2b7-9d13411a94f7" (UID: "92d72487-02a6-4172-a2b7-9d13411a94f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:37:02.798908 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:02.798890 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f997da5-e10f-4926-92c0-f5559fb13567-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:02.798968 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:02.798910 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92d72487-02a6-4172-a2b7-9d13411a94f7-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:02.800371 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:02.800350 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f997da5-e10f-4926-92c0-f5559fb13567-kube-api-access-8lp5p" pod "0f997da5-e10f-4926-92c0-f5559fb13567" (UID: "0f997da5-e10f-4926-92c0-f5559fb13567"). InnerVolumeSpecName "kube-api-access-8lp5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:37:02.800436 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:02.800372 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92d72487-02a6-4172-a2b7-9d13411a94f7-kube-api-access-p86n7" pod "92d72487-02a6-4172-a2b7-9d13411a94f7" (UID: "92d72487-02a6-4172-a2b7-9d13411a94f7"). InnerVolumeSpecName "kube-api-access-p86n7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:37:02.889658 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:02.888422 5979 generic.go:356] "Generic (PLEG): container finished" podID="e6506f7e-7151-4cf3-9f2d-fefbba0a3783" containerID="5bf168b7972cf903e6f1931a05395880327d5e83f2420bf885f22f194e1942e3" exitCode=0 Nov 03 13:37:02.889658 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:02.888706 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa2568-d5dd645b7-pvp9f" event={"ID":"e6506f7e-7151-4cf3-9f2d-fefbba0a3783","Type":"ContainerDied","Data":"5bf168b7972cf903e6f1931a05395880327d5e83f2420bf885f22f194e1942e3"} Nov 03 13:37:02.889658 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:02.888769 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa2568-d5dd645b7-pvp9f" event={"ID":"e6506f7e-7151-4cf3-9f2d-fefbba0a3783","Type":"ContainerDied","Data":"9634b582ce02c399b4383432dc6fe3f1b25c3d8724adaf97e1b81482c80f2dd4"} Nov 03 13:37:02.889658 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:02.888804 5979 scope.go:122] "RemoveContainer" containerID="5bf168b7972cf903e6f1931a05395880327d5e83f2420bf885f22f194e1942e3" Nov 03 13:37:02.889658 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:02.889009 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa2568-d5dd645b7-pvp9f" Nov 03 13:37:02.948551 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:02.924227 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lp5p\" (UniqueName: \"kubernetes.io/projected/0f997da5-e10f-4926-92c0-f5559fb13567-kube-api-access-8lp5p\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:02.948551 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:02.924265 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p86n7\" (UniqueName: \"kubernetes.io/projected/92d72487-02a6-4172-a2b7-9d13411a94f7-kube-api-access-p86n7\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:03.028214 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:03.019816 5979 generic.go:356] "Generic (PLEG): container finished" podID="92d72487-02a6-4172-a2b7-9d13411a94f7" containerID="2630b423d6fe9f67b3684bdb13cf6f0bae5e9b4868c00fc9540e3aaf4dbb3124" exitCode=0 Nov 03 13:37:03.028214 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:03.019985 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa2568-6cbd46f946-6fn4v" event={"ID":"92d72487-02a6-4172-a2b7-9d13411a94f7","Type":"ContainerDied","Data":"2630b423d6fe9f67b3684bdb13cf6f0bae5e9b4868c00fc9540e3aaf4dbb3124"} Nov 03 13:37:03.028214 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:03.020044 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa2568-6cbd46f946-6fn4v" event={"ID":"92d72487-02a6-4172-a2b7-9d13411a94f7","Type":"ContainerDied","Data":"0d5a8ee220b3962423684259f2f93f25cb183ce0085f79f4be4787448b336dce"} Nov 03 13:37:03.028214 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:03.020143 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fa2568-6cbd46f946-6fn4v" Nov 03 13:37:03.033806 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:03.033458 5979 scope.go:122] "RemoveContainer" containerID="5bf168b7972cf903e6f1931a05395880327d5e83f2420bf885f22f194e1942e3" Nov 03 13:37:03.034547 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:37:03.034515 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bf168b7972cf903e6f1931a05395880327d5e83f2420bf885f22f194e1942e3\": container with ID starting with 5bf168b7972cf903e6f1931a05395880327d5e83f2420bf885f22f194e1942e3 not found: ID does not exist" containerID="5bf168b7972cf903e6f1931a05395880327d5e83f2420bf885f22f194e1942e3" Nov 03 13:37:03.034751 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:03.034713 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bf168b7972cf903e6f1931a05395880327d5e83f2420bf885f22f194e1942e3"} err="failed to get container status \"5bf168b7972cf903e6f1931a05395880327d5e83f2420bf885f22f194e1942e3\": rpc error: code = NotFound desc = could not find container \"5bf168b7972cf903e6f1931a05395880327d5e83f2420bf885f22f194e1942e3\": container with ID starting with 5bf168b7972cf903e6f1931a05395880327d5e83f2420bf885f22f194e1942e3 not found: ID does not exist" Nov 03 13:37:03.034935 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:03.034902 5979 scope.go:122] "RemoveContainer" containerID="2630b423d6fe9f67b3684bdb13cf6f0bae5e9b4868c00fc9540e3aaf4dbb3124" Nov 03 13:37:03.036523 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:03.036354 5979 generic.go:356] "Generic (PLEG): container finished" podID="0f997da5-e10f-4926-92c0-f5559fb13567" containerID="06d3c34da84ccbfb8a95fe4040a7d23503b61b18c2517ca9648dae28f4d13abc" exitCode=0 Nov 03 13:37:03.036523 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:03.036413 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa2568-67ffc88795-t7jsv" event={"ID":"0f997da5-e10f-4926-92c0-f5559fb13567","Type":"ContainerDied","Data":"06d3c34da84ccbfb8a95fe4040a7d23503b61b18c2517ca9648dae28f4d13abc"} Nov 03 13:37:03.036523 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:03.036450 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa2568-67ffc88795-t7jsv" event={"ID":"0f997da5-e10f-4926-92c0-f5559fb13567","Type":"ContainerDied","Data":"d587384f98d3997ee2433ff5208082daf5d6650975402e59d5e3de8ae6532210"} Nov 03 13:37:03.036705 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:03.036536 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fa2568-67ffc88795-t7jsv" Nov 03 13:37:03.051661 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:03.051634 5979 scope.go:122] "RemoveContainer" containerID="2630b423d6fe9f67b3684bdb13cf6f0bae5e9b4868c00fc9540e3aaf4dbb3124" Nov 03 13:37:03.052377 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:37:03.052357 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2630b423d6fe9f67b3684bdb13cf6f0bae5e9b4868c00fc9540e3aaf4dbb3124\": container with ID starting with 2630b423d6fe9f67b3684bdb13cf6f0bae5e9b4868c00fc9540e3aaf4dbb3124 not found: ID does not exist" containerID="2630b423d6fe9f67b3684bdb13cf6f0bae5e9b4868c00fc9540e3aaf4dbb3124" Nov 03 13:37:03.052533 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:03.052492 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2630b423d6fe9f67b3684bdb13cf6f0bae5e9b4868c00fc9540e3aaf4dbb3124"} err="failed to get container status \"2630b423d6fe9f67b3684bdb13cf6f0bae5e9b4868c00fc9540e3aaf4dbb3124\": rpc error: code = NotFound desc = could not find container \"2630b423d6fe9f67b3684bdb13cf6f0bae5e9b4868c00fc9540e3aaf4dbb3124\": container with ID starting with 2630b423d6fe9f67b3684bdb13cf6f0bae5e9b4868c00fc9540e3aaf4dbb3124 not found: ID does not exist" Nov 03 13:37:03.052533 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:03.052531 5979 scope.go:122] "RemoveContainer" containerID="06d3c34da84ccbfb8a95fe4040a7d23503b61b18c2517ca9648dae28f4d13abc" Nov 03 13:37:03.066455 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:03.066430 5979 scope.go:122] "RemoveContainer" containerID="06d3c34da84ccbfb8a95fe4040a7d23503b61b18c2517ca9648dae28f4d13abc" Nov 03 13:37:03.066841 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:37:03.066814 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06d3c34da84ccbfb8a95fe4040a7d23503b61b18c2517ca9648dae28f4d13abc\": container with ID starting with 06d3c34da84ccbfb8a95fe4040a7d23503b61b18c2517ca9648dae28f4d13abc not found: ID does not exist" containerID="06d3c34da84ccbfb8a95fe4040a7d23503b61b18c2517ca9648dae28f4d13abc" Nov 03 13:37:03.066927 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:03.066843 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06d3c34da84ccbfb8a95fe4040a7d23503b61b18c2517ca9648dae28f4d13abc"} err="failed to get container status \"06d3c34da84ccbfb8a95fe4040a7d23503b61b18c2517ca9648dae28f4d13abc\": rpc error: code = NotFound desc = could not find container \"06d3c34da84ccbfb8a95fe4040a7d23503b61b18c2517ca9648dae28f4d13abc\": container with ID starting with 06d3c34da84ccbfb8a95fe4040a7d23503b61b18c2517ca9648dae28f4d13abc not found: ID does not exist" Nov 03 13:37:03.182497 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:03.179820 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa2568-d5dd645b7-pvp9f"] Nov 03 13:37:03.187097 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:03.187061 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fa2568-d5dd645b7-pvp9f"] Nov 03 13:37:03.289405 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:03.289321 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fa2568-6cbd46f946-6fn4v"] Nov 03 13:37:03.294176 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:03.294148 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fa2568-6cbd46f946-6fn4v"] Nov 03 13:37:03.397440 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:03.397396 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fa2568-67ffc88795-t7jsv"] Nov 03 13:37:03.401786 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:03.401754 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fa2568-67ffc88795-t7jsv"] Nov 03 13:37:04.115811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:04.115774 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f997da5-e10f-4926-92c0-f5559fb13567" path="/var/lib/kubelet/pods/0f997da5-e10f-4926-92c0-f5559fb13567/volumes" Nov 03 13:37:04.116401 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:04.116379 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92d72487-02a6-4172-a2b7-9d13411a94f7" path="/var/lib/kubelet/pods/92d72487-02a6-4172-a2b7-9d13411a94f7/volumes" Nov 03 13:37:04.116921 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:04.116904 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6506f7e-7151-4cf3-9f2d-fefbba0a3783" path="/var/lib/kubelet/pods/e6506f7e-7151-4cf3-9f2d-fefbba0a3783/volumes" Nov 03 13:37:06.467238 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:06.467199 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s"] Nov 03 13:37:06.467665 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:06.467420 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s" podUID="80fa8184-94c4-4ce1-8354-92681a69a6bf" containerName="istio-proxy" containerID="cri-o://03374afcced2e1fa408e68b3ec85774ff85263d64eef1a300e9ff4c12ecc9ea9" gracePeriod=30 Nov 03 13:37:07.235439 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:37:07.235401 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:37:09.784998 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:09.784960 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f6f9ed-6dc5f9df7f-pk6dn"] Nov 03 13:37:09.785593 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:09.785577 5979 cpu_manager.go:414] "containerMap: removing container" podUID="10c68fea-ba03-4f0c-85a9-01f113c0f65d" containerName="payload-pre-processing" Nov 03 13:37:09.785660 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:09.785593 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10c68fea-ba03-4f0c-85a9-01f113c0f65d" containerName="payload-pre-processing" Nov 03 13:37:09.785660 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:09.785618 5979 cpu_manager.go:414] "containerMap: removing container" podUID="e6506f7e-7151-4cf3-9f2d-fefbba0a3783" containerName="maas-api" Nov 03 13:37:09.785660 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:09.785623 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6506f7e-7151-4cf3-9f2d-fefbba0a3783" containerName="maas-api" Nov 03 13:37:09.785660 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:09.785639 5979 cpu_manager.go:414] "containerMap: removing container" podUID="92d72487-02a6-4172-a2b7-9d13411a94f7" containerName="payload-processing" Nov 03 13:37:09.785660 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:09.785644 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92d72487-02a6-4172-a2b7-9d13411a94f7" containerName="payload-processing" Nov 03 13:37:09.785660 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:09.785654 5979 cpu_manager.go:414] "containerMap: removing container" podUID="bd3ebbea-ef03-4183-8e8e-cd89fb9cae36" containerName="payload-processing" Nov 03 13:37:09.785660 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:09.785659 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd3ebbea-ef03-4183-8e8e-cd89fb9cae36" containerName="payload-processing" Nov 03 13:37:09.785894 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:09.785665 5979 cpu_manager.go:414] "containerMap: removing container" podUID="0f997da5-e10f-4926-92c0-f5559fb13567" containerName="payload-pre-processing" Nov 03 13:37:09.785894 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:09.785670 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f997da5-e10f-4926-92c0-f5559fb13567" containerName="payload-pre-processing" Nov 03 13:37:09.785894 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:09.785683 5979 cpu_manager.go:414] "containerMap: removing container" podUID="ca5d6e86-e31a-4a7f-b71b-54216e92e668" containerName="payload-processing" Nov 03 13:37:09.785894 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:09.785688 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca5d6e86-e31a-4a7f-b71b-54216e92e668" containerName="payload-processing" Nov 03 13:37:09.785894 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:09.785695 5979 cpu_manager.go:414] "containerMap: removing container" podUID="c72f8e91-a138-46d6-8440-94cac7793866" containerName="payload-pre-processing" Nov 03 13:37:09.785894 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:09.785700 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c72f8e91-a138-46d6-8440-94cac7793866" containerName="payload-pre-processing" Nov 03 13:37:09.785894 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:09.785781 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6506f7e-7151-4cf3-9f2d-fefbba0a3783" containerName="maas-api" Nov 03 13:37:09.785894 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:09.785792 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="92d72487-02a6-4172-a2b7-9d13411a94f7" containerName="payload-processing" Nov 03 13:37:09.785894 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:09.785798 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd3ebbea-ef03-4183-8e8e-cd89fb9cae36" containerName="payload-processing" Nov 03 13:37:09.785894 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:09.785804 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f997da5-e10f-4926-92c0-f5559fb13567" containerName="payload-pre-processing" Nov 03 13:37:09.785894 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:09.785812 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca5d6e86-e31a-4a7f-b71b-54216e92e668" containerName="payload-processing" Nov 03 13:37:09.785894 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:09.785819 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="c72f8e91-a138-46d6-8440-94cac7793866" containerName="payload-pre-processing" Nov 03 13:37:09.785894 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:09.785829 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="10c68fea-ba03-4f0c-85a9-01f113c0f65d" containerName="payload-pre-processing" Nov 03 13:37:09.791103 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:09.791081 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6f9ed-6dc5f9df7f-pk6dn" Nov 03 13:37:09.837961 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:09.837923 5979 patch_prober.go:29] interesting pod/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 13:37:09.838131 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:09.837983 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s" podUID="80fa8184-94c4-4ce1-8354-92681a69a6bf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 13:37:09.858307 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:09.858266 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck7fj\" (UniqueName: \"kubernetes.io/projected/aaab21c8-09c2-442a-8d48-e70bcff953b2-kube-api-access-ck7fj\") pod \"payload-pre-processing-e2e-ipp-a-f6f9ed-6dc5f9df7f-pk6dn\" (UID: \"aaab21c8-09c2-442a-8d48-e70bcff953b2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6f9ed-6dc5f9df7f-pk6dn" Nov 03 13:37:09.858481 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:09.858442 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaab21c8-09c2-442a-8d48-e70bcff953b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f6f9ed-6dc5f9df7f-pk6dn\" (UID: \"aaab21c8-09c2-442a-8d48-e70bcff953b2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6f9ed-6dc5f9df7f-pk6dn" Nov 03 13:37:09.886745 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:09.886704 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f6f9ed\"" Nov 03 13:37:09.886920 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:09.886821 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f6f9ed-dockercfg-6qmlk\"" Nov 03 13:37:09.889450 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:09.889426 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f6f9ed-6dc5f9df7f-pk6dn"] Nov 03 13:37:09.959735 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:09.959691 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck7fj\" (UniqueName: \"kubernetes.io/projected/aaab21c8-09c2-442a-8d48-e70bcff953b2-kube-api-access-ck7fj\") pod \"payload-pre-processing-e2e-ipp-a-f6f9ed-6dc5f9df7f-pk6dn\" (UID: \"aaab21c8-09c2-442a-8d48-e70bcff953b2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6f9ed-6dc5f9df7f-pk6dn" Nov 03 13:37:09.959918 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:09.959791 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaab21c8-09c2-442a-8d48-e70bcff953b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f6f9ed-6dc5f9df7f-pk6dn\" (UID: \"aaab21c8-09c2-442a-8d48-e70bcff953b2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6f9ed-6dc5f9df7f-pk6dn" Nov 03 13:37:09.960531 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:09.960512 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaab21c8-09c2-442a-8d48-e70bcff953b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f6f9ed-6dc5f9df7f-pk6dn\" (UID: \"aaab21c8-09c2-442a-8d48-e70bcff953b2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6f9ed-6dc5f9df7f-pk6dn" Nov 03 13:37:10.014959 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:10.014928 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck7fj\" (UniqueName: \"kubernetes.io/projected/aaab21c8-09c2-442a-8d48-e70bcff953b2-kube-api-access-ck7fj\") pod \"payload-pre-processing-e2e-ipp-a-f6f9ed-6dc5f9df7f-pk6dn\" (UID: \"aaab21c8-09c2-442a-8d48-e70bcff953b2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6f9ed-6dc5f9df7f-pk6dn" Nov 03 13:37:10.105237 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:10.105144 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6f9ed-6dc5f9df7f-pk6dn" Nov 03 13:37:10.450565 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:10.450534 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f6f9ed-7d67775554-8cvc9"] Nov 03 13:37:10.455368 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:10.455342 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6f9ed-7d67775554-8cvc9" Nov 03 13:37:10.481375 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:10.481344 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f6f9ed-6dc5f9df7f-pk6dn"] Nov 03 13:37:10.510811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:10.510781 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f6f9ed-7d67775554-8cvc9"] Nov 03 13:37:10.530705 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:37:10.530669 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaab21c8_09c2_442a_8d48_e70bcff953b2.slice/crio-b506365513209c09d516ccc78c800e5bb451d43a4825d6eea9631b5860e5e4e0 WatchSource:0}: Error finding container b506365513209c09d516ccc78c800e5bb451d43a4825d6eea9631b5860e5e4e0: Status 404 returned error can't find the container with id b506365513209c09d516ccc78c800e5bb451d43a4825d6eea9631b5860e5e4e0 Nov 03 13:37:10.570770 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:10.570737 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhv4v\" (UniqueName: \"kubernetes.io/projected/01dd2dc9-f37f-4f9f-926a-ef0679393cd0-kube-api-access-nhv4v\") pod \"payload-processing-e2e-ipp-a-f6f9ed-7d67775554-8cvc9\" (UID: \"01dd2dc9-f37f-4f9f-926a-ef0679393cd0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f6f9ed-7d67775554-8cvc9" Nov 03 13:37:10.570897 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:10.570858 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01dd2dc9-f37f-4f9f-926a-ef0679393cd0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f6f9ed-7d67775554-8cvc9\" (UID: \"01dd2dc9-f37f-4f9f-926a-ef0679393cd0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f6f9ed-7d67775554-8cvc9" Nov 03 13:37:10.672263 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:10.672228 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhv4v\" (UniqueName: \"kubernetes.io/projected/01dd2dc9-f37f-4f9f-926a-ef0679393cd0-kube-api-access-nhv4v\") pod \"payload-processing-e2e-ipp-a-f6f9ed-7d67775554-8cvc9\" (UID: \"01dd2dc9-f37f-4f9f-926a-ef0679393cd0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f6f9ed-7d67775554-8cvc9" Nov 03 13:37:10.672500 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:10.672308 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01dd2dc9-f37f-4f9f-926a-ef0679393cd0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f6f9ed-7d67775554-8cvc9\" (UID: \"01dd2dc9-f37f-4f9f-926a-ef0679393cd0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f6f9ed-7d67775554-8cvc9" Nov 03 13:37:10.673071 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:10.673046 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01dd2dc9-f37f-4f9f-926a-ef0679393cd0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f6f9ed-7d67775554-8cvc9\" (UID: \"01dd2dc9-f37f-4f9f-926a-ef0679393cd0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f6f9ed-7d67775554-8cvc9" Nov 03 13:37:10.726882 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:10.726804 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhv4v\" (UniqueName: \"kubernetes.io/projected/01dd2dc9-f37f-4f9f-926a-ef0679393cd0-kube-api-access-nhv4v\") pod \"payload-processing-e2e-ipp-a-f6f9ed-7d67775554-8cvc9\" (UID: \"01dd2dc9-f37f-4f9f-926a-ef0679393cd0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f6f9ed-7d67775554-8cvc9" Nov 03 13:37:10.771737 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:10.771703 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6f9ed-7d67775554-8cvc9" Nov 03 13:37:11.096409 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:11.096304 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6f9ed-6dc5f9df7f-pk6dn" event={"ID":"aaab21c8-09c2-442a-8d48-e70bcff953b2","Type":"ContainerStarted","Data":"dde6c10b47578f410c75698b8b769fcb96ba506137403fb3feedb22f5d3183cf"} Nov 03 13:37:11.096862 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:11.096409 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6f9ed-6dc5f9df7f-pk6dn" Nov 03 13:37:11.096862 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:11.096432 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6f9ed-6dc5f9df7f-pk6dn" event={"ID":"aaab21c8-09c2-442a-8d48-e70bcff953b2","Type":"ContainerStarted","Data":"b506365513209c09d516ccc78c800e5bb451d43a4825d6eea9631b5860e5e4e0"} Nov 03 13:37:11.156233 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:11.156191 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f6f9ed-7d67775554-8cvc9"] Nov 03 13:37:11.163956 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:37:11.163929 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01dd2dc9_f37f_4f9f_926a_ef0679393cd0.slice/crio-00db85c4d772abecd3e9865702c3e8fd4e001060765f8b9a60df9c7951e9e7c0 WatchSource:0}: Error finding container 00db85c4d772abecd3e9865702c3e8fd4e001060765f8b9a60df9c7951e9e7c0: Status 404 returned error can't find the container with id 00db85c4d772abecd3e9865702c3e8fd4e001060765f8b9a60df9c7951e9e7c0 Nov 03 13:37:11.209151 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:11.209081 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6f9ed-6dc5f9df7f-pk6dn" podStartSLOduration=2.209063725 podStartE2EDuration="2.209063725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:37:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:37:11.206525902 +0000 UTC m=+1913.217214145" watchObservedRunningTime="2026-11-03 13:37:11.209063725 +0000 UTC m=+1913.219751967" Nov 03 13:37:11.527944 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:11.527903 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-02f3a0-5b5967f877-c652k"] Nov 03 13:37:11.532205 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:11.532177 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-02f3a0-5b5967f877-c652k" Nov 03 13:37:11.582394 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:11.582365 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-02f3a0-serving-cert\"" Nov 03 13:37:11.593220 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:11.593189 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-02f3a0-5b5967f877-c652k"] Nov 03 13:37:11.686126 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:11.686094 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3cd2275-2289-4acc-8966-5b6515421385-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-02f3a0-5b5967f877-c652k\" (UID: \"a3cd2275-2289-4acc-8966-5b6515421385\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-02f3a0-5b5967f877-c652k" Nov 03 13:37:11.686338 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:11.686168 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3cd2275-2289-4acc-8966-5b6515421385-metrics-certs\") pod \"maas-api-e2e-ipp-a-02f3a0-5b5967f877-c652k\" (UID: \"a3cd2275-2289-4acc-8966-5b6515421385\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-02f3a0-5b5967f877-c652k" Nov 03 13:37:11.686338 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:11.686236 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsl58\" (UniqueName: \"kubernetes.io/projected/a3cd2275-2289-4acc-8966-5b6515421385-kube-api-access-vsl58\") pod \"maas-api-e2e-ipp-a-02f3a0-5b5967f877-c652k\" (UID: \"a3cd2275-2289-4acc-8966-5b6515421385\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-02f3a0-5b5967f877-c652k" Nov 03 13:37:11.686338 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:11.686294 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3cd2275-2289-4acc-8966-5b6515421385-maas-api-tls\") pod \"maas-api-e2e-ipp-a-02f3a0-5b5967f877-c652k\" (UID: \"a3cd2275-2289-4acc-8966-5b6515421385\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-02f3a0-5b5967f877-c652k" Nov 03 13:37:11.762932 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:11.762906 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s" Nov 03 13:37:11.787491 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:11.787436 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3cd2275-2289-4acc-8966-5b6515421385-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-02f3a0-5b5967f877-c652k\" (UID: \"a3cd2275-2289-4acc-8966-5b6515421385\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-02f3a0-5b5967f877-c652k" Nov 03 13:37:11.787653 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:11.787507 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3cd2275-2289-4acc-8966-5b6515421385-metrics-certs\") pod \"maas-api-e2e-ipp-a-02f3a0-5b5967f877-c652k\" (UID: \"a3cd2275-2289-4acc-8966-5b6515421385\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-02f3a0-5b5967f877-c652k" Nov 03 13:37:11.787653 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:11.787548 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsl58\" (UniqueName: \"kubernetes.io/projected/a3cd2275-2289-4acc-8966-5b6515421385-kube-api-access-vsl58\") pod \"maas-api-e2e-ipp-a-02f3a0-5b5967f877-c652k\" (UID: \"a3cd2275-2289-4acc-8966-5b6515421385\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-02f3a0-5b5967f877-c652k" Nov 03 13:37:11.787772 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:11.787697 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3cd2275-2289-4acc-8966-5b6515421385-maas-api-tls\") pod \"maas-api-e2e-ipp-a-02f3a0-5b5967f877-c652k\" (UID: \"a3cd2275-2289-4acc-8966-5b6515421385\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-02f3a0-5b5967f877-c652k" Nov 03 13:37:11.788253 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:11.788228 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3cd2275-2289-4acc-8966-5b6515421385-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-02f3a0-5b5967f877-c652k\" (UID: \"a3cd2275-2289-4acc-8966-5b6515421385\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-02f3a0-5b5967f877-c652k" Nov 03 13:37:11.790099 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:11.790076 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3cd2275-2289-4acc-8966-5b6515421385-metrics-certs\") pod \"maas-api-e2e-ipp-a-02f3a0-5b5967f877-c652k\" (UID: \"a3cd2275-2289-4acc-8966-5b6515421385\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-02f3a0-5b5967f877-c652k" Nov 03 13:37:11.790181 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:11.790105 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3cd2275-2289-4acc-8966-5b6515421385-maas-api-tls\") pod \"maas-api-e2e-ipp-a-02f3a0-5b5967f877-c652k\" (UID: \"a3cd2275-2289-4acc-8966-5b6515421385\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-02f3a0-5b5967f877-c652k" Nov 03 13:37:11.845348 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:11.845315 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsl58\" (UniqueName: \"kubernetes.io/projected/a3cd2275-2289-4acc-8966-5b6515421385-kube-api-access-vsl58\") pod \"maas-api-e2e-ipp-a-02f3a0-5b5967f877-c652k\" (UID: \"a3cd2275-2289-4acc-8966-5b6515421385\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-02f3a0-5b5967f877-c652k" Nov 03 13:37:11.869796 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:11.869757 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-02f3a0-5b5967f877-c652k" Nov 03 13:37:11.888404 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:11.888376 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80fa8184-94c4-4ce1-8354-92681a69a6bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80fa8184-94c4-4ce1-8354-92681a69a6bf-istio-envoy\") pod \"80fa8184-94c4-4ce1-8354-92681a69a6bf\" (UID: \"80fa8184-94c4-4ce1-8354-92681a69a6bf\") " Nov 03 13:37:11.888522 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:11.888437 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80fa8184-94c4-4ce1-8354-92681a69a6bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80fa8184-94c4-4ce1-8354-92681a69a6bf-workload-certs\") pod \"80fa8184-94c4-4ce1-8354-92681a69a6bf\" (UID: \"80fa8184-94c4-4ce1-8354-92681a69a6bf\") " Nov 03 13:37:11.888522 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:11.888479 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80fa8184-94c4-4ce1-8354-92681a69a6bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80fa8184-94c4-4ce1-8354-92681a69a6bf-credential-socket\") pod \"80fa8184-94c4-4ce1-8354-92681a69a6bf\" (UID: \"80fa8184-94c4-4ce1-8354-92681a69a6bf\") " Nov 03 13:37:11.888635 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:11.888524 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80fa8184-94c4-4ce1-8354-92681a69a6bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80fa8184-94c4-4ce1-8354-92681a69a6bf-istiod-ca-cert\") pod \"80fa8184-94c4-4ce1-8354-92681a69a6bf\" (UID: \"80fa8184-94c4-4ce1-8354-92681a69a6bf\") " Nov 03 13:37:11.888863 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:11.888829 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80fa8184-94c4-4ce1-8354-92681a69a6bf-credential-socket" pod "80fa8184-94c4-4ce1-8354-92681a69a6bf" (UID: "80fa8184-94c4-4ce1-8354-92681a69a6bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:37:11.888962 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:11.888943 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80fa8184-94c4-4ce1-8354-92681a69a6bf-istio-token\" (UniqueName: \"kubernetes.io/projected/80fa8184-94c4-4ce1-8354-92681a69a6bf-istio-token\") pod \"80fa8184-94c4-4ce1-8354-92681a69a6bf\" (UID: \"80fa8184-94c4-4ce1-8354-92681a69a6bf\") " Nov 03 13:37:11.889194 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:11.889164 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80fa8184-94c4-4ce1-8354-92681a69a6bf-istiod-ca-cert" pod "80fa8184-94c4-4ce1-8354-92681a69a6bf" (UID: "80fa8184-94c4-4ce1-8354-92681a69a6bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:37:11.889337 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:11.889316 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80fa8184-94c4-4ce1-8354-92681a69a6bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80fa8184-94c4-4ce1-8354-92681a69a6bf-istio-data\") pod \"80fa8184-94c4-4ce1-8354-92681a69a6bf\" (UID: \"80fa8184-94c4-4ce1-8354-92681a69a6bf\") " Nov 03 13:37:11.889535 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:11.889512 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80fa8184-94c4-4ce1-8354-92681a69a6bf-istio-data" pod "80fa8184-94c4-4ce1-8354-92681a69a6bf" (UID: "80fa8184-94c4-4ce1-8354-92681a69a6bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:37:11.889632 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:11.889609 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80fa8184-94c4-4ce1-8354-92681a69a6bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80fa8184-94c4-4ce1-8354-92681a69a6bf-istio-podinfo\") pod \"80fa8184-94c4-4ce1-8354-92681a69a6bf\" (UID: \"80fa8184-94c4-4ce1-8354-92681a69a6bf\") " Nov 03 13:37:11.890010 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:11.889987 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80fa8184-94c4-4ce1-8354-92681a69a6bf-kube-api-access-s5c25\" (UniqueName: \"kubernetes.io/projected/80fa8184-94c4-4ce1-8354-92681a69a6bf-kube-api-access-s5c25\") pod \"80fa8184-94c4-4ce1-8354-92681a69a6bf\" (UID: \"80fa8184-94c4-4ce1-8354-92681a69a6bf\") " Nov 03 13:37:11.890078 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:11.890044 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80fa8184-94c4-4ce1-8354-92681a69a6bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80fa8184-94c4-4ce1-8354-92681a69a6bf-workload-socket\") pod \"80fa8184-94c4-4ce1-8354-92681a69a6bf\" (UID: \"80fa8184-94c4-4ce1-8354-92681a69a6bf\") " Nov 03 13:37:11.890703 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:11.890676 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80fa8184-94c4-4ce1-8354-92681a69a6bf-workload-socket" pod "80fa8184-94c4-4ce1-8354-92681a69a6bf" (UID: "80fa8184-94c4-4ce1-8354-92681a69a6bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:37:11.890800 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:11.890705 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80fa8184-94c4-4ce1-8354-92681a69a6bf-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:11.890800 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:11.890774 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80fa8184-94c4-4ce1-8354-92681a69a6bf-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:11.890800 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:11.890791 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80fa8184-94c4-4ce1-8354-92681a69a6bf-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:11.890996 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:11.890974 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80fa8184-94c4-4ce1-8354-92681a69a6bf-workload-certs" pod "80fa8184-94c4-4ce1-8354-92681a69a6bf" (UID: "80fa8184-94c4-4ce1-8354-92681a69a6bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:37:11.891680 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:11.891641 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80fa8184-94c4-4ce1-8354-92681a69a6bf-istio-token" pod "80fa8184-94c4-4ce1-8354-92681a69a6bf" (UID: "80fa8184-94c4-4ce1-8354-92681a69a6bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:37:11.891984 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:11.891957 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80fa8184-94c4-4ce1-8354-92681a69a6bf-istio-envoy" pod "80fa8184-94c4-4ce1-8354-92681a69a6bf" (UID: "80fa8184-94c4-4ce1-8354-92681a69a6bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:37:11.893201 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:11.893179 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80fa8184-94c4-4ce1-8354-92681a69a6bf-istio-podinfo" pod "80fa8184-94c4-4ce1-8354-92681a69a6bf" (UID: "80fa8184-94c4-4ce1-8354-92681a69a6bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:37:11.893308 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:11.893233 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80fa8184-94c4-4ce1-8354-92681a69a6bf-kube-api-access-s5c25" pod "80fa8184-94c4-4ce1-8354-92681a69a6bf" (UID: "80fa8184-94c4-4ce1-8354-92681a69a6bf"). InnerVolumeSpecName "kube-api-access-s5c25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:37:12.012543 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:11.999684 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80fa8184-94c4-4ce1-8354-92681a69a6bf-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:12.012543 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:11.999723 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5c25\" (UniqueName: \"kubernetes.io/projected/80fa8184-94c4-4ce1-8354-92681a69a6bf-kube-api-access-s5c25\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:12.012543 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:11.999739 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80fa8184-94c4-4ce1-8354-92681a69a6bf-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:12.012543 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:11.999755 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80fa8184-94c4-4ce1-8354-92681a69a6bf-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:12.012543 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:11.999769 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80fa8184-94c4-4ce1-8354-92681a69a6bf-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:12.012543 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:11.999781 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80fa8184-94c4-4ce1-8354-92681a69a6bf-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:12.106822 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:12.106785 5979 generic.go:356] "Generic (PLEG): container finished" podID="80fa8184-94c4-4ce1-8354-92681a69a6bf" containerID="03374afcced2e1fa408e68b3ec85774ff85263d64eef1a300e9ff4c12ecc9ea9" exitCode=0 Nov 03 13:37:12.107251 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:12.106841 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s" Nov 03 13:37:12.107251 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:12.106861 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s" event={"ID":"80fa8184-94c4-4ce1-8354-92681a69a6bf","Type":"ContainerDied","Data":"03374afcced2e1fa408e68b3ec85774ff85263d64eef1a300e9ff4c12ecc9ea9"} Nov 03 13:37:12.107251 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:12.106915 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s" event={"ID":"80fa8184-94c4-4ce1-8354-92681a69a6bf","Type":"ContainerDied","Data":"d2a8feb67827203babc5a40232622c272c897df25aab03e8dbf5d814ebfefa65"} Nov 03 13:37:12.107251 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:12.106943 5979 scope.go:122] "RemoveContainer" containerID="03374afcced2e1fa408e68b3ec85774ff85263d64eef1a300e9ff4c12ecc9ea9" Nov 03 13:37:12.120420 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:12.120390 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6f9ed-7d67775554-8cvc9" event={"ID":"01dd2dc9-f37f-4f9f-926a-ef0679393cd0","Type":"ContainerStarted","Data":"0b20210355d9ab1a98cfbfb21af83060a10f4960e48974f72341c8102f7ffb5a"} Nov 03 13:37:12.120651 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:12.120436 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6f9ed-7d67775554-8cvc9" event={"ID":"01dd2dc9-f37f-4f9f-926a-ef0679393cd0","Type":"ContainerStarted","Data":"00db85c4d772abecd3e9865702c3e8fd4e001060765f8b9a60df9c7951e9e7c0"} Nov 03 13:37:12.120651 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:12.120580 5979 scope.go:122] "RemoveContainer" containerID="4d964944b8213e907ff08f64d1e48efce13f570076c250aa322688d326ad21c3" Nov 03 13:37:12.130182 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:12.130157 5979 scope.go:122] "RemoveContainer" containerID="03374afcced2e1fa408e68b3ec85774ff85263d64eef1a300e9ff4c12ecc9ea9" Nov 03 13:37:12.130482 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:37:12.130449 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03374afcced2e1fa408e68b3ec85774ff85263d64eef1a300e9ff4c12ecc9ea9\": container with ID starting with 03374afcced2e1fa408e68b3ec85774ff85263d64eef1a300e9ff4c12ecc9ea9 not found: ID does not exist" containerID="03374afcced2e1fa408e68b3ec85774ff85263d64eef1a300e9ff4c12ecc9ea9" Nov 03 13:37:12.130580 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:12.130511 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03374afcced2e1fa408e68b3ec85774ff85263d64eef1a300e9ff4c12ecc9ea9"} err="failed to get container status \"03374afcced2e1fa408e68b3ec85774ff85263d64eef1a300e9ff4c12ecc9ea9\": rpc error: code = NotFound desc = could not find container \"03374afcced2e1fa408e68b3ec85774ff85263d64eef1a300e9ff4c12ecc9ea9\": container with ID starting with 03374afcced2e1fa408e68b3ec85774ff85263d64eef1a300e9ff4c12ecc9ea9 not found: ID does not exist" Nov 03 13:37:12.130580 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:12.130538 5979 scope.go:122] "RemoveContainer" containerID="4d964944b8213e907ff08f64d1e48efce13f570076c250aa322688d326ad21c3" Nov 03 13:37:12.130808 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:37:12.130791 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d964944b8213e907ff08f64d1e48efce13f570076c250aa322688d326ad21c3\": container with ID starting with 4d964944b8213e907ff08f64d1e48efce13f570076c250aa322688d326ad21c3 not found: ID does not exist" containerID="4d964944b8213e907ff08f64d1e48efce13f570076c250aa322688d326ad21c3" Nov 03 13:37:12.130857 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:12.130813 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d964944b8213e907ff08f64d1e48efce13f570076c250aa322688d326ad21c3"} err="failed to get container status \"4d964944b8213e907ff08f64d1e48efce13f570076c250aa322688d326ad21c3\": rpc error: code = NotFound desc = could not find container \"4d964944b8213e907ff08f64d1e48efce13f570076c250aa322688d326ad21c3\": container with ID starting with 4d964944b8213e907ff08f64d1e48efce13f570076c250aa322688d326ad21c3 not found: ID does not exist" Nov 03 13:37:12.255450 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:12.255411 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-02f3a0-5b5967f877-c652k"] Nov 03 13:37:12.263014 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:37:12.262941 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3cd2275_2289_4acc_8966_5b6515421385.slice/crio-75fb7067d031071bb36500ed90fdee01420ab65ae5bbac6d47212bba6fd6b131 WatchSource:0}: Error finding container 75fb7067d031071bb36500ed90fdee01420ab65ae5bbac6d47212bba6fd6b131: Status 404 returned error can't find the container with id 75fb7067d031071bb36500ed90fdee01420ab65ae5bbac6d47212bba6fd6b131 Nov 03 13:37:12.263694 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:12.263640 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s"] Nov 03 13:37:12.302454 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:12.302424 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-fa2568-openshift-default-55dc8f4cd7-zh64s"] Nov 03 13:37:12.409711 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:12.409650 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6f9ed-7d67775554-8cvc9" podStartSLOduration=2.409632987 podStartE2EDuration="2.409632987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:37:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:37:12.408731967 +0000 UTC m=+1914.419420209" watchObservedRunningTime="2026-11-03 13:37:12.409632987 +0000 UTC m=+1914.420321229" Nov 03 13:37:13.119534 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:13.119494 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-02f3a0-5b5967f877-c652k" event={"ID":"a3cd2275-2289-4acc-8966-5b6515421385","Type":"ContainerStarted","Data":"75fb7067d031071bb36500ed90fdee01420ab65ae5bbac6d47212bba6fd6b131"} Nov 03 13:37:13.119928 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:13.119584 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6f9ed-7d67775554-8cvc9" Nov 03 13:37:14.121778 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:14.121738 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80fa8184-94c4-4ce1-8354-92681a69a6bf" path="/var/lib/kubelet/pods/80fa8184-94c4-4ce1-8354-92681a69a6bf/volumes" Nov 03 13:37:14.136122 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:14.136087 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-02f3a0-5b5967f877-c652k" event={"ID":"a3cd2275-2289-4acc-8966-5b6515421385","Type":"ContainerStarted","Data":"ebc88bd82454bd54097a33883fa8cd26665a67bb02197764c2138761c00baf66"} Nov 03 13:37:14.248140 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:14.248085 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-02f3a0-5b5967f877-c652k" podStartSLOduration=2.293021644 podStartE2EDuration="3.248070963s" totalImagesPullingTime="955.049319ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:37:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:37:14.246044424 +0000 UTC m=+1916.256732665" watchObservedRunningTime="2026-11-03 13:37:14.248070963 +0000 UTC m=+1916.258759214" Nov 03 13:37:15.142352 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:15.142315 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-02f3a0-5b5967f877-c652k" Nov 03 13:37:19.446896 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:19.446860 5979 scope.go:122] "RemoveContainer" containerID="89e0e2d1884b48753ce12355a85f607bd069a97190b666803bca5f1ed5225809" Nov 03 13:37:19.456585 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:19.456566 5979 scope.go:122] "RemoveContainer" containerID="76eeebf7398bc59db926c5f1d80974c839f9e1eece61f8808de63574975c4b5c" Nov 03 13:37:21.154825 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:21.154796 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-02f3a0-5b5967f877-c652k" Nov 03 13:37:22.119924 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:22.119887 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6f9ed-6dc5f9df7f-pk6dn" Nov 03 13:37:22.217981 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:22.217950 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c8192-55bbd4cd78-8jfgv"] Nov 03 13:37:22.218384 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:22.218136 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c8192-55bbd4cd78-8jfgv" podUID="e13199dc-0be9-4635-92f7-61706f259156" containerName="maas-api" containerID="cri-o://ccf8eb24a6436b1056bc39276c0ad1053ef98443753507d21ce816047403b676" gracePeriod=30 Nov 03 13:37:22.373594 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:22.373499 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c8192-55bbd4cd78-8jfgv" podUID="e13199dc-0be9-4635-92f7-61706f259156" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.174:8443/health\": dial tcp 10.134.0.174:8443: connect: connection refused" Nov 03 13:37:22.536080 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:22.536053 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c8192-55bbd4cd78-8jfgv" Nov 03 13:37:22.665381 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:22.665335 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e13199dc-0be9-4635-92f7-61706f259156-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e13199dc-0be9-4635-92f7-61706f259156-metrics-certs\") pod \"e13199dc-0be9-4635-92f7-61706f259156\" (UID: \"e13199dc-0be9-4635-92f7-61706f259156\") " Nov 03 13:37:22.665618 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:22.665543 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e13199dc-0be9-4635-92f7-61706f259156-kube-api-access-kp9t5\" (UniqueName: \"kubernetes.io/projected/e13199dc-0be9-4635-92f7-61706f259156-kube-api-access-kp9t5\") pod \"e13199dc-0be9-4635-92f7-61706f259156\" (UID: \"e13199dc-0be9-4635-92f7-61706f259156\") " Nov 03 13:37:22.665790 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:22.665770 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e13199dc-0be9-4635-92f7-61706f259156-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e13199dc-0be9-4635-92f7-61706f259156-openshift-service-ca\") pod \"e13199dc-0be9-4635-92f7-61706f259156\" (UID: \"e13199dc-0be9-4635-92f7-61706f259156\") " Nov 03 13:37:22.665874 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:22.665827 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e13199dc-0be9-4635-92f7-61706f259156-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e13199dc-0be9-4635-92f7-61706f259156-maas-api-tls\") pod \"e13199dc-0be9-4635-92f7-61706f259156\" (UID: \"e13199dc-0be9-4635-92f7-61706f259156\") " Nov 03 13:37:22.669231 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:22.669202 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e13199dc-0be9-4635-92f7-61706f259156-maas-api-tls" pod "e13199dc-0be9-4635-92f7-61706f259156" (UID: "e13199dc-0be9-4635-92f7-61706f259156"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:37:22.669513 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:22.669460 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e13199dc-0be9-4635-92f7-61706f259156-openshift-service-ca" pod "e13199dc-0be9-4635-92f7-61706f259156" (UID: "e13199dc-0be9-4635-92f7-61706f259156"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:37:22.672713 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:22.671624 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e13199dc-0be9-4635-92f7-61706f259156-metrics-certs" pod "e13199dc-0be9-4635-92f7-61706f259156" (UID: "e13199dc-0be9-4635-92f7-61706f259156"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:37:22.673887 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:22.673861 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e13199dc-0be9-4635-92f7-61706f259156-kube-api-access-kp9t5" pod "e13199dc-0be9-4635-92f7-61706f259156" (UID: "e13199dc-0be9-4635-92f7-61706f259156"). InnerVolumeSpecName "kube-api-access-kp9t5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:37:22.768153 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:22.768116 5979 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e13199dc-0be9-4635-92f7-61706f259156-openshift-service-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:22.768153 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:22.768157 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e13199dc-0be9-4635-92f7-61706f259156-maas-api-tls\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:22.768365 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:22.768173 5979 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e13199dc-0be9-4635-92f7-61706f259156-metrics-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:22.768365 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:22.768185 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kp9t5\" (UniqueName: \"kubernetes.io/projected/e13199dc-0be9-4635-92f7-61706f259156-kube-api-access-kp9t5\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:23.253453 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:23.253401 5979 generic.go:356] "Generic (PLEG): container finished" podID="e13199dc-0be9-4635-92f7-61706f259156" containerID="ccf8eb24a6436b1056bc39276c0ad1053ef98443753507d21ce816047403b676" exitCode=0 Nov 03 13:37:23.253987 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:23.253667 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c8192-55bbd4cd78-8jfgv" event={"ID":"e13199dc-0be9-4635-92f7-61706f259156","Type":"ContainerDied","Data":"ccf8eb24a6436b1056bc39276c0ad1053ef98443753507d21ce816047403b676"} Nov 03 13:37:23.253987 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:23.253731 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c8192-55bbd4cd78-8jfgv" event={"ID":"e13199dc-0be9-4635-92f7-61706f259156","Type":"ContainerDied","Data":"378e1e98a280252dc7a15a36d3a2dc8161a7625fa9f80906619141b798977f48"} Nov 03 13:37:23.253987 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:23.253766 5979 scope.go:122] "RemoveContainer" containerID="ccf8eb24a6436b1056bc39276c0ad1053ef98443753507d21ce816047403b676" Nov 03 13:37:23.253987 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:23.253940 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c8192-55bbd4cd78-8jfgv" Nov 03 13:37:23.278410 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:23.278383 5979 scope.go:122] "RemoveContainer" containerID="ccf8eb24a6436b1056bc39276c0ad1053ef98443753507d21ce816047403b676" Nov 03 13:37:23.279041 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:37:23.279005 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccf8eb24a6436b1056bc39276c0ad1053ef98443753507d21ce816047403b676\": container with ID starting with ccf8eb24a6436b1056bc39276c0ad1053ef98443753507d21ce816047403b676 not found: ID does not exist" containerID="ccf8eb24a6436b1056bc39276c0ad1053ef98443753507d21ce816047403b676" Nov 03 13:37:23.279245 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:23.279048 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccf8eb24a6436b1056bc39276c0ad1053ef98443753507d21ce816047403b676"} err="failed to get container status \"ccf8eb24a6436b1056bc39276c0ad1053ef98443753507d21ce816047403b676\": rpc error: code = NotFound desc = could not find container \"ccf8eb24a6436b1056bc39276c0ad1053ef98443753507d21ce816047403b676\": container with ID starting with ccf8eb24a6436b1056bc39276c0ad1053ef98443753507d21ce816047403b676 not found: ID does not exist" Nov 03 13:37:23.414513 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:23.414467 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c8192-55bbd4cd78-8jfgv"] Nov 03 13:37:23.423552 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:23.423522 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5c8192-55bbd4cd78-8jfgv"] Nov 03 13:37:23.823076 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:23.823040 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5474b7-75c7f99fd7-lvxls"] Nov 03 13:37:23.823736 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:23.823712 5979 cpu_manager.go:414] "containerMap: removing container" podUID="80fa8184-94c4-4ce1-8354-92681a69a6bf" containerName="istio-proxy" Nov 03 13:37:23.823736 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:23.823730 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80fa8184-94c4-4ce1-8354-92681a69a6bf" containerName="istio-proxy" Nov 03 13:37:23.823845 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:23.823767 5979 cpu_manager.go:414] "containerMap: removing container" podUID="80fa8184-94c4-4ce1-8354-92681a69a6bf" containerName="istio-proxy" Nov 03 13:37:23.823845 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:23.823773 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80fa8184-94c4-4ce1-8354-92681a69a6bf" containerName="istio-proxy" Nov 03 13:37:23.823845 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:23.823795 5979 cpu_manager.go:414] "containerMap: removing container" podUID="e13199dc-0be9-4635-92f7-61706f259156" containerName="maas-api" Nov 03 13:37:23.823845 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:23.823801 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e13199dc-0be9-4635-92f7-61706f259156" containerName="maas-api" Nov 03 13:37:23.823988 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:23.823885 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="80fa8184-94c4-4ce1-8354-92681a69a6bf" containerName="istio-proxy" Nov 03 13:37:23.823988 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:23.823903 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="e13199dc-0be9-4635-92f7-61706f259156" containerName="maas-api" Nov 03 13:37:23.823988 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:23.823913 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="80fa8184-94c4-4ce1-8354-92681a69a6bf" containerName="istio-proxy" Nov 03 13:37:23.828763 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:23.828739 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5474b7-75c7f99fd7-lvxls" Nov 03 13:37:23.879275 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:23.879242 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-5474b7-serving-cert\"" Nov 03 13:37:23.883261 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:23.883236 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5474b7-75c7f99fd7-lvxls"] Nov 03 13:37:23.886557 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:23.886527 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e90f590-e39e-4e48-9aa7-215be078347b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5474b7-75c7f99fd7-lvxls\" (UID: \"0e90f590-e39e-4e48-9aa7-215be078347b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5474b7-75c7f99fd7-lvxls" Nov 03 13:37:23.886698 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:23.886563 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e90f590-e39e-4e48-9aa7-215be078347b-metrics-certs\") pod \"maas-api-e2e-ipp-b-5474b7-75c7f99fd7-lvxls\" (UID: \"0e90f590-e39e-4e48-9aa7-215be078347b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5474b7-75c7f99fd7-lvxls" Nov 03 13:37:23.886698 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:23.886680 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e90f590-e39e-4e48-9aa7-215be078347b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5474b7-75c7f99fd7-lvxls\" (UID: \"0e90f590-e39e-4e48-9aa7-215be078347b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5474b7-75c7f99fd7-lvxls" Nov 03 13:37:23.886835 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:23.886736 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts6wc\" (UniqueName: \"kubernetes.io/projected/0e90f590-e39e-4e48-9aa7-215be078347b-kube-api-access-ts6wc\") pod \"maas-api-e2e-ipp-b-5474b7-75c7f99fd7-lvxls\" (UID: \"0e90f590-e39e-4e48-9aa7-215be078347b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5474b7-75c7f99fd7-lvxls" Nov 03 13:37:23.988226 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:23.988189 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e90f590-e39e-4e48-9aa7-215be078347b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5474b7-75c7f99fd7-lvxls\" (UID: \"0e90f590-e39e-4e48-9aa7-215be078347b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5474b7-75c7f99fd7-lvxls" Nov 03 13:37:23.988226 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:23.988230 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e90f590-e39e-4e48-9aa7-215be078347b-metrics-certs\") pod \"maas-api-e2e-ipp-b-5474b7-75c7f99fd7-lvxls\" (UID: \"0e90f590-e39e-4e48-9aa7-215be078347b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5474b7-75c7f99fd7-lvxls" Nov 03 13:37:23.988483 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:23.988294 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e90f590-e39e-4e48-9aa7-215be078347b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5474b7-75c7f99fd7-lvxls\" (UID: \"0e90f590-e39e-4e48-9aa7-215be078347b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5474b7-75c7f99fd7-lvxls" Nov 03 13:37:23.988483 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:23.988337 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts6wc\" (UniqueName: \"kubernetes.io/projected/0e90f590-e39e-4e48-9aa7-215be078347b-kube-api-access-ts6wc\") pod \"maas-api-e2e-ipp-b-5474b7-75c7f99fd7-lvxls\" (UID: \"0e90f590-e39e-4e48-9aa7-215be078347b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5474b7-75c7f99fd7-lvxls" Nov 03 13:37:23.988710 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:37:23.988682 5979 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-5474b7-serving-cert: secret "maas-api-e2e-ipp-b-5474b7-serving-cert" not found Nov 03 13:37:23.988849 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:37:23.988775 5979 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e90f590-e39e-4e48-9aa7-215be078347b-maas-api-tls podName:0e90f590-e39e-4e48-9aa7-215be078347b nodeName:}" failed. No retries permitted until 2026-11-03 13:37:24.48875428 +0000 UTC m=+1926.499442504 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0e90f590-e39e-4e48-9aa7-215be078347b-maas-api-tls") pod "maas-api-e2e-ipp-b-5474b7-75c7f99fd7-lvxls" (UID: "0e90f590-e39e-4e48-9aa7-215be078347b") : secret "maas-api-e2e-ipp-b-5474b7-serving-cert" not found Nov 03 13:37:23.989136 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:23.989111 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e90f590-e39e-4e48-9aa7-215be078347b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5474b7-75c7f99fd7-lvxls\" (UID: \"0e90f590-e39e-4e48-9aa7-215be078347b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5474b7-75c7f99fd7-lvxls" Nov 03 13:37:23.990914 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:23.990893 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e90f590-e39e-4e48-9aa7-215be078347b-metrics-certs\") pod \"maas-api-e2e-ipp-b-5474b7-75c7f99fd7-lvxls\" (UID: \"0e90f590-e39e-4e48-9aa7-215be078347b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5474b7-75c7f99fd7-lvxls" Nov 03 13:37:24.044395 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:24.044359 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts6wc\" (UniqueName: \"kubernetes.io/projected/0e90f590-e39e-4e48-9aa7-215be078347b-kube-api-access-ts6wc\") pod \"maas-api-e2e-ipp-b-5474b7-75c7f99fd7-lvxls\" (UID: \"0e90f590-e39e-4e48-9aa7-215be078347b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5474b7-75c7f99fd7-lvxls" Nov 03 13:37:24.158494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:24.158446 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e13199dc-0be9-4635-92f7-61706f259156" path="/var/lib/kubelet/pods/e13199dc-0be9-4635-92f7-61706f259156/volumes" Nov 03 13:37:24.158933 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:24.158908 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6f9ed-7d67775554-8cvc9" Nov 03 13:37:24.497555 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:24.497440 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e90f590-e39e-4e48-9aa7-215be078347b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5474b7-75c7f99fd7-lvxls\" (UID: \"0e90f590-e39e-4e48-9aa7-215be078347b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5474b7-75c7f99fd7-lvxls" Nov 03 13:37:24.499858 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:24.499837 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e90f590-e39e-4e48-9aa7-215be078347b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5474b7-75c7f99fd7-lvxls\" (UID: \"0e90f590-e39e-4e48-9aa7-215be078347b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5474b7-75c7f99fd7-lvxls" Nov 03 13:37:24.742487 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:24.742445 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5474b7-75c7f99fd7-lvxls" Nov 03 13:37:25.126902 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:37:25.126850 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e90f590_e39e_4e48_9aa7_215be078347b.slice/crio-5bd44f6d9a0900c5d74f43d0f7142886b9aee3c482e5dea73e0ab34b2172e430 WatchSource:0}: Error finding container 5bd44f6d9a0900c5d74f43d0f7142886b9aee3c482e5dea73e0ab34b2172e430: Status 404 returned error can't find the container with id 5bd44f6d9a0900c5d74f43d0f7142886b9aee3c482e5dea73e0ab34b2172e430 Nov 03 13:37:25.132109 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:25.132084 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5474b7-75c7f99fd7-lvxls"] Nov 03 13:37:25.295543 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:25.295501 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5474b7-75c7f99fd7-lvxls" event={"ID":"0e90f590-e39e-4e48-9aa7-215be078347b","Type":"ContainerStarted","Data":"5bd44f6d9a0900c5d74f43d0f7142886b9aee3c482e5dea73e0ab34b2172e430"} Nov 03 13:37:26.304606 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:26.304502 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5474b7-75c7f99fd7-lvxls" event={"ID":"0e90f590-e39e-4e48-9aa7-215be078347b","Type":"ContainerStarted","Data":"bb7ab0470c9fbe796aa14715f32f5915a213b185ba11286a599e9a5abab29583"} Nov 03 13:37:26.305071 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:26.304689 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5474b7-75c7f99fd7-lvxls" Nov 03 13:37:26.420098 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:26.420033 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5474b7-75c7f99fd7-lvxls" podStartSLOduration=2.583855494 podStartE2EDuration="3.420015409s" totalImagesPullingTime="836.159915ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:37:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:37:26.417385934 +0000 UTC m=+1928.428074231" watchObservedRunningTime="2026-11-03 13:37:26.420015409 +0000 UTC m=+1928.430703653" Nov 03 13:37:31.303462 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:31.303427 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr"] Nov 03 13:37:31.304007 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:31.303674 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr" podUID="7fd6dca0-648c-4476-a1a5-8a7407bf598f" containerName="istio-proxy" containerID="cri-o://b3e45af3b10566cc23f0ca3f1a964ad58edef29584104bd0730f57fe418d4c72" gracePeriod=30 Nov 03 13:37:31.584655 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:31.583569 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8"] Nov 03 13:37:31.584655 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:31.583856 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8" podUID="337bd1a1-184a-4942-a921-b6a7a0df9130" containerName="istio-proxy" containerID="cri-o://d110700aec531b08e67013b3016382c18cc02599adfba48b658ce7f088e7a1ce" gracePeriod=30 Nov 03 13:37:31.596266 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:31.594572 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st"] Nov 03 13:37:31.596266 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:31.594860 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st" podUID="be094f6c-c6d9-4a53-8c71-33acdca60856" containerName="istio-proxy" containerID="cri-o://ea98d4a6d893c9e781d1c4307b043cd6d67333b62972779fd65eb089738a092e" gracePeriod=30 Nov 03 13:37:32.317411 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:32.317382 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5474b7-75c7f99fd7-lvxls" Nov 03 13:37:36.590764 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:36.590737 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr" Nov 03 13:37:36.702429 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:36.702397 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fd6dca0-648c-4476-a1a5-8a7407bf598f-istio-token\" (UniqueName: \"kubernetes.io/projected/7fd6dca0-648c-4476-a1a5-8a7407bf598f-istio-token\") pod \"7fd6dca0-648c-4476-a1a5-8a7407bf598f\" (UID: \"7fd6dca0-648c-4476-a1a5-8a7407bf598f\") " Nov 03 13:37:36.702638 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:36.702459 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fd6dca0-648c-4476-a1a5-8a7407bf598f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fd6dca0-648c-4476-a1a5-8a7407bf598f-workload-certs\") pod \"7fd6dca0-648c-4476-a1a5-8a7407bf598f\" (UID: \"7fd6dca0-648c-4476-a1a5-8a7407bf598f\") " Nov 03 13:37:36.702638 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:36.702556 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7fd6dca0-648c-4476-a1a5-8a7407bf598f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fd6dca0-648c-4476-a1a5-8a7407bf598f-istio-podinfo\") pod \"7fd6dca0-648c-4476-a1a5-8a7407bf598f\" (UID: \"7fd6dca0-648c-4476-a1a5-8a7407bf598f\") " Nov 03 13:37:36.702788 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:36.702639 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fd6dca0-648c-4476-a1a5-8a7407bf598f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fd6dca0-648c-4476-a1a5-8a7407bf598f-credential-socket\") pod \"7fd6dca0-648c-4476-a1a5-8a7407bf598f\" (UID: \"7fd6dca0-648c-4476-a1a5-8a7407bf598f\") " Nov 03 13:37:36.702788 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:36.702698 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fd6dca0-648c-4476-a1a5-8a7407bf598f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fd6dca0-648c-4476-a1a5-8a7407bf598f-istio-data\") pod \"7fd6dca0-648c-4476-a1a5-8a7407bf598f\" (UID: \"7fd6dca0-648c-4476-a1a5-8a7407bf598f\") " Nov 03 13:37:36.702788 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:36.702740 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fd6dca0-648c-4476-a1a5-8a7407bf598f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fd6dca0-648c-4476-a1a5-8a7407bf598f-istio-envoy\") pod \"7fd6dca0-648c-4476-a1a5-8a7407bf598f\" (UID: \"7fd6dca0-648c-4476-a1a5-8a7407bf598f\") " Nov 03 13:37:36.702958 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:36.702834 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fd6dca0-648c-4476-a1a5-8a7407bf598f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fd6dca0-648c-4476-a1a5-8a7407bf598f-workload-socket\") pod \"7fd6dca0-648c-4476-a1a5-8a7407bf598f\" (UID: \"7fd6dca0-648c-4476-a1a5-8a7407bf598f\") " Nov 03 13:37:36.702958 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:36.702889 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fd6dca0-648c-4476-a1a5-8a7407bf598f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fd6dca0-648c-4476-a1a5-8a7407bf598f-istiod-ca-cert\") pod \"7fd6dca0-648c-4476-a1a5-8a7407bf598f\" (UID: \"7fd6dca0-648c-4476-a1a5-8a7407bf598f\") " Nov 03 13:37:36.702958 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:36.702935 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fd6dca0-648c-4476-a1a5-8a7407bf598f-kube-api-access-zc8p2\" (UniqueName: \"kubernetes.io/projected/7fd6dca0-648c-4476-a1a5-8a7407bf598f-kube-api-access-zc8p2\") pod \"7fd6dca0-648c-4476-a1a5-8a7407bf598f\" (UID: \"7fd6dca0-648c-4476-a1a5-8a7407bf598f\") " Nov 03 13:37:36.703112 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:36.702827 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fd6dca0-648c-4476-a1a5-8a7407bf598f-workload-certs" pod "7fd6dca0-648c-4476-a1a5-8a7407bf598f" (UID: "7fd6dca0-648c-4476-a1a5-8a7407bf598f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:37:36.703112 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:36.702912 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fd6dca0-648c-4476-a1a5-8a7407bf598f-credential-socket" pod "7fd6dca0-648c-4476-a1a5-8a7407bf598f" (UID: "7fd6dca0-648c-4476-a1a5-8a7407bf598f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:37:36.703112 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:36.703034 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fd6dca0-648c-4476-a1a5-8a7407bf598f-istio-data" pod "7fd6dca0-648c-4476-a1a5-8a7407bf598f" (UID: "7fd6dca0-648c-4476-a1a5-8a7407bf598f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:37:36.703326 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:36.703302 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fd6dca0-648c-4476-a1a5-8a7407bf598f-workload-socket" pod "7fd6dca0-648c-4476-a1a5-8a7407bf598f" (UID: "7fd6dca0-648c-4476-a1a5-8a7407bf598f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:37:36.703734 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:36.703621 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fd6dca0-648c-4476-a1a5-8a7407bf598f-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:36.703734 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:36.703646 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fd6dca0-648c-4476-a1a5-8a7407bf598f-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:36.703734 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:36.703660 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fd6dca0-648c-4476-a1a5-8a7407bf598f-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:36.703734 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:36.703675 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fd6dca0-648c-4476-a1a5-8a7407bf598f-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:36.703734 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:36.703703 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fd6dca0-648c-4476-a1a5-8a7407bf598f-istiod-ca-cert" pod "7fd6dca0-648c-4476-a1a5-8a7407bf598f" (UID: "7fd6dca0-648c-4476-a1a5-8a7407bf598f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:37:36.705013 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:36.704988 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fd6dca0-648c-4476-a1a5-8a7407bf598f-istio-token" pod "7fd6dca0-648c-4476-a1a5-8a7407bf598f" (UID: "7fd6dca0-648c-4476-a1a5-8a7407bf598f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:37:36.705133 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:36.705071 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7fd6dca0-648c-4476-a1a5-8a7407bf598f-istio-podinfo" pod "7fd6dca0-648c-4476-a1a5-8a7407bf598f" (UID: "7fd6dca0-648c-4476-a1a5-8a7407bf598f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:37:36.705374 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:36.705351 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fd6dca0-648c-4476-a1a5-8a7407bf598f-istio-envoy" pod "7fd6dca0-648c-4476-a1a5-8a7407bf598f" (UID: "7fd6dca0-648c-4476-a1a5-8a7407bf598f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:37:36.705606 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:36.705592 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fd6dca0-648c-4476-a1a5-8a7407bf598f-kube-api-access-zc8p2" pod "7fd6dca0-648c-4476-a1a5-8a7407bf598f" (UID: "7fd6dca0-648c-4476-a1a5-8a7407bf598f"). InnerVolumeSpecName "kube-api-access-zc8p2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:37:36.805812 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:36.805776 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fd6dca0-648c-4476-a1a5-8a7407bf598f-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:36.805948 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:36.805816 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fd6dca0-648c-4476-a1a5-8a7407bf598f-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:36.805948 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:36.805830 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fd6dca0-648c-4476-a1a5-8a7407bf598f-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:36.805948 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:36.805843 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc8p2\" (UniqueName: \"kubernetes.io/projected/7fd6dca0-648c-4476-a1a5-8a7407bf598f-kube-api-access-zc8p2\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:36.805948 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:36.805894 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fd6dca0-648c-4476-a1a5-8a7407bf598f-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:36.867835 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:36.867806 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8" Nov 03 13:37:36.903980 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:36.903956 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st" Nov 03 13:37:37.009456 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.009410 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be094f6c-c6d9-4a53-8c71-33acdca60856-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be094f6c-c6d9-4a53-8c71-33acdca60856-workload-certs\") pod \"be094f6c-c6d9-4a53-8c71-33acdca60856\" (UID: \"be094f6c-c6d9-4a53-8c71-33acdca60856\") " Nov 03 13:37:37.009691 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.009547 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/337bd1a1-184a-4942-a921-b6a7a0df9130-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/337bd1a1-184a-4942-a921-b6a7a0df9130-istio-podinfo\") pod \"337bd1a1-184a-4942-a921-b6a7a0df9130\" (UID: \"337bd1a1-184a-4942-a921-b6a7a0df9130\") " Nov 03 13:37:37.009691 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.009627 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be094f6c-c6d9-4a53-8c71-33acdca60856-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be094f6c-c6d9-4a53-8c71-33acdca60856-istio-podinfo\") pod \"be094f6c-c6d9-4a53-8c71-33acdca60856\" (UID: \"be094f6c-c6d9-4a53-8c71-33acdca60856\") " Nov 03 13:37:37.009691 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.009665 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/337bd1a1-184a-4942-a921-b6a7a0df9130-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/337bd1a1-184a-4942-a921-b6a7a0df9130-credential-socket\") pod \"337bd1a1-184a-4942-a921-b6a7a0df9130\" (UID: \"337bd1a1-184a-4942-a921-b6a7a0df9130\") " Nov 03 13:37:37.009877 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.009705 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/337bd1a1-184a-4942-a921-b6a7a0df9130-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/337bd1a1-184a-4942-a921-b6a7a0df9130-workload-certs\") pod \"337bd1a1-184a-4942-a921-b6a7a0df9130\" (UID: \"337bd1a1-184a-4942-a921-b6a7a0df9130\") " Nov 03 13:37:37.009877 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.009754 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/337bd1a1-184a-4942-a921-b6a7a0df9130-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/337bd1a1-184a-4942-a921-b6a7a0df9130-istiod-ca-cert\") pod \"337bd1a1-184a-4942-a921-b6a7a0df9130\" (UID: \"337bd1a1-184a-4942-a921-b6a7a0df9130\") " Nov 03 13:37:37.009877 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.009754 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be094f6c-c6d9-4a53-8c71-33acdca60856-workload-certs" pod "be094f6c-c6d9-4a53-8c71-33acdca60856" (UID: "be094f6c-c6d9-4a53-8c71-33acdca60856"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:37:37.009877 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.009780 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/337bd1a1-184a-4942-a921-b6a7a0df9130-kube-api-access-rdsmh\" (UniqueName: \"kubernetes.io/projected/337bd1a1-184a-4942-a921-b6a7a0df9130-kube-api-access-rdsmh\") pod \"337bd1a1-184a-4942-a921-b6a7a0df9130\" (UID: \"337bd1a1-184a-4942-a921-b6a7a0df9130\") " Nov 03 13:37:37.009877 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.009798 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be094f6c-c6d9-4a53-8c71-33acdca60856-istio-token\" (UniqueName: \"kubernetes.io/projected/be094f6c-c6d9-4a53-8c71-33acdca60856-istio-token\") pod \"be094f6c-c6d9-4a53-8c71-33acdca60856\" (UID: \"be094f6c-c6d9-4a53-8c71-33acdca60856\") " Nov 03 13:37:37.009877 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.009841 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be094f6c-c6d9-4a53-8c71-33acdca60856-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be094f6c-c6d9-4a53-8c71-33acdca60856-istio-envoy\") pod \"be094f6c-c6d9-4a53-8c71-33acdca60856\" (UID: \"be094f6c-c6d9-4a53-8c71-33acdca60856\") " Nov 03 13:37:37.009877 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.009870 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/337bd1a1-184a-4942-a921-b6a7a0df9130-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/337bd1a1-184a-4942-a921-b6a7a0df9130-istio-envoy\") pod \"337bd1a1-184a-4942-a921-b6a7a0df9130\" (UID: \"337bd1a1-184a-4942-a921-b6a7a0df9130\") " Nov 03 13:37:37.010263 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.009922 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/337bd1a1-184a-4942-a921-b6a7a0df9130-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/337bd1a1-184a-4942-a921-b6a7a0df9130-istio-data\") pod \"337bd1a1-184a-4942-a921-b6a7a0df9130\" (UID: \"337bd1a1-184a-4942-a921-b6a7a0df9130\") " Nov 03 13:37:37.010263 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.009978 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be094f6c-c6d9-4a53-8c71-33acdca60856-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be094f6c-c6d9-4a53-8c71-33acdca60856-workload-socket\") pod \"be094f6c-c6d9-4a53-8c71-33acdca60856\" (UID: \"be094f6c-c6d9-4a53-8c71-33acdca60856\") " Nov 03 13:37:37.010263 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.010013 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be094f6c-c6d9-4a53-8c71-33acdca60856-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be094f6c-c6d9-4a53-8c71-33acdca60856-istio-data\") pod \"be094f6c-c6d9-4a53-8c71-33acdca60856\" (UID: \"be094f6c-c6d9-4a53-8c71-33acdca60856\") " Nov 03 13:37:37.010263 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.010073 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be094f6c-c6d9-4a53-8c71-33acdca60856-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be094f6c-c6d9-4a53-8c71-33acdca60856-credential-socket\") pod \"be094f6c-c6d9-4a53-8c71-33acdca60856\" (UID: \"be094f6c-c6d9-4a53-8c71-33acdca60856\") " Nov 03 13:37:37.010263 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.010122 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be094f6c-c6d9-4a53-8c71-33acdca60856-kube-api-access-wxvjl\" (UniqueName: \"kubernetes.io/projected/be094f6c-c6d9-4a53-8c71-33acdca60856-kube-api-access-wxvjl\") pod \"be094f6c-c6d9-4a53-8c71-33acdca60856\" (UID: \"be094f6c-c6d9-4a53-8c71-33acdca60856\") " Nov 03 13:37:37.010263 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.010152 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/337bd1a1-184a-4942-a921-b6a7a0df9130-istio-token\" (UniqueName: \"kubernetes.io/projected/337bd1a1-184a-4942-a921-b6a7a0df9130-istio-token\") pod \"337bd1a1-184a-4942-a921-b6a7a0df9130\" (UID: \"337bd1a1-184a-4942-a921-b6a7a0df9130\") " Nov 03 13:37:37.010263 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.010204 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/337bd1a1-184a-4942-a921-b6a7a0df9130-credential-socket" pod "337bd1a1-184a-4942-a921-b6a7a0df9130" (UID: "337bd1a1-184a-4942-a921-b6a7a0df9130"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:37:37.010263 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.010227 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be094f6c-c6d9-4a53-8c71-33acdca60856-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be094f6c-c6d9-4a53-8c71-33acdca60856-istiod-ca-cert\") pod \"be094f6c-c6d9-4a53-8c71-33acdca60856\" (UID: \"be094f6c-c6d9-4a53-8c71-33acdca60856\") " Nov 03 13:37:37.010723 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.010328 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/337bd1a1-184a-4942-a921-b6a7a0df9130-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/337bd1a1-184a-4942-a921-b6a7a0df9130-workload-socket\") pod \"337bd1a1-184a-4942-a921-b6a7a0df9130\" (UID: \"337bd1a1-184a-4942-a921-b6a7a0df9130\") " Nov 03 13:37:37.010723 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.010340 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/337bd1a1-184a-4942-a921-b6a7a0df9130-istio-data" pod "337bd1a1-184a-4942-a921-b6a7a0df9130" (UID: "337bd1a1-184a-4942-a921-b6a7a0df9130"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:37:37.010723 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.010427 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be094f6c-c6d9-4a53-8c71-33acdca60856-credential-socket" pod "be094f6c-c6d9-4a53-8c71-33acdca60856" (UID: "be094f6c-c6d9-4a53-8c71-33acdca60856"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:37:37.010723 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.010680 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be094f6c-c6d9-4a53-8c71-33acdca60856-workload-socket" pod "be094f6c-c6d9-4a53-8c71-33acdca60856" (UID: "be094f6c-c6d9-4a53-8c71-33acdca60856"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:37:37.011041 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.011010 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be094f6c-c6d9-4a53-8c71-33acdca60856-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:37.011041 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.011043 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be094f6c-c6d9-4a53-8c71-33acdca60856-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:37.011237 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.011059 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/337bd1a1-184a-4942-a921-b6a7a0df9130-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:37.011237 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.011073 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/337bd1a1-184a-4942-a921-b6a7a0df9130-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:37.011237 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.011087 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be094f6c-c6d9-4a53-8c71-33acdca60856-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:37.011410 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.011331 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be094f6c-c6d9-4a53-8c71-33acdca60856-istio-data" pod "be094f6c-c6d9-4a53-8c71-33acdca60856" (UID: "be094f6c-c6d9-4a53-8c71-33acdca60856"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:37:37.011689 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.011665 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be094f6c-c6d9-4a53-8c71-33acdca60856-istiod-ca-cert" pod "be094f6c-c6d9-4a53-8c71-33acdca60856" (UID: "be094f6c-c6d9-4a53-8c71-33acdca60856"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:37:37.012253 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.012228 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/337bd1a1-184a-4942-a921-b6a7a0df9130-workload-certs" pod "337bd1a1-184a-4942-a921-b6a7a0df9130" (UID: "337bd1a1-184a-4942-a921-b6a7a0df9130"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:37:37.012655 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.012627 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/337bd1a1-184a-4942-a921-b6a7a0df9130-workload-socket" pod "337bd1a1-184a-4942-a921-b6a7a0df9130" (UID: "337bd1a1-184a-4942-a921-b6a7a0df9130"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:37:37.012887 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.012865 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/337bd1a1-184a-4942-a921-b6a7a0df9130-istiod-ca-cert" pod "337bd1a1-184a-4942-a921-b6a7a0df9130" (UID: "337bd1a1-184a-4942-a921-b6a7a0df9130"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:37:37.013018 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.012957 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be094f6c-c6d9-4a53-8c71-33acdca60856-istio-podinfo" pod "be094f6c-c6d9-4a53-8c71-33acdca60856" (UID: "be094f6c-c6d9-4a53-8c71-33acdca60856"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:37:37.013018 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.013009 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/337bd1a1-184a-4942-a921-b6a7a0df9130-kube-api-access-rdsmh" pod "337bd1a1-184a-4942-a921-b6a7a0df9130" (UID: "337bd1a1-184a-4942-a921-b6a7a0df9130"). InnerVolumeSpecName "kube-api-access-rdsmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:37:37.013764 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.013737 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/337bd1a1-184a-4942-a921-b6a7a0df9130-istio-token" pod "337bd1a1-184a-4942-a921-b6a7a0df9130" (UID: "337bd1a1-184a-4942-a921-b6a7a0df9130"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:37:37.014349 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.014324 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be094f6c-c6d9-4a53-8c71-33acdca60856-istio-envoy" pod "be094f6c-c6d9-4a53-8c71-33acdca60856" (UID: "be094f6c-c6d9-4a53-8c71-33acdca60856"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:37:37.014534 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.014511 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/337bd1a1-184a-4942-a921-b6a7a0df9130-istio-envoy" pod "337bd1a1-184a-4942-a921-b6a7a0df9130" (UID: "337bd1a1-184a-4942-a921-b6a7a0df9130"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:37:37.015589 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.015457 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/337bd1a1-184a-4942-a921-b6a7a0df9130-istio-podinfo" pod "337bd1a1-184a-4942-a921-b6a7a0df9130" (UID: "337bd1a1-184a-4942-a921-b6a7a0df9130"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:37:37.015589 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.015556 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be094f6c-c6d9-4a53-8c71-33acdca60856-istio-token" pod "be094f6c-c6d9-4a53-8c71-33acdca60856" (UID: "be094f6c-c6d9-4a53-8c71-33acdca60856"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:37:37.015751 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.015594 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be094f6c-c6d9-4a53-8c71-33acdca60856-kube-api-access-wxvjl" pod "be094f6c-c6d9-4a53-8c71-33acdca60856" (UID: "be094f6c-c6d9-4a53-8c71-33acdca60856"). InnerVolumeSpecName "kube-api-access-wxvjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:37:37.112074 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.111988 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxvjl\" (UniqueName: \"kubernetes.io/projected/be094f6c-c6d9-4a53-8c71-33acdca60856-kube-api-access-wxvjl\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:37.112074 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.112019 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/337bd1a1-184a-4942-a921-b6a7a0df9130-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:37.112074 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.112029 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be094f6c-c6d9-4a53-8c71-33acdca60856-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:37.112074 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.112038 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/337bd1a1-184a-4942-a921-b6a7a0df9130-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:37.112074 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.112047 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/337bd1a1-184a-4942-a921-b6a7a0df9130-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:37.112074 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.112056 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be094f6c-c6d9-4a53-8c71-33acdca60856-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:37.112074 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.112064 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/337bd1a1-184a-4942-a921-b6a7a0df9130-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:37.112074 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.112072 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/337bd1a1-184a-4942-a921-b6a7a0df9130-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:37.112074 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.112080 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdsmh\" (UniqueName: \"kubernetes.io/projected/337bd1a1-184a-4942-a921-b6a7a0df9130-kube-api-access-rdsmh\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:37.112788 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.112088 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be094f6c-c6d9-4a53-8c71-33acdca60856-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:37.112788 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.112096 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be094f6c-c6d9-4a53-8c71-33acdca60856-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:37.112788 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.112104 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/337bd1a1-184a-4942-a921-b6a7a0df9130-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:37.112788 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.112112 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be094f6c-c6d9-4a53-8c71-33acdca60856-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:37:37.264346 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:37:37.264308 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:37:37.378968 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.378884 5979 generic.go:356] "Generic (PLEG): container finished" podID="be094f6c-c6d9-4a53-8c71-33acdca60856" containerID="ea98d4a6d893c9e781d1c4307b043cd6d67333b62972779fd65eb089738a092e" exitCode=0 Nov 03 13:37:37.378968 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.378946 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st" Nov 03 13:37:37.379212 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.378964 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st" event={"ID":"be094f6c-c6d9-4a53-8c71-33acdca60856","Type":"ContainerDied","Data":"ea98d4a6d893c9e781d1c4307b043cd6d67333b62972779fd65eb089738a092e"} Nov 03 13:37:37.379212 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.379021 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st" event={"ID":"be094f6c-c6d9-4a53-8c71-33acdca60856","Type":"ContainerDied","Data":"76706258da6bb1473e3f53d716b8150a13d083d100098cb85356a706604a20ca"} Nov 03 13:37:37.379212 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.379053 5979 scope.go:122] "RemoveContainer" containerID="ea98d4a6d893c9e781d1c4307b043cd6d67333b62972779fd65eb089738a092e" Nov 03 13:37:37.380974 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.380950 5979 generic.go:356] "Generic (PLEG): container finished" podID="7fd6dca0-648c-4476-a1a5-8a7407bf598f" containerID="b3e45af3b10566cc23f0ca3f1a964ad58edef29584104bd0730f57fe418d4c72" exitCode=0 Nov 03 13:37:37.381101 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.381018 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr" Nov 03 13:37:37.381101 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.381037 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr" event={"ID":"7fd6dca0-648c-4476-a1a5-8a7407bf598f","Type":"ContainerDied","Data":"b3e45af3b10566cc23f0ca3f1a964ad58edef29584104bd0730f57fe418d4c72"} Nov 03 13:37:37.381101 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.381086 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr" event={"ID":"7fd6dca0-648c-4476-a1a5-8a7407bf598f","Type":"ContainerDied","Data":"f7803d2e0d6c26f3bcbb042204d6d3c571d6e7f4b852eb7c4770c0828b17e072"} Nov 03 13:37:37.382780 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.382756 5979 generic.go:356] "Generic (PLEG): container finished" podID="337bd1a1-184a-4942-a921-b6a7a0df9130" containerID="d110700aec531b08e67013b3016382c18cc02599adfba48b658ce7f088e7a1ce" exitCode=0 Nov 03 13:37:37.382916 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.382807 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8" event={"ID":"337bd1a1-184a-4942-a921-b6a7a0df9130","Type":"ContainerDied","Data":"d110700aec531b08e67013b3016382c18cc02599adfba48b658ce7f088e7a1ce"} Nov 03 13:37:37.382916 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.382828 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8" Nov 03 13:37:37.382916 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.382843 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8" event={"ID":"337bd1a1-184a-4942-a921-b6a7a0df9130","Type":"ContainerDied","Data":"61bfd55a0268e6fc6447332a108cb7cfdf309c936d03f11ad1975de114a63509"} Nov 03 13:37:37.395733 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.395707 5979 scope.go:122] "RemoveContainer" containerID="ea98d4a6d893c9e781d1c4307b043cd6d67333b62972779fd65eb089738a092e" Nov 03 13:37:37.396103 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:37:37.396077 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea98d4a6d893c9e781d1c4307b043cd6d67333b62972779fd65eb089738a092e\": container with ID starting with ea98d4a6d893c9e781d1c4307b043cd6d67333b62972779fd65eb089738a092e not found: ID does not exist" containerID="ea98d4a6d893c9e781d1c4307b043cd6d67333b62972779fd65eb089738a092e" Nov 03 13:37:37.396204 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.396113 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea98d4a6d893c9e781d1c4307b043cd6d67333b62972779fd65eb089738a092e"} err="failed to get container status \"ea98d4a6d893c9e781d1c4307b043cd6d67333b62972779fd65eb089738a092e\": rpc error: code = NotFound desc = could not find container \"ea98d4a6d893c9e781d1c4307b043cd6d67333b62972779fd65eb089738a092e\": container with ID starting with ea98d4a6d893c9e781d1c4307b043cd6d67333b62972779fd65eb089738a092e not found: ID does not exist" Nov 03 13:37:37.396204 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.396130 5979 scope.go:122] "RemoveContainer" containerID="b3e45af3b10566cc23f0ca3f1a964ad58edef29584104bd0730f57fe418d4c72" Nov 03 13:37:37.406520 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.406497 5979 scope.go:122] "RemoveContainer" containerID="c0696ee584a93c5517435d6824775f80ed90c756889e69f557c8b761b8295daf" Nov 03 13:37:37.415792 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.415767 5979 scope.go:122] "RemoveContainer" containerID="b3e45af3b10566cc23f0ca3f1a964ad58edef29584104bd0730f57fe418d4c72" Nov 03 13:37:37.416110 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:37:37.416085 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3e45af3b10566cc23f0ca3f1a964ad58edef29584104bd0730f57fe418d4c72\": container with ID starting with b3e45af3b10566cc23f0ca3f1a964ad58edef29584104bd0730f57fe418d4c72 not found: ID does not exist" containerID="b3e45af3b10566cc23f0ca3f1a964ad58edef29584104bd0730f57fe418d4c72" Nov 03 13:37:37.416185 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.416124 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3e45af3b10566cc23f0ca3f1a964ad58edef29584104bd0730f57fe418d4c72"} err="failed to get container status \"b3e45af3b10566cc23f0ca3f1a964ad58edef29584104bd0730f57fe418d4c72\": rpc error: code = NotFound desc = could not find container \"b3e45af3b10566cc23f0ca3f1a964ad58edef29584104bd0730f57fe418d4c72\": container with ID starting with b3e45af3b10566cc23f0ca3f1a964ad58edef29584104bd0730f57fe418d4c72 not found: ID does not exist" Nov 03 13:37:37.416185 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.416149 5979 scope.go:122] "RemoveContainer" containerID="c0696ee584a93c5517435d6824775f80ed90c756889e69f557c8b761b8295daf" Nov 03 13:37:37.416433 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:37:37.416415 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0696ee584a93c5517435d6824775f80ed90c756889e69f557c8b761b8295daf\": container with ID starting with c0696ee584a93c5517435d6824775f80ed90c756889e69f557c8b761b8295daf not found: ID does not exist" containerID="c0696ee584a93c5517435d6824775f80ed90c756889e69f557c8b761b8295daf" Nov 03 13:37:37.416520 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.416443 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0696ee584a93c5517435d6824775f80ed90c756889e69f557c8b761b8295daf"} err="failed to get container status \"c0696ee584a93c5517435d6824775f80ed90c756889e69f557c8b761b8295daf\": rpc error: code = NotFound desc = could not find container \"c0696ee584a93c5517435d6824775f80ed90c756889e69f557c8b761b8295daf\": container with ID starting with c0696ee584a93c5517435d6824775f80ed90c756889e69f557c8b761b8295daf not found: ID does not exist" Nov 03 13:37:37.416520 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.416459 5979 scope.go:122] "RemoveContainer" containerID="d110700aec531b08e67013b3016382c18cc02599adfba48b658ce7f088e7a1ce" Nov 03 13:37:37.426573 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.426553 5979 scope.go:122] "RemoveContainer" containerID="d110700aec531b08e67013b3016382c18cc02599adfba48b658ce7f088e7a1ce" Nov 03 13:37:37.426897 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:37:37.426874 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d110700aec531b08e67013b3016382c18cc02599adfba48b658ce7f088e7a1ce\": container with ID starting with d110700aec531b08e67013b3016382c18cc02599adfba48b658ce7f088e7a1ce not found: ID does not exist" containerID="d110700aec531b08e67013b3016382c18cc02599adfba48b658ce7f088e7a1ce" Nov 03 13:37:37.426995 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.426900 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d110700aec531b08e67013b3016382c18cc02599adfba48b658ce7f088e7a1ce"} err="failed to get container status \"d110700aec531b08e67013b3016382c18cc02599adfba48b658ce7f088e7a1ce\": rpc error: code = NotFound desc = could not find container \"d110700aec531b08e67013b3016382c18cc02599adfba48b658ce7f088e7a1ce\": container with ID starting with d110700aec531b08e67013b3016382c18cc02599adfba48b658ce7f088e7a1ce not found: ID does not exist" Nov 03 13:37:37.535903 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.535860 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr"] Nov 03 13:37:37.540957 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.540927 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-5c8192-openshift-default-5ff47bb7b9-68nnr"] Nov 03 13:37:37.643620 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.643584 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st"] Nov 03 13:37:37.649466 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.649440 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-6l5st"] Nov 03 13:37:37.753348 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.753303 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8"] Nov 03 13:37:37.755875 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:37.755848 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-c4wd8"] Nov 03 13:37:38.117059 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:38.116966 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="337bd1a1-184a-4942-a921-b6a7a0df9130" path="/var/lib/kubelet/pods/337bd1a1-184a-4942-a921-b6a7a0df9130/volumes" Nov 03 13:37:38.117570 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:38.117550 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fd6dca0-648c-4476-a1a5-8a7407bf598f" path="/var/lib/kubelet/pods/7fd6dca0-648c-4476-a1a5-8a7407bf598f/volumes" Nov 03 13:37:38.118055 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:38.118035 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be094f6c-c6d9-4a53-8c71-33acdca60856" path="/var/lib/kubelet/pods/be094f6c-c6d9-4a53-8c71-33acdca60856/volumes" Nov 03 13:37:41.919859 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:41.919819 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-21f111-5f9499d695-qpxbk"] Nov 03 13:37:41.920463 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:41.920442 5979 cpu_manager.go:414] "containerMap: removing container" podUID="be094f6c-c6d9-4a53-8c71-33acdca60856" containerName="istio-proxy" Nov 03 13:37:41.920463 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:41.920461 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be094f6c-c6d9-4a53-8c71-33acdca60856" containerName="istio-proxy" Nov 03 13:37:41.920672 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:41.920508 5979 cpu_manager.go:414] "containerMap: removing container" podUID="7fd6dca0-648c-4476-a1a5-8a7407bf598f" containerName="istio-proxy" Nov 03 13:37:41.920672 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:41.920513 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fd6dca0-648c-4476-a1a5-8a7407bf598f" containerName="istio-proxy" Nov 03 13:37:41.920672 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:41.920526 5979 cpu_manager.go:414] "containerMap: removing container" podUID="7fd6dca0-648c-4476-a1a5-8a7407bf598f" containerName="istio-proxy" Nov 03 13:37:41.920672 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:41.920533 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fd6dca0-648c-4476-a1a5-8a7407bf598f" containerName="istio-proxy" Nov 03 13:37:41.920672 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:41.920545 5979 cpu_manager.go:414] "containerMap: removing container" podUID="337bd1a1-184a-4942-a921-b6a7a0df9130" containerName="istio-proxy" Nov 03 13:37:41.920672 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:41.920549 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="337bd1a1-184a-4942-a921-b6a7a0df9130" containerName="istio-proxy" Nov 03 13:37:41.920672 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:41.920647 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="be094f6c-c6d9-4a53-8c71-33acdca60856" containerName="istio-proxy" Nov 03 13:37:41.920672 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:41.920661 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fd6dca0-648c-4476-a1a5-8a7407bf598f" containerName="istio-proxy" Nov 03 13:37:41.920672 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:41.920668 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fd6dca0-648c-4476-a1a5-8a7407bf598f" containerName="istio-proxy" Nov 03 13:37:41.920672 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:41.920675 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="337bd1a1-184a-4942-a921-b6a7a0df9130" containerName="istio-proxy" Nov 03 13:37:41.924138 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:41.924118 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-21f111-5f9499d695-qpxbk" Nov 03 13:37:41.974881 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:41.974846 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c61c5a2c-8b5d-4c88-96d3-add6b701eb0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-21f111-5f9499d695-qpxbk\" (UID: \"c61c5a2c-8b5d-4c88-96d3-add6b701eb0f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-21f111-5f9499d695-qpxbk" Nov 03 13:37:41.975086 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:41.974963 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6rpj\" (UniqueName: \"kubernetes.io/projected/c61c5a2c-8b5d-4c88-96d3-add6b701eb0f-kube-api-access-g6rpj\") pod \"payload-pre-processing-e2e-praxis-a-21f111-5f9499d695-qpxbk\" (UID: \"c61c5a2c-8b5d-4c88-96d3-add6b701eb0f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-21f111-5f9499d695-qpxbk" Nov 03 13:37:42.046537 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:42.021843 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-21f111-dockercfg-nzbbp\"" Nov 03 13:37:42.046892 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:42.038990 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-21f111\"" Nov 03 13:37:42.078109 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:42.078054 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c61c5a2c-8b5d-4c88-96d3-add6b701eb0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-21f111-5f9499d695-qpxbk\" (UID: \"c61c5a2c-8b5d-4c88-96d3-add6b701eb0f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-21f111-5f9499d695-qpxbk" Nov 03 13:37:42.078487 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:42.078442 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6rpj\" (UniqueName: \"kubernetes.io/projected/c61c5a2c-8b5d-4c88-96d3-add6b701eb0f-kube-api-access-g6rpj\") pod \"payload-pre-processing-e2e-praxis-a-21f111-5f9499d695-qpxbk\" (UID: \"c61c5a2c-8b5d-4c88-96d3-add6b701eb0f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-21f111-5f9499d695-qpxbk" Nov 03 13:37:42.079102 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:42.078625 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c61c5a2c-8b5d-4c88-96d3-add6b701eb0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-21f111-5f9499d695-qpxbk\" (UID: \"c61c5a2c-8b5d-4c88-96d3-add6b701eb0f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-21f111-5f9499d695-qpxbk" Nov 03 13:37:42.101522 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:42.092223 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-21f111-5f9499d695-qpxbk"] Nov 03 13:37:42.101522 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:42.100997 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-21f111-5dc5cbd779-vkggd"] Nov 03 13:37:42.106738 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:42.106704 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-21f111-5dc5cbd779-vkggd" Nov 03 13:37:42.130908 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:42.130873 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-21f111-5dc5cbd779-vkggd"] Nov 03 13:37:42.134963 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:42.134935 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6rpj\" (UniqueName: \"kubernetes.io/projected/c61c5a2c-8b5d-4c88-96d3-add6b701eb0f-kube-api-access-g6rpj\") pod \"payload-pre-processing-e2e-praxis-a-21f111-5f9499d695-qpxbk\" (UID: \"c61c5a2c-8b5d-4c88-96d3-add6b701eb0f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-21f111-5f9499d695-qpxbk" Nov 03 13:37:42.180751 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:42.180654 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c355a440-18fd-4f9f-8016-8c679e8f1aa4-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-21f111-5dc5cbd779-vkggd\" (UID: \"c355a440-18fd-4f9f-8016-8c679e8f1aa4\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-21f111-5dc5cbd779-vkggd" Nov 03 13:37:42.180947 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:42.180912 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvm68\" (UniqueName: \"kubernetes.io/projected/c355a440-18fd-4f9f-8016-8c679e8f1aa4-kube-api-access-kvm68\") pod \"payload-processing-e2e-praxis-a-21f111-5dc5cbd779-vkggd\" (UID: \"c355a440-18fd-4f9f-8016-8c679e8f1aa4\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-21f111-5dc5cbd779-vkggd" Nov 03 13:37:42.237718 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:42.237682 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-21f111-5f9499d695-qpxbk" Nov 03 13:37:42.281752 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:42.281713 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvm68\" (UniqueName: \"kubernetes.io/projected/c355a440-18fd-4f9f-8016-8c679e8f1aa4-kube-api-access-kvm68\") pod \"payload-processing-e2e-praxis-a-21f111-5dc5cbd779-vkggd\" (UID: \"c355a440-18fd-4f9f-8016-8c679e8f1aa4\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-21f111-5dc5cbd779-vkggd" Nov 03 13:37:42.281930 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:42.281761 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c355a440-18fd-4f9f-8016-8c679e8f1aa4-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-21f111-5dc5cbd779-vkggd\" (UID: \"c355a440-18fd-4f9f-8016-8c679e8f1aa4\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-21f111-5dc5cbd779-vkggd" Nov 03 13:37:42.282889 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:42.282860 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c355a440-18fd-4f9f-8016-8c679e8f1aa4-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-21f111-5dc5cbd779-vkggd\" (UID: \"c355a440-18fd-4f9f-8016-8c679e8f1aa4\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-21f111-5dc5cbd779-vkggd" Nov 03 13:37:42.339255 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:42.339213 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvm68\" (UniqueName: \"kubernetes.io/projected/c355a440-18fd-4f9f-8016-8c679e8f1aa4-kube-api-access-kvm68\") pod \"payload-processing-e2e-praxis-a-21f111-5dc5cbd779-vkggd\" (UID: \"c355a440-18fd-4f9f-8016-8c679e8f1aa4\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-21f111-5dc5cbd779-vkggd" Nov 03 13:37:42.425411 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:42.425382 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-21f111-5dc5cbd779-vkggd" Nov 03 13:37:42.632524 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:42.632483 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-21f111-5f9499d695-qpxbk"] Nov 03 13:37:42.638673 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:37:42.638647 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc61c5a2c_8b5d_4c88_96d3_add6b701eb0f.slice/crio-a5c02cf54d24abc9162822cd40e40c167eba2bbd10617a02a716a44b9a17dca3 WatchSource:0}: Error finding container a5c02cf54d24abc9162822cd40e40c167eba2bbd10617a02a716a44b9a17dca3: Status 404 returned error can't find the container with id a5c02cf54d24abc9162822cd40e40c167eba2bbd10617a02a716a44b9a17dca3 Nov 03 13:37:42.800621 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:42.800582 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-21f111-5dc5cbd779-vkggd"] Nov 03 13:37:42.807209 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:37:42.807179 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc355a440_18fd_4f9f_8016_8c679e8f1aa4.slice/crio-52f577d5c68fef7ab312353be8d7da0ffa88e821faa0dda23301f8d7bfd22c1d WatchSource:0}: Error finding container 52f577d5c68fef7ab312353be8d7da0ffa88e821faa0dda23301f8d7bfd22c1d: Status 404 returned error can't find the container with id 52f577d5c68fef7ab312353be8d7da0ffa88e821faa0dda23301f8d7bfd22c1d Nov 03 13:37:43.426813 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:43.426778 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-21f111-5dc5cbd779-vkggd" event={"ID":"c355a440-18fd-4f9f-8016-8c679e8f1aa4","Type":"ContainerStarted","Data":"7c60e6140cb78cb263e770bbd6b90670115a44aa54954f06a92a9438577ad364"} Nov 03 13:37:43.427320 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:43.426835 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-21f111-5dc5cbd779-vkggd" event={"ID":"c355a440-18fd-4f9f-8016-8c679e8f1aa4","Type":"ContainerStarted","Data":"52f577d5c68fef7ab312353be8d7da0ffa88e821faa0dda23301f8d7bfd22c1d"} Nov 03 13:37:43.427320 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:43.426893 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-21f111-5dc5cbd779-vkggd" Nov 03 13:37:43.428845 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:43.428820 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-21f111-5f9499d695-qpxbk" event={"ID":"c61c5a2c-8b5d-4c88-96d3-add6b701eb0f","Type":"ContainerStarted","Data":"28bd3d3542ce2a069072316e8e2923698d622d8c15fe3eb58aeb0967ea52b26f"} Nov 03 13:37:43.428967 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:43.428867 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-21f111-5f9499d695-qpxbk" event={"ID":"c61c5a2c-8b5d-4c88-96d3-add6b701eb0f","Type":"ContainerStarted","Data":"a5c02cf54d24abc9162822cd40e40c167eba2bbd10617a02a716a44b9a17dca3"} Nov 03 13:37:43.428967 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:43.428896 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-21f111-5f9499d695-qpxbk" Nov 03 13:37:43.535687 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:43.535616 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-21f111-5dc5cbd779-vkggd" podStartSLOduration=1.535598631 podStartE2EDuration="1.535598631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:37:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:37:43.534312957 +0000 UTC m=+1945.545001199" watchObservedRunningTime="2026-11-03 13:37:43.535598631 +0000 UTC m=+1945.546286870" Nov 03 13:37:43.645122 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:43.645063 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-21f111-5f9499d695-qpxbk" podStartSLOduration=2.645048742 podStartE2EDuration="2.645048742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:37:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:37:43.643502566 +0000 UTC m=+1945.654190799" watchObservedRunningTime="2026-11-03 13:37:43.645048742 +0000 UTC m=+1945.655736983" Nov 03 13:37:47.788680 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:47.788648 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:37:54.440648 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:54.440613 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-21f111-5dc5cbd779-vkggd" Nov 03 13:37:54.441021 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:54.440695 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-21f111-5f9499d695-qpxbk" Nov 03 13:37:57.146248 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:57.146187 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a770b9-65459cd457-28vvf"] Nov 03 13:37:57.253444 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:57.253408 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a770b9-5fbcd499ff-2xsw7"] Nov 03 13:37:58.793636 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:58.792752 5979 patch_prober.go:29] interesting pod/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": read tcp 10.134.0.2:50734->10.134.0.181:15021: read: connection reset by peer" start-of-body= Nov 03 13:37:58.793636 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:58.792814 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf" podUID="efd2a382-b6ea-419f-b33a-9ebfdf0bd61b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": read tcp 10.134.0.2:50734->10.134.0.181:15021: read: connection reset by peer" Nov 03 13:37:58.945035 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:58.945001 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d29e07-5585b5fc95-2hf65"] Nov 03 13:37:58.948594 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:58.948571 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d29e07-5585b5fc95-2hf65" Nov 03 13:37:59.020552 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:59.016954 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-d29e07\"" Nov 03 13:37:59.020552 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:59.018256 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d29e07-5585b5fc95-2hf65"] Nov 03 13:37:59.038085 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:59.038043 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzh9v\" (UniqueName: \"kubernetes.io/projected/7346ca1b-5758-4634-b0b7-19f8b2a52cfb-kube-api-access-bzh9v\") pod \"payload-pre-processing-e2e-ipp-b-d29e07-5585b5fc95-2hf65\" (UID: \"7346ca1b-5758-4634-b0b7-19f8b2a52cfb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d29e07-5585b5fc95-2hf65" Nov 03 13:37:59.038275 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:59.038250 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7346ca1b-5758-4634-b0b7-19f8b2a52cfb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d29e07-5585b5fc95-2hf65\" (UID: \"7346ca1b-5758-4634-b0b7-19f8b2a52cfb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d29e07-5585b5fc95-2hf65" Nov 03 13:37:59.055811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:59.055728 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-d29e07-dockercfg-kjh7x\"" Nov 03 13:37:59.139637 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:59.139593 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7346ca1b-5758-4634-b0b7-19f8b2a52cfb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d29e07-5585b5fc95-2hf65\" (UID: \"7346ca1b-5758-4634-b0b7-19f8b2a52cfb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d29e07-5585b5fc95-2hf65" Nov 03 13:37:59.139908 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:59.139743 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzh9v\" (UniqueName: \"kubernetes.io/projected/7346ca1b-5758-4634-b0b7-19f8b2a52cfb-kube-api-access-bzh9v\") pod \"payload-pre-processing-e2e-ipp-b-d29e07-5585b5fc95-2hf65\" (UID: \"7346ca1b-5758-4634-b0b7-19f8b2a52cfb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d29e07-5585b5fc95-2hf65" Nov 03 13:37:59.140382 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:59.140360 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7346ca1b-5758-4634-b0b7-19f8b2a52cfb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d29e07-5585b5fc95-2hf65\" (UID: \"7346ca1b-5758-4634-b0b7-19f8b2a52cfb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d29e07-5585b5fc95-2hf65" Nov 03 13:37:59.196180 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:59.196141 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzh9v\" (UniqueName: \"kubernetes.io/projected/7346ca1b-5758-4634-b0b7-19f8b2a52cfb-kube-api-access-bzh9v\") pod \"payload-pre-processing-e2e-ipp-b-d29e07-5585b5fc95-2hf65\" (UID: \"7346ca1b-5758-4634-b0b7-19f8b2a52cfb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d29e07-5585b5fc95-2hf65" Nov 03 13:37:59.297955 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:59.297924 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d29e07-5585b5fc95-2hf65" Nov 03 13:37:59.569389 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:59.569309 5979 generic.go:356] "Generic (PLEG): container finished" podID="efd2a382-b6ea-419f-b33a-9ebfdf0bd61b" containerID="d714840db823ea2dea8bdda3ef53b5e66a20c5bc6204dc3ca8b5e2bb2a4b2125" exitCode=0 Nov 03 13:37:59.569548 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:59.569381 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf" event={"ID":"efd2a382-b6ea-419f-b33a-9ebfdf0bd61b","Type":"ContainerDied","Data":"d714840db823ea2dea8bdda3ef53b5e66a20c5bc6204dc3ca8b5e2bb2a4b2125"} Nov 03 13:37:59.569829 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:59.569794 5979 scope.go:122] "RemoveContainer" containerID="d714840db823ea2dea8bdda3ef53b5e66a20c5bc6204dc3ca8b5e2bb2a4b2125" Nov 03 13:37:59.571247 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:59.571219 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:37:59.571325 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:59.571283 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:37:59.571325 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:59.571311 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:37:59.677231 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:59.677196 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d29e07-5585b5fc95-2hf65"] Nov 03 13:37:59.726798 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:59.726568 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d29e07-6b54b7bcd6-lqtkq"] Nov 03 13:37:59.731042 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:59.730905 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d29e07-6b54b7bcd6-lqtkq" Nov 03 13:37:59.793899 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:59.793816 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d29e07-6b54b7bcd6-lqtkq"] Nov 03 13:37:59.849973 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:59.849873 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jdt9\" (UniqueName: \"kubernetes.io/projected/f32a7513-df8c-4286-b484-0b67f7a5329e-kube-api-access-6jdt9\") pod \"payload-processing-e2e-ipp-b-d29e07-6b54b7bcd6-lqtkq\" (UID: \"f32a7513-df8c-4286-b484-0b67f7a5329e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d29e07-6b54b7bcd6-lqtkq" Nov 03 13:37:59.850155 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:59.850047 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f32a7513-df8c-4286-b484-0b67f7a5329e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d29e07-6b54b7bcd6-lqtkq\" (UID: \"f32a7513-df8c-4286-b484-0b67f7a5329e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d29e07-6b54b7bcd6-lqtkq" Nov 03 13:37:59.950994 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:59.950949 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f32a7513-df8c-4286-b484-0b67f7a5329e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d29e07-6b54b7bcd6-lqtkq\" (UID: \"f32a7513-df8c-4286-b484-0b67f7a5329e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d29e07-6b54b7bcd6-lqtkq" Nov 03 13:37:59.951181 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:59.951141 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jdt9\" (UniqueName: \"kubernetes.io/projected/f32a7513-df8c-4286-b484-0b67f7a5329e-kube-api-access-6jdt9\") pod \"payload-processing-e2e-ipp-b-d29e07-6b54b7bcd6-lqtkq\" (UID: \"f32a7513-df8c-4286-b484-0b67f7a5329e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d29e07-6b54b7bcd6-lqtkq" Nov 03 13:37:59.951750 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:37:59.951728 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f32a7513-df8c-4286-b484-0b67f7a5329e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d29e07-6b54b7bcd6-lqtkq\" (UID: \"f32a7513-df8c-4286-b484-0b67f7a5329e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d29e07-6b54b7bcd6-lqtkq" Nov 03 13:38:00.004987 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:00.004959 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jdt9\" (UniqueName: \"kubernetes.io/projected/f32a7513-df8c-4286-b484-0b67f7a5329e-kube-api-access-6jdt9\") pod \"payload-processing-e2e-ipp-b-d29e07-6b54b7bcd6-lqtkq\" (UID: \"f32a7513-df8c-4286-b484-0b67f7a5329e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d29e07-6b54b7bcd6-lqtkq" Nov 03 13:38:00.072220 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:00.072187 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d29e07-6b54b7bcd6-lqtkq" Nov 03 13:38:00.476359 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:00.476266 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d29e07-6b54b7bcd6-lqtkq"] Nov 03 13:38:00.481961 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:38:00.481931 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf32a7513_df8c_4286_b484_0b67f7a5329e.slice/crio-6e829bc99bf1d05ffc0d130161a6948a73bf70e45c9519f6ddc1b5867e3e2be3 WatchSource:0}: Error finding container 6e829bc99bf1d05ffc0d130161a6948a73bf70e45c9519f6ddc1b5867e3e2be3: Status 404 returned error can't find the container with id 6e829bc99bf1d05ffc0d130161a6948a73bf70e45c9519f6ddc1b5867e3e2be3 Nov 03 13:38:00.586541 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:00.586508 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d29e07-5585b5fc95-2hf65" event={"ID":"7346ca1b-5758-4634-b0b7-19f8b2a52cfb","Type":"ContainerStarted","Data":"0ccea9b4faf4a0ab70ddbee861b6c0378c3f5a1020f051b436e31b0ae44873eb"} Nov 03 13:38:00.586750 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:00.586730 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d29e07-5585b5fc95-2hf65" event={"ID":"7346ca1b-5758-4634-b0b7-19f8b2a52cfb","Type":"ContainerStarted","Data":"af891a67a686c3c368ae18ec9110d2e0c68d05f5c1916ed719eb2acc83fb1d88"} Nov 03 13:38:00.586884 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:00.586870 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d29e07-5585b5fc95-2hf65" Nov 03 13:38:00.589525 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:00.588802 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf" event={"ID":"efd2a382-b6ea-419f-b33a-9ebfdf0bd61b","Type":"ContainerStarted","Data":"89dc0798d357859d7d39d3dcae3278d83018164920f3d2ca4344f29b3fcb459d"} Nov 03 13:38:00.590663 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:00.590638 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d29e07-6b54b7bcd6-lqtkq" event={"ID":"f32a7513-df8c-4286-b484-0b67f7a5329e","Type":"ContainerStarted","Data":"6e829bc99bf1d05ffc0d130161a6948a73bf70e45c9519f6ddc1b5867e3e2be3"} Nov 03 13:38:00.697394 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:00.697343 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d29e07-5585b5fc95-2hf65" podStartSLOduration=2.697329587 podStartE2EDuration="2.697329587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:37:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:38:00.695872464 +0000 UTC m=+1962.706560706" watchObservedRunningTime="2026-11-03 13:38:00.697329587 +0000 UTC m=+1962.708017829" Nov 03 13:38:01.485815 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:01.485772 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf" Nov 03 13:38:01.486236 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:01.486003 5979 patch_prober.go:29] interesting pod/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Nov 03 13:38:01.486236 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:01.486056 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf" podUID="efd2a382-b6ea-419f-b33a-9ebfdf0bd61b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Nov 03 13:38:01.600804 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:01.600759 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d29e07-6b54b7bcd6-lqtkq" event={"ID":"f32a7513-df8c-4286-b484-0b67f7a5329e","Type":"ContainerStarted","Data":"4284951c53fff3f5071588d6c09e6d23d5f6d3da8394c21276e67966d88e65c1"} Nov 03 13:38:01.600994 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:01.600829 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d29e07-6b54b7bcd6-lqtkq" Nov 03 13:38:01.717832 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:01.717765 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-d29e07-6b54b7bcd6-lqtkq" podStartSLOduration=2.717750268 podStartE2EDuration="2.717750268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:37:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:38:01.71668687 +0000 UTC m=+1963.727375112" watchObservedRunningTime="2026-11-03 13:38:01.717750268 +0000 UTC m=+1963.728438551" Nov 03 13:38:02.156496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:02.153308 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-a770b9-65459cd457-28vvf" podUID="489c2cad-0beb-4090-8a88-16dce56f5471" containerName="payload-processing" containerID="cri-o://218ebf061f3ce3b153493cbfee49fa293ab7dc7be95165833a4bdb29ee43ae98" gracePeriod=25 Nov 03 13:38:02.269527 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:02.266537 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a770b9-5fbcd499ff-2xsw7" podUID="6be3240b-9b5b-4e17-ada4-676bd02c18ff" containerName="payload-pre-processing" containerID="cri-o://8519cad9e861e3b13f6a4a29d5cd67c0e41fefc95e7c75e71e7203b0c6e84b59" gracePeriod=25 Nov 03 13:38:02.485213 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:02.485127 5979 patch_prober.go:29] interesting pod/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Nov 03 13:38:02.485213 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:02.485188 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf" podUID="efd2a382-b6ea-419f-b33a-9ebfdf0bd61b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Nov 03 13:38:02.492790 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:02.492759 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a770b9-65459cd457-28vvf" Nov 03 13:38:02.605207 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:02.605181 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a770b9-5fbcd499ff-2xsw7" Nov 03 13:38:02.607335 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:02.607309 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/489c2cad-0beb-4090-8a88-16dce56f5471-kube-api-access-9ds6z\" (UniqueName: \"kubernetes.io/projected/489c2cad-0beb-4090-8a88-16dce56f5471-kube-api-access-9ds6z\") pod \"489c2cad-0beb-4090-8a88-16dce56f5471\" (UID: \"489c2cad-0beb-4090-8a88-16dce56f5471\") " Nov 03 13:38:02.607610 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:02.607592 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/489c2cad-0beb-4090-8a88-16dce56f5471-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/489c2cad-0beb-4090-8a88-16dce56f5471-plugins-config-volume\") pod \"489c2cad-0beb-4090-8a88-16dce56f5471\" (UID: \"489c2cad-0beb-4090-8a88-16dce56f5471\") " Nov 03 13:38:02.607983 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:02.607952 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/489c2cad-0beb-4090-8a88-16dce56f5471-plugins-config-volume" pod "489c2cad-0beb-4090-8a88-16dce56f5471" (UID: "489c2cad-0beb-4090-8a88-16dce56f5471"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:38:02.608109 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:02.608089 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/489c2cad-0beb-4090-8a88-16dce56f5471-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:02.609175 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:02.609151 5979 generic.go:356] "Generic (PLEG): container finished" podID="6be3240b-9b5b-4e17-ada4-676bd02c18ff" containerID="8519cad9e861e3b13f6a4a29d5cd67c0e41fefc95e7c75e71e7203b0c6e84b59" exitCode=0 Nov 03 13:38:02.609272 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:02.609192 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a770b9-5fbcd499ff-2xsw7" Nov 03 13:38:02.609272 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:02.609236 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a770b9-5fbcd499ff-2xsw7" event={"ID":"6be3240b-9b5b-4e17-ada4-676bd02c18ff","Type":"ContainerDied","Data":"8519cad9e861e3b13f6a4a29d5cd67c0e41fefc95e7c75e71e7203b0c6e84b59"} Nov 03 13:38:02.609404 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:02.609319 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a770b9-5fbcd499ff-2xsw7" event={"ID":"6be3240b-9b5b-4e17-ada4-676bd02c18ff","Type":"ContainerDied","Data":"88891655528a0703595162ebacd4c128ee86896e72de4fed5fcfb0b9df9e8441"} Nov 03 13:38:02.609404 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:02.609354 5979 scope.go:122] "RemoveContainer" containerID="8519cad9e861e3b13f6a4a29d5cd67c0e41fefc95e7c75e71e7203b0c6e84b59" Nov 03 13:38:02.609720 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:02.609701 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/489c2cad-0beb-4090-8a88-16dce56f5471-kube-api-access-9ds6z" pod "489c2cad-0beb-4090-8a88-16dce56f5471" (UID: "489c2cad-0beb-4090-8a88-16dce56f5471"). InnerVolumeSpecName "kube-api-access-9ds6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:38:02.610562 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:02.610543 5979 generic.go:356] "Generic (PLEG): container finished" podID="489c2cad-0beb-4090-8a88-16dce56f5471" containerID="218ebf061f3ce3b153493cbfee49fa293ab7dc7be95165833a4bdb29ee43ae98" exitCode=0 Nov 03 13:38:02.610626 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:02.610593 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a770b9-65459cd457-28vvf" Nov 03 13:38:02.610626 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:02.610597 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a770b9-65459cd457-28vvf" event={"ID":"489c2cad-0beb-4090-8a88-16dce56f5471","Type":"ContainerDied","Data":"218ebf061f3ce3b153493cbfee49fa293ab7dc7be95165833a4bdb29ee43ae98"} Nov 03 13:38:02.610711 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:02.610641 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a770b9-65459cd457-28vvf" event={"ID":"489c2cad-0beb-4090-8a88-16dce56f5471","Type":"ContainerDied","Data":"04d07e886543b55a4b1a403bfad0adccc754343e4213761e564f8c2faad25054"} Nov 03 13:38:02.620937 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:02.620918 5979 scope.go:122] "RemoveContainer" containerID="8519cad9e861e3b13f6a4a29d5cd67c0e41fefc95e7c75e71e7203b0c6e84b59" Nov 03 13:38:02.621280 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:38:02.621258 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8519cad9e861e3b13f6a4a29d5cd67c0e41fefc95e7c75e71e7203b0c6e84b59\": container with ID starting with 8519cad9e861e3b13f6a4a29d5cd67c0e41fefc95e7c75e71e7203b0c6e84b59 not found: ID does not exist" containerID="8519cad9e861e3b13f6a4a29d5cd67c0e41fefc95e7c75e71e7203b0c6e84b59" Nov 03 13:38:02.621387 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:02.621295 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8519cad9e861e3b13f6a4a29d5cd67c0e41fefc95e7c75e71e7203b0c6e84b59"} err="failed to get container status \"8519cad9e861e3b13f6a4a29d5cd67c0e41fefc95e7c75e71e7203b0c6e84b59\": rpc error: code = NotFound desc = could not find container \"8519cad9e861e3b13f6a4a29d5cd67c0e41fefc95e7c75e71e7203b0c6e84b59\": container with ID starting with 8519cad9e861e3b13f6a4a29d5cd67c0e41fefc95e7c75e71e7203b0c6e84b59 not found: ID does not exist" Nov 03 13:38:02.621387 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:02.621317 5979 scope.go:122] "RemoveContainer" containerID="218ebf061f3ce3b153493cbfee49fa293ab7dc7be95165833a4bdb29ee43ae98" Nov 03 13:38:02.631712 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:02.631686 5979 scope.go:122] "RemoveContainer" containerID="218ebf061f3ce3b153493cbfee49fa293ab7dc7be95165833a4bdb29ee43ae98" Nov 03 13:38:02.632013 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:38:02.631993 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"218ebf061f3ce3b153493cbfee49fa293ab7dc7be95165833a4bdb29ee43ae98\": container with ID starting with 218ebf061f3ce3b153493cbfee49fa293ab7dc7be95165833a4bdb29ee43ae98 not found: ID does not exist" containerID="218ebf061f3ce3b153493cbfee49fa293ab7dc7be95165833a4bdb29ee43ae98" Nov 03 13:38:02.632091 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:02.632025 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"218ebf061f3ce3b153493cbfee49fa293ab7dc7be95165833a4bdb29ee43ae98"} err="failed to get container status \"218ebf061f3ce3b153493cbfee49fa293ab7dc7be95165833a4bdb29ee43ae98\": rpc error: code = NotFound desc = could not find container \"218ebf061f3ce3b153493cbfee49fa293ab7dc7be95165833a4bdb29ee43ae98\": container with ID starting with 218ebf061f3ce3b153493cbfee49fa293ab7dc7be95165833a4bdb29ee43ae98 not found: ID does not exist" Nov 03 13:38:02.709390 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:02.709300 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6be3240b-9b5b-4e17-ada4-676bd02c18ff-kube-api-access-lxj9f\" (UniqueName: \"kubernetes.io/projected/6be3240b-9b5b-4e17-ada4-676bd02c18ff-kube-api-access-lxj9f\") pod \"6be3240b-9b5b-4e17-ada4-676bd02c18ff\" (UID: \"6be3240b-9b5b-4e17-ada4-676bd02c18ff\") " Nov 03 13:38:02.709596 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:02.709438 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6be3240b-9b5b-4e17-ada4-676bd02c18ff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6be3240b-9b5b-4e17-ada4-676bd02c18ff-plugins-config-volume\") pod \"6be3240b-9b5b-4e17-ada4-676bd02c18ff\" (UID: \"6be3240b-9b5b-4e17-ada4-676bd02c18ff\") " Nov 03 13:38:02.709852 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:02.709828 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6be3240b-9b5b-4e17-ada4-676bd02c18ff-plugins-config-volume" pod "6be3240b-9b5b-4e17-ada4-676bd02c18ff" (UID: "6be3240b-9b5b-4e17-ada4-676bd02c18ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:38:02.710087 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:02.710065 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ds6z\" (UniqueName: \"kubernetes.io/projected/489c2cad-0beb-4090-8a88-16dce56f5471-kube-api-access-9ds6z\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:02.710164 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:02.710095 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6be3240b-9b5b-4e17-ada4-676bd02c18ff-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:02.711509 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:02.711486 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6be3240b-9b5b-4e17-ada4-676bd02c18ff-kube-api-access-lxj9f" pod "6be3240b-9b5b-4e17-ada4-676bd02c18ff" (UID: "6be3240b-9b5b-4e17-ada4-676bd02c18ff"). InnerVolumeSpecName "kube-api-access-lxj9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:38:02.771762 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:02.771711 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a770b9-65459cd457-28vvf"] Nov 03 13:38:02.778883 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:02.778850 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a770b9-65459cd457-28vvf"] Nov 03 13:38:02.811810 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:02.811763 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxj9f\" (UniqueName: \"kubernetes.io/projected/6be3240b-9b5b-4e17-ada4-676bd02c18ff-kube-api-access-lxj9f\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:03.029568 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:03.029484 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a770b9-5fbcd499ff-2xsw7"] Nov 03 13:38:03.037994 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:03.037957 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a770b9-5fbcd499ff-2xsw7"] Nov 03 13:38:03.486079 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:03.486042 5979 patch_prober.go:29] interesting pod/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Nov 03 13:38:03.486269 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:03.486098 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf" podUID="efd2a382-b6ea-419f-b33a-9ebfdf0bd61b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Nov 03 13:38:04.115712 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:04.115677 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="489c2cad-0beb-4090-8a88-16dce56f5471" path="/var/lib/kubelet/pods/489c2cad-0beb-4090-8a88-16dce56f5471/volumes" Nov 03 13:38:04.116132 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:04.116118 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6be3240b-9b5b-4e17-ada4-676bd02c18ff" path="/var/lib/kubelet/pods/6be3240b-9b5b-4e17-ada4-676bd02c18ff/volumes" Nov 03 13:38:04.485276 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:04.485244 5979 patch_prober.go:29] interesting pod/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Nov 03 13:38:04.485483 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:04.485299 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf" podUID="efd2a382-b6ea-419f-b33a-9ebfdf0bd61b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Nov 03 13:38:05.485944 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:05.485914 5979 patch_prober.go:29] interesting pod/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Nov 03 13:38:05.486338 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:05.485986 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf" podUID="efd2a382-b6ea-419f-b33a-9ebfdf0bd61b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Nov 03 13:38:06.489839 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:06.489806 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf" Nov 03 13:38:06.490342 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:06.490195 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf" Nov 03 13:38:06.490911 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:06.490892 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf" Nov 03 13:38:07.234778 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:38:07.234744 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:38:11.603189 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:11.603161 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d29e07-5585b5fc95-2hf65" Nov 03 13:38:12.613493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:12.613449 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d29e07-6b54b7bcd6-lqtkq" Nov 03 13:38:19.522302 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:19.522264 5979 scope.go:122] "RemoveContainer" containerID="ba3b18c0434f4adb2184bed7e48c444e620195f7c39a2800171d59fd8bc8c4f5" Nov 03 13:38:33.999006 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:33.998970 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-02f3a0-57f7b565b5-xg9v9"] Nov 03 13:38:34.120851 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:34.120809 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-02f3a0-66bc949bbc-dk2q5"] Nov 03 13:38:37.235224 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:38:37.235184 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:38:38.999606 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:38.999567 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-02f3a0-57f7b565b5-xg9v9" podUID="daa7244d-0a66-4e91-acb5-3a604fb83aa7" containerName="payload-processing" containerID="cri-o://068fee9d9edc4d9dfede971afdbf01c32c27378b48f5973e052ea0caf1dda203" gracePeriod=25 Nov 03 13:38:39.121550 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:39.121515 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02f3a0-66bc949bbc-dk2q5" podUID="23c0c7c1-7712-49e1-9dfe-f7eb396bdaa0" containerName="payload-pre-processing" containerID="cri-o://cd31b70e93fa1dec703c23eec631868fb238f87aa71f44c1e979b470a8adadb4" gracePeriod=25 Nov 03 13:38:39.274639 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:39.274611 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-02f3a0-57f7b565b5-xg9v9" Nov 03 13:38:39.379657 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:39.379633 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02f3a0-66bc949bbc-dk2q5" Nov 03 13:38:39.396948 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:39.396920 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daa7244d-0a66-4e91-acb5-3a604fb83aa7-kube-api-access-nqkfb\" (UniqueName: \"kubernetes.io/projected/daa7244d-0a66-4e91-acb5-3a604fb83aa7-kube-api-access-nqkfb\") pod \"daa7244d-0a66-4e91-acb5-3a604fb83aa7\" (UID: \"daa7244d-0a66-4e91-acb5-3a604fb83aa7\") " Nov 03 13:38:39.397260 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:39.397234 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/daa7244d-0a66-4e91-acb5-3a604fb83aa7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daa7244d-0a66-4e91-acb5-3a604fb83aa7-plugins-config-volume\") pod \"daa7244d-0a66-4e91-acb5-3a604fb83aa7\" (UID: \"daa7244d-0a66-4e91-acb5-3a604fb83aa7\") " Nov 03 13:38:39.397676 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:39.397645 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daa7244d-0a66-4e91-acb5-3a604fb83aa7-plugins-config-volume" pod "daa7244d-0a66-4e91-acb5-3a604fb83aa7" (UID: "daa7244d-0a66-4e91-acb5-3a604fb83aa7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:38:39.397902 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:39.397880 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daa7244d-0a66-4e91-acb5-3a604fb83aa7-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:39.399113 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:39.399091 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daa7244d-0a66-4e91-acb5-3a604fb83aa7-kube-api-access-nqkfb" pod "daa7244d-0a66-4e91-acb5-3a604fb83aa7" (UID: "daa7244d-0a66-4e91-acb5-3a604fb83aa7"). InnerVolumeSpecName "kube-api-access-nqkfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:38:39.499083 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:39.499050 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23c0c7c1-7712-49e1-9dfe-f7eb396bdaa0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23c0c7c1-7712-49e1-9dfe-f7eb396bdaa0-plugins-config-volume\") pod \"23c0c7c1-7712-49e1-9dfe-f7eb396bdaa0\" (UID: \"23c0c7c1-7712-49e1-9dfe-f7eb396bdaa0\") " Nov 03 13:38:39.499273 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:39.499136 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23c0c7c1-7712-49e1-9dfe-f7eb396bdaa0-kube-api-access-xh6wz\" (UniqueName: \"kubernetes.io/projected/23c0c7c1-7712-49e1-9dfe-f7eb396bdaa0-kube-api-access-xh6wz\") pod \"23c0c7c1-7712-49e1-9dfe-f7eb396bdaa0\" (UID: \"23c0c7c1-7712-49e1-9dfe-f7eb396bdaa0\") " Nov 03 13:38:39.499459 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:39.499435 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23c0c7c1-7712-49e1-9dfe-f7eb396bdaa0-plugins-config-volume" pod "23c0c7c1-7712-49e1-9dfe-f7eb396bdaa0" (UID: "23c0c7c1-7712-49e1-9dfe-f7eb396bdaa0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:38:39.499683 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:39.499664 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqkfb\" (UniqueName: \"kubernetes.io/projected/daa7244d-0a66-4e91-acb5-3a604fb83aa7-kube-api-access-nqkfb\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:39.499742 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:39.499684 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23c0c7c1-7712-49e1-9dfe-f7eb396bdaa0-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:39.501314 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:39.501293 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23c0c7c1-7712-49e1-9dfe-f7eb396bdaa0-kube-api-access-xh6wz" pod "23c0c7c1-7712-49e1-9dfe-f7eb396bdaa0" (UID: "23c0c7c1-7712-49e1-9dfe-f7eb396bdaa0"). InnerVolumeSpecName "kube-api-access-xh6wz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:38:39.601169 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:39.601080 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh6wz\" (UniqueName: \"kubernetes.io/projected/23c0c7c1-7712-49e1-9dfe-f7eb396bdaa0-kube-api-access-xh6wz\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:39.930865 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:39.930718 5979 generic.go:356] "Generic (PLEG): container finished" podID="23c0c7c1-7712-49e1-9dfe-f7eb396bdaa0" containerID="cd31b70e93fa1dec703c23eec631868fb238f87aa71f44c1e979b470a8adadb4" exitCode=0 Nov 03 13:38:39.931072 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:39.930910 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02f3a0-66bc949bbc-dk2q5" event={"ID":"23c0c7c1-7712-49e1-9dfe-f7eb396bdaa0","Type":"ContainerDied","Data":"cd31b70e93fa1dec703c23eec631868fb238f87aa71f44c1e979b470a8adadb4"} Nov 03 13:38:39.931072 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:39.930983 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02f3a0-66bc949bbc-dk2q5" event={"ID":"23c0c7c1-7712-49e1-9dfe-f7eb396bdaa0","Type":"ContainerDied","Data":"dfe95619d9c0282be12a320978c4c036618aa7c60df6f760c0498d38cb56ed29"} Nov 03 13:38:39.931072 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:39.931015 5979 scope.go:122] "RemoveContainer" containerID="cd31b70e93fa1dec703c23eec631868fb238f87aa71f44c1e979b470a8adadb4" Nov 03 13:38:39.931246 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:39.931187 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02f3a0-66bc949bbc-dk2q5" Nov 03 13:38:39.950700 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:39.950650 5979 generic.go:356] "Generic (PLEG): container finished" podID="daa7244d-0a66-4e91-acb5-3a604fb83aa7" containerID="068fee9d9edc4d9dfede971afdbf01c32c27378b48f5973e052ea0caf1dda203" exitCode=0 Nov 03 13:38:39.950961 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:39.950933 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-02f3a0-57f7b565b5-xg9v9" event={"ID":"daa7244d-0a66-4e91-acb5-3a604fb83aa7","Type":"ContainerDied","Data":"068fee9d9edc4d9dfede971afdbf01c32c27378b48f5973e052ea0caf1dda203"} Nov 03 13:38:39.951045 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:39.951002 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-02f3a0-57f7b565b5-xg9v9" event={"ID":"daa7244d-0a66-4e91-acb5-3a604fb83aa7","Type":"ContainerDied","Data":"d911f4af23de083317ab00f784237670c747e5a81e7a6e5012228c886f78b579"} Nov 03 13:38:39.951104 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:39.951087 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-02f3a0-57f7b565b5-xg9v9" Nov 03 13:38:39.960644 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:39.960619 5979 scope.go:122] "RemoveContainer" containerID="cd31b70e93fa1dec703c23eec631868fb238f87aa71f44c1e979b470a8adadb4" Nov 03 13:38:39.961096 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:38:39.961071 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd31b70e93fa1dec703c23eec631868fb238f87aa71f44c1e979b470a8adadb4\": container with ID starting with cd31b70e93fa1dec703c23eec631868fb238f87aa71f44c1e979b470a8adadb4 not found: ID does not exist" containerID="cd31b70e93fa1dec703c23eec631868fb238f87aa71f44c1e979b470a8adadb4" Nov 03 13:38:39.961195 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:39.961109 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd31b70e93fa1dec703c23eec631868fb238f87aa71f44c1e979b470a8adadb4"} err="failed to get container status \"cd31b70e93fa1dec703c23eec631868fb238f87aa71f44c1e979b470a8adadb4\": rpc error: code = NotFound desc = could not find container \"cd31b70e93fa1dec703c23eec631868fb238f87aa71f44c1e979b470a8adadb4\": container with ID starting with cd31b70e93fa1dec703c23eec631868fb238f87aa71f44c1e979b470a8adadb4 not found: ID does not exist" Nov 03 13:38:39.961195 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:39.961128 5979 scope.go:122] "RemoveContainer" containerID="068fee9d9edc4d9dfede971afdbf01c32c27378b48f5973e052ea0caf1dda203" Nov 03 13:38:39.976961 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:39.976929 5979 scope.go:122] "RemoveContainer" containerID="068fee9d9edc4d9dfede971afdbf01c32c27378b48f5973e052ea0caf1dda203" Nov 03 13:38:39.977312 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:38:39.977284 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"068fee9d9edc4d9dfede971afdbf01c32c27378b48f5973e052ea0caf1dda203\": container with ID starting with 068fee9d9edc4d9dfede971afdbf01c32c27378b48f5973e052ea0caf1dda203 not found: ID does not exist" containerID="068fee9d9edc4d9dfede971afdbf01c32c27378b48f5973e052ea0caf1dda203" Nov 03 13:38:39.977458 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:39.977323 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"068fee9d9edc4d9dfede971afdbf01c32c27378b48f5973e052ea0caf1dda203"} err="failed to get container status \"068fee9d9edc4d9dfede971afdbf01c32c27378b48f5973e052ea0caf1dda203\": rpc error: code = NotFound desc = could not find container \"068fee9d9edc4d9dfede971afdbf01c32c27378b48f5973e052ea0caf1dda203\": container with ID starting with 068fee9d9edc4d9dfede971afdbf01c32c27378b48f5973e052ea0caf1dda203 not found: ID does not exist" Nov 03 13:38:40.090184 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:40.090149 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-02f3a0-57f7b565b5-xg9v9"] Nov 03 13:38:40.136688 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:40.136656 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-02f3a0-57f7b565b5-xg9v9"] Nov 03 13:38:40.251051 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:40.250968 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-02f3a0-66bc949bbc-dk2q5"] Nov 03 13:38:40.254922 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:40.254897 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-02f3a0-66bc949bbc-dk2q5"] Nov 03 13:38:40.449361 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:40.449331 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-be8d0a-58864d5c94-vcxtd"] Nov 03 13:38:40.449623 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:40.449594 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-be8d0a-58864d5c94-vcxtd" podUID="fc3ef1e4-0333-411b-a762-9a4f7cc8a0db" containerName="payload-processing" containerID="cri-o://363bda6db308dd86f0cb172b97b88b6333337b4133f12e2b962eb9bb7723249f" gracePeriod=30 Nov 03 13:38:40.503976 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:40.503885 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-be8d0a-7bf74dc5f-lkxx5"] Nov 03 13:38:40.504164 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:40.504118 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be8d0a-7bf74dc5f-lkxx5" podUID="3d59ed0e-e153-4d82-93ad-dd7adf1a3cd7" containerName="payload-pre-processing" containerID="cri-o://cd60dc983ef8b94418ba0ab27203fd6de8d36f93b94e3a252be3654a14d19379" gracePeriod=30 Nov 03 13:38:40.786961 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:40.786931 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-be8d0a-58864d5c94-vcxtd" Nov 03 13:38:40.804438 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:40.804413 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be8d0a-7bf74dc5f-lkxx5" Nov 03 13:38:40.817641 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:40.817616 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d59ed0e-e153-4d82-93ad-dd7adf1a3cd7-kube-api-access-9pfbj\" (UniqueName: \"kubernetes.io/projected/3d59ed0e-e153-4d82-93ad-dd7adf1a3cd7-kube-api-access-9pfbj\") pod \"3d59ed0e-e153-4d82-93ad-dd7adf1a3cd7\" (UID: \"3d59ed0e-e153-4d82-93ad-dd7adf1a3cd7\") " Nov 03 13:38:40.817780 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:40.817716 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d59ed0e-e153-4d82-93ad-dd7adf1a3cd7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d59ed0e-e153-4d82-93ad-dd7adf1a3cd7-plugins-config-volume\") pod \"3d59ed0e-e153-4d82-93ad-dd7adf1a3cd7\" (UID: \"3d59ed0e-e153-4d82-93ad-dd7adf1a3cd7\") " Nov 03 13:38:40.817780 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:40.817768 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc3ef1e4-0333-411b-a762-9a4f7cc8a0db-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc3ef1e4-0333-411b-a762-9a4f7cc8a0db-plugins-config-volume\") pod \"fc3ef1e4-0333-411b-a762-9a4f7cc8a0db\" (UID: \"fc3ef1e4-0333-411b-a762-9a4f7cc8a0db\") " Nov 03 13:38:40.817869 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:40.817822 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc3ef1e4-0333-411b-a762-9a4f7cc8a0db-kube-api-access-vhfqk\" (UniqueName: \"kubernetes.io/projected/fc3ef1e4-0333-411b-a762-9a4f7cc8a0db-kube-api-access-vhfqk\") pod \"fc3ef1e4-0333-411b-a762-9a4f7cc8a0db\" (UID: \"fc3ef1e4-0333-411b-a762-9a4f7cc8a0db\") " Nov 03 13:38:40.818113 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:40.818086 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d59ed0e-e153-4d82-93ad-dd7adf1a3cd7-plugins-config-volume" pod "3d59ed0e-e153-4d82-93ad-dd7adf1a3cd7" (UID: "3d59ed0e-e153-4d82-93ad-dd7adf1a3cd7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:38:40.818223 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:40.818144 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc3ef1e4-0333-411b-a762-9a4f7cc8a0db-plugins-config-volume" pod "fc3ef1e4-0333-411b-a762-9a4f7cc8a0db" (UID: "fc3ef1e4-0333-411b-a762-9a4f7cc8a0db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:38:40.818269 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:40.818221 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d59ed0e-e153-4d82-93ad-dd7adf1a3cd7-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:40.819841 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:40.819818 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d59ed0e-e153-4d82-93ad-dd7adf1a3cd7-kube-api-access-9pfbj" pod "3d59ed0e-e153-4d82-93ad-dd7adf1a3cd7" (UID: "3d59ed0e-e153-4d82-93ad-dd7adf1a3cd7"). InnerVolumeSpecName "kube-api-access-9pfbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:38:40.820119 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:40.820104 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc3ef1e4-0333-411b-a762-9a4f7cc8a0db-kube-api-access-vhfqk" pod "fc3ef1e4-0333-411b-a762-9a4f7cc8a0db" (UID: "fc3ef1e4-0333-411b-a762-9a4f7cc8a0db"). InnerVolumeSpecName "kube-api-access-vhfqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:38:40.919928 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:40.919895 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc3ef1e4-0333-411b-a762-9a4f7cc8a0db-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:40.919928 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:40.919922 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhfqk\" (UniqueName: \"kubernetes.io/projected/fc3ef1e4-0333-411b-a762-9a4f7cc8a0db-kube-api-access-vhfqk\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:40.919928 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:40.919932 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pfbj\" (UniqueName: \"kubernetes.io/projected/3d59ed0e-e153-4d82-93ad-dd7adf1a3cd7-kube-api-access-9pfbj\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:40.978201 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:40.966948 5979 generic.go:356] "Generic (PLEG): container finished" podID="3d59ed0e-e153-4d82-93ad-dd7adf1a3cd7" containerID="cd60dc983ef8b94418ba0ab27203fd6de8d36f93b94e3a252be3654a14d19379" exitCode=0 Nov 03 13:38:40.978201 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:40.967060 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be8d0a-7bf74dc5f-lkxx5" event={"ID":"3d59ed0e-e153-4d82-93ad-dd7adf1a3cd7","Type":"ContainerDied","Data":"cd60dc983ef8b94418ba0ab27203fd6de8d36f93b94e3a252be3654a14d19379"} Nov 03 13:38:40.978201 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:40.967128 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be8d0a-7bf74dc5f-lkxx5" event={"ID":"3d59ed0e-e153-4d82-93ad-dd7adf1a3cd7","Type":"ContainerDied","Data":"fc4de57eb4824780c8ad1ae4bfa73da081cf76bdedede4ca3502909963e92f18"} Nov 03 13:38:40.978201 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:40.967168 5979 scope.go:122] "RemoveContainer" containerID="cd60dc983ef8b94418ba0ab27203fd6de8d36f93b94e3a252be3654a14d19379" Nov 03 13:38:41.003092 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:40.994179 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-be8d0a-7bf74dc5f-lkxx5" Nov 03 13:38:41.053511 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:41.053446 5979 generic.go:356] "Generic (PLEG): container finished" podID="fc3ef1e4-0333-411b-a762-9a4f7cc8a0db" containerID="363bda6db308dd86f0cb172b97b88b6333337b4133f12e2b962eb9bb7723249f" exitCode=0 Nov 03 13:38:41.053689 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:41.053599 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-be8d0a-58864d5c94-vcxtd" event={"ID":"fc3ef1e4-0333-411b-a762-9a4f7cc8a0db","Type":"ContainerDied","Data":"363bda6db308dd86f0cb172b97b88b6333337b4133f12e2b962eb9bb7723249f"} Nov 03 13:38:41.053689 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:41.053651 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-be8d0a-58864d5c94-vcxtd" event={"ID":"fc3ef1e4-0333-411b-a762-9a4f7cc8a0db","Type":"ContainerDied","Data":"96a21f004ef9f53bdf0e56b03e30445ffdfb0dd653d32dfe0d9a7446a1679374"} Nov 03 13:38:41.053816 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:41.053733 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-be8d0a-58864d5c94-vcxtd" Nov 03 13:38:41.094059 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:41.094037 5979 scope.go:122] "RemoveContainer" containerID="cd60dc983ef8b94418ba0ab27203fd6de8d36f93b94e3a252be3654a14d19379" Nov 03 13:38:41.094417 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:38:41.094317 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd60dc983ef8b94418ba0ab27203fd6de8d36f93b94e3a252be3654a14d19379\": container with ID starting with cd60dc983ef8b94418ba0ab27203fd6de8d36f93b94e3a252be3654a14d19379 not found: ID does not exist" containerID="cd60dc983ef8b94418ba0ab27203fd6de8d36f93b94e3a252be3654a14d19379" Nov 03 13:38:41.094417 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:41.094342 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd60dc983ef8b94418ba0ab27203fd6de8d36f93b94e3a252be3654a14d19379"} err="failed to get container status \"cd60dc983ef8b94418ba0ab27203fd6de8d36f93b94e3a252be3654a14d19379\": rpc error: code = NotFound desc = could not find container \"cd60dc983ef8b94418ba0ab27203fd6de8d36f93b94e3a252be3654a14d19379\": container with ID starting with cd60dc983ef8b94418ba0ab27203fd6de8d36f93b94e3a252be3654a14d19379 not found: ID does not exist" Nov 03 13:38:41.094417 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:41.094358 5979 scope.go:122] "RemoveContainer" containerID="363bda6db308dd86f0cb172b97b88b6333337b4133f12e2b962eb9bb7723249f" Nov 03 13:38:41.106612 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:41.106584 5979 scope.go:122] "RemoveContainer" containerID="363bda6db308dd86f0cb172b97b88b6333337b4133f12e2b962eb9bb7723249f" Nov 03 13:38:41.107029 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:38:41.107002 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"363bda6db308dd86f0cb172b97b88b6333337b4133f12e2b962eb9bb7723249f\": container with ID starting with 363bda6db308dd86f0cb172b97b88b6333337b4133f12e2b962eb9bb7723249f not found: ID does not exist" containerID="363bda6db308dd86f0cb172b97b88b6333337b4133f12e2b962eb9bb7723249f" Nov 03 13:38:41.107127 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:41.107036 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"363bda6db308dd86f0cb172b97b88b6333337b4133f12e2b962eb9bb7723249f"} err="failed to get container status \"363bda6db308dd86f0cb172b97b88b6333337b4133f12e2b962eb9bb7723249f\": rpc error: code = NotFound desc = could not find container \"363bda6db308dd86f0cb172b97b88b6333337b4133f12e2b962eb9bb7723249f\": container with ID starting with 363bda6db308dd86f0cb172b97b88b6333337b4133f12e2b962eb9bb7723249f not found: ID does not exist" Nov 03 13:38:41.238598 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:41.238541 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-be8d0a-7bf74dc5f-lkxx5"] Nov 03 13:38:41.242077 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:41.242051 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-be8d0a-7bf74dc5f-lkxx5"] Nov 03 13:38:41.344227 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:41.344150 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-be8d0a-58864d5c94-vcxtd"] Nov 03 13:38:41.348909 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:41.348879 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-be8d0a-58864d5c94-vcxtd"] Nov 03 13:38:41.806978 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:41.806939 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-21f111-9f4fb987-jb2sv"] Nov 03 13:38:41.807625 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:41.807606 5979 cpu_manager.go:414] "containerMap: removing container" podUID="3d59ed0e-e153-4d82-93ad-dd7adf1a3cd7" containerName="payload-pre-processing" Nov 03 13:38:41.807625 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:41.807624 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d59ed0e-e153-4d82-93ad-dd7adf1a3cd7" containerName="payload-pre-processing" Nov 03 13:38:41.807767 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:41.807636 5979 cpu_manager.go:414] "containerMap: removing container" podUID="daa7244d-0a66-4e91-acb5-3a604fb83aa7" containerName="payload-processing" Nov 03 13:38:41.807767 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:41.807641 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daa7244d-0a66-4e91-acb5-3a604fb83aa7" containerName="payload-processing" Nov 03 13:38:41.807767 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:41.807723 5979 cpu_manager.go:414] "containerMap: removing container" podUID="6be3240b-9b5b-4e17-ada4-676bd02c18ff" containerName="payload-pre-processing" Nov 03 13:38:41.807767 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:41.807729 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6be3240b-9b5b-4e17-ada4-676bd02c18ff" containerName="payload-pre-processing" Nov 03 13:38:41.807767 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:41.807766 5979 cpu_manager.go:414] "containerMap: removing container" podUID="fc3ef1e4-0333-411b-a762-9a4f7cc8a0db" containerName="payload-processing" Nov 03 13:38:41.808023 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:41.807773 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc3ef1e4-0333-411b-a762-9a4f7cc8a0db" containerName="payload-processing" Nov 03 13:38:41.808023 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:41.807783 5979 cpu_manager.go:414] "containerMap: removing container" podUID="23c0c7c1-7712-49e1-9dfe-f7eb396bdaa0" containerName="payload-pre-processing" Nov 03 13:38:41.808023 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:41.807790 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23c0c7c1-7712-49e1-9dfe-f7eb396bdaa0" containerName="payload-pre-processing" Nov 03 13:38:41.808023 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:41.807797 5979 cpu_manager.go:414] "containerMap: removing container" podUID="489c2cad-0beb-4090-8a88-16dce56f5471" containerName="payload-processing" Nov 03 13:38:41.808023 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:41.807802 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="489c2cad-0beb-4090-8a88-16dce56f5471" containerName="payload-processing" Nov 03 13:38:41.808023 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:41.807945 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d59ed0e-e153-4d82-93ad-dd7adf1a3cd7" containerName="payload-pre-processing" Nov 03 13:38:41.808023 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:41.807955 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="daa7244d-0a66-4e91-acb5-3a604fb83aa7" containerName="payload-processing" Nov 03 13:38:41.808023 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:41.807970 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="6be3240b-9b5b-4e17-ada4-676bd02c18ff" containerName="payload-pre-processing" Nov 03 13:38:41.808023 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:41.807981 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc3ef1e4-0333-411b-a762-9a4f7cc8a0db" containerName="payload-processing" Nov 03 13:38:41.808023 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:41.807988 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="23c0c7c1-7712-49e1-9dfe-f7eb396bdaa0" containerName="payload-pre-processing" Nov 03 13:38:41.808023 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:41.807995 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="489c2cad-0beb-4090-8a88-16dce56f5471" containerName="payload-processing" Nov 03 13:38:41.812729 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:41.812709 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-21f111-9f4fb987-jb2sv" Nov 03 13:38:41.830925 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:41.830900 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66067047-9f8c-499e-af1c-1245b56b39f8-maas-api-tls\") pod \"maas-api-e2e-praxis-a-21f111-9f4fb987-jb2sv\" (UID: \"66067047-9f8c-499e-af1c-1245b56b39f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-21f111-9f4fb987-jb2sv" Nov 03 13:38:41.831051 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:41.830950 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66067047-9f8c-499e-af1c-1245b56b39f8-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-21f111-9f4fb987-jb2sv\" (UID: \"66067047-9f8c-499e-af1c-1245b56b39f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-21f111-9f4fb987-jb2sv" Nov 03 13:38:41.831098 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:41.831049 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66067047-9f8c-499e-af1c-1245b56b39f8-metrics-certs\") pod \"maas-api-e2e-praxis-a-21f111-9f4fb987-jb2sv\" (UID: \"66067047-9f8c-499e-af1c-1245b56b39f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-21f111-9f4fb987-jb2sv" Nov 03 13:38:41.831195 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:41.831177 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4jdf\" (UniqueName: \"kubernetes.io/projected/66067047-9f8c-499e-af1c-1245b56b39f8-kube-api-access-h4jdf\") pod \"maas-api-e2e-praxis-a-21f111-9f4fb987-jb2sv\" (UID: \"66067047-9f8c-499e-af1c-1245b56b39f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-21f111-9f4fb987-jb2sv" Nov 03 13:38:41.863908 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:41.863870 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-21f111-serving-cert\"" Nov 03 13:38:41.866541 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:41.866509 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-21f111-9f4fb987-jb2sv"] Nov 03 13:38:41.932716 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:41.932683 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4jdf\" (UniqueName: \"kubernetes.io/projected/66067047-9f8c-499e-af1c-1245b56b39f8-kube-api-access-h4jdf\") pod \"maas-api-e2e-praxis-a-21f111-9f4fb987-jb2sv\" (UID: \"66067047-9f8c-499e-af1c-1245b56b39f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-21f111-9f4fb987-jb2sv" Nov 03 13:38:41.932896 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:41.932811 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66067047-9f8c-499e-af1c-1245b56b39f8-maas-api-tls\") pod \"maas-api-e2e-praxis-a-21f111-9f4fb987-jb2sv\" (UID: \"66067047-9f8c-499e-af1c-1245b56b39f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-21f111-9f4fb987-jb2sv" Nov 03 13:38:41.932896 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:41.932867 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66067047-9f8c-499e-af1c-1245b56b39f8-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-21f111-9f4fb987-jb2sv\" (UID: \"66067047-9f8c-499e-af1c-1245b56b39f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-21f111-9f4fb987-jb2sv" Nov 03 13:38:41.933006 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:41.932908 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66067047-9f8c-499e-af1c-1245b56b39f8-metrics-certs\") pod \"maas-api-e2e-praxis-a-21f111-9f4fb987-jb2sv\" (UID: \"66067047-9f8c-499e-af1c-1245b56b39f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-21f111-9f4fb987-jb2sv" Nov 03 13:38:41.933006 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:38:41.932946 5979 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-21f111-serving-cert: secret "maas-api-e2e-praxis-a-21f111-serving-cert" not found Nov 03 13:38:41.933119 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:38:41.933024 5979 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66067047-9f8c-499e-af1c-1245b56b39f8-maas-api-tls podName:66067047-9f8c-499e-af1c-1245b56b39f8 nodeName:}" failed. No retries permitted until 2026-11-03 13:38:42.433005563 +0000 UTC m=+2004.443693795 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/66067047-9f8c-499e-af1c-1245b56b39f8-maas-api-tls") pod "maas-api-e2e-praxis-a-21f111-9f4fb987-jb2sv" (UID: "66067047-9f8c-499e-af1c-1245b56b39f8") : secret "maas-api-e2e-praxis-a-21f111-serving-cert" not found Nov 03 13:38:41.933639 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:41.933617 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66067047-9f8c-499e-af1c-1245b56b39f8-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-21f111-9f4fb987-jb2sv\" (UID: \"66067047-9f8c-499e-af1c-1245b56b39f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-21f111-9f4fb987-jb2sv" Nov 03 13:38:41.935432 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:41.935410 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66067047-9f8c-499e-af1c-1245b56b39f8-metrics-certs\") pod \"maas-api-e2e-praxis-a-21f111-9f4fb987-jb2sv\" (UID: \"66067047-9f8c-499e-af1c-1245b56b39f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-21f111-9f4fb987-jb2sv" Nov 03 13:38:41.987239 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:41.987204 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4jdf\" (UniqueName: \"kubernetes.io/projected/66067047-9f8c-499e-af1c-1245b56b39f8-kube-api-access-h4jdf\") pod \"maas-api-e2e-praxis-a-21f111-9f4fb987-jb2sv\" (UID: \"66067047-9f8c-499e-af1c-1245b56b39f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-21f111-9f4fb987-jb2sv" Nov 03 13:38:42.115326 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:42.115247 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23c0c7c1-7712-49e1-9dfe-f7eb396bdaa0" path="/var/lib/kubelet/pods/23c0c7c1-7712-49e1-9dfe-f7eb396bdaa0/volumes" Nov 03 13:38:42.115772 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:42.115739 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d59ed0e-e153-4d82-93ad-dd7adf1a3cd7" path="/var/lib/kubelet/pods/3d59ed0e-e153-4d82-93ad-dd7adf1a3cd7/volumes" Nov 03 13:38:42.116127 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:42.116113 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="daa7244d-0a66-4e91-acb5-3a604fb83aa7" path="/var/lib/kubelet/pods/daa7244d-0a66-4e91-acb5-3a604fb83aa7/volumes" Nov 03 13:38:42.116495 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:42.116465 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc3ef1e4-0333-411b-a762-9a4f7cc8a0db" path="/var/lib/kubelet/pods/fc3ef1e4-0333-411b-a762-9a4f7cc8a0db/volumes" Nov 03 13:38:42.438528 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:42.438492 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66067047-9f8c-499e-af1c-1245b56b39f8-maas-api-tls\") pod \"maas-api-e2e-praxis-a-21f111-9f4fb987-jb2sv\" (UID: \"66067047-9f8c-499e-af1c-1245b56b39f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-21f111-9f4fb987-jb2sv" Nov 03 13:38:42.440929 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:42.440903 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66067047-9f8c-499e-af1c-1245b56b39f8-maas-api-tls\") pod \"maas-api-e2e-praxis-a-21f111-9f4fb987-jb2sv\" (UID: \"66067047-9f8c-499e-af1c-1245b56b39f8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-21f111-9f4fb987-jb2sv" Nov 03 13:38:42.726796 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:42.726716 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-21f111-9f4fb987-jb2sv" Nov 03 13:38:43.107089 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:43.107013 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-21f111-9f4fb987-jb2sv"] Nov 03 13:38:43.114267 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:38:43.114230 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66067047_9f8c_499e_af1c_1245b56b39f8.slice/crio-05f4e76d52ebc1bda56dae424371081761a49b5ac21741bf89ef68be81496bb0 WatchSource:0}: Error finding container 05f4e76d52ebc1bda56dae424371081761a49b5ac21741bf89ef68be81496bb0: Status 404 returned error can't find the container with id 05f4e76d52ebc1bda56dae424371081761a49b5ac21741bf89ef68be81496bb0 Nov 03 13:38:43.115620 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:43.115598 5979 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 13:38:44.115740 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:44.115710 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-21f111-9f4fb987-jb2sv" event={"ID":"66067047-9f8c-499e-af1c-1245b56b39f8","Type":"ContainerStarted","Data":"05f4e76d52ebc1bda56dae424371081761a49b5ac21741bf89ef68be81496bb0"} Nov 03 13:38:44.692917 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:44.692885 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5474b7-7496845f45-6ncjj"] Nov 03 13:38:44.843824 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:44.843785 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5474b7-ccbcc59f-hlkd5"] Nov 03 13:38:45.178171 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:45.178113 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-21f111-9f4fb987-jb2sv" event={"ID":"66067047-9f8c-499e-af1c-1245b56b39f8","Type":"ContainerStarted","Data":"e3d439da8081444a1aa0e817d175aed7ca9501f87f98082bf4333ff524a49332"} Nov 03 13:38:45.178683 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:45.178203 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-21f111-9f4fb987-jb2sv" Nov 03 13:38:48.860332 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:48.860277 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-21f111-9f4fb987-jb2sv" podStartSLOduration=6.945394675 podStartE2EDuration="7.860261613s" totalImagesPullingTime="914.866938ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:38:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:38:45.303393141 +0000 UTC m=+2007.314081407" watchObservedRunningTime="2026-11-03 13:38:48.860261613 +0000 UTC m=+2010.870949855" Nov 03 13:38:48.861084 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:48.861065 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s"] Nov 03 13:38:48.861283 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:48.861262 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s" podUID="80f447c3-327a-4726-b976-8ce7adcd92fa" containerName="istio-proxy" containerID="cri-o://c7db051d1389b916fd8cefea8d7b19b247ad25bcbba96a1755555a3a1888375e" gracePeriod=30 Nov 03 13:38:49.693509 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:49.693437 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-5474b7-7496845f45-6ncjj" podUID="e22aa23e-9a2a-4499-bba1-4da8208e7386" containerName="payload-processing" containerID="cri-o://250546879e63765f67a74a640f85d7484aa1591d7693464e3f614f2a3a7c8a58" gracePeriod=25 Nov 03 13:38:49.844502 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:49.844437 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5474b7-ccbcc59f-hlkd5" podUID="f27c783a-fa71-4db1-a3bb-4ed989e72d08" containerName="payload-pre-processing" containerID="cri-o://1f0b697479115d5629d5b5871550d76e288907c9d11bc661ea6f4662247fb2bb" gracePeriod=25 Nov 03 13:38:49.984312 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:49.984272 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5474b7-7496845f45-6ncjj" Nov 03 13:38:50.053367 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:50.053327 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e22aa23e-9a2a-4499-bba1-4da8208e7386-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e22aa23e-9a2a-4499-bba1-4da8208e7386-plugins-config-volume\") pod \"e22aa23e-9a2a-4499-bba1-4da8208e7386\" (UID: \"e22aa23e-9a2a-4499-bba1-4da8208e7386\") " Nov 03 13:38:50.053660 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:50.053432 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e22aa23e-9a2a-4499-bba1-4da8208e7386-kube-api-access-kpwq2\" (UniqueName: \"kubernetes.io/projected/e22aa23e-9a2a-4499-bba1-4da8208e7386-kube-api-access-kpwq2\") pod \"e22aa23e-9a2a-4499-bba1-4da8208e7386\" (UID: \"e22aa23e-9a2a-4499-bba1-4da8208e7386\") " Nov 03 13:38:50.053782 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:50.053754 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e22aa23e-9a2a-4499-bba1-4da8208e7386-plugins-config-volume" pod "e22aa23e-9a2a-4499-bba1-4da8208e7386" (UID: "e22aa23e-9a2a-4499-bba1-4da8208e7386"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:38:50.054225 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:50.054197 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e22aa23e-9a2a-4499-bba1-4da8208e7386-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:50.056722 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:50.056679 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e22aa23e-9a2a-4499-bba1-4da8208e7386-kube-api-access-kpwq2" pod "e22aa23e-9a2a-4499-bba1-4da8208e7386" (UID: "e22aa23e-9a2a-4499-bba1-4da8208e7386"). InnerVolumeSpecName "kube-api-access-kpwq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:38:50.099537 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:50.099505 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5474b7-ccbcc59f-hlkd5" Nov 03 13:38:50.155505 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:50.155462 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f27c783a-fa71-4db1-a3bb-4ed989e72d08-kube-api-access-s56nj\" (UniqueName: \"kubernetes.io/projected/f27c783a-fa71-4db1-a3bb-4ed989e72d08-kube-api-access-s56nj\") pod \"f27c783a-fa71-4db1-a3bb-4ed989e72d08\" (UID: \"f27c783a-fa71-4db1-a3bb-4ed989e72d08\") " Nov 03 13:38:50.155667 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:50.155533 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f27c783a-fa71-4db1-a3bb-4ed989e72d08-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f27c783a-fa71-4db1-a3bb-4ed989e72d08-plugins-config-volume\") pod \"f27c783a-fa71-4db1-a3bb-4ed989e72d08\" (UID: \"f27c783a-fa71-4db1-a3bb-4ed989e72d08\") " Nov 03 13:38:50.155921 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:50.155886 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f27c783a-fa71-4db1-a3bb-4ed989e72d08-plugins-config-volume" pod "f27c783a-fa71-4db1-a3bb-4ed989e72d08" (UID: "f27c783a-fa71-4db1-a3bb-4ed989e72d08"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:38:50.156079 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:50.155944 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpwq2\" (UniqueName: \"kubernetes.io/projected/e22aa23e-9a2a-4499-bba1-4da8208e7386-kube-api-access-kpwq2\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:50.157654 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:50.157636 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f27c783a-fa71-4db1-a3bb-4ed989e72d08-kube-api-access-s56nj" pod "f27c783a-fa71-4db1-a3bb-4ed989e72d08" (UID: "f27c783a-fa71-4db1-a3bb-4ed989e72d08"). InnerVolumeSpecName "kube-api-access-s56nj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:38:50.257276 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:50.257183 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s56nj\" (UniqueName: \"kubernetes.io/projected/f27c783a-fa71-4db1-a3bb-4ed989e72d08-kube-api-access-s56nj\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:50.257276 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:50.257222 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f27c783a-fa71-4db1-a3bb-4ed989e72d08-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:50.262051 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:50.262019 5979 generic.go:356] "Generic (PLEG): container finished" podID="f27c783a-fa71-4db1-a3bb-4ed989e72d08" containerID="1f0b697479115d5629d5b5871550d76e288907c9d11bc661ea6f4662247fb2bb" exitCode=0 Nov 03 13:38:50.262200 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:50.262077 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5474b7-ccbcc59f-hlkd5" Nov 03 13:38:50.262200 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:50.262098 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5474b7-ccbcc59f-hlkd5" event={"ID":"f27c783a-fa71-4db1-a3bb-4ed989e72d08","Type":"ContainerDied","Data":"1f0b697479115d5629d5b5871550d76e288907c9d11bc661ea6f4662247fb2bb"} Nov 03 13:38:50.262200 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:50.262162 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5474b7-ccbcc59f-hlkd5" event={"ID":"f27c783a-fa71-4db1-a3bb-4ed989e72d08","Type":"ContainerDied","Data":"9275ff900b86f213d8a18e1148813c46f4fa7a72b2de4256ab048e8c0c9dbcb1"} Nov 03 13:38:50.262200 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:50.262194 5979 scope.go:122] "RemoveContainer" containerID="1f0b697479115d5629d5b5871550d76e288907c9d11bc661ea6f4662247fb2bb" Nov 03 13:38:50.263769 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:50.263739 5979 generic.go:356] "Generic (PLEG): container finished" podID="e22aa23e-9a2a-4499-bba1-4da8208e7386" containerID="250546879e63765f67a74a640f85d7484aa1591d7693464e3f614f2a3a7c8a58" exitCode=0 Nov 03 13:38:50.263885 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:50.263781 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5474b7-7496845f45-6ncjj" event={"ID":"e22aa23e-9a2a-4499-bba1-4da8208e7386","Type":"ContainerDied","Data":"250546879e63765f67a74a640f85d7484aa1591d7693464e3f614f2a3a7c8a58"} Nov 03 13:38:50.263885 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:50.263820 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5474b7-7496845f45-6ncjj" event={"ID":"e22aa23e-9a2a-4499-bba1-4da8208e7386","Type":"ContainerDied","Data":"f539d3e77508f72358be5df322617857a5bd29e1a8dfe18635868331a5181d83"} Nov 03 13:38:50.263885 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:50.263833 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5474b7-7496845f45-6ncjj" Nov 03 13:38:50.273926 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:50.273908 5979 scope.go:122] "RemoveContainer" containerID="1f0b697479115d5629d5b5871550d76e288907c9d11bc661ea6f4662247fb2bb" Nov 03 13:38:50.274183 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:38:50.274164 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f0b697479115d5629d5b5871550d76e288907c9d11bc661ea6f4662247fb2bb\": container with ID starting with 1f0b697479115d5629d5b5871550d76e288907c9d11bc661ea6f4662247fb2bb not found: ID does not exist" containerID="1f0b697479115d5629d5b5871550d76e288907c9d11bc661ea6f4662247fb2bb" Nov 03 13:38:50.274261 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:50.274196 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f0b697479115d5629d5b5871550d76e288907c9d11bc661ea6f4662247fb2bb"} err="failed to get container status \"1f0b697479115d5629d5b5871550d76e288907c9d11bc661ea6f4662247fb2bb\": rpc error: code = NotFound desc = could not find container \"1f0b697479115d5629d5b5871550d76e288907c9d11bc661ea6f4662247fb2bb\": container with ID starting with 1f0b697479115d5629d5b5871550d76e288907c9d11bc661ea6f4662247fb2bb not found: ID does not exist" Nov 03 13:38:50.274261 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:50.274221 5979 scope.go:122] "RemoveContainer" containerID="250546879e63765f67a74a640f85d7484aa1591d7693464e3f614f2a3a7c8a58" Nov 03 13:38:50.282812 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:50.282796 5979 scope.go:122] "RemoveContainer" containerID="250546879e63765f67a74a640f85d7484aa1591d7693464e3f614f2a3a7c8a58" Nov 03 13:38:50.283062 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:38:50.283044 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"250546879e63765f67a74a640f85d7484aa1591d7693464e3f614f2a3a7c8a58\": container with ID starting with 250546879e63765f67a74a640f85d7484aa1591d7693464e3f614f2a3a7c8a58 not found: ID does not exist" containerID="250546879e63765f67a74a640f85d7484aa1591d7693464e3f614f2a3a7c8a58" Nov 03 13:38:50.283111 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:50.283068 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"250546879e63765f67a74a640f85d7484aa1591d7693464e3f614f2a3a7c8a58"} err="failed to get container status \"250546879e63765f67a74a640f85d7484aa1591d7693464e3f614f2a3a7c8a58\": rpc error: code = NotFound desc = could not find container \"250546879e63765f67a74a640f85d7484aa1591d7693464e3f614f2a3a7c8a58\": container with ID starting with 250546879e63765f67a74a640f85d7484aa1591d7693464e3f614f2a3a7c8a58 not found: ID does not exist" Nov 03 13:38:50.420347 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:50.420316 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5474b7-ccbcc59f-hlkd5"] Nov 03 13:38:50.427611 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:50.427579 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5474b7-ccbcc59f-hlkd5"] Nov 03 13:38:50.539365 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:50.539272 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5474b7-7496845f45-6ncjj"] Nov 03 13:38:50.545708 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:50.545681 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5474b7-7496845f45-6ncjj"] Nov 03 13:38:51.239897 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:51.239863 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-21f111-9f4fb987-jb2sv" Nov 03 13:38:52.115977 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:52.115948 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e22aa23e-9a2a-4499-bba1-4da8208e7386" path="/var/lib/kubelet/pods/e22aa23e-9a2a-4499-bba1-4da8208e7386/volumes" Nov 03 13:38:52.116374 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:52.116361 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f27c783a-fa71-4db1-a3bb-4ed989e72d08" path="/var/lib/kubelet/pods/f27c783a-fa71-4db1-a3bb-4ed989e72d08/volumes" Nov 03 13:38:54.145851 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:54.145825 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s" Nov 03 13:38:54.224159 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:54.224121 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80f447c3-327a-4726-b976-8ce7adcd92fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80f447c3-327a-4726-b976-8ce7adcd92fa-istio-podinfo\") pod \"80f447c3-327a-4726-b976-8ce7adcd92fa\" (UID: \"80f447c3-327a-4726-b976-8ce7adcd92fa\") " Nov 03 13:38:54.224332 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:54.224174 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80f447c3-327a-4726-b976-8ce7adcd92fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80f447c3-327a-4726-b976-8ce7adcd92fa-workload-certs\") pod \"80f447c3-327a-4726-b976-8ce7adcd92fa\" (UID: \"80f447c3-327a-4726-b976-8ce7adcd92fa\") " Nov 03 13:38:54.224332 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:54.224223 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80f447c3-327a-4726-b976-8ce7adcd92fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80f447c3-327a-4726-b976-8ce7adcd92fa-istio-envoy\") pod \"80f447c3-327a-4726-b976-8ce7adcd92fa\" (UID: \"80f447c3-327a-4726-b976-8ce7adcd92fa\") " Nov 03 13:38:54.224332 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:54.224304 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80f447c3-327a-4726-b976-8ce7adcd92fa-kube-api-access-76t66\" (UniqueName: \"kubernetes.io/projected/80f447c3-327a-4726-b976-8ce7adcd92fa-kube-api-access-76t66\") pod \"80f447c3-327a-4726-b976-8ce7adcd92fa\" (UID: \"80f447c3-327a-4726-b976-8ce7adcd92fa\") " Nov 03 13:38:54.224332 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:54.224328 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80f447c3-327a-4726-b976-8ce7adcd92fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80f447c3-327a-4726-b976-8ce7adcd92fa-credential-socket\") pod \"80f447c3-327a-4726-b976-8ce7adcd92fa\" (UID: \"80f447c3-327a-4726-b976-8ce7adcd92fa\") " Nov 03 13:38:54.224620 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:54.224385 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80f447c3-327a-4726-b976-8ce7adcd92fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80f447c3-327a-4726-b976-8ce7adcd92fa-istio-data\") pod \"80f447c3-327a-4726-b976-8ce7adcd92fa\" (UID: \"80f447c3-327a-4726-b976-8ce7adcd92fa\") " Nov 03 13:38:54.224620 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:54.224437 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80f447c3-327a-4726-b976-8ce7adcd92fa-istio-token\" (UniqueName: \"kubernetes.io/projected/80f447c3-327a-4726-b976-8ce7adcd92fa-istio-token\") pod \"80f447c3-327a-4726-b976-8ce7adcd92fa\" (UID: \"80f447c3-327a-4726-b976-8ce7adcd92fa\") " Nov 03 13:38:54.224620 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:54.224558 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80f447c3-327a-4726-b976-8ce7adcd92fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80f447c3-327a-4726-b976-8ce7adcd92fa-istiod-ca-cert\") pod \"80f447c3-327a-4726-b976-8ce7adcd92fa\" (UID: \"80f447c3-327a-4726-b976-8ce7adcd92fa\") " Nov 03 13:38:54.224620 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:54.224583 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80f447c3-327a-4726-b976-8ce7adcd92fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80f447c3-327a-4726-b976-8ce7adcd92fa-workload-socket\") pod \"80f447c3-327a-4726-b976-8ce7adcd92fa\" (UID: \"80f447c3-327a-4726-b976-8ce7adcd92fa\") " Nov 03 13:38:54.224620 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:54.224602 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80f447c3-327a-4726-b976-8ce7adcd92fa-workload-certs" pod "80f447c3-327a-4726-b976-8ce7adcd92fa" (UID: "80f447c3-327a-4726-b976-8ce7adcd92fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:38:54.224897 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:54.224755 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80f447c3-327a-4726-b976-8ce7adcd92fa-credential-socket" pod "80f447c3-327a-4726-b976-8ce7adcd92fa" (UID: "80f447c3-327a-4726-b976-8ce7adcd92fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:38:54.224897 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:54.224834 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80f447c3-327a-4726-b976-8ce7adcd92fa-istio-data" pod "80f447c3-327a-4726-b976-8ce7adcd92fa" (UID: "80f447c3-327a-4726-b976-8ce7adcd92fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:38:54.225020 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:54.224977 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80f447c3-327a-4726-b976-8ce7adcd92fa-workload-socket" pod "80f447c3-327a-4726-b976-8ce7adcd92fa" (UID: "80f447c3-327a-4726-b976-8ce7adcd92fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:38:54.225178 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:54.225156 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80f447c3-327a-4726-b976-8ce7adcd92fa-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:54.225254 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:54.225183 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80f447c3-327a-4726-b976-8ce7adcd92fa-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:54.225254 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:54.225201 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80f447c3-327a-4726-b976-8ce7adcd92fa-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:54.225254 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:54.225216 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80f447c3-327a-4726-b976-8ce7adcd92fa-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:54.225254 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:54.225198 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80f447c3-327a-4726-b976-8ce7adcd92fa-istiod-ca-cert" pod "80f447c3-327a-4726-b976-8ce7adcd92fa" (UID: "80f447c3-327a-4726-b976-8ce7adcd92fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:38:54.226552 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:54.226525 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80f447c3-327a-4726-b976-8ce7adcd92fa-istio-podinfo" pod "80f447c3-327a-4726-b976-8ce7adcd92fa" (UID: "80f447c3-327a-4726-b976-8ce7adcd92fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:38:54.226954 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:54.226931 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80f447c3-327a-4726-b976-8ce7adcd92fa-istio-envoy" pod "80f447c3-327a-4726-b976-8ce7adcd92fa" (UID: "80f447c3-327a-4726-b976-8ce7adcd92fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:38:54.227013 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:54.226963 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80f447c3-327a-4726-b976-8ce7adcd92fa-istio-token" pod "80f447c3-327a-4726-b976-8ce7adcd92fa" (UID: "80f447c3-327a-4726-b976-8ce7adcd92fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:38:54.227071 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:54.227059 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80f447c3-327a-4726-b976-8ce7adcd92fa-kube-api-access-76t66" pod "80f447c3-327a-4726-b976-8ce7adcd92fa" (UID: "80f447c3-327a-4726-b976-8ce7adcd92fa"). InnerVolumeSpecName "kube-api-access-76t66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:38:54.292370 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:54.292333 5979 generic.go:356] "Generic (PLEG): container finished" podID="80f447c3-327a-4726-b976-8ce7adcd92fa" containerID="c7db051d1389b916fd8cefea8d7b19b247ad25bcbba96a1755555a3a1888375e" exitCode=0 Nov 03 13:38:54.292569 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:54.292391 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s" Nov 03 13:38:54.292569 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:54.292393 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s" event={"ID":"80f447c3-327a-4726-b976-8ce7adcd92fa","Type":"ContainerDied","Data":"c7db051d1389b916fd8cefea8d7b19b247ad25bcbba96a1755555a3a1888375e"} Nov 03 13:38:54.292569 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:54.292533 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s" event={"ID":"80f447c3-327a-4726-b976-8ce7adcd92fa","Type":"ContainerDied","Data":"14f13776cfca46fb84d4fc1270efa0248036247334865bf8e42d597d1c9db941"} Nov 03 13:38:54.292569 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:54.292562 5979 scope.go:122] "RemoveContainer" containerID="c7db051d1389b916fd8cefea8d7b19b247ad25bcbba96a1755555a3a1888375e" Nov 03 13:38:54.303054 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:54.303033 5979 scope.go:122] "RemoveContainer" containerID="0d6b04afa5b09b783bbb268c1ce484933b3ea0b2010040be34269f66ef17ef95" Nov 03 13:38:54.313760 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:54.313740 5979 scope.go:122] "RemoveContainer" containerID="c7db051d1389b916fd8cefea8d7b19b247ad25bcbba96a1755555a3a1888375e" Nov 03 13:38:54.314074 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:38:54.314054 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7db051d1389b916fd8cefea8d7b19b247ad25bcbba96a1755555a3a1888375e\": container with ID starting with c7db051d1389b916fd8cefea8d7b19b247ad25bcbba96a1755555a3a1888375e not found: ID does not exist" containerID="c7db051d1389b916fd8cefea8d7b19b247ad25bcbba96a1755555a3a1888375e" Nov 03 13:38:54.314126 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:54.314084 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7db051d1389b916fd8cefea8d7b19b247ad25bcbba96a1755555a3a1888375e"} err="failed to get container status \"c7db051d1389b916fd8cefea8d7b19b247ad25bcbba96a1755555a3a1888375e\": rpc error: code = NotFound desc = could not find container \"c7db051d1389b916fd8cefea8d7b19b247ad25bcbba96a1755555a3a1888375e\": container with ID starting with c7db051d1389b916fd8cefea8d7b19b247ad25bcbba96a1755555a3a1888375e not found: ID does not exist" Nov 03 13:38:54.314126 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:54.314102 5979 scope.go:122] "RemoveContainer" containerID="0d6b04afa5b09b783bbb268c1ce484933b3ea0b2010040be34269f66ef17ef95" Nov 03 13:38:54.314365 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:38:54.314346 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d6b04afa5b09b783bbb268c1ce484933b3ea0b2010040be34269f66ef17ef95\": container with ID starting with 0d6b04afa5b09b783bbb268c1ce484933b3ea0b2010040be34269f66ef17ef95 not found: ID does not exist" containerID="0d6b04afa5b09b783bbb268c1ce484933b3ea0b2010040be34269f66ef17ef95" Nov 03 13:38:54.314436 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:54.314374 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d6b04afa5b09b783bbb268c1ce484933b3ea0b2010040be34269f66ef17ef95"} err="failed to get container status \"0d6b04afa5b09b783bbb268c1ce484933b3ea0b2010040be34269f66ef17ef95\": rpc error: code = NotFound desc = could not find container \"0d6b04afa5b09b783bbb268c1ce484933b3ea0b2010040be34269f66ef17ef95\": container with ID starting with 0d6b04afa5b09b783bbb268c1ce484933b3ea0b2010040be34269f66ef17ef95 not found: ID does not exist" Nov 03 13:38:54.326172 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:54.326145 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80f447c3-327a-4726-b976-8ce7adcd92fa-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:54.326172 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:54.326172 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80f447c3-327a-4726-b976-8ce7adcd92fa-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:54.326172 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:54.326181 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80f447c3-327a-4726-b976-8ce7adcd92fa-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:54.326432 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:54.326189 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76t66\" (UniqueName: \"kubernetes.io/projected/80f447c3-327a-4726-b976-8ce7adcd92fa-kube-api-access-76t66\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:54.326432 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:54.326200 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80f447c3-327a-4726-b976-8ce7adcd92fa-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:38:54.408280 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:54.408247 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s"] Nov 03 13:38:54.414421 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:54.414396 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-be8d0a-openshift-default-7c766bb7c9-d952s"] Nov 03 13:38:56.116142 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:56.116106 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80f447c3-327a-4726-b976-8ce7adcd92fa" path="/var/lib/kubelet/pods/80f447c3-327a-4726-b976-8ce7adcd92fa/volumes" Nov 03 13:38:56.510525 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:38:56.510489 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:39:04.978246 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:04.978206 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a770b9-695d4dd84c-lzblq"] Nov 03 13:39:04.978864 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:04.978848 5979 cpu_manager.go:414] "containerMap: removing container" podUID="80f447c3-327a-4726-b976-8ce7adcd92fa" containerName="istio-proxy" Nov 03 13:39:04.978914 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:04.978865 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80f447c3-327a-4726-b976-8ce7adcd92fa" containerName="istio-proxy" Nov 03 13:39:04.978914 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:04.978883 5979 cpu_manager.go:414] "containerMap: removing container" podUID="e22aa23e-9a2a-4499-bba1-4da8208e7386" containerName="payload-processing" Nov 03 13:39:04.978914 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:04.978889 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e22aa23e-9a2a-4499-bba1-4da8208e7386" containerName="payload-processing" Nov 03 13:39:04.978914 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:04.978897 5979 cpu_manager.go:414] "containerMap: removing container" podUID="80f447c3-327a-4726-b976-8ce7adcd92fa" containerName="istio-proxy" Nov 03 13:39:04.978914 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:04.978903 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80f447c3-327a-4726-b976-8ce7adcd92fa" containerName="istio-proxy" Nov 03 13:39:04.978914 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:04.978912 5979 cpu_manager.go:414] "containerMap: removing container" podUID="f27c783a-fa71-4db1-a3bb-4ed989e72d08" containerName="payload-pre-processing" Nov 03 13:39:04.979101 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:04.978918 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f27c783a-fa71-4db1-a3bb-4ed989e72d08" containerName="payload-pre-processing" Nov 03 13:39:04.979101 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:04.979012 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="80f447c3-327a-4726-b976-8ce7adcd92fa" containerName="istio-proxy" Nov 03 13:39:04.979101 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:04.979022 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="e22aa23e-9a2a-4499-bba1-4da8208e7386" containerName="payload-processing" Nov 03 13:39:04.979101 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:04.979028 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="80f447c3-327a-4726-b976-8ce7adcd92fa" containerName="istio-proxy" Nov 03 13:39:04.979101 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:04.979033 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="f27c783a-fa71-4db1-a3bb-4ed989e72d08" containerName="payload-pre-processing" Nov 03 13:39:04.984153 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:04.984132 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a770b9-695d4dd84c-lzblq" Nov 03 13:39:05.068854 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:05.068811 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6vc6\" (UniqueName: \"kubernetes.io/projected/9af873cf-0227-40cc-ba24-6df2b62f4676-kube-api-access-b6vc6\") pod \"payload-pre-processing-e2e-ipp-a-a770b9-695d4dd84c-lzblq\" (UID: \"9af873cf-0227-40cc-ba24-6df2b62f4676\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a770b9-695d4dd84c-lzblq" Nov 03 13:39:05.069013 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:05.068880 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9af873cf-0227-40cc-ba24-6df2b62f4676-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a770b9-695d4dd84c-lzblq\" (UID: \"9af873cf-0227-40cc-ba24-6df2b62f4676\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a770b9-695d4dd84c-lzblq" Nov 03 13:39:05.080991 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:05.080962 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a770b9\"" Nov 03 13:39:05.081151 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:05.081006 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-a770b9-dockercfg-c7lxw\"" Nov 03 13:39:05.083818 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:05.083795 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a770b9-695d4dd84c-lzblq"] Nov 03 13:39:05.169985 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:05.169945 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6vc6\" (UniqueName: \"kubernetes.io/projected/9af873cf-0227-40cc-ba24-6df2b62f4676-kube-api-access-b6vc6\") pod \"payload-pre-processing-e2e-ipp-a-a770b9-695d4dd84c-lzblq\" (UID: \"9af873cf-0227-40cc-ba24-6df2b62f4676\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a770b9-695d4dd84c-lzblq" Nov 03 13:39:05.170209 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:05.170185 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9af873cf-0227-40cc-ba24-6df2b62f4676-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a770b9-695d4dd84c-lzblq\" (UID: \"9af873cf-0227-40cc-ba24-6df2b62f4676\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a770b9-695d4dd84c-lzblq" Nov 03 13:39:05.170837 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:05.170811 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9af873cf-0227-40cc-ba24-6df2b62f4676-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a770b9-695d4dd84c-lzblq\" (UID: \"9af873cf-0227-40cc-ba24-6df2b62f4676\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a770b9-695d4dd84c-lzblq" Nov 03 13:39:05.224331 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:05.224295 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6vc6\" (UniqueName: \"kubernetes.io/projected/9af873cf-0227-40cc-ba24-6df2b62f4676-kube-api-access-b6vc6\") pod \"payload-pre-processing-e2e-ipp-a-a770b9-695d4dd84c-lzblq\" (UID: \"9af873cf-0227-40cc-ba24-6df2b62f4676\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a770b9-695d4dd84c-lzblq" Nov 03 13:39:05.297760 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:05.297660 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a770b9-695d4dd84c-lzblq" Nov 03 13:39:05.657787 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:05.657751 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a770b9-774d768db-9zksn"] Nov 03 13:39:05.663112 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:05.663084 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a770b9-774d768db-9zksn" Nov 03 13:39:05.672003 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:05.671980 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a770b9-695d4dd84c-lzblq"] Nov 03 13:39:05.676119 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:05.676095 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2gr4\" (UniqueName: \"kubernetes.io/projected/5a9531dd-b724-4801-ac0d-71d42b7b3868-kube-api-access-t2gr4\") pod \"payload-processing-e2e-ipp-a-a770b9-774d768db-9zksn\" (UID: \"5a9531dd-b724-4801-ac0d-71d42b7b3868\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a770b9-774d768db-9zksn" Nov 03 13:39:05.676207 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:05.676137 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a9531dd-b724-4801-ac0d-71d42b7b3868-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a770b9-774d768db-9zksn\" (UID: \"5a9531dd-b724-4801-ac0d-71d42b7b3868\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a770b9-774d768db-9zksn" Nov 03 13:39:05.679810 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:39:05.679787 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9af873cf_0227_40cc_ba24_6df2b62f4676.slice/crio-dfc44eeb2543024552f4ea9ecdf82184c5c19854fd137af9a550fc9722043b75 WatchSource:0}: Error finding container dfc44eeb2543024552f4ea9ecdf82184c5c19854fd137af9a550fc9722043b75: Status 404 returned error can't find the container with id dfc44eeb2543024552f4ea9ecdf82184c5c19854fd137af9a550fc9722043b75 Nov 03 13:39:05.717976 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:05.717851 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a770b9-774d768db-9zksn"] Nov 03 13:39:05.777914 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:05.777875 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2gr4\" (UniqueName: \"kubernetes.io/projected/5a9531dd-b724-4801-ac0d-71d42b7b3868-kube-api-access-t2gr4\") pod \"payload-processing-e2e-ipp-a-a770b9-774d768db-9zksn\" (UID: \"5a9531dd-b724-4801-ac0d-71d42b7b3868\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a770b9-774d768db-9zksn" Nov 03 13:39:05.778075 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:05.778025 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a9531dd-b724-4801-ac0d-71d42b7b3868-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a770b9-774d768db-9zksn\" (UID: \"5a9531dd-b724-4801-ac0d-71d42b7b3868\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a770b9-774d768db-9zksn" Nov 03 13:39:05.778947 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:05.778922 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a9531dd-b724-4801-ac0d-71d42b7b3868-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a770b9-774d768db-9zksn\" (UID: \"5a9531dd-b724-4801-ac0d-71d42b7b3868\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a770b9-774d768db-9zksn" Nov 03 13:39:05.833302 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:05.833272 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2gr4\" (UniqueName: \"kubernetes.io/projected/5a9531dd-b724-4801-ac0d-71d42b7b3868-kube-api-access-t2gr4\") pod \"payload-processing-e2e-ipp-a-a770b9-774d768db-9zksn\" (UID: \"5a9531dd-b724-4801-ac0d-71d42b7b3868\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a770b9-774d768db-9zksn" Nov 03 13:39:05.978390 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:05.978310 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a770b9-774d768db-9zksn" Nov 03 13:39:06.358860 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:06.358782 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a770b9-774d768db-9zksn"] Nov 03 13:39:06.365903 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:39:06.365872 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a9531dd_b724_4801_ac0d_71d42b7b3868.slice/crio-0ea4d941755b689daeeb114604e744965eb6c30d7334cd43ecb5bf736804b339 WatchSource:0}: Error finding container 0ea4d941755b689daeeb114604e744965eb6c30d7334cd43ecb5bf736804b339: Status 404 returned error can't find the container with id 0ea4d941755b689daeeb114604e744965eb6c30d7334cd43ecb5bf736804b339 Nov 03 13:39:06.405127 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:06.405102 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d29e07-7bcf49d5f9-fxr6d"] Nov 03 13:39:06.405299 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:06.405279 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d29e07-7bcf49d5f9-fxr6d" podUID="ede9e063-9b8f-4ad9-80a1-2d841732c694" containerName="maas-api" containerID="cri-o://f95f512f06fd4218f13dbfb4bba5870dbdff2068e7cf8b45e73e5c7c7df99770" gracePeriod=30 Nov 03 13:39:06.419819 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:06.419784 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a770b9-695d4dd84c-lzblq" event={"ID":"9af873cf-0227-40cc-ba24-6df2b62f4676","Type":"ContainerStarted","Data":"a17fe2964ccdf910bc4838e42a185d6be9bdad884146e220954fe40a11f22b0d"} Nov 03 13:39:06.419968 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:06.419851 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a770b9-695d4dd84c-lzblq" event={"ID":"9af873cf-0227-40cc-ba24-6df2b62f4676","Type":"ContainerStarted","Data":"dfc44eeb2543024552f4ea9ecdf82184c5c19854fd137af9a550fc9722043b75"} Nov 03 13:39:06.420522 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:06.420489 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a770b9-695d4dd84c-lzblq" Nov 03 13:39:06.421993 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:06.421969 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a770b9-774d768db-9zksn" event={"ID":"5a9531dd-b724-4801-ac0d-71d42b7b3868","Type":"ContainerStarted","Data":"0ea4d941755b689daeeb114604e744965eb6c30d7334cd43ecb5bf736804b339"} Nov 03 13:39:06.632857 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:06.629676 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a770b9-695d4dd84c-lzblq" podStartSLOduration=2.62965774 podStartE2EDuration="2.62965774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:39:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:39:06.627883205 +0000 UTC m=+2028.638571449" watchObservedRunningTime="2026-11-03 13:39:06.62965774 +0000 UTC m=+2028.640345984" Nov 03 13:39:06.722674 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:06.722650 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d29e07-7bcf49d5f9-fxr6d" Nov 03 13:39:06.789443 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:06.789410 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ede9e063-9b8f-4ad9-80a1-2d841732c694-kube-api-access-q5nqz\" (UniqueName: \"kubernetes.io/projected/ede9e063-9b8f-4ad9-80a1-2d841732c694-kube-api-access-q5nqz\") pod \"ede9e063-9b8f-4ad9-80a1-2d841732c694\" (UID: \"ede9e063-9b8f-4ad9-80a1-2d841732c694\") " Nov 03 13:39:06.789629 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:06.789553 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ede9e063-9b8f-4ad9-80a1-2d841732c694-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ede9e063-9b8f-4ad9-80a1-2d841732c694-maas-api-tls\") pod \"ede9e063-9b8f-4ad9-80a1-2d841732c694\" (UID: \"ede9e063-9b8f-4ad9-80a1-2d841732c694\") " Nov 03 13:39:06.789720 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:06.789625 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ede9e063-9b8f-4ad9-80a1-2d841732c694-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ede9e063-9b8f-4ad9-80a1-2d841732c694-metrics-certs\") pod \"ede9e063-9b8f-4ad9-80a1-2d841732c694\" (UID: \"ede9e063-9b8f-4ad9-80a1-2d841732c694\") " Nov 03 13:39:06.789809 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:06.789788 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ede9e063-9b8f-4ad9-80a1-2d841732c694-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ede9e063-9b8f-4ad9-80a1-2d841732c694-openshift-service-ca\") pod \"ede9e063-9b8f-4ad9-80a1-2d841732c694\" (UID: \"ede9e063-9b8f-4ad9-80a1-2d841732c694\") " Nov 03 13:39:06.790208 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:06.790166 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ede9e063-9b8f-4ad9-80a1-2d841732c694-openshift-service-ca" pod "ede9e063-9b8f-4ad9-80a1-2d841732c694" (UID: "ede9e063-9b8f-4ad9-80a1-2d841732c694"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:39:06.790548 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:06.790525 5979 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ede9e063-9b8f-4ad9-80a1-2d841732c694-openshift-service-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:06.791697 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:06.791673 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ede9e063-9b8f-4ad9-80a1-2d841732c694-kube-api-access-q5nqz" pod "ede9e063-9b8f-4ad9-80a1-2d841732c694" (UID: "ede9e063-9b8f-4ad9-80a1-2d841732c694"). InnerVolumeSpecName "kube-api-access-q5nqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:39:06.792034 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:06.791997 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ede9e063-9b8f-4ad9-80a1-2d841732c694-maas-api-tls" pod "ede9e063-9b8f-4ad9-80a1-2d841732c694" (UID: "ede9e063-9b8f-4ad9-80a1-2d841732c694"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:39:06.792034 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:06.792017 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ede9e063-9b8f-4ad9-80a1-2d841732c694-metrics-certs" pod "ede9e063-9b8f-4ad9-80a1-2d841732c694" (UID: "ede9e063-9b8f-4ad9-80a1-2d841732c694"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:39:06.901155 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:06.895855 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5nqz\" (UniqueName: \"kubernetes.io/projected/ede9e063-9b8f-4ad9-80a1-2d841732c694-kube-api-access-q5nqz\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:06.901155 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:06.895908 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ede9e063-9b8f-4ad9-80a1-2d841732c694-maas-api-tls\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:06.901155 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:06.895923 5979 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ede9e063-9b8f-4ad9-80a1-2d841732c694-metrics-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:06.935011 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:06.934972 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d29e07-6b54b7bcd6-lqtkq"] Nov 03 13:39:06.935196 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:06.935171 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-d29e07-6b54b7bcd6-lqtkq" podUID="f32a7513-df8c-4286-b484-0b67f7a5329e" containerName="payload-processing" containerID="cri-o://4284951c53fff3f5071588d6c09e6d23d5f6d3da8394c21276e67966d88e65c1" gracePeriod=30 Nov 03 13:39:06.993606 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:06.993434 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d29e07-5585b5fc95-2hf65"] Nov 03 13:39:06.994163 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:06.993770 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d29e07-5585b5fc95-2hf65" podUID="7346ca1b-5758-4634-b0b7-19f8b2a52cfb" containerName="payload-pre-processing" containerID="cri-o://0ccea9b4faf4a0ab70ddbee861b6c0378c3f5a1020f051b436e31b0ae44873eb" gracePeriod=30 Nov 03 13:39:07.253141 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:39:07.253099 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:39:07.288307 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:07.288004 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d29e07-6b54b7bcd6-lqtkq" Nov 03 13:39:07.300448 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:07.300425 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f32a7513-df8c-4286-b484-0b67f7a5329e-kube-api-access-6jdt9\" (UniqueName: \"kubernetes.io/projected/f32a7513-df8c-4286-b484-0b67f7a5329e-kube-api-access-6jdt9\") pod \"f32a7513-df8c-4286-b484-0b67f7a5329e\" (UID: \"f32a7513-df8c-4286-b484-0b67f7a5329e\") " Nov 03 13:39:07.300528 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:07.300510 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f32a7513-df8c-4286-b484-0b67f7a5329e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f32a7513-df8c-4286-b484-0b67f7a5329e-plugins-config-volume\") pod \"f32a7513-df8c-4286-b484-0b67f7a5329e\" (UID: \"f32a7513-df8c-4286-b484-0b67f7a5329e\") " Nov 03 13:39:07.301007 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:07.300981 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f32a7513-df8c-4286-b484-0b67f7a5329e-plugins-config-volume" pod "f32a7513-df8c-4286-b484-0b67f7a5329e" (UID: "f32a7513-df8c-4286-b484-0b67f7a5329e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:39:07.302576 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:07.302549 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f32a7513-df8c-4286-b484-0b67f7a5329e-kube-api-access-6jdt9" pod "f32a7513-df8c-4286-b484-0b67f7a5329e" (UID: "f32a7513-df8c-4286-b484-0b67f7a5329e"). InnerVolumeSpecName "kube-api-access-6jdt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:39:07.322136 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:07.322114 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d29e07-5585b5fc95-2hf65" Nov 03 13:39:07.401682 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:07.401647 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7346ca1b-5758-4634-b0b7-19f8b2a52cfb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7346ca1b-5758-4634-b0b7-19f8b2a52cfb-plugins-config-volume\") pod \"7346ca1b-5758-4634-b0b7-19f8b2a52cfb\" (UID: \"7346ca1b-5758-4634-b0b7-19f8b2a52cfb\") " Nov 03 13:39:07.401902 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:07.401825 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7346ca1b-5758-4634-b0b7-19f8b2a52cfb-kube-api-access-bzh9v\" (UniqueName: \"kubernetes.io/projected/7346ca1b-5758-4634-b0b7-19f8b2a52cfb-kube-api-access-bzh9v\") pod \"7346ca1b-5758-4634-b0b7-19f8b2a52cfb\" (UID: \"7346ca1b-5758-4634-b0b7-19f8b2a52cfb\") " Nov 03 13:39:07.402100 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:07.402076 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7346ca1b-5758-4634-b0b7-19f8b2a52cfb-plugins-config-volume" pod "7346ca1b-5758-4634-b0b7-19f8b2a52cfb" (UID: "7346ca1b-5758-4634-b0b7-19f8b2a52cfb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:39:07.402255 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:07.402238 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jdt9\" (UniqueName: \"kubernetes.io/projected/f32a7513-df8c-4286-b484-0b67f7a5329e-kube-api-access-6jdt9\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:07.402323 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:07.402256 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7346ca1b-5758-4634-b0b7-19f8b2a52cfb-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:07.402323 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:07.402266 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f32a7513-df8c-4286-b484-0b67f7a5329e-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:07.404393 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:07.404358 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7346ca1b-5758-4634-b0b7-19f8b2a52cfb-kube-api-access-bzh9v" pod "7346ca1b-5758-4634-b0b7-19f8b2a52cfb" (UID: "7346ca1b-5758-4634-b0b7-19f8b2a52cfb"). InnerVolumeSpecName "kube-api-access-bzh9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:39:07.431116 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:07.431034 5979 generic.go:356] "Generic (PLEG): container finished" podID="ede9e063-9b8f-4ad9-80a1-2d841732c694" containerID="f95f512f06fd4218f13dbfb4bba5870dbdff2068e7cf8b45e73e5c7c7df99770" exitCode=0 Nov 03 13:39:07.431116 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:07.431085 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d29e07-7bcf49d5f9-fxr6d" Nov 03 13:39:07.431318 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:07.431128 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d29e07-7bcf49d5f9-fxr6d" event={"ID":"ede9e063-9b8f-4ad9-80a1-2d841732c694","Type":"ContainerDied","Data":"f95f512f06fd4218f13dbfb4bba5870dbdff2068e7cf8b45e73e5c7c7df99770"} Nov 03 13:39:07.431318 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:07.431197 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d29e07-7bcf49d5f9-fxr6d" event={"ID":"ede9e063-9b8f-4ad9-80a1-2d841732c694","Type":"ContainerDied","Data":"d04a8a7f88c64a61e7a7ff234c96ee4b44062c0ef44dad1f4d7adaf575ca8d93"} Nov 03 13:39:07.431318 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:07.431229 5979 scope.go:122] "RemoveContainer" containerID="f95f512f06fd4218f13dbfb4bba5870dbdff2068e7cf8b45e73e5c7c7df99770" Nov 03 13:39:07.432740 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:07.432606 5979 generic.go:356] "Generic (PLEG): container finished" podID="f32a7513-df8c-4286-b484-0b67f7a5329e" containerID="4284951c53fff3f5071588d6c09e6d23d5f6d3da8394c21276e67966d88e65c1" exitCode=0 Nov 03 13:39:07.432740 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:07.432649 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d29e07-6b54b7bcd6-lqtkq" Nov 03 13:39:07.432740 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:07.432684 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d29e07-6b54b7bcd6-lqtkq" event={"ID":"f32a7513-df8c-4286-b484-0b67f7a5329e","Type":"ContainerDied","Data":"4284951c53fff3f5071588d6c09e6d23d5f6d3da8394c21276e67966d88e65c1"} Nov 03 13:39:07.432740 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:07.432735 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d29e07-6b54b7bcd6-lqtkq" event={"ID":"f32a7513-df8c-4286-b484-0b67f7a5329e","Type":"ContainerDied","Data":"6e829bc99bf1d05ffc0d130161a6948a73bf70e45c9519f6ddc1b5867e3e2be3"} Nov 03 13:39:07.434443 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:07.434416 5979 generic.go:356] "Generic (PLEG): container finished" podID="7346ca1b-5758-4634-b0b7-19f8b2a52cfb" containerID="0ccea9b4faf4a0ab70ddbee861b6c0378c3f5a1020f051b436e31b0ae44873eb" exitCode=0 Nov 03 13:39:07.434599 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:07.434526 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d29e07-5585b5fc95-2hf65" event={"ID":"7346ca1b-5758-4634-b0b7-19f8b2a52cfb","Type":"ContainerDied","Data":"0ccea9b4faf4a0ab70ddbee861b6c0378c3f5a1020f051b436e31b0ae44873eb"} Nov 03 13:39:07.434599 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:07.434564 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d29e07-5585b5fc95-2hf65" event={"ID":"7346ca1b-5758-4634-b0b7-19f8b2a52cfb","Type":"ContainerDied","Data":"af891a67a686c3c368ae18ec9110d2e0c68d05f5c1916ed719eb2acc83fb1d88"} Nov 03 13:39:07.434732 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:07.434627 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d29e07-5585b5fc95-2hf65" Nov 03 13:39:07.439838 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:07.439811 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a770b9-774d768db-9zksn" event={"ID":"5a9531dd-b724-4801-ac0d-71d42b7b3868","Type":"ContainerStarted","Data":"037612024f52fde38513e0b0b3e6ac6721c15cd3043c91488e9411e2ad4e9cd9"} Nov 03 13:39:07.439974 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:07.439862 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a770b9-774d768db-9zksn" Nov 03 13:39:07.445261 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:07.445240 5979 scope.go:122] "RemoveContainer" containerID="f95f512f06fd4218f13dbfb4bba5870dbdff2068e7cf8b45e73e5c7c7df99770" Nov 03 13:39:07.445586 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:39:07.445565 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f95f512f06fd4218f13dbfb4bba5870dbdff2068e7cf8b45e73e5c7c7df99770\": container with ID starting with f95f512f06fd4218f13dbfb4bba5870dbdff2068e7cf8b45e73e5c7c7df99770 not found: ID does not exist" containerID="f95f512f06fd4218f13dbfb4bba5870dbdff2068e7cf8b45e73e5c7c7df99770" Nov 03 13:39:07.445665 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:07.445594 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f95f512f06fd4218f13dbfb4bba5870dbdff2068e7cf8b45e73e5c7c7df99770"} err="failed to get container status \"f95f512f06fd4218f13dbfb4bba5870dbdff2068e7cf8b45e73e5c7c7df99770\": rpc error: code = NotFound desc = could not find container \"f95f512f06fd4218f13dbfb4bba5870dbdff2068e7cf8b45e73e5c7c7df99770\": container with ID starting with f95f512f06fd4218f13dbfb4bba5870dbdff2068e7cf8b45e73e5c7c7df99770 not found: ID does not exist" Nov 03 13:39:07.445665 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:07.445610 5979 scope.go:122] "RemoveContainer" containerID="4284951c53fff3f5071588d6c09e6d23d5f6d3da8394c21276e67966d88e65c1" Nov 03 13:39:07.454785 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:07.454767 5979 scope.go:122] "RemoveContainer" containerID="4284951c53fff3f5071588d6c09e6d23d5f6d3da8394c21276e67966d88e65c1" Nov 03 13:39:07.455050 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:39:07.455031 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4284951c53fff3f5071588d6c09e6d23d5f6d3da8394c21276e67966d88e65c1\": container with ID starting with 4284951c53fff3f5071588d6c09e6d23d5f6d3da8394c21276e67966d88e65c1 not found: ID does not exist" containerID="4284951c53fff3f5071588d6c09e6d23d5f6d3da8394c21276e67966d88e65c1" Nov 03 13:39:07.455138 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:07.455061 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4284951c53fff3f5071588d6c09e6d23d5f6d3da8394c21276e67966d88e65c1"} err="failed to get container status \"4284951c53fff3f5071588d6c09e6d23d5f6d3da8394c21276e67966d88e65c1\": rpc error: code = NotFound desc = could not find container \"4284951c53fff3f5071588d6c09e6d23d5f6d3da8394c21276e67966d88e65c1\": container with ID starting with 4284951c53fff3f5071588d6c09e6d23d5f6d3da8394c21276e67966d88e65c1 not found: ID does not exist" Nov 03 13:39:07.455138 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:07.455084 5979 scope.go:122] "RemoveContainer" containerID="0ccea9b4faf4a0ab70ddbee861b6c0378c3f5a1020f051b436e31b0ae44873eb" Nov 03 13:39:07.486717 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:07.470525 5979 scope.go:122] "RemoveContainer" containerID="0ccea9b4faf4a0ab70ddbee861b6c0378c3f5a1020f051b436e31b0ae44873eb" Nov 03 13:39:07.486717 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:39:07.470937 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ccea9b4faf4a0ab70ddbee861b6c0378c3f5a1020f051b436e31b0ae44873eb\": container with ID starting with 0ccea9b4faf4a0ab70ddbee861b6c0378c3f5a1020f051b436e31b0ae44873eb not found: ID does not exist" containerID="0ccea9b4faf4a0ab70ddbee861b6c0378c3f5a1020f051b436e31b0ae44873eb" Nov 03 13:39:07.486717 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:07.470973 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ccea9b4faf4a0ab70ddbee861b6c0378c3f5a1020f051b436e31b0ae44873eb"} err="failed to get container status \"0ccea9b4faf4a0ab70ddbee861b6c0378c3f5a1020f051b436e31b0ae44873eb\": rpc error: code = NotFound desc = could not find container \"0ccea9b4faf4a0ab70ddbee861b6c0378c3f5a1020f051b436e31b0ae44873eb\": container with ID starting with 0ccea9b4faf4a0ab70ddbee861b6c0378c3f5a1020f051b436e31b0ae44873eb not found: ID does not exist" Nov 03 13:39:07.507494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:07.503558 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzh9v\" (UniqueName: \"kubernetes.io/projected/7346ca1b-5758-4634-b0b7-19f8b2a52cfb-kube-api-access-bzh9v\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:07.641536 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:07.641497 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d29e07-6b54b7bcd6-lqtkq"] Nov 03 13:39:07.644207 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:07.644184 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d29e07-6b54b7bcd6-lqtkq"] Nov 03 13:39:07.752154 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:07.752050 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-a770b9-774d768db-9zksn" podStartSLOduration=2.752038175 podStartE2EDuration="2.752038175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:39:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:39:07.750650671 +0000 UTC m=+2029.761338913" watchObservedRunningTime="2026-11-03 13:39:07.752038175 +0000 UTC m=+2029.762726417" Nov 03 13:39:07.863275 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:07.861394 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d29e07-5585b5fc95-2hf65"] Nov 03 13:39:07.896013 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:07.868106 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d29e07-5585b5fc95-2hf65"] Nov 03 13:39:07.974504 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:07.974455 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d29e07-7bcf49d5f9-fxr6d"] Nov 03 13:39:07.976616 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:07.976589 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d29e07-7bcf49d5f9-fxr6d"] Nov 03 13:39:08.116570 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:08.116460 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7346ca1b-5758-4634-b0b7-19f8b2a52cfb" path="/var/lib/kubelet/pods/7346ca1b-5758-4634-b0b7-19f8b2a52cfb/volumes" Nov 03 13:39:08.117087 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:08.117068 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ede9e063-9b8f-4ad9-80a1-2d841732c694" path="/var/lib/kubelet/pods/ede9e063-9b8f-4ad9-80a1-2d841732c694/volumes" Nov 03 13:39:08.117519 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:08.117498 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f32a7513-df8c-4286-b484-0b67f7a5329e" path="/var/lib/kubelet/pods/f32a7513-df8c-4286-b484-0b67f7a5329e/volumes" Nov 03 13:39:09.608703 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:09.608660 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d29e07-7bcf49d5f9-fxr6d" podUID="ede9e063-9b8f-4ad9-80a1-2d841732c694" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.178:8443/health\": context deadline exceeded" Nov 03 13:39:12.946524 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:12.946490 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l"] Nov 03 13:39:12.946963 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:12.946722 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l" podUID="ddd1f97b-814c-43b5-a81b-201bd1ef2c9b" containerName="istio-proxy" containerID="cri-o://3948467f6687f08d0693e3c03d844c4a4db3c45208c7a9243eac83fc3b2d6d1e" gracePeriod=30 Nov 03 13:39:16.211831 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:16.211795 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686"] Nov 03 13:39:16.212628 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:16.212441 5979 cpu_manager.go:414] "containerMap: removing container" podUID="7346ca1b-5758-4634-b0b7-19f8b2a52cfb" containerName="payload-pre-processing" Nov 03 13:39:16.212628 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:16.212457 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7346ca1b-5758-4634-b0b7-19f8b2a52cfb" containerName="payload-pre-processing" Nov 03 13:39:16.212628 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:16.212507 5979 cpu_manager.go:414] "containerMap: removing container" podUID="ede9e063-9b8f-4ad9-80a1-2d841732c694" containerName="maas-api" Nov 03 13:39:16.212628 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:16.212516 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ede9e063-9b8f-4ad9-80a1-2d841732c694" containerName="maas-api" Nov 03 13:39:16.212628 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:16.212554 5979 cpu_manager.go:414] "containerMap: removing container" podUID="f32a7513-df8c-4286-b484-0b67f7a5329e" containerName="payload-processing" Nov 03 13:39:16.212628 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:16.212562 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f32a7513-df8c-4286-b484-0b67f7a5329e" containerName="payload-processing" Nov 03 13:39:16.212932 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:16.212681 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="f32a7513-df8c-4286-b484-0b67f7a5329e" containerName="payload-processing" Nov 03 13:39:16.212932 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:16.212700 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="7346ca1b-5758-4634-b0b7-19f8b2a52cfb" containerName="payload-pre-processing" Nov 03 13:39:16.212932 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:16.212717 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="ede9e063-9b8f-4ad9-80a1-2d841732c694" containerName="maas-api" Nov 03 13:39:16.215850 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:16.215828 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686" Nov 03 13:39:16.273858 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:16.273820 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-8d8af2-openshift-default-dockercfg-slskt\"" Nov 03 13:39:16.338150 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:16.338106 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686"] Nov 03 13:39:16.346969 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:16.346928 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87552227-fce6-474b-9b1b-fe167d1f6547-istio-data\") pod \"e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686\" (UID: \"87552227-fce6-474b-9b1b-fe167d1f6547\") " pod="openshift-ingress/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686" Nov 03 13:39:16.353726 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:16.353694 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87552227-fce6-474b-9b1b-fe167d1f6547-istiod-ca-cert\") pod \"e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686\" (UID: \"87552227-fce6-474b-9b1b-fe167d1f6547\") " pod="openshift-ingress/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686" Nov 03 13:39:16.353891 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:16.353842 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87552227-fce6-474b-9b1b-fe167d1f6547-istio-podinfo\") pod \"e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686\" (UID: \"87552227-fce6-474b-9b1b-fe167d1f6547\") " pod="openshift-ingress/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686" Nov 03 13:39:16.353997 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:16.353966 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87552227-fce6-474b-9b1b-fe167d1f6547-workload-socket\") pod \"e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686\" (UID: \"87552227-fce6-474b-9b1b-fe167d1f6547\") " pod="openshift-ingress/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686" Nov 03 13:39:16.354073 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:16.354021 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87552227-fce6-474b-9b1b-fe167d1f6547-istio-token\") pod \"e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686\" (UID: \"87552227-fce6-474b-9b1b-fe167d1f6547\") " pod="openshift-ingress/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686" Nov 03 13:39:16.354134 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:16.354071 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87552227-fce6-474b-9b1b-fe167d1f6547-credential-socket\") pod \"e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686\" (UID: \"87552227-fce6-474b-9b1b-fe167d1f6547\") " pod="openshift-ingress/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686" Nov 03 13:39:16.354134 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:16.354112 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87552227-fce6-474b-9b1b-fe167d1f6547-istio-envoy\") pod \"e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686\" (UID: \"87552227-fce6-474b-9b1b-fe167d1f6547\") " pod="openshift-ingress/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686" Nov 03 13:39:16.354251 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:16.354180 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hdb2\" (UniqueName: \"kubernetes.io/projected/87552227-fce6-474b-9b1b-fe167d1f6547-kube-api-access-8hdb2\") pod \"e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686\" (UID: \"87552227-fce6-474b-9b1b-fe167d1f6547\") " pod="openshift-ingress/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686" Nov 03 13:39:16.354251 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:16.354224 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87552227-fce6-474b-9b1b-fe167d1f6547-workload-certs\") pod \"e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686\" (UID: \"87552227-fce6-474b-9b1b-fe167d1f6547\") " pod="openshift-ingress/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686" Nov 03 13:39:16.455556 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:16.455524 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87552227-fce6-474b-9b1b-fe167d1f6547-workload-socket\") pod \"e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686\" (UID: \"87552227-fce6-474b-9b1b-fe167d1f6547\") " pod="openshift-ingress/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686" Nov 03 13:39:16.455735 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:16.455563 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87552227-fce6-474b-9b1b-fe167d1f6547-istio-token\") pod \"e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686\" (UID: \"87552227-fce6-474b-9b1b-fe167d1f6547\") " pod="openshift-ingress/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686" Nov 03 13:39:16.455735 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:16.455595 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87552227-fce6-474b-9b1b-fe167d1f6547-credential-socket\") pod \"e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686\" (UID: \"87552227-fce6-474b-9b1b-fe167d1f6547\") " pod="openshift-ingress/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686" Nov 03 13:39:16.455735 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:16.455639 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87552227-fce6-474b-9b1b-fe167d1f6547-istio-envoy\") pod \"e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686\" (UID: \"87552227-fce6-474b-9b1b-fe167d1f6547\") " pod="openshift-ingress/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686" Nov 03 13:39:16.455735 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:16.455723 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hdb2\" (UniqueName: \"kubernetes.io/projected/87552227-fce6-474b-9b1b-fe167d1f6547-kube-api-access-8hdb2\") pod \"e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686\" (UID: \"87552227-fce6-474b-9b1b-fe167d1f6547\") " pod="openshift-ingress/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686" Nov 03 13:39:16.456012 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:16.455772 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87552227-fce6-474b-9b1b-fe167d1f6547-workload-certs\") pod \"e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686\" (UID: \"87552227-fce6-474b-9b1b-fe167d1f6547\") " pod="openshift-ingress/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686" Nov 03 13:39:16.456012 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:16.455858 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87552227-fce6-474b-9b1b-fe167d1f6547-istio-data\") pod \"e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686\" (UID: \"87552227-fce6-474b-9b1b-fe167d1f6547\") " pod="openshift-ingress/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686" Nov 03 13:39:16.456012 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:16.455883 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87552227-fce6-474b-9b1b-fe167d1f6547-istiod-ca-cert\") pod \"e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686\" (UID: \"87552227-fce6-474b-9b1b-fe167d1f6547\") " pod="openshift-ingress/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686" Nov 03 13:39:16.456012 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:16.455935 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87552227-fce6-474b-9b1b-fe167d1f6547-workload-socket\") pod \"e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686\" (UID: \"87552227-fce6-474b-9b1b-fe167d1f6547\") " pod="openshift-ingress/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686" Nov 03 13:39:16.456012 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:16.455948 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87552227-fce6-474b-9b1b-fe167d1f6547-istio-podinfo\") pod \"e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686\" (UID: \"87552227-fce6-474b-9b1b-fe167d1f6547\") " pod="openshift-ingress/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686" Nov 03 13:39:16.456313 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:16.456056 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87552227-fce6-474b-9b1b-fe167d1f6547-credential-socket\") pod \"e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686\" (UID: \"87552227-fce6-474b-9b1b-fe167d1f6547\") " pod="openshift-ingress/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686" Nov 03 13:39:16.456313 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:16.456157 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87552227-fce6-474b-9b1b-fe167d1f6547-workload-certs\") pod \"e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686\" (UID: \"87552227-fce6-474b-9b1b-fe167d1f6547\") " pod="openshift-ingress/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686" Nov 03 13:39:16.456313 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:16.456289 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87552227-fce6-474b-9b1b-fe167d1f6547-istio-data\") pod \"e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686\" (UID: \"87552227-fce6-474b-9b1b-fe167d1f6547\") " pod="openshift-ingress/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686" Nov 03 13:39:16.456830 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:16.456793 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87552227-fce6-474b-9b1b-fe167d1f6547-istiod-ca-cert\") pod \"e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686\" (UID: \"87552227-fce6-474b-9b1b-fe167d1f6547\") " pod="openshift-ingress/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686" Nov 03 13:39:16.458350 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:16.458326 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87552227-fce6-474b-9b1b-fe167d1f6547-istio-envoy\") pod \"e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686\" (UID: \"87552227-fce6-474b-9b1b-fe167d1f6547\") " pod="openshift-ingress/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686" Nov 03 13:39:16.458456 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:16.458430 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87552227-fce6-474b-9b1b-fe167d1f6547-istio-podinfo\") pod \"e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686\" (UID: \"87552227-fce6-474b-9b1b-fe167d1f6547\") " pod="openshift-ingress/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686" Nov 03 13:39:16.513308 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:16.513233 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87552227-fce6-474b-9b1b-fe167d1f6547-istio-token\") pod \"e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686\" (UID: \"87552227-fce6-474b-9b1b-fe167d1f6547\") " pod="openshift-ingress/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686" Nov 03 13:39:16.513537 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:16.513515 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hdb2\" (UniqueName: \"kubernetes.io/projected/87552227-fce6-474b-9b1b-fe167d1f6547-kube-api-access-8hdb2\") pod \"e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686\" (UID: \"87552227-fce6-474b-9b1b-fe167d1f6547\") " pod="openshift-ingress/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686" Nov 03 13:39:16.530217 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:16.530190 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686" Nov 03 13:39:16.991012 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:16.990974 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686"] Nov 03 13:39:16.997482 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:39:16.997446 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87552227_fce6_474b_9b1b_fe167d1f6547.slice/crio-717b25c78539d81f95df98d2710b46a0bae20cedb3b26a507d1fd99a9bd22f9e WatchSource:0}: Error finding container 717b25c78539d81f95df98d2710b46a0bae20cedb3b26a507d1fd99a9bd22f9e: Status 404 returned error can't find the container with id 717b25c78539d81f95df98d2710b46a0bae20cedb3b26a507d1fd99a9bd22f9e Nov 03 13:39:17.000044 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:17.000002 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:39:17.000132 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:17.000094 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:39:17.000172 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:17.000142 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:39:17.532865 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:17.532827 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686" event={"ID":"87552227-fce6-474b-9b1b-fe167d1f6547","Type":"ContainerStarted","Data":"1d343e63cb3beedf3c9030d3ec7edaba1256dd00b507a492462edf1af793c031"} Nov 03 13:39:17.533274 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:17.532886 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686" event={"ID":"87552227-fce6-474b-9b1b-fe167d1f6547","Type":"ContainerStarted","Data":"717b25c78539d81f95df98d2710b46a0bae20cedb3b26a507d1fd99a9bd22f9e"} Nov 03 13:39:17.644241 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:17.644182 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686" podStartSLOduration=1.644169177 podStartE2EDuration="1.644169177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:39:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:39:17.641844192 +0000 UTC m=+2039.652532434" watchObservedRunningTime="2026-11-03 13:39:17.644169177 +0000 UTC m=+2039.654857419" Nov 03 13:39:18.211902 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:18.211835 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l" Nov 03 13:39:18.284002 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:18.283974 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-istio-envoy\") pod \"ddd1f97b-814c-43b5-a81b-201bd1ef2c9b\" (UID: \"ddd1f97b-814c-43b5-a81b-201bd1ef2c9b\") " Nov 03 13:39:18.284168 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:18.284008 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-istiod-ca-cert\") pod \"ddd1f97b-814c-43b5-a81b-201bd1ef2c9b\" (UID: \"ddd1f97b-814c-43b5-a81b-201bd1ef2c9b\") " Nov 03 13:39:18.284168 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:18.284040 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-workload-socket\") pod \"ddd1f97b-814c-43b5-a81b-201bd1ef2c9b\" (UID: \"ddd1f97b-814c-43b5-a81b-201bd1ef2c9b\") " Nov 03 13:39:18.284168 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:18.284062 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-credential-socket\") pod \"ddd1f97b-814c-43b5-a81b-201bd1ef2c9b\" (UID: \"ddd1f97b-814c-43b5-a81b-201bd1ef2c9b\") " Nov 03 13:39:18.284168 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:18.284107 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-workload-certs\") pod \"ddd1f97b-814c-43b5-a81b-201bd1ef2c9b\" (UID: \"ddd1f97b-814c-43b5-a81b-201bd1ef2c9b\") " Nov 03 13:39:18.284358 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:18.284211 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-istio-token\" (UniqueName: \"kubernetes.io/projected/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-istio-token\") pod \"ddd1f97b-814c-43b5-a81b-201bd1ef2c9b\" (UID: \"ddd1f97b-814c-43b5-a81b-201bd1ef2c9b\") " Nov 03 13:39:18.284358 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:18.284254 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-kube-api-access-ht9mb\" (UniqueName: \"kubernetes.io/projected/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-kube-api-access-ht9mb\") pod \"ddd1f97b-814c-43b5-a81b-201bd1ef2c9b\" (UID: \"ddd1f97b-814c-43b5-a81b-201bd1ef2c9b\") " Nov 03 13:39:18.284358 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:18.284331 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-workload-socket" pod "ddd1f97b-814c-43b5-a81b-201bd1ef2c9b" (UID: "ddd1f97b-814c-43b5-a81b-201bd1ef2c9b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:39:18.284554 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:18.284343 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-istio-podinfo\") pod \"ddd1f97b-814c-43b5-a81b-201bd1ef2c9b\" (UID: \"ddd1f97b-814c-43b5-a81b-201bd1ef2c9b\") " Nov 03 13:39:18.284554 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:18.284390 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-credential-socket" pod "ddd1f97b-814c-43b5-a81b-201bd1ef2c9b" (UID: "ddd1f97b-814c-43b5-a81b-201bd1ef2c9b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:39:18.284554 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:18.284493 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-istiod-ca-cert" pod "ddd1f97b-814c-43b5-a81b-201bd1ef2c9b" (UID: "ddd1f97b-814c-43b5-a81b-201bd1ef2c9b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:39:18.284554 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:18.284516 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-workload-certs" pod "ddd1f97b-814c-43b5-a81b-201bd1ef2c9b" (UID: "ddd1f97b-814c-43b5-a81b-201bd1ef2c9b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:39:18.284554 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:18.284526 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-istio-data\") pod \"ddd1f97b-814c-43b5-a81b-201bd1ef2c9b\" (UID: \"ddd1f97b-814c-43b5-a81b-201bd1ef2c9b\") " Nov 03 13:39:18.284848 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:18.284755 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-istio-data" pod "ddd1f97b-814c-43b5-a81b-201bd1ef2c9b" (UID: "ddd1f97b-814c-43b5-a81b-201bd1ef2c9b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:39:18.285136 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:18.285113 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:18.285227 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:18.285140 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:18.285227 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:18.285156 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:18.285227 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:18.285170 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:18.285227 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:18.285185 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:18.286211 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:18.286185 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-istio-envoy" pod "ddd1f97b-814c-43b5-a81b-201bd1ef2c9b" (UID: "ddd1f97b-814c-43b5-a81b-201bd1ef2c9b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:39:18.286453 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:18.286434 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-istio-podinfo" pod "ddd1f97b-814c-43b5-a81b-201bd1ef2c9b" (UID: "ddd1f97b-814c-43b5-a81b-201bd1ef2c9b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:39:18.286549 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:18.286489 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-istio-token" pod "ddd1f97b-814c-43b5-a81b-201bd1ef2c9b" (UID: "ddd1f97b-814c-43b5-a81b-201bd1ef2c9b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:39:18.286549 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:18.286523 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-kube-api-access-ht9mb" pod "ddd1f97b-814c-43b5-a81b-201bd1ef2c9b" (UID: "ddd1f97b-814c-43b5-a81b-201bd1ef2c9b"). InnerVolumeSpecName "kube-api-access-ht9mb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:39:18.386151 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:18.386121 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:18.386151 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:18.386149 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht9mb\" (UniqueName: \"kubernetes.io/projected/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-kube-api-access-ht9mb\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:18.386151 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:18.386159 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:18.386395 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:18.386167 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:18.476365 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:18.476293 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a770b9-774d768db-9zksn" Nov 03 13:39:18.477055 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:18.477023 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a770b9-695d4dd84c-lzblq" Nov 03 13:39:18.531028 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:18.531001 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686" Nov 03 13:39:18.537155 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:18.537133 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686" Nov 03 13:39:18.542744 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:18.542715 5979 generic.go:356] "Generic (PLEG): container finished" podID="ddd1f97b-814c-43b5-a81b-201bd1ef2c9b" containerID="3948467f6687f08d0693e3c03d844c4a4db3c45208c7a9243eac83fc3b2d6d1e" exitCode=0 Nov 03 13:39:18.542906 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:18.542776 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l" Nov 03 13:39:18.542906 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:18.542798 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l" event={"ID":"ddd1f97b-814c-43b5-a81b-201bd1ef2c9b","Type":"ContainerDied","Data":"3948467f6687f08d0693e3c03d844c4a4db3c45208c7a9243eac83fc3b2d6d1e"} Nov 03 13:39:18.542906 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:18.542854 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l" event={"ID":"ddd1f97b-814c-43b5-a81b-201bd1ef2c9b","Type":"ContainerDied","Data":"c2d057d9669a2a56c70df97d8ad5305714f27774e81ce71d462c432561892262"} Nov 03 13:39:18.542906 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:18.542885 5979 scope.go:122] "RemoveContainer" containerID="3948467f6687f08d0693e3c03d844c4a4db3c45208c7a9243eac83fc3b2d6d1e" Nov 03 13:39:18.544742 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:18.543736 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686" Nov 03 13:39:18.544943 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:18.544827 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686" Nov 03 13:39:18.561687 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:18.561669 5979 scope.go:122] "RemoveContainer" containerID="01ff368eaf37eeb14f397fd8731124fe04ee652a6271303d9a55e7228f7b00fa" Nov 03 13:39:18.570972 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:18.570954 5979 scope.go:122] "RemoveContainer" containerID="3948467f6687f08d0693e3c03d844c4a4db3c45208c7a9243eac83fc3b2d6d1e" Nov 03 13:39:18.571250 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:39:18.571222 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3948467f6687f08d0693e3c03d844c4a4db3c45208c7a9243eac83fc3b2d6d1e\": container with ID starting with 3948467f6687f08d0693e3c03d844c4a4db3c45208c7a9243eac83fc3b2d6d1e not found: ID does not exist" containerID="3948467f6687f08d0693e3c03d844c4a4db3c45208c7a9243eac83fc3b2d6d1e" Nov 03 13:39:18.571368 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:18.571253 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3948467f6687f08d0693e3c03d844c4a4db3c45208c7a9243eac83fc3b2d6d1e"} err="failed to get container status \"3948467f6687f08d0693e3c03d844c4a4db3c45208c7a9243eac83fc3b2d6d1e\": rpc error: code = NotFound desc = could not find container \"3948467f6687f08d0693e3c03d844c4a4db3c45208c7a9243eac83fc3b2d6d1e\": container with ID starting with 3948467f6687f08d0693e3c03d844c4a4db3c45208c7a9243eac83fc3b2d6d1e not found: ID does not exist" Nov 03 13:39:18.571368 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:18.571272 5979 scope.go:122] "RemoveContainer" containerID="01ff368eaf37eeb14f397fd8731124fe04ee652a6271303d9a55e7228f7b00fa" Nov 03 13:39:18.571534 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:39:18.571462 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01ff368eaf37eeb14f397fd8731124fe04ee652a6271303d9a55e7228f7b00fa\": container with ID starting with 01ff368eaf37eeb14f397fd8731124fe04ee652a6271303d9a55e7228f7b00fa not found: ID does not exist" containerID="01ff368eaf37eeb14f397fd8731124fe04ee652a6271303d9a55e7228f7b00fa" Nov 03 13:39:18.571534 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:18.571502 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01ff368eaf37eeb14f397fd8731124fe04ee652a6271303d9a55e7228f7b00fa"} err="failed to get container status \"01ff368eaf37eeb14f397fd8731124fe04ee652a6271303d9a55e7228f7b00fa\": rpc error: code = NotFound desc = could not find container \"01ff368eaf37eeb14f397fd8731124fe04ee652a6271303d9a55e7228f7b00fa\": container with ID starting with 01ff368eaf37eeb14f397fd8731124fe04ee652a6271303d9a55e7228f7b00fa not found: ID does not exist" Nov 03 13:39:18.913309 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:18.913274 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l"] Nov 03 13:39:18.918111 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:18.918080 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-d29e07-openshift-default-57ddcf8bf9-4lk2l"] Nov 03 13:39:20.116070 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:20.116037 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddd1f97b-814c-43b5-a81b-201bd1ef2c9b" path="/var/lib/kubelet/pods/ddd1f97b-814c-43b5-a81b-201bd1ef2c9b/volumes" Nov 03 13:39:21.088767 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:21.088734 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x"] Nov 03 13:39:21.089266 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:21.089253 5979 cpu_manager.go:414] "containerMap: removing container" podUID="ddd1f97b-814c-43b5-a81b-201bd1ef2c9b" containerName="istio-proxy" Nov 03 13:39:21.089313 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:21.089267 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddd1f97b-814c-43b5-a81b-201bd1ef2c9b" containerName="istio-proxy" Nov 03 13:39:21.089407 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:21.089397 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddd1f97b-814c-43b5-a81b-201bd1ef2c9b" containerName="istio-proxy" Nov 03 13:39:21.089448 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:21.089409 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddd1f97b-814c-43b5-a81b-201bd1ef2c9b" containerName="istio-proxy" Nov 03 13:39:21.094890 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:21.094869 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x" Nov 03 13:39:21.149355 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:21.149320 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-32bec3-openshift-default-dockercfg-ttzsk\"" Nov 03 13:39:21.195924 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:21.195880 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x"] Nov 03 13:39:21.220397 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:21.220362 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdfr8\" (UniqueName: \"kubernetes.io/projected/f1705a6f-0727-437b-af88-8fb5448050ac-kube-api-access-mdfr8\") pod \"e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x\" (UID: \"f1705a6f-0727-437b-af88-8fb5448050ac\") " pod="openshift-ingress/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x" Nov 03 13:39:21.220580 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:21.220405 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1705a6f-0727-437b-af88-8fb5448050ac-istio-podinfo\") pod \"e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x\" (UID: \"f1705a6f-0727-437b-af88-8fb5448050ac\") " pod="openshift-ingress/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x" Nov 03 13:39:21.220580 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:21.220526 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1705a6f-0727-437b-af88-8fb5448050ac-istiod-ca-cert\") pod \"e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x\" (UID: \"f1705a6f-0727-437b-af88-8fb5448050ac\") " pod="openshift-ingress/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x" Nov 03 13:39:21.220659 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:21.220592 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1705a6f-0727-437b-af88-8fb5448050ac-istio-data\") pod \"e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x\" (UID: \"f1705a6f-0727-437b-af88-8fb5448050ac\") " pod="openshift-ingress/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x" Nov 03 13:39:21.220713 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:21.220688 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1705a6f-0727-437b-af88-8fb5448050ac-workload-certs\") pod \"e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x\" (UID: \"f1705a6f-0727-437b-af88-8fb5448050ac\") " pod="openshift-ingress/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x" Nov 03 13:39:21.220784 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:21.220767 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1705a6f-0727-437b-af88-8fb5448050ac-credential-socket\") pod \"e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x\" (UID: \"f1705a6f-0727-437b-af88-8fb5448050ac\") " pod="openshift-ingress/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x" Nov 03 13:39:21.220870 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:21.220855 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1705a6f-0727-437b-af88-8fb5448050ac-istio-envoy\") pod \"e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x\" (UID: \"f1705a6f-0727-437b-af88-8fb5448050ac\") " pod="openshift-ingress/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x" Nov 03 13:39:21.221278 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:21.220995 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1705a6f-0727-437b-af88-8fb5448050ac-istio-token\") pod \"e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x\" (UID: \"f1705a6f-0727-437b-af88-8fb5448050ac\") " pod="openshift-ingress/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x" Nov 03 13:39:21.221278 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:21.221203 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1705a6f-0727-437b-af88-8fb5448050ac-workload-socket\") pod \"e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x\" (UID: \"f1705a6f-0727-437b-af88-8fb5448050ac\") " pod="openshift-ingress/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x" Nov 03 13:39:21.332328 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:21.329522 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1705a6f-0727-437b-af88-8fb5448050ac-istio-envoy\") pod \"e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x\" (UID: \"f1705a6f-0727-437b-af88-8fb5448050ac\") " pod="openshift-ingress/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x" Nov 03 13:39:21.332328 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:21.329618 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1705a6f-0727-437b-af88-8fb5448050ac-istio-token\") pod \"e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x\" (UID: \"f1705a6f-0727-437b-af88-8fb5448050ac\") " pod="openshift-ingress/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x" Nov 03 13:39:21.332328 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:21.329714 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1705a6f-0727-437b-af88-8fb5448050ac-workload-socket\") pod \"e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x\" (UID: \"f1705a6f-0727-437b-af88-8fb5448050ac\") " pod="openshift-ingress/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x" Nov 03 13:39:21.332328 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:21.329752 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdfr8\" (UniqueName: \"kubernetes.io/projected/f1705a6f-0727-437b-af88-8fb5448050ac-kube-api-access-mdfr8\") pod \"e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x\" (UID: \"f1705a6f-0727-437b-af88-8fb5448050ac\") " pod="openshift-ingress/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x" Nov 03 13:39:21.332328 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:21.329799 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1705a6f-0727-437b-af88-8fb5448050ac-istio-podinfo\") pod \"e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x\" (UID: \"f1705a6f-0727-437b-af88-8fb5448050ac\") " pod="openshift-ingress/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x" Nov 03 13:39:21.332328 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:21.329876 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1705a6f-0727-437b-af88-8fb5448050ac-istiod-ca-cert\") pod \"e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x\" (UID: \"f1705a6f-0727-437b-af88-8fb5448050ac\") " pod="openshift-ingress/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x" Nov 03 13:39:21.332328 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:21.329942 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1705a6f-0727-437b-af88-8fb5448050ac-istio-data\") pod \"e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x\" (UID: \"f1705a6f-0727-437b-af88-8fb5448050ac\") " pod="openshift-ingress/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x" Nov 03 13:39:21.332328 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:21.330001 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1705a6f-0727-437b-af88-8fb5448050ac-workload-certs\") pod \"e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x\" (UID: \"f1705a6f-0727-437b-af88-8fb5448050ac\") " pod="openshift-ingress/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x" Nov 03 13:39:21.332328 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:21.330038 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1705a6f-0727-437b-af88-8fb5448050ac-credential-socket\") pod \"e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x\" (UID: \"f1705a6f-0727-437b-af88-8fb5448050ac\") " pod="openshift-ingress/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x" Nov 03 13:39:21.332328 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:21.330461 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1705a6f-0727-437b-af88-8fb5448050ac-credential-socket\") pod \"e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x\" (UID: \"f1705a6f-0727-437b-af88-8fb5448050ac\") " pod="openshift-ingress/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x" Nov 03 13:39:21.332328 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:21.331167 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1705a6f-0727-437b-af88-8fb5448050ac-workload-socket\") pod \"e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x\" (UID: \"f1705a6f-0727-437b-af88-8fb5448050ac\") " pod="openshift-ingress/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x" Nov 03 13:39:21.349549 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:21.339223 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1705a6f-0727-437b-af88-8fb5448050ac-istio-data\") pod \"e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x\" (UID: \"f1705a6f-0727-437b-af88-8fb5448050ac\") " pod="openshift-ingress/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x" Nov 03 13:39:21.349549 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:21.339937 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1705a6f-0727-437b-af88-8fb5448050ac-istiod-ca-cert\") pod \"e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x\" (UID: \"f1705a6f-0727-437b-af88-8fb5448050ac\") " pod="openshift-ingress/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x" Nov 03 13:39:21.349549 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:21.340180 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1705a6f-0727-437b-af88-8fb5448050ac-workload-certs\") pod \"e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x\" (UID: \"f1705a6f-0727-437b-af88-8fb5448050ac\") " pod="openshift-ingress/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x" Nov 03 13:39:21.353802 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:21.353770 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1705a6f-0727-437b-af88-8fb5448050ac-istio-envoy\") pod \"e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x\" (UID: \"f1705a6f-0727-437b-af88-8fb5448050ac\") " pod="openshift-ingress/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x" Nov 03 13:39:21.361455 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:21.361411 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1705a6f-0727-437b-af88-8fb5448050ac-istio-podinfo\") pod \"e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x\" (UID: \"f1705a6f-0727-437b-af88-8fb5448050ac\") " pod="openshift-ingress/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x" Nov 03 13:39:21.397380 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:21.397208 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdfr8\" (UniqueName: \"kubernetes.io/projected/f1705a6f-0727-437b-af88-8fb5448050ac-kube-api-access-mdfr8\") pod \"e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x\" (UID: \"f1705a6f-0727-437b-af88-8fb5448050ac\") " pod="openshift-ingress/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x" Nov 03 13:39:21.404836 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:21.402216 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1705a6f-0727-437b-af88-8fb5448050ac-istio-token\") pod \"e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x\" (UID: \"f1705a6f-0727-437b-af88-8fb5448050ac\") " pod="openshift-ingress/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x" Nov 03 13:39:21.411458 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:21.411427 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x" Nov 03 13:39:21.860225 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:21.860191 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x"] Nov 03 13:39:21.901753 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:39:21.901723 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1705a6f_0727_437b_af88_8fb5448050ac.slice/crio-5699936c1c50e15da059505660b97e57837daacfbd337810ab3a8b0395bf15b4 WatchSource:0}: Error finding container 5699936c1c50e15da059505660b97e57837daacfbd337810ab3a8b0395bf15b4: Status 404 returned error can't find the container with id 5699936c1c50e15da059505660b97e57837daacfbd337810ab3a8b0395bf15b4 Nov 03 13:39:21.904429 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:21.904393 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:39:21.904561 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:21.904462 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:39:21.904561 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:21.904508 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:39:22.571814 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:22.571771 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x" event={"ID":"f1705a6f-0727-437b-af88-8fb5448050ac","Type":"ContainerStarted","Data":"62d75487cbfbe1dd189530177ccc545a6ecbdeebe59d06f61632f1d1eee49d48"} Nov 03 13:39:22.572235 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:22.571840 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x" event={"ID":"f1705a6f-0727-437b-af88-8fb5448050ac","Type":"ContainerStarted","Data":"5699936c1c50e15da059505660b97e57837daacfbd337810ab3a8b0395bf15b4"} Nov 03 13:39:22.781191 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:22.781139 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x" podStartSLOduration=1.781125405 podStartE2EDuration="1.781125405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:39:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:39:22.733636688 +0000 UTC m=+2044.744324968" watchObservedRunningTime="2026-11-03 13:39:22.781125405 +0000 UTC m=+2044.791813647" Nov 03 13:39:23.412231 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:23.412198 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x" Nov 03 13:39:23.417767 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:23.417742 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x" Nov 03 13:39:23.579997 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:23.579961 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x" Nov 03 13:39:23.580995 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:23.580973 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x" Nov 03 13:39:24.155399 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:24.155360 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-8d8af2-6979795ddc-f4x64"] Nov 03 13:39:24.156185 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:24.156162 5979 cpu_manager.go:414] "containerMap: removing container" podUID="ddd1f97b-814c-43b5-a81b-201bd1ef2c9b" containerName="istio-proxy" Nov 03 13:39:24.156185 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:24.156183 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddd1f97b-814c-43b5-a81b-201bd1ef2c9b" containerName="istio-proxy" Nov 03 13:39:24.167460 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:24.167435 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8d8af2-6979795ddc-f4x64" Nov 03 13:39:24.213159 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:24.213129 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-8d8af2-6979795ddc-f4x64"] Nov 03 13:39:24.216194 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:24.216173 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-8d8af2\"" Nov 03 13:39:24.216338 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:24.216244 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-8d8af2-dockercfg-x2clf\"" Nov 03 13:39:24.274510 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:24.271240 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jcnl\" (UniqueName: \"kubernetes.io/projected/b92a7bfb-f886-4b87-9c00-e07d8fb43732-kube-api-access-9jcnl\") pod \"payload-pre-processing-e2e-praxis-a-8d8af2-6979795ddc-f4x64\" (UID: \"b92a7bfb-f886-4b87-9c00-e07d8fb43732\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8d8af2-6979795ddc-f4x64" Nov 03 13:39:24.276511 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:24.276462 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b92a7bfb-f886-4b87-9c00-e07d8fb43732-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-8d8af2-6979795ddc-f4x64\" (UID: \"b92a7bfb-f886-4b87-9c00-e07d8fb43732\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8d8af2-6979795ddc-f4x64" Nov 03 13:39:24.309491 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:24.309436 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-8d8af2-6778d74d77-f5xjw"] Nov 03 13:39:24.317611 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:24.317314 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-8d8af2-6778d74d77-f5xjw" Nov 03 13:39:24.361788 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:24.361752 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-8d8af2-6778d74d77-f5xjw"] Nov 03 13:39:24.378518 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:24.378485 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b92a7bfb-f886-4b87-9c00-e07d8fb43732-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-8d8af2-6979795ddc-f4x64\" (UID: \"b92a7bfb-f886-4b87-9c00-e07d8fb43732\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8d8af2-6979795ddc-f4x64" Nov 03 13:39:24.378799 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:24.378771 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jcnl\" (UniqueName: \"kubernetes.io/projected/b92a7bfb-f886-4b87-9c00-e07d8fb43732-kube-api-access-9jcnl\") pod \"payload-pre-processing-e2e-praxis-a-8d8af2-6979795ddc-f4x64\" (UID: \"b92a7bfb-f886-4b87-9c00-e07d8fb43732\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8d8af2-6979795ddc-f4x64" Nov 03 13:39:24.379217 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:24.379196 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b92a7bfb-f886-4b87-9c00-e07d8fb43732-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-8d8af2-6979795ddc-f4x64\" (UID: \"b92a7bfb-f886-4b87-9c00-e07d8fb43732\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8d8af2-6979795ddc-f4x64" Nov 03 13:39:24.432731 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:24.432662 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jcnl\" (UniqueName: \"kubernetes.io/projected/b92a7bfb-f886-4b87-9c00-e07d8fb43732-kube-api-access-9jcnl\") pod \"payload-pre-processing-e2e-praxis-a-8d8af2-6979795ddc-f4x64\" (UID: \"b92a7bfb-f886-4b87-9c00-e07d8fb43732\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8d8af2-6979795ddc-f4x64" Nov 03 13:39:24.480057 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:24.480027 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9kv2\" (UniqueName: \"kubernetes.io/projected/f5c4766a-e455-497c-ba06-716c55b6b8aa-kube-api-access-q9kv2\") pod \"payload-processing-e2e-praxis-a-8d8af2-6778d74d77-f5xjw\" (UID: \"f5c4766a-e455-497c-ba06-716c55b6b8aa\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-8d8af2-6778d74d77-f5xjw" Nov 03 13:39:24.480235 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:24.480101 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5c4766a-e455-497c-ba06-716c55b6b8aa-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-8d8af2-6778d74d77-f5xjw\" (UID: \"f5c4766a-e455-497c-ba06-716c55b6b8aa\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-8d8af2-6778d74d77-f5xjw" Nov 03 13:39:24.481636 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:24.481606 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8d8af2-6979795ddc-f4x64" Nov 03 13:39:24.565081 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:24.564911 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6f9ed-7f68fdb665-jhx2v"] Nov 03 13:39:24.565506 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:24.565397 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6f9ed-7f68fdb665-jhx2v" podUID="4839945c-72ce-4a48-9879-1d94da9c3b48" containerName="maas-api" containerID="cri-o://b74f6155ecedac22889eae636711d091301b3855657ec2c65ed7e2e0a058f824" gracePeriod=30 Nov 03 13:39:24.583991 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:24.582970 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9kv2\" (UniqueName: \"kubernetes.io/projected/f5c4766a-e455-497c-ba06-716c55b6b8aa-kube-api-access-q9kv2\") pod \"payload-processing-e2e-praxis-a-8d8af2-6778d74d77-f5xjw\" (UID: \"f5c4766a-e455-497c-ba06-716c55b6b8aa\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-8d8af2-6778d74d77-f5xjw" Nov 03 13:39:24.583991 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:24.583062 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5c4766a-e455-497c-ba06-716c55b6b8aa-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-8d8af2-6778d74d77-f5xjw\" (UID: \"f5c4766a-e455-497c-ba06-716c55b6b8aa\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-8d8af2-6778d74d77-f5xjw" Nov 03 13:39:24.583991 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:24.583931 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5c4766a-e455-497c-ba06-716c55b6b8aa-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-8d8af2-6778d74d77-f5xjw\" (UID: \"f5c4766a-e455-497c-ba06-716c55b6b8aa\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-8d8af2-6778d74d77-f5xjw" Nov 03 13:39:24.639163 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:24.639127 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9kv2\" (UniqueName: \"kubernetes.io/projected/f5c4766a-e455-497c-ba06-716c55b6b8aa-kube-api-access-q9kv2\") pod \"payload-processing-e2e-praxis-a-8d8af2-6778d74d77-f5xjw\" (UID: \"f5c4766a-e455-497c-ba06-716c55b6b8aa\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-8d8af2-6778d74d77-f5xjw" Nov 03 13:39:24.859553 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:24.859518 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6f9ed-7f68fdb665-jhx2v" Nov 03 13:39:24.868136 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:24.868111 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-8d8af2-6979795ddc-f4x64"] Nov 03 13:39:24.875159 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:39:24.875131 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb92a7bfb_f886_4b87_9c00_e07d8fb43732.slice/crio-017959fb8bc481a27a72afef2c31b971c7627c6268819fb12de9531855cc5091 WatchSource:0}: Error finding container 017959fb8bc481a27a72afef2c31b971c7627c6268819fb12de9531855cc5091: Status 404 returned error can't find the container with id 017959fb8bc481a27a72afef2c31b971c7627c6268819fb12de9531855cc5091 Nov 03 13:39:24.936421 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:24.936361 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-8d8af2-6778d74d77-f5xjw" Nov 03 13:39:24.992149 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:24.990040 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4839945c-72ce-4a48-9879-1d94da9c3b48-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4839945c-72ce-4a48-9879-1d94da9c3b48-openshift-service-ca\") pod \"4839945c-72ce-4a48-9879-1d94da9c3b48\" (UID: \"4839945c-72ce-4a48-9879-1d94da9c3b48\") " Nov 03 13:39:24.992149 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:24.990134 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4839945c-72ce-4a48-9879-1d94da9c3b48-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4839945c-72ce-4a48-9879-1d94da9c3b48-metrics-certs\") pod \"4839945c-72ce-4a48-9879-1d94da9c3b48\" (UID: \"4839945c-72ce-4a48-9879-1d94da9c3b48\") " Nov 03 13:39:24.992149 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:24.990208 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4839945c-72ce-4a48-9879-1d94da9c3b48-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4839945c-72ce-4a48-9879-1d94da9c3b48-maas-api-tls\") pod \"4839945c-72ce-4a48-9879-1d94da9c3b48\" (UID: \"4839945c-72ce-4a48-9879-1d94da9c3b48\") " Nov 03 13:39:24.992149 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:24.990399 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4839945c-72ce-4a48-9879-1d94da9c3b48-kube-api-access-vnzkz\" (UniqueName: \"kubernetes.io/projected/4839945c-72ce-4a48-9879-1d94da9c3b48-kube-api-access-vnzkz\") pod \"4839945c-72ce-4a48-9879-1d94da9c3b48\" (UID: \"4839945c-72ce-4a48-9879-1d94da9c3b48\") " Nov 03 13:39:24.992149 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:24.992103 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4839945c-72ce-4a48-9879-1d94da9c3b48-openshift-service-ca" pod "4839945c-72ce-4a48-9879-1d94da9c3b48" (UID: "4839945c-72ce-4a48-9879-1d94da9c3b48"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:39:24.993161 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:24.993121 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4839945c-72ce-4a48-9879-1d94da9c3b48-kube-api-access-vnzkz" pod "4839945c-72ce-4a48-9879-1d94da9c3b48" (UID: "4839945c-72ce-4a48-9879-1d94da9c3b48"). InnerVolumeSpecName "kube-api-access-vnzkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:39:24.993662 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:24.993634 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4839945c-72ce-4a48-9879-1d94da9c3b48-metrics-certs" pod "4839945c-72ce-4a48-9879-1d94da9c3b48" (UID: "4839945c-72ce-4a48-9879-1d94da9c3b48"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:39:24.996056 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:24.996034 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4839945c-72ce-4a48-9879-1d94da9c3b48-maas-api-tls" pod "4839945c-72ce-4a48-9879-1d94da9c3b48" (UID: "4839945c-72ce-4a48-9879-1d94da9c3b48"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:39:25.091563 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:25.091524 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f6f9ed-7d67775554-8cvc9"] Nov 03 13:39:25.091799 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:25.091752 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6f9ed-7d67775554-8cvc9" podUID="01dd2dc9-f37f-4f9f-926a-ef0679393cd0" containerName="payload-processing" containerID="cri-o://0b20210355d9ab1a98cfbfb21af83060a10f4960e48974f72341c8102f7ffb5a" gracePeriod=30 Nov 03 13:39:25.092811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:25.092784 5979 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4839945c-72ce-4a48-9879-1d94da9c3b48-openshift-service-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:25.092811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:25.092813 5979 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4839945c-72ce-4a48-9879-1d94da9c3b48-metrics-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:25.092962 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:25.092822 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4839945c-72ce-4a48-9879-1d94da9c3b48-maas-api-tls\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:25.092962 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:25.092832 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnzkz\" (UniqueName: \"kubernetes.io/projected/4839945c-72ce-4a48-9879-1d94da9c3b48-kube-api-access-vnzkz\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:25.148750 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:25.148707 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f6f9ed-6dc5f9df7f-pk6dn"] Nov 03 13:39:25.148955 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:25.148931 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6f9ed-6dc5f9df7f-pk6dn" podUID="aaab21c8-09c2-442a-8d48-e70bcff953b2" containerName="payload-pre-processing" containerID="cri-o://dde6c10b47578f410c75698b8b769fcb96ba506137403fb3feedb22f5d3183cf" gracePeriod=30 Nov 03 13:39:25.315229 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:25.315194 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-8d8af2-6778d74d77-f5xjw"] Nov 03 13:39:25.409023 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:25.408994 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6f9ed-7d67775554-8cvc9" Nov 03 13:39:25.478227 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:25.478199 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6f9ed-6dc5f9df7f-pk6dn" Nov 03 13:39:25.597613 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:25.597504 5979 generic.go:356] "Generic (PLEG): container finished" podID="aaab21c8-09c2-442a-8d48-e70bcff953b2" containerID="dde6c10b47578f410c75698b8b769fcb96ba506137403fb3feedb22f5d3183cf" exitCode=0 Nov 03 13:39:25.597613 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:25.597568 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6f9ed-6dc5f9df7f-pk6dn" Nov 03 13:39:25.597613 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:25.597592 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6f9ed-6dc5f9df7f-pk6dn" event={"ID":"aaab21c8-09c2-442a-8d48-e70bcff953b2","Type":"ContainerDied","Data":"dde6c10b47578f410c75698b8b769fcb96ba506137403fb3feedb22f5d3183cf"} Nov 03 13:39:25.598154 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:25.597649 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f6f9ed-6dc5f9df7f-pk6dn" event={"ID":"aaab21c8-09c2-442a-8d48-e70bcff953b2","Type":"ContainerDied","Data":"b506365513209c09d516ccc78c800e5bb451d43a4825d6eea9631b5860e5e4e0"} Nov 03 13:39:25.598154 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:25.597679 5979 scope.go:122] "RemoveContainer" containerID="dde6c10b47578f410c75698b8b769fcb96ba506137403fb3feedb22f5d3183cf" Nov 03 13:39:25.599639 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:25.599614 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01dd2dc9-f37f-4f9f-926a-ef0679393cd0-kube-api-access-nhv4v\" (UniqueName: \"kubernetes.io/projected/01dd2dc9-f37f-4f9f-926a-ef0679393cd0-kube-api-access-nhv4v\") pod \"01dd2dc9-f37f-4f9f-926a-ef0679393cd0\" (UID: \"01dd2dc9-f37f-4f9f-926a-ef0679393cd0\") " Nov 03 13:39:25.599763 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:25.599666 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaab21c8-09c2-442a-8d48-e70bcff953b2-kube-api-access-ck7fj\" (UniqueName: \"kubernetes.io/projected/aaab21c8-09c2-442a-8d48-e70bcff953b2-kube-api-access-ck7fj\") pod \"aaab21c8-09c2-442a-8d48-e70bcff953b2\" (UID: \"aaab21c8-09c2-442a-8d48-e70bcff953b2\") " Nov 03 13:39:25.599763 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:25.599723 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaab21c8-09c2-442a-8d48-e70bcff953b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaab21c8-09c2-442a-8d48-e70bcff953b2-plugins-config-volume\") pod \"aaab21c8-09c2-442a-8d48-e70bcff953b2\" (UID: \"aaab21c8-09c2-442a-8d48-e70bcff953b2\") " Nov 03 13:39:25.599858 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:25.599771 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01dd2dc9-f37f-4f9f-926a-ef0679393cd0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01dd2dc9-f37f-4f9f-926a-ef0679393cd0-plugins-config-volume\") pod \"01dd2dc9-f37f-4f9f-926a-ef0679393cd0\" (UID: \"01dd2dc9-f37f-4f9f-926a-ef0679393cd0\") " Nov 03 13:39:25.600018 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:25.599992 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8d8af2-6979795ddc-f4x64" event={"ID":"b92a7bfb-f886-4b87-9c00-e07d8fb43732","Type":"ContainerStarted","Data":"e3f0f7227cd6991da98ed826d98930eae2bce202c676a8e0b81b14f0e7fcde3f"} Nov 03 13:39:25.600149 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:25.600045 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8d8af2-6979795ddc-f4x64" Nov 03 13:39:25.600149 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:25.600070 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8d8af2-6979795ddc-f4x64" event={"ID":"b92a7bfb-f886-4b87-9c00-e07d8fb43732","Type":"ContainerStarted","Data":"017959fb8bc481a27a72afef2c31b971c7627c6268819fb12de9531855cc5091"} Nov 03 13:39:25.600812 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:25.600753 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaab21c8-09c2-442a-8d48-e70bcff953b2-plugins-config-volume" pod "aaab21c8-09c2-442a-8d48-e70bcff953b2" (UID: "aaab21c8-09c2-442a-8d48-e70bcff953b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:39:25.600940 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:25.600828 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01dd2dc9-f37f-4f9f-926a-ef0679393cd0-plugins-config-volume" pod "01dd2dc9-f37f-4f9f-926a-ef0679393cd0" (UID: "01dd2dc9-f37f-4f9f-926a-ef0679393cd0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:39:25.602372 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:25.602121 5979 generic.go:356] "Generic (PLEG): container finished" podID="4839945c-72ce-4a48-9879-1d94da9c3b48" containerID="b74f6155ecedac22889eae636711d091301b3855657ec2c65ed7e2e0a058f824" exitCode=0 Nov 03 13:39:25.602372 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:25.602179 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6f9ed-7f68fdb665-jhx2v" Nov 03 13:39:25.602372 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:25.602278 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6f9ed-7f68fdb665-jhx2v" event={"ID":"4839945c-72ce-4a48-9879-1d94da9c3b48","Type":"ContainerDied","Data":"b74f6155ecedac22889eae636711d091301b3855657ec2c65ed7e2e0a058f824"} Nov 03 13:39:25.602372 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:25.602323 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6f9ed-7f68fdb665-jhx2v" event={"ID":"4839945c-72ce-4a48-9879-1d94da9c3b48","Type":"ContainerDied","Data":"710bc84e605e936e6740b28bb5350db9f2b1eaf8d1068a0df22a2612a4b773df"} Nov 03 13:39:25.602849 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:25.602823 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaab21c8-09c2-442a-8d48-e70bcff953b2-kube-api-access-ck7fj" pod "aaab21c8-09c2-442a-8d48-e70bcff953b2" (UID: "aaab21c8-09c2-442a-8d48-e70bcff953b2"). InnerVolumeSpecName "kube-api-access-ck7fj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:39:25.603153 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:25.603128 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01dd2dc9-f37f-4f9f-926a-ef0679393cd0-kube-api-access-nhv4v" pod "01dd2dc9-f37f-4f9f-926a-ef0679393cd0" (UID: "01dd2dc9-f37f-4f9f-926a-ef0679393cd0"). InnerVolumeSpecName "kube-api-access-nhv4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:39:25.609177 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:25.609148 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-8d8af2-6778d74d77-f5xjw" event={"ID":"f5c4766a-e455-497c-ba06-716c55b6b8aa","Type":"ContainerStarted","Data":"5d6ea38e37ce9cfefdd5912bc6d2809296b66f809c29e1c9bbadaa535a14757c"} Nov 03 13:39:25.609336 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:25.609201 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-8d8af2-6778d74d77-f5xjw" event={"ID":"f5c4766a-e455-497c-ba06-716c55b6b8aa","Type":"ContainerStarted","Data":"cf22dff6df37aa9f1ebc0e4268920e9e9705101bbd65d97dcf3dbdde1cead22f"} Nov 03 13:39:25.609821 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:25.609801 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-8d8af2-6778d74d77-f5xjw" Nov 03 13:39:25.615816 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:25.612126 5979 generic.go:356] "Generic (PLEG): container finished" podID="01dd2dc9-f37f-4f9f-926a-ef0679393cd0" containerID="0b20210355d9ab1a98cfbfb21af83060a10f4960e48974f72341c8102f7ffb5a" exitCode=0 Nov 03 13:39:25.615816 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:25.612929 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6f9ed-7d67775554-8cvc9" Nov 03 13:39:25.615816 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:25.615557 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6f9ed-7d67775554-8cvc9" event={"ID":"01dd2dc9-f37f-4f9f-926a-ef0679393cd0","Type":"ContainerDied","Data":"0b20210355d9ab1a98cfbfb21af83060a10f4960e48974f72341c8102f7ffb5a"} Nov 03 13:39:25.615816 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:25.615649 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f6f9ed-7d67775554-8cvc9" event={"ID":"01dd2dc9-f37f-4f9f-926a-ef0679393cd0","Type":"ContainerDied","Data":"00db85c4d772abecd3e9865702c3e8fd4e001060765f8b9a60df9c7951e9e7c0"} Nov 03 13:39:25.698656 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:25.698463 5979 scope.go:122] "RemoveContainer" containerID="dde6c10b47578f410c75698b8b769fcb96ba506137403fb3feedb22f5d3183cf" Nov 03 13:39:25.702315 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:39:25.698881 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dde6c10b47578f410c75698b8b769fcb96ba506137403fb3feedb22f5d3183cf\": container with ID starting with dde6c10b47578f410c75698b8b769fcb96ba506137403fb3feedb22f5d3183cf not found: ID does not exist" containerID="dde6c10b47578f410c75698b8b769fcb96ba506137403fb3feedb22f5d3183cf" Nov 03 13:39:25.702315 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:25.698918 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dde6c10b47578f410c75698b8b769fcb96ba506137403fb3feedb22f5d3183cf"} err="failed to get container status \"dde6c10b47578f410c75698b8b769fcb96ba506137403fb3feedb22f5d3183cf\": rpc error: code = NotFound desc = could not find container \"dde6c10b47578f410c75698b8b769fcb96ba506137403fb3feedb22f5d3183cf\": container with ID starting with dde6c10b47578f410c75698b8b769fcb96ba506137403fb3feedb22f5d3183cf not found: ID does not exist" Nov 03 13:39:25.702315 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:25.698941 5979 scope.go:122] "RemoveContainer" containerID="b74f6155ecedac22889eae636711d091301b3855657ec2c65ed7e2e0a058f824" Nov 03 13:39:25.702315 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:25.701191 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01dd2dc9-f37f-4f9f-926a-ef0679393cd0-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:25.702315 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:25.701215 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhv4v\" (UniqueName: \"kubernetes.io/projected/01dd2dc9-f37f-4f9f-926a-ef0679393cd0-kube-api-access-nhv4v\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:25.702315 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:25.701230 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ck7fj\" (UniqueName: \"kubernetes.io/projected/aaab21c8-09c2-442a-8d48-e70bcff953b2-kube-api-access-ck7fj\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:25.702315 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:25.701243 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaab21c8-09c2-442a-8d48-e70bcff953b2-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:25.723236 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:25.722790 5979 scope.go:122] "RemoveContainer" containerID="b74f6155ecedac22889eae636711d091301b3855657ec2c65ed7e2e0a058f824" Nov 03 13:39:25.723236 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:39:25.723172 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b74f6155ecedac22889eae636711d091301b3855657ec2c65ed7e2e0a058f824\": container with ID starting with b74f6155ecedac22889eae636711d091301b3855657ec2c65ed7e2e0a058f824 not found: ID does not exist" containerID="b74f6155ecedac22889eae636711d091301b3855657ec2c65ed7e2e0a058f824" Nov 03 13:39:25.723236 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:25.723206 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b74f6155ecedac22889eae636711d091301b3855657ec2c65ed7e2e0a058f824"} err="failed to get container status \"b74f6155ecedac22889eae636711d091301b3855657ec2c65ed7e2e0a058f824\": rpc error: code = NotFound desc = could not find container \"b74f6155ecedac22889eae636711d091301b3855657ec2c65ed7e2e0a058f824\": container with ID starting with b74f6155ecedac22889eae636711d091301b3855657ec2c65ed7e2e0a058f824 not found: ID does not exist" Nov 03 13:39:25.723236 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:25.723230 5979 scope.go:122] "RemoveContainer" containerID="0b20210355d9ab1a98cfbfb21af83060a10f4960e48974f72341c8102f7ffb5a" Nov 03 13:39:25.737623 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:25.737601 5979 scope.go:122] "RemoveContainer" containerID="0b20210355d9ab1a98cfbfb21af83060a10f4960e48974f72341c8102f7ffb5a" Nov 03 13:39:25.737965 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:39:25.737939 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b20210355d9ab1a98cfbfb21af83060a10f4960e48974f72341c8102f7ffb5a\": container with ID starting with 0b20210355d9ab1a98cfbfb21af83060a10f4960e48974f72341c8102f7ffb5a not found: ID does not exist" containerID="0b20210355d9ab1a98cfbfb21af83060a10f4960e48974f72341c8102f7ffb5a" Nov 03 13:39:25.738056 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:25.737972 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b20210355d9ab1a98cfbfb21af83060a10f4960e48974f72341c8102f7ffb5a"} err="failed to get container status \"0b20210355d9ab1a98cfbfb21af83060a10f4960e48974f72341c8102f7ffb5a\": rpc error: code = NotFound desc = could not find container \"0b20210355d9ab1a98cfbfb21af83060a10f4960e48974f72341c8102f7ffb5a\": container with ID starting with 0b20210355d9ab1a98cfbfb21af83060a10f4960e48974f72341c8102f7ffb5a not found: ID does not exist" Nov 03 13:39:25.798003 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:25.797951 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8d8af2-6979795ddc-f4x64" podStartSLOduration=1.797935716 podStartE2EDuration="1.797935716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:39:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:39:25.797122989 +0000 UTC m=+2047.807811231" watchObservedRunningTime="2026-11-03 13:39:25.797935716 +0000 UTC m=+2047.808623957" Nov 03 13:39:25.969175 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:25.969120 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-8d8af2-6778d74d77-f5xjw" podStartSLOduration=1.969104792 podStartE2EDuration="1.969104792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:39:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:39:25.966748101 +0000 UTC m=+2047.977436342" watchObservedRunningTime="2026-11-03 13:39:25.969104792 +0000 UTC m=+2047.979793034" Nov 03 13:39:26.138617 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:26.138505 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f6f9ed-7d67775554-8cvc9"] Nov 03 13:39:26.138617 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:26.138561 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f6f9ed-7d67775554-8cvc9"] Nov 03 13:39:26.229240 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:26.229163 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f6f9ed-6dc5f9df7f-pk6dn"] Nov 03 13:39:26.233530 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:26.233498 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f6f9ed-6dc5f9df7f-pk6dn"] Nov 03 13:39:26.336165 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:26.336133 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6f9ed-7f68fdb665-jhx2v"] Nov 03 13:39:26.341390 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:26.341364 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6f9ed-7f68fdb665-jhx2v"] Nov 03 13:39:28.085142 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:28.085102 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:39:28.115745 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:28.115706 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01dd2dc9-f37f-4f9f-926a-ef0679393cd0" path="/var/lib/kubelet/pods/01dd2dc9-f37f-4f9f-926a-ef0679393cd0/volumes" Nov 03 13:39:28.116272 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:28.116250 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4839945c-72ce-4a48-9879-1d94da9c3b48" path="/var/lib/kubelet/pods/4839945c-72ce-4a48-9879-1d94da9c3b48/volumes" Nov 03 13:39:28.116715 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:28.116696 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaab21c8-09c2-442a-8d48-e70bcff953b2" path="/var/lib/kubelet/pods/aaab21c8-09c2-442a-8d48-e70bcff953b2/volumes" Nov 03 13:39:29.608350 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:29.608315 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-32bec3-6c4596d787-tsnlp"] Nov 03 13:39:29.609233 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:29.609213 5979 cpu_manager.go:414] "containerMap: removing container" podUID="4839945c-72ce-4a48-9879-1d94da9c3b48" containerName="maas-api" Nov 03 13:39:29.609275 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:29.609236 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4839945c-72ce-4a48-9879-1d94da9c3b48" containerName="maas-api" Nov 03 13:39:29.609275 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:29.609252 5979 cpu_manager.go:414] "containerMap: removing container" podUID="aaab21c8-09c2-442a-8d48-e70bcff953b2" containerName="payload-pre-processing" Nov 03 13:39:29.609275 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:29.609260 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaab21c8-09c2-442a-8d48-e70bcff953b2" containerName="payload-pre-processing" Nov 03 13:39:29.609394 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:29.609292 5979 cpu_manager.go:414] "containerMap: removing container" podUID="01dd2dc9-f37f-4f9f-926a-ef0679393cd0" containerName="payload-processing" Nov 03 13:39:29.609394 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:29.609301 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01dd2dc9-f37f-4f9f-926a-ef0679393cd0" containerName="payload-processing" Nov 03 13:39:29.609459 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:29.609442 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="4839945c-72ce-4a48-9879-1d94da9c3b48" containerName="maas-api" Nov 03 13:39:29.609459 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:29.609454 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaab21c8-09c2-442a-8d48-e70bcff953b2" containerName="payload-pre-processing" Nov 03 13:39:29.609559 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:29.609465 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="01dd2dc9-f37f-4f9f-926a-ef0679393cd0" containerName="payload-processing" Nov 03 13:39:29.614386 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:29.614364 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-32bec3-6c4596d787-tsnlp" Nov 03 13:39:29.676689 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:29.676641 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4239490-ae74-411a-8b11-237b853b70e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-32bec3-6c4596d787-tsnlp\" (UID: \"c4239490-ae74-411a-8b11-237b853b70e1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-32bec3-6c4596d787-tsnlp" Nov 03 13:39:29.676863 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:29.676784 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hgrr\" (UniqueName: \"kubernetes.io/projected/c4239490-ae74-411a-8b11-237b853b70e1-kube-api-access-6hgrr\") pod \"payload-pre-processing-e2e-ipp-b-32bec3-6c4596d787-tsnlp\" (UID: \"c4239490-ae74-411a-8b11-237b853b70e1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-32bec3-6c4596d787-tsnlp" Nov 03 13:39:29.726496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:29.724782 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-32bec3\"" Nov 03 13:39:29.726496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:29.724896 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-32bec3-dockercfg-qm4pf\"" Nov 03 13:39:29.801829 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:29.798872 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4239490-ae74-411a-8b11-237b853b70e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-32bec3-6c4596d787-tsnlp\" (UID: \"c4239490-ae74-411a-8b11-237b853b70e1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-32bec3-6c4596d787-tsnlp" Nov 03 13:39:29.801829 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:29.799029 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hgrr\" (UniqueName: \"kubernetes.io/projected/c4239490-ae74-411a-8b11-237b853b70e1-kube-api-access-6hgrr\") pod \"payload-pre-processing-e2e-ipp-b-32bec3-6c4596d787-tsnlp\" (UID: \"c4239490-ae74-411a-8b11-237b853b70e1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-32bec3-6c4596d787-tsnlp" Nov 03 13:39:29.801829 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:29.801647 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4239490-ae74-411a-8b11-237b853b70e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-32bec3-6c4596d787-tsnlp\" (UID: \"c4239490-ae74-411a-8b11-237b853b70e1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-32bec3-6c4596d787-tsnlp" Nov 03 13:39:29.808716 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:29.808687 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-32bec3-6c4596d787-tsnlp"] Nov 03 13:39:29.814060 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:29.814035 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-32bec3-654f99d4f9-m9m9c"] Nov 03 13:39:29.818156 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:29.818141 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-32bec3-654f99d4f9-m9m9c" Nov 03 13:39:29.835327 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:29.835296 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-32bec3-654f99d4f9-m9m9c"] Nov 03 13:39:29.853878 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:29.853850 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hgrr\" (UniqueName: \"kubernetes.io/projected/c4239490-ae74-411a-8b11-237b853b70e1-kube-api-access-6hgrr\") pod \"payload-pre-processing-e2e-ipp-b-32bec3-6c4596d787-tsnlp\" (UID: \"c4239490-ae74-411a-8b11-237b853b70e1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-32bec3-6c4596d787-tsnlp" Nov 03 13:39:29.928668 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:29.928636 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-32bec3-6c4596d787-tsnlp" Nov 03 13:39:30.002206 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:30.002174 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk97s\" (UniqueName: \"kubernetes.io/projected/247afc70-fb0c-4092-91f0-ecd7d4f3a03c-kube-api-access-nk97s\") pod \"payload-processing-e2e-ipp-b-32bec3-654f99d4f9-m9m9c\" (UID: \"247afc70-fb0c-4092-91f0-ecd7d4f3a03c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-32bec3-654f99d4f9-m9m9c" Nov 03 13:39:30.002423 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:30.002402 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/247afc70-fb0c-4092-91f0-ecd7d4f3a03c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-32bec3-654f99d4f9-m9m9c\" (UID: \"247afc70-fb0c-4092-91f0-ecd7d4f3a03c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-32bec3-654f99d4f9-m9m9c" Nov 03 13:39:30.103112 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:30.103073 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/247afc70-fb0c-4092-91f0-ecd7d4f3a03c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-32bec3-654f99d4f9-m9m9c\" (UID: \"247afc70-fb0c-4092-91f0-ecd7d4f3a03c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-32bec3-654f99d4f9-m9m9c" Nov 03 13:39:30.103330 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:30.103171 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk97s\" (UniqueName: \"kubernetes.io/projected/247afc70-fb0c-4092-91f0-ecd7d4f3a03c-kube-api-access-nk97s\") pod \"payload-processing-e2e-ipp-b-32bec3-654f99d4f9-m9m9c\" (UID: \"247afc70-fb0c-4092-91f0-ecd7d4f3a03c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-32bec3-654f99d4f9-m9m9c" Nov 03 13:39:30.103878 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:30.103853 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/247afc70-fb0c-4092-91f0-ecd7d4f3a03c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-32bec3-654f99d4f9-m9m9c\" (UID: \"247afc70-fb0c-4092-91f0-ecd7d4f3a03c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-32bec3-654f99d4f9-m9m9c" Nov 03 13:39:30.157432 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:30.157395 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk97s\" (UniqueName: \"kubernetes.io/projected/247afc70-fb0c-4092-91f0-ecd7d4f3a03c-kube-api-access-nk97s\") pod \"payload-processing-e2e-ipp-b-32bec3-654f99d4f9-m9m9c\" (UID: \"247afc70-fb0c-4092-91f0-ecd7d4f3a03c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-32bec3-654f99d4f9-m9m9c" Nov 03 13:39:30.301311 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:30.301225 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-32bec3-6c4596d787-tsnlp"] Nov 03 13:39:30.308775 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:39:30.308746 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4239490_ae74_411a_8b11_237b853b70e1.slice/crio-3c1b0b02a98f5b20f30d3dd915eb352f1d27d8b35d02bb4ea96320da216274b7 WatchSource:0}: Error finding container 3c1b0b02a98f5b20f30d3dd915eb352f1d27d8b35d02bb4ea96320da216274b7: Status 404 returned error can't find the container with id 3c1b0b02a98f5b20f30d3dd915eb352f1d27d8b35d02bb4ea96320da216274b7 Nov 03 13:39:30.431914 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:30.431871 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-32bec3-654f99d4f9-m9m9c" Nov 03 13:39:30.647967 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:30.647927 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-32bec3-6c4596d787-tsnlp" event={"ID":"c4239490-ae74-411a-8b11-237b853b70e1","Type":"ContainerStarted","Data":"4c963100dec989f0edb9c8168f668efb32b66f9d6d0520657022897143f2b3be"} Nov 03 13:39:30.648384 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:30.648021 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-32bec3-6c4596d787-tsnlp" event={"ID":"c4239490-ae74-411a-8b11-237b853b70e1","Type":"ContainerStarted","Data":"3c1b0b02a98f5b20f30d3dd915eb352f1d27d8b35d02bb4ea96320da216274b7"} Nov 03 13:39:30.648384 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:30.648053 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-32bec3-6c4596d787-tsnlp" Nov 03 13:39:30.762937 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:30.762883 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-32bec3-6c4596d787-tsnlp" podStartSLOduration=1.762869243 podStartE2EDuration="1.762869243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:39:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:39:30.762679442 +0000 UTC m=+2052.773367685" watchObservedRunningTime="2026-11-03 13:39:30.762869243 +0000 UTC m=+2052.773557482" Nov 03 13:39:30.809650 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:30.809612 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-32bec3-654f99d4f9-m9m9c"] Nov 03 13:39:30.818337 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:39:30.818307 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod247afc70_fb0c_4092_91f0_ecd7d4f3a03c.slice/crio-7ba275a5547112c178667234402c498da462af94e6e4824af08a569cde8414fa WatchSource:0}: Error finding container 7ba275a5547112c178667234402c498da462af94e6e4824af08a569cde8414fa: Status 404 returned error can't find the container with id 7ba275a5547112c178667234402c498da462af94e6e4824af08a569cde8414fa Nov 03 13:39:31.656529 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:31.656492 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-32bec3-654f99d4f9-m9m9c" event={"ID":"247afc70-fb0c-4092-91f0-ecd7d4f3a03c","Type":"ContainerStarted","Data":"37637c2fe572152a87f0cd0af692b003aac20f9d6fdc80a53880371b2d84844a"} Nov 03 13:39:31.656953 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:31.656558 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-32bec3-654f99d4f9-m9m9c" event={"ID":"247afc70-fb0c-4092-91f0-ecd7d4f3a03c","Type":"ContainerStarted","Data":"7ba275a5547112c178667234402c498da462af94e6e4824af08a569cde8414fa"} Nov 03 13:39:31.656953 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:31.656856 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-32bec3-654f99d4f9-m9m9c" Nov 03 13:39:31.769531 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:31.769482 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-32bec3-654f99d4f9-m9m9c" podStartSLOduration=2.769453559 podStartE2EDuration="2.769453559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:39:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:39:31.766992685 +0000 UTC m=+2053.777680926" watchObservedRunningTime="2026-11-03 13:39:31.769453559 +0000 UTC m=+2053.780141801" Nov 03 13:39:32.952130 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:32.952088 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-02f3a0-68759d857f-9c2dn"] Nov 03 13:39:32.957285 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:32.957258 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02f3a0-68759d857f-9c2dn" Nov 03 13:39:33.007213 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:33.007184 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-02f3a0\"" Nov 03 13:39:33.049563 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:33.049526 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kzqq\" (UniqueName: \"kubernetes.io/projected/fd43ea90-19f2-4109-85a7-07098fc3da7c-kube-api-access-2kzqq\") pod \"payload-pre-processing-e2e-ipp-a-02f3a0-68759d857f-9c2dn\" (UID: \"fd43ea90-19f2-4109-85a7-07098fc3da7c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02f3a0-68759d857f-9c2dn" Nov 03 13:39:33.049806 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:33.049785 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd43ea90-19f2-4109-85a7-07098fc3da7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-02f3a0-68759d857f-9c2dn\" (UID: \"fd43ea90-19f2-4109-85a7-07098fc3da7c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02f3a0-68759d857f-9c2dn" Nov 03 13:39:33.053563 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:33.053540 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-02f3a0-dockercfg-bgmp9\"" Nov 03 13:39:33.055928 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:33.055908 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-02f3a0-68759d857f-9c2dn"] Nov 03 13:39:33.150664 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:33.150634 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd43ea90-19f2-4109-85a7-07098fc3da7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-02f3a0-68759d857f-9c2dn\" (UID: \"fd43ea90-19f2-4109-85a7-07098fc3da7c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02f3a0-68759d857f-9c2dn" Nov 03 13:39:33.150807 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:33.150700 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kzqq\" (UniqueName: \"kubernetes.io/projected/fd43ea90-19f2-4109-85a7-07098fc3da7c-kube-api-access-2kzqq\") pod \"payload-pre-processing-e2e-ipp-a-02f3a0-68759d857f-9c2dn\" (UID: \"fd43ea90-19f2-4109-85a7-07098fc3da7c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02f3a0-68759d857f-9c2dn" Nov 03 13:39:33.151367 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:33.151346 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd43ea90-19f2-4109-85a7-07098fc3da7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-02f3a0-68759d857f-9c2dn\" (UID: \"fd43ea90-19f2-4109-85a7-07098fc3da7c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02f3a0-68759d857f-9c2dn" Nov 03 13:39:33.212350 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:33.212279 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kzqq\" (UniqueName: \"kubernetes.io/projected/fd43ea90-19f2-4109-85a7-07098fc3da7c-kube-api-access-2kzqq\") pod \"payload-pre-processing-e2e-ipp-a-02f3a0-68759d857f-9c2dn\" (UID: \"fd43ea90-19f2-4109-85a7-07098fc3da7c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02f3a0-68759d857f-9c2dn" Nov 03 13:39:33.274195 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:33.274166 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02f3a0-68759d857f-9c2dn" Nov 03 13:39:33.636085 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:33.636056 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-02f3a0-c66d764b6-qwl7q"] Nov 03 13:39:33.641505 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:33.641464 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-02f3a0-c66d764b6-qwl7q" Nov 03 13:39:33.651184 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:33.651161 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-02f3a0-68759d857f-9c2dn"] Nov 03 13:39:33.654682 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:33.654659 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbdj6\" (UniqueName: \"kubernetes.io/projected/5faad0ec-c254-4ed2-a88c-8f1012747b57-kube-api-access-lbdj6\") pod \"payload-processing-e2e-ipp-a-02f3a0-c66d764b6-qwl7q\" (UID: \"5faad0ec-c254-4ed2-a88c-8f1012747b57\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-02f3a0-c66d764b6-qwl7q" Nov 03 13:39:33.654747 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:33.654735 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5faad0ec-c254-4ed2-a88c-8f1012747b57-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-02f3a0-c66d764b6-qwl7q\" (UID: \"5faad0ec-c254-4ed2-a88c-8f1012747b57\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-02f3a0-c66d764b6-qwl7q" Nov 03 13:39:33.657367 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:39:33.657339 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd43ea90_19f2_4109_85a7_07098fc3da7c.slice/crio-b63d61b8bd623f918c527c5e0434bd967b842c4c469b647eee0d5b6ca40317c0 WatchSource:0}: Error finding container b63d61b8bd623f918c527c5e0434bd967b842c4c469b647eee0d5b6ca40317c0: Status 404 returned error can't find the container with id b63d61b8bd623f918c527c5e0434bd967b842c4c469b647eee0d5b6ca40317c0 Nov 03 13:39:33.670919 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:33.670886 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02f3a0-68759d857f-9c2dn" event={"ID":"fd43ea90-19f2-4109-85a7-07098fc3da7c","Type":"ContainerStarted","Data":"b63d61b8bd623f918c527c5e0434bd967b842c4c469b647eee0d5b6ca40317c0"} Nov 03 13:39:33.697500 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:33.697441 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-02f3a0-c66d764b6-qwl7q"] Nov 03 13:39:33.755964 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:33.755927 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbdj6\" (UniqueName: \"kubernetes.io/projected/5faad0ec-c254-4ed2-a88c-8f1012747b57-kube-api-access-lbdj6\") pod \"payload-processing-e2e-ipp-a-02f3a0-c66d764b6-qwl7q\" (UID: \"5faad0ec-c254-4ed2-a88c-8f1012747b57\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-02f3a0-c66d764b6-qwl7q" Nov 03 13:39:33.756125 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:33.756020 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5faad0ec-c254-4ed2-a88c-8f1012747b57-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-02f3a0-c66d764b6-qwl7q\" (UID: \"5faad0ec-c254-4ed2-a88c-8f1012747b57\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-02f3a0-c66d764b6-qwl7q" Nov 03 13:39:33.756896 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:33.756869 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5faad0ec-c254-4ed2-a88c-8f1012747b57-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-02f3a0-c66d764b6-qwl7q\" (UID: \"5faad0ec-c254-4ed2-a88c-8f1012747b57\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-02f3a0-c66d764b6-qwl7q" Nov 03 13:39:33.810594 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:33.810561 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbdj6\" (UniqueName: \"kubernetes.io/projected/5faad0ec-c254-4ed2-a88c-8f1012747b57-kube-api-access-lbdj6\") pod \"payload-processing-e2e-ipp-a-02f3a0-c66d764b6-qwl7q\" (UID: \"5faad0ec-c254-4ed2-a88c-8f1012747b57\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-02f3a0-c66d764b6-qwl7q" Nov 03 13:39:33.957066 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:33.956978 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-02f3a0-c66d764b6-qwl7q" Nov 03 13:39:34.332653 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:34.332566 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-02f3a0-c66d764b6-qwl7q"] Nov 03 13:39:34.338939 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:39:34.338910 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5faad0ec_c254_4ed2_a88c_8f1012747b57.slice/crio-078650ba1345da7fedf10f08f22c0d24131f552db9c489425bbafc5d3670aca2 WatchSource:0}: Error finding container 078650ba1345da7fedf10f08f22c0d24131f552db9c489425bbafc5d3670aca2: Status 404 returned error can't find the container with id 078650ba1345da7fedf10f08f22c0d24131f552db9c489425bbafc5d3670aca2 Nov 03 13:39:34.679098 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:34.679059 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02f3a0-68759d857f-9c2dn" event={"ID":"fd43ea90-19f2-4109-85a7-07098fc3da7c","Type":"ContainerStarted","Data":"58dbd0850b4491b94e7bd4602d6ddf0f326f3f8a1b76f0b71f7d1a4f1415a10e"} Nov 03 13:39:34.679300 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:34.679268 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02f3a0-68759d857f-9c2dn" Nov 03 13:39:34.680931 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:34.680898 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-02f3a0-c66d764b6-qwl7q" event={"ID":"5faad0ec-c254-4ed2-a88c-8f1012747b57","Type":"ContainerStarted","Data":"b11aaf27837c2d79839bb790d39ad92003cba1659ba89ad25a3147ceb493c781"} Nov 03 13:39:34.681075 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:34.680954 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-02f3a0-c66d764b6-qwl7q" event={"ID":"5faad0ec-c254-4ed2-a88c-8f1012747b57","Type":"ContainerStarted","Data":"078650ba1345da7fedf10f08f22c0d24131f552db9c489425bbafc5d3670aca2"} Nov 03 13:39:34.681075 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:34.680991 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-02f3a0-c66d764b6-qwl7q" Nov 03 13:39:34.797284 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:34.797233 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02f3a0-68759d857f-9c2dn" podStartSLOduration=2.7972192270000003 podStartE2EDuration="2.797219227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:39:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:39:34.794081685 +0000 UTC m=+2056.804769926" watchObservedRunningTime="2026-11-03 13:39:34.797219227 +0000 UTC m=+2056.807907463" Nov 03 13:39:34.910880 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:34.910684 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-02f3a0-c66d764b6-qwl7q" podStartSLOduration=1.910665413 podStartE2EDuration="1.910665413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:39:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:39:34.906589026 +0000 UTC m=+2056.917277267" watchObservedRunningTime="2026-11-03 13:39:34.910665413 +0000 UTC m=+2056.921353658" Nov 03 13:39:36.624859 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:36.624826 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8d8af2-6979795ddc-f4x64" Nov 03 13:39:37.176647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:37.176610 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s"] Nov 03 13:39:37.176838 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:37.176803 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s" podUID="09fccbb2-307c-4752-a4d7-f9986eb5f61c" containerName="istio-proxy" containerID="cri-o://05b18fb973820ac2c88f3c93162fc00bccd839f3959871d39e86492c6898af84" gracePeriod=30 Nov 03 13:39:37.235465 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:39:37.235427 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:39:37.631510 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:37.631390 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-8d8af2-6778d74d77-f5xjw" Nov 03 13:39:40.186542 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:40.186503 5979 patch_prober.go:29] interesting pod/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 13:39:40.187079 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:40.186557 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s" podUID="09fccbb2-307c-4752-a4d7-f9986eb5f61c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 13:39:41.660521 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:41.660451 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-32bec3-6c4596d787-tsnlp" Nov 03 13:39:42.445782 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:42.445722 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s" Nov 03 13:39:42.556829 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:42.556796 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09fccbb2-307c-4752-a4d7-f9986eb5f61c-istio-token\" (UniqueName: \"kubernetes.io/projected/09fccbb2-307c-4752-a4d7-f9986eb5f61c-istio-token\") pod \"09fccbb2-307c-4752-a4d7-f9986eb5f61c\" (UID: \"09fccbb2-307c-4752-a4d7-f9986eb5f61c\") " Nov 03 13:39:42.557000 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:42.556854 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09fccbb2-307c-4752-a4d7-f9986eb5f61c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09fccbb2-307c-4752-a4d7-f9986eb5f61c-istio-podinfo\") pod \"09fccbb2-307c-4752-a4d7-f9986eb5f61c\" (UID: \"09fccbb2-307c-4752-a4d7-f9986eb5f61c\") " Nov 03 13:39:42.557000 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:42.556957 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09fccbb2-307c-4752-a4d7-f9986eb5f61c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09fccbb2-307c-4752-a4d7-f9986eb5f61c-istio-envoy\") pod \"09fccbb2-307c-4752-a4d7-f9986eb5f61c\" (UID: \"09fccbb2-307c-4752-a4d7-f9986eb5f61c\") " Nov 03 13:39:42.557000 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:42.556990 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09fccbb2-307c-4752-a4d7-f9986eb5f61c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09fccbb2-307c-4752-a4d7-f9986eb5f61c-workload-socket\") pod \"09fccbb2-307c-4752-a4d7-f9986eb5f61c\" (UID: \"09fccbb2-307c-4752-a4d7-f9986eb5f61c\") " Nov 03 13:39:42.557145 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:42.557023 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09fccbb2-307c-4752-a4d7-f9986eb5f61c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09fccbb2-307c-4752-a4d7-f9986eb5f61c-istiod-ca-cert\") pod \"09fccbb2-307c-4752-a4d7-f9986eb5f61c\" (UID: \"09fccbb2-307c-4752-a4d7-f9986eb5f61c\") " Nov 03 13:39:42.557145 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:42.557110 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09fccbb2-307c-4752-a4d7-f9986eb5f61c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09fccbb2-307c-4752-a4d7-f9986eb5f61c-credential-socket\") pod \"09fccbb2-307c-4752-a4d7-f9986eb5f61c\" (UID: \"09fccbb2-307c-4752-a4d7-f9986eb5f61c\") " Nov 03 13:39:42.557268 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:42.557142 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09fccbb2-307c-4752-a4d7-f9986eb5f61c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09fccbb2-307c-4752-a4d7-f9986eb5f61c-workload-certs\") pod \"09fccbb2-307c-4752-a4d7-f9986eb5f61c\" (UID: \"09fccbb2-307c-4752-a4d7-f9986eb5f61c\") " Nov 03 13:39:42.557268 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:42.557181 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09fccbb2-307c-4752-a4d7-f9986eb5f61c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09fccbb2-307c-4752-a4d7-f9986eb5f61c-istio-data\") pod \"09fccbb2-307c-4752-a4d7-f9986eb5f61c\" (UID: \"09fccbb2-307c-4752-a4d7-f9986eb5f61c\") " Nov 03 13:39:42.557268 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:42.557238 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09fccbb2-307c-4752-a4d7-f9986eb5f61c-workload-socket" pod "09fccbb2-307c-4752-a4d7-f9986eb5f61c" (UID: "09fccbb2-307c-4752-a4d7-f9986eb5f61c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:39:42.557439 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:42.557309 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09fccbb2-307c-4752-a4d7-f9986eb5f61c-kube-api-access-r72tr\" (UniqueName: \"kubernetes.io/projected/09fccbb2-307c-4752-a4d7-f9986eb5f61c-kube-api-access-r72tr\") pod \"09fccbb2-307c-4752-a4d7-f9986eb5f61c\" (UID: \"09fccbb2-307c-4752-a4d7-f9986eb5f61c\") " Nov 03 13:39:42.557517 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:42.557435 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09fccbb2-307c-4752-a4d7-f9986eb5f61c-istiod-ca-cert" pod "09fccbb2-307c-4752-a4d7-f9986eb5f61c" (UID: "09fccbb2-307c-4752-a4d7-f9986eb5f61c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:39:42.557517 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:42.557483 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09fccbb2-307c-4752-a4d7-f9986eb5f61c-workload-certs" pod "09fccbb2-307c-4752-a4d7-f9986eb5f61c" (UID: "09fccbb2-307c-4752-a4d7-f9986eb5f61c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:39:42.557649 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:42.557618 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09fccbb2-307c-4752-a4d7-f9986eb5f61c-istio-data" pod "09fccbb2-307c-4752-a4d7-f9986eb5f61c" (UID: "09fccbb2-307c-4752-a4d7-f9986eb5f61c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:39:42.557834 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:42.557815 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09fccbb2-307c-4752-a4d7-f9986eb5f61c-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:42.557912 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:42.557837 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09fccbb2-307c-4752-a4d7-f9986eb5f61c-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:42.557912 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:42.557850 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09fccbb2-307c-4752-a4d7-f9986eb5f61c-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:42.557912 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:42.557863 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09fccbb2-307c-4752-a4d7-f9986eb5f61c-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:42.558086 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:42.557916 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09fccbb2-307c-4752-a4d7-f9986eb5f61c-credential-socket" pod "09fccbb2-307c-4752-a4d7-f9986eb5f61c" (UID: "09fccbb2-307c-4752-a4d7-f9986eb5f61c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:39:42.559129 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:42.559095 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09fccbb2-307c-4752-a4d7-f9986eb5f61c-istio-podinfo" pod "09fccbb2-307c-4752-a4d7-f9986eb5f61c" (UID: "09fccbb2-307c-4752-a4d7-f9986eb5f61c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:39:42.559225 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:42.559137 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09fccbb2-307c-4752-a4d7-f9986eb5f61c-istio-envoy" pod "09fccbb2-307c-4752-a4d7-f9986eb5f61c" (UID: "09fccbb2-307c-4752-a4d7-f9986eb5f61c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:39:42.559434 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:42.559401 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09fccbb2-307c-4752-a4d7-f9986eb5f61c-kube-api-access-r72tr" pod "09fccbb2-307c-4752-a4d7-f9986eb5f61c" (UID: "09fccbb2-307c-4752-a4d7-f9986eb5f61c"). InnerVolumeSpecName "kube-api-access-r72tr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:39:42.559544 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:42.559481 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09fccbb2-307c-4752-a4d7-f9986eb5f61c-istio-token" pod "09fccbb2-307c-4752-a4d7-f9986eb5f61c" (UID: "09fccbb2-307c-4752-a4d7-f9986eb5f61c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:39:42.659493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:42.659433 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09fccbb2-307c-4752-a4d7-f9986eb5f61c-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:42.659493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:42.659463 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09fccbb2-307c-4752-a4d7-f9986eb5f61c-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:42.659692 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:42.659501 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r72tr\" (UniqueName: \"kubernetes.io/projected/09fccbb2-307c-4752-a4d7-f9986eb5f61c-kube-api-access-r72tr\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:42.659692 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:42.659513 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09fccbb2-307c-4752-a4d7-f9986eb5f61c-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:42.659692 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:42.659521 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09fccbb2-307c-4752-a4d7-f9986eb5f61c-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:39:42.665610 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:42.665587 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-32bec3-654f99d4f9-m9m9c" Nov 03 13:39:42.731353 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:42.731258 5979 generic.go:356] "Generic (PLEG): container finished" podID="09fccbb2-307c-4752-a4d7-f9986eb5f61c" containerID="05b18fb973820ac2c88f3c93162fc00bccd839f3959871d39e86492c6898af84" exitCode=0 Nov 03 13:39:42.731353 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:42.731320 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s" Nov 03 13:39:42.731616 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:42.731368 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s" event={"ID":"09fccbb2-307c-4752-a4d7-f9986eb5f61c","Type":"ContainerDied","Data":"05b18fb973820ac2c88f3c93162fc00bccd839f3959871d39e86492c6898af84"} Nov 03 13:39:42.731616 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:42.731419 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s" event={"ID":"09fccbb2-307c-4752-a4d7-f9986eb5f61c","Type":"ContainerDied","Data":"d061e9ab10270fb74cbf3d8a074bc5530062ae4e2a89642de1b27d7eb93ee166"} Nov 03 13:39:42.731616 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:42.731448 5979 scope.go:122] "RemoveContainer" containerID="05b18fb973820ac2c88f3c93162fc00bccd839f3959871d39e86492c6898af84" Nov 03 13:39:42.741874 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:42.741855 5979 scope.go:122] "RemoveContainer" containerID="5188e424135679f01285add632aa930ddcc42d7f7793c56b89a020827a80ff5c" Nov 03 13:39:42.751327 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:42.751306 5979 scope.go:122] "RemoveContainer" containerID="05b18fb973820ac2c88f3c93162fc00bccd839f3959871d39e86492c6898af84" Nov 03 13:39:42.751623 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:39:42.751605 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05b18fb973820ac2c88f3c93162fc00bccd839f3959871d39e86492c6898af84\": container with ID starting with 05b18fb973820ac2c88f3c93162fc00bccd839f3959871d39e86492c6898af84 not found: ID does not exist" containerID="05b18fb973820ac2c88f3c93162fc00bccd839f3959871d39e86492c6898af84" Nov 03 13:39:42.751693 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:42.751631 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05b18fb973820ac2c88f3c93162fc00bccd839f3959871d39e86492c6898af84"} err="failed to get container status \"05b18fb973820ac2c88f3c93162fc00bccd839f3959871d39e86492c6898af84\": rpc error: code = NotFound desc = could not find container \"05b18fb973820ac2c88f3c93162fc00bccd839f3959871d39e86492c6898af84\": container with ID starting with 05b18fb973820ac2c88f3c93162fc00bccd839f3959871d39e86492c6898af84 not found: ID does not exist" Nov 03 13:39:42.751693 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:42.751647 5979 scope.go:122] "RemoveContainer" containerID="5188e424135679f01285add632aa930ddcc42d7f7793c56b89a020827a80ff5c" Nov 03 13:39:42.751927 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:39:42.751907 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5188e424135679f01285add632aa930ddcc42d7f7793c56b89a020827a80ff5c\": container with ID starting with 5188e424135679f01285add632aa930ddcc42d7f7793c56b89a020827a80ff5c not found: ID does not exist" containerID="5188e424135679f01285add632aa930ddcc42d7f7793c56b89a020827a80ff5c" Nov 03 13:39:42.751971 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:42.751934 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5188e424135679f01285add632aa930ddcc42d7f7793c56b89a020827a80ff5c"} err="failed to get container status \"5188e424135679f01285add632aa930ddcc42d7f7793c56b89a020827a80ff5c\": rpc error: code = NotFound desc = could not find container \"5188e424135679f01285add632aa930ddcc42d7f7793c56b89a020827a80ff5c\": container with ID starting with 5188e424135679f01285add632aa930ddcc42d7f7793c56b89a020827a80ff5c not found: ID does not exist" Nov 03 13:39:42.882627 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:42.882595 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s"] Nov 03 13:39:42.891259 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:42.891228 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f6f9ed-openshift-default-78b6dbdddc-rvh4s"] Nov 03 13:39:44.116311 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:44.116279 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09fccbb2-307c-4752-a4d7-f9986eb5f61c" path="/var/lib/kubelet/pods/09fccbb2-307c-4752-a4d7-f9986eb5f61c/volumes" Nov 03 13:39:45.689629 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:45.689600 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-02f3a0-c66d764b6-qwl7q" Nov 03 13:39:45.690061 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:45.689677 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02f3a0-68759d857f-9c2dn" Nov 03 13:39:46.972916 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:46.972888 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5474b7-547b5c768c-kkfdb"] Nov 03 13:39:46.973483 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:46.973458 5979 cpu_manager.go:414] "containerMap: removing container" podUID="09fccbb2-307c-4752-a4d7-f9986eb5f61c" containerName="istio-proxy" Nov 03 13:39:46.973552 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:46.973485 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09fccbb2-307c-4752-a4d7-f9986eb5f61c" containerName="istio-proxy" Nov 03 13:39:46.973552 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:46.973520 5979 cpu_manager.go:414] "containerMap: removing container" podUID="09fccbb2-307c-4752-a4d7-f9986eb5f61c" containerName="istio-proxy" Nov 03 13:39:46.973552 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:46.973526 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09fccbb2-307c-4752-a4d7-f9986eb5f61c" containerName="istio-proxy" Nov 03 13:39:46.973667 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:46.973625 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="09fccbb2-307c-4752-a4d7-f9986eb5f61c" containerName="istio-proxy" Nov 03 13:39:46.978246 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:46.978227 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5474b7-547b5c768c-kkfdb" Nov 03 13:39:47.019208 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:47.019179 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csg96\" (UniqueName: \"kubernetes.io/projected/3b504f5b-6fd2-47fd-9e95-ef445a0775ce-kube-api-access-csg96\") pod \"payload-pre-processing-e2e-ipp-b-5474b7-547b5c768c-kkfdb\" (UID: \"3b504f5b-6fd2-47fd-9e95-ef445a0775ce\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5474b7-547b5c768c-kkfdb" Nov 03 13:39:47.019367 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:47.019333 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b504f5b-6fd2-47fd-9e95-ef445a0775ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5474b7-547b5c768c-kkfdb\" (UID: \"3b504f5b-6fd2-47fd-9e95-ef445a0775ce\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5474b7-547b5c768c-kkfdb" Nov 03 13:39:47.079460 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:47.079426 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-5474b7-dockercfg-4xqlj\"" Nov 03 13:39:47.079650 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:47.079496 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-5474b7\"" Nov 03 13:39:47.082225 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:47.082199 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5474b7-547b5c768c-kkfdb"] Nov 03 13:39:47.120659 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:47.120624 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b504f5b-6fd2-47fd-9e95-ef445a0775ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5474b7-547b5c768c-kkfdb\" (UID: \"3b504f5b-6fd2-47fd-9e95-ef445a0775ce\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5474b7-547b5c768c-kkfdb" Nov 03 13:39:47.120871 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:47.120797 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csg96\" (UniqueName: \"kubernetes.io/projected/3b504f5b-6fd2-47fd-9e95-ef445a0775ce-kube-api-access-csg96\") pod \"payload-pre-processing-e2e-ipp-b-5474b7-547b5c768c-kkfdb\" (UID: \"3b504f5b-6fd2-47fd-9e95-ef445a0775ce\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5474b7-547b5c768c-kkfdb" Nov 03 13:39:47.121338 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:47.121316 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b504f5b-6fd2-47fd-9e95-ef445a0775ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5474b7-547b5c768c-kkfdb\" (UID: \"3b504f5b-6fd2-47fd-9e95-ef445a0775ce\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5474b7-547b5c768c-kkfdb" Nov 03 13:39:47.194068 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:47.194030 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csg96\" (UniqueName: \"kubernetes.io/projected/3b504f5b-6fd2-47fd-9e95-ef445a0775ce-kube-api-access-csg96\") pod \"payload-pre-processing-e2e-ipp-b-5474b7-547b5c768c-kkfdb\" (UID: \"3b504f5b-6fd2-47fd-9e95-ef445a0775ce\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5474b7-547b5c768c-kkfdb" Nov 03 13:39:47.292111 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:47.292027 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5474b7-547b5c768c-kkfdb" Nov 03 13:39:47.596245 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:47.596161 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5474b7-cdb6b4598-t5szx"] Nov 03 13:39:47.596963 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:47.596945 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="09fccbb2-307c-4752-a4d7-f9986eb5f61c" containerName="istio-proxy" Nov 03 13:39:47.601159 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:47.601131 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5474b7-cdb6b4598-t5szx" Nov 03 13:39:47.664004 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:47.663974 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5474b7-cdb6b4598-t5szx"] Nov 03 13:39:47.728647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:47.728613 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fbb116e-cbe7-4b76-b379-15537008e6a9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5474b7-cdb6b4598-t5szx\" (UID: \"4fbb116e-cbe7-4b76-b379-15537008e6a9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5474b7-cdb6b4598-t5szx" Nov 03 13:39:47.728859 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:47.728677 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5b6d\" (UniqueName: \"kubernetes.io/projected/4fbb116e-cbe7-4b76-b379-15537008e6a9-kube-api-access-h5b6d\") pod \"payload-processing-e2e-ipp-b-5474b7-cdb6b4598-t5szx\" (UID: \"4fbb116e-cbe7-4b76-b379-15537008e6a9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5474b7-cdb6b4598-t5szx" Nov 03 13:39:47.738416 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:47.738382 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5474b7-547b5c768c-kkfdb"] Nov 03 13:39:47.775607 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:39:47.775578 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b504f5b_6fd2_47fd_9e95_ef445a0775ce.slice/crio-77d4f0a5eb700ff9c47b742387a13a6996007742ebf0ccc82306f10c49fa08a4 WatchSource:0}: Error finding container 77d4f0a5eb700ff9c47b742387a13a6996007742ebf0ccc82306f10c49fa08a4: Status 404 returned error can't find the container with id 77d4f0a5eb700ff9c47b742387a13a6996007742ebf0ccc82306f10c49fa08a4 Nov 03 13:39:47.830348 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:47.830315 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fbb116e-cbe7-4b76-b379-15537008e6a9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5474b7-cdb6b4598-t5szx\" (UID: \"4fbb116e-cbe7-4b76-b379-15537008e6a9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5474b7-cdb6b4598-t5szx" Nov 03 13:39:47.830516 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:47.830373 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5b6d\" (UniqueName: \"kubernetes.io/projected/4fbb116e-cbe7-4b76-b379-15537008e6a9-kube-api-access-h5b6d\") pod \"payload-processing-e2e-ipp-b-5474b7-cdb6b4598-t5szx\" (UID: \"4fbb116e-cbe7-4b76-b379-15537008e6a9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5474b7-cdb6b4598-t5szx" Nov 03 13:39:47.831016 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:47.830995 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fbb116e-cbe7-4b76-b379-15537008e6a9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5474b7-cdb6b4598-t5szx\" (UID: \"4fbb116e-cbe7-4b76-b379-15537008e6a9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5474b7-cdb6b4598-t5szx" Nov 03 13:39:47.883860 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:47.883829 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5b6d\" (UniqueName: \"kubernetes.io/projected/4fbb116e-cbe7-4b76-b379-15537008e6a9-kube-api-access-h5b6d\") pod \"payload-processing-e2e-ipp-b-5474b7-cdb6b4598-t5szx\" (UID: \"4fbb116e-cbe7-4b76-b379-15537008e6a9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5474b7-cdb6b4598-t5szx" Nov 03 13:39:47.915090 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:47.915059 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5474b7-cdb6b4598-t5szx" Nov 03 13:39:48.326853 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:48.326772 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5474b7-cdb6b4598-t5szx"] Nov 03 13:39:48.333508 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:39:48.333463 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fbb116e_cbe7_4b76_b379_15537008e6a9.slice/crio-d38a24c9068f10db43d3b4cbd19fb2d0ee449c5d6789f5ef69a00414ce2ff0b4 WatchSource:0}: Error finding container d38a24c9068f10db43d3b4cbd19fb2d0ee449c5d6789f5ef69a00414ce2ff0b4: Status 404 returned error can't find the container with id d38a24c9068f10db43d3b4cbd19fb2d0ee449c5d6789f5ef69a00414ce2ff0b4 Nov 03 13:39:48.780275 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:48.780235 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5474b7-547b5c768c-kkfdb" event={"ID":"3b504f5b-6fd2-47fd-9e95-ef445a0775ce","Type":"ContainerStarted","Data":"34a2b24e1e183e3ef397535bc7f6db743b4b6248b27174d2be0f41c217308d54"} Nov 03 13:39:48.780511 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:48.780300 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5474b7-547b5c768c-kkfdb" event={"ID":"3b504f5b-6fd2-47fd-9e95-ef445a0775ce","Type":"ContainerStarted","Data":"77d4f0a5eb700ff9c47b742387a13a6996007742ebf0ccc82306f10c49fa08a4"} Nov 03 13:39:48.780511 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:48.780327 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5474b7-547b5c768c-kkfdb" Nov 03 13:39:48.782143 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:48.782115 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5474b7-cdb6b4598-t5szx" event={"ID":"4fbb116e-cbe7-4b76-b379-15537008e6a9","Type":"ContainerStarted","Data":"68add378ec3313b3934426cf7bdc9676113195f5e8f816c4c47cffc8c4517ff5"} Nov 03 13:39:48.782266 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:48.782163 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5474b7-cdb6b4598-t5szx" event={"ID":"4fbb116e-cbe7-4b76-b379-15537008e6a9","Type":"ContainerStarted","Data":"d38a24c9068f10db43d3b4cbd19fb2d0ee449c5d6789f5ef69a00414ce2ff0b4"} Nov 03 13:39:48.782266 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:48.782213 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5474b7-cdb6b4598-t5szx" Nov 03 13:39:48.925542 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:48.925462 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5474b7-547b5c768c-kkfdb" podStartSLOduration=2.925443927 podStartE2EDuration="2.925443927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:39:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:39:48.922168214 +0000 UTC m=+2070.932856544" watchObservedRunningTime="2026-11-03 13:39:48.925443927 +0000 UTC m=+2070.936132169" Nov 03 13:39:57.974079 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:57.973989 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-5474b7-cdb6b4598-t5szx" podStartSLOduration=10.973974888 podStartE2EDuration="10.973974888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:39:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:39:49.085656146 +0000 UTC m=+2071.096344390" watchObservedRunningTime="2026-11-03 13:39:57.973974888 +0000 UTC m=+2079.984663131" Nov 03 13:39:57.976422 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:57.976398 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6bd5c1-78bb66789f-pjv6n"] Nov 03 13:39:57.982112 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:57.982093 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6bd5c1-78bb66789f-pjv6n" Nov 03 13:39:58.036772 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:58.036726 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6bd5c1-78bb66789f-pjv6n"] Nov 03 13:39:58.110507 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:58.110103 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-6bd5c1-dockercfg-kwdhj\"" Nov 03 13:39:58.110507 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:58.110363 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-6bd5c1\"" Nov 03 13:39:58.128529 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:58.125650 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/122981ad-160e-4bd3-a78d-de1649e61667-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6bd5c1-78bb66789f-pjv6n\" (UID: \"122981ad-160e-4bd3-a78d-de1649e61667\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6bd5c1-78bb66789f-pjv6n" Nov 03 13:39:58.128529 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:58.125726 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt8wt\" (UniqueName: \"kubernetes.io/projected/122981ad-160e-4bd3-a78d-de1649e61667-kube-api-access-qt8wt\") pod \"payload-pre-processing-e2e-ipp-b-6bd5c1-78bb66789f-pjv6n\" (UID: \"122981ad-160e-4bd3-a78d-de1649e61667\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6bd5c1-78bb66789f-pjv6n" Nov 03 13:39:58.146309 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:58.146276 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6bd5c1-fd8fdd4bf-2p55l"] Nov 03 13:39:58.151572 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:58.151537 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6bd5c1-fd8fdd4bf-2p55l" Nov 03 13:39:58.175034 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:58.175001 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6bd5c1-fd8fdd4bf-2p55l"] Nov 03 13:39:58.226930 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:58.226841 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/122981ad-160e-4bd3-a78d-de1649e61667-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6bd5c1-78bb66789f-pjv6n\" (UID: \"122981ad-160e-4bd3-a78d-de1649e61667\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6bd5c1-78bb66789f-pjv6n" Nov 03 13:39:58.226930 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:58.226882 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ca552c2-8806-4fd9-b015-759d3aac38bf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6bd5c1-fd8fdd4bf-2p55l\" (UID: \"8ca552c2-8806-4fd9-b015-759d3aac38bf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6bd5c1-fd8fdd4bf-2p55l" Nov 03 13:39:58.227141 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:58.226998 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt8wt\" (UniqueName: \"kubernetes.io/projected/122981ad-160e-4bd3-a78d-de1649e61667-kube-api-access-qt8wt\") pod \"payload-pre-processing-e2e-ipp-b-6bd5c1-78bb66789f-pjv6n\" (UID: \"122981ad-160e-4bd3-a78d-de1649e61667\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6bd5c1-78bb66789f-pjv6n" Nov 03 13:39:58.227187 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:58.227167 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shnkj\" (UniqueName: \"kubernetes.io/projected/8ca552c2-8806-4fd9-b015-759d3aac38bf-kube-api-access-shnkj\") pod \"payload-processing-e2e-ipp-b-6bd5c1-fd8fdd4bf-2p55l\" (UID: \"8ca552c2-8806-4fd9-b015-759d3aac38bf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6bd5c1-fd8fdd4bf-2p55l" Nov 03 13:39:58.227744 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:58.227724 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/122981ad-160e-4bd3-a78d-de1649e61667-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6bd5c1-78bb66789f-pjv6n\" (UID: \"122981ad-160e-4bd3-a78d-de1649e61667\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6bd5c1-78bb66789f-pjv6n" Nov 03 13:39:58.281064 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:58.281024 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt8wt\" (UniqueName: \"kubernetes.io/projected/122981ad-160e-4bd3-a78d-de1649e61667-kube-api-access-qt8wt\") pod \"payload-pre-processing-e2e-ipp-b-6bd5c1-78bb66789f-pjv6n\" (UID: \"122981ad-160e-4bd3-a78d-de1649e61667\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6bd5c1-78bb66789f-pjv6n" Nov 03 13:39:58.297261 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:58.297227 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6bd5c1-78bb66789f-pjv6n" Nov 03 13:39:58.328926 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:58.328894 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shnkj\" (UniqueName: \"kubernetes.io/projected/8ca552c2-8806-4fd9-b015-759d3aac38bf-kube-api-access-shnkj\") pod \"payload-processing-e2e-ipp-b-6bd5c1-fd8fdd4bf-2p55l\" (UID: \"8ca552c2-8806-4fd9-b015-759d3aac38bf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6bd5c1-fd8fdd4bf-2p55l" Nov 03 13:39:58.329106 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:58.329087 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ca552c2-8806-4fd9-b015-759d3aac38bf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6bd5c1-fd8fdd4bf-2p55l\" (UID: \"8ca552c2-8806-4fd9-b015-759d3aac38bf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6bd5c1-fd8fdd4bf-2p55l" Nov 03 13:39:58.329830 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:58.329810 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ca552c2-8806-4fd9-b015-759d3aac38bf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6bd5c1-fd8fdd4bf-2p55l\" (UID: \"8ca552c2-8806-4fd9-b015-759d3aac38bf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6bd5c1-fd8fdd4bf-2p55l" Nov 03 13:39:58.387251 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:58.387220 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shnkj\" (UniqueName: \"kubernetes.io/projected/8ca552c2-8806-4fd9-b015-759d3aac38bf-kube-api-access-shnkj\") pod \"payload-processing-e2e-ipp-b-6bd5c1-fd8fdd4bf-2p55l\" (UID: \"8ca552c2-8806-4fd9-b015-759d3aac38bf\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6bd5c1-fd8fdd4bf-2p55l" Nov 03 13:39:58.470939 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:58.470896 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6bd5c1-fd8fdd4bf-2p55l" Nov 03 13:39:58.675188 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:58.675156 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6bd5c1-78bb66789f-pjv6n"] Nov 03 13:39:58.866455 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:58.866420 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6bd5c1-fd8fdd4bf-2p55l"] Nov 03 13:39:58.873985 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:39:58.873958 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ca552c2_8806_4fd9_b015_759d3aac38bf.slice/crio-5ec21083f9ec3910fec7e65564b2284e9dbdea68c4da7e01b68ec1e336f16252 WatchSource:0}: Error finding container 5ec21083f9ec3910fec7e65564b2284e9dbdea68c4da7e01b68ec1e336f16252: Status 404 returned error can't find the container with id 5ec21083f9ec3910fec7e65564b2284e9dbdea68c4da7e01b68ec1e336f16252 Nov 03 13:39:58.887977 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:58.887944 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6bd5c1-78bb66789f-pjv6n" event={"ID":"122981ad-160e-4bd3-a78d-de1649e61667","Type":"ContainerStarted","Data":"4328f08b2b2c1894d36e64ede7ec2925da2e3d0c17555361778762a74097fd8a"} Nov 03 13:39:58.888104 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:58.887998 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6bd5c1-78bb66789f-pjv6n" event={"ID":"122981ad-160e-4bd3-a78d-de1649e61667","Type":"ContainerStarted","Data":"bef40e603803eb06e0a7675e9e62cde25ef1250af1a29a51ae242bd8f5add58c"} Nov 03 13:39:58.888104 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:58.888060 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6bd5c1-78bb66789f-pjv6n" Nov 03 13:39:58.996896 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:58.996780 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6bd5c1-78bb66789f-pjv6n" podStartSLOduration=1.996764735 podStartE2EDuration="1.996764735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:39:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:39:58.996216577 +0000 UTC m=+2081.006904821" watchObservedRunningTime="2026-11-03 13:39:58.996764735 +0000 UTC m=+2081.007452976" Nov 03 13:39:59.790667 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:59.790634 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5474b7-cdb6b4598-t5szx" Nov 03 13:39:59.790869 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:59.790716 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5474b7-547b5c768c-kkfdb" Nov 03 13:39:59.896307 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:59.896262 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6bd5c1-fd8fdd4bf-2p55l" event={"ID":"8ca552c2-8806-4fd9-b015-759d3aac38bf","Type":"ContainerStarted","Data":"7028016cd60bca7175e8f5c20ed3e58963a5bc190ea2dc503eaee7ad9bf3f655"} Nov 03 13:39:59.896518 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:59.896327 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6bd5c1-fd8fdd4bf-2p55l" event={"ID":"8ca552c2-8806-4fd9-b015-759d3aac38bf","Type":"ContainerStarted","Data":"5ec21083f9ec3910fec7e65564b2284e9dbdea68c4da7e01b68ec1e336f16252"} Nov 03 13:39:59.896518 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:39:59.896369 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6bd5c1-fd8fdd4bf-2p55l" Nov 03 13:40:00.146694 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:00.146639 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-6bd5c1-fd8fdd4bf-2p55l" podStartSLOduration=2.146624709 podStartE2EDuration="2.146624709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:39:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:40:00.145405052 +0000 UTC m=+2082.156093295" watchObservedRunningTime="2026-11-03 13:40:00.146624709 +0000 UTC m=+2082.157312951" Nov 03 13:40:07.235194 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:40:07.235151 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:40:07.671509 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:07.671463 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-007b0d-7bcff96687-b8kvb"] Nov 03 13:40:07.677562 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:07.677534 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-007b0d-7bcff96687-b8kvb" Nov 03 13:40:07.815494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:07.790318 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-007b0d-7bcff96687-b8kvb"] Nov 03 13:40:07.815847 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:07.815816 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-007b0d\"" Nov 03 13:40:07.816001 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:07.815911 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-007b0d-dockercfg-vtg27\"" Nov 03 13:40:07.818997 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:07.818966 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjkh5\" (UniqueName: \"kubernetes.io/projected/f9454ca7-0414-4da3-a8c6-98b41c84a16e-kube-api-access-pjkh5\") pod \"payload-pre-processing-e2e-praxis-b-007b0d-7bcff96687-b8kvb\" (UID: \"f9454ca7-0414-4da3-a8c6-98b41c84a16e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-007b0d-7bcff96687-b8kvb" Nov 03 13:40:07.819207 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:07.819188 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9454ca7-0414-4da3-a8c6-98b41c84a16e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-007b0d-7bcff96687-b8kvb\" (UID: \"f9454ca7-0414-4da3-a8c6-98b41c84a16e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-007b0d-7bcff96687-b8kvb" Nov 03 13:40:07.839693 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:07.839663 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-007b0d-54546cd87f-5b5kz"] Nov 03 13:40:07.845021 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:07.844992 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-007b0d-54546cd87f-5b5kz" Nov 03 13:40:07.901979 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:07.901940 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-007b0d-54546cd87f-5b5kz"] Nov 03 13:40:07.919975 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:07.919941 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p26ff\" (UniqueName: \"kubernetes.io/projected/84b314b4-86d3-4849-b0a1-35fcd77f2747-kube-api-access-p26ff\") pod \"payload-processing-e2e-praxis-b-007b0d-54546cd87f-5b5kz\" (UID: \"84b314b4-86d3-4849-b0a1-35fcd77f2747\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-007b0d-54546cd87f-5b5kz" Nov 03 13:40:07.920124 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:07.920054 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjkh5\" (UniqueName: \"kubernetes.io/projected/f9454ca7-0414-4da3-a8c6-98b41c84a16e-kube-api-access-pjkh5\") pod \"payload-pre-processing-e2e-praxis-b-007b0d-7bcff96687-b8kvb\" (UID: \"f9454ca7-0414-4da3-a8c6-98b41c84a16e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-007b0d-7bcff96687-b8kvb" Nov 03 13:40:07.920124 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:07.920112 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9454ca7-0414-4da3-a8c6-98b41c84a16e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-007b0d-7bcff96687-b8kvb\" (UID: \"f9454ca7-0414-4da3-a8c6-98b41c84a16e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-007b0d-7bcff96687-b8kvb" Nov 03 13:40:07.920257 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:07.920235 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84b314b4-86d3-4849-b0a1-35fcd77f2747-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-007b0d-54546cd87f-5b5kz\" (UID: \"84b314b4-86d3-4849-b0a1-35fcd77f2747\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-007b0d-54546cd87f-5b5kz" Nov 03 13:40:07.920813 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:07.920791 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9454ca7-0414-4da3-a8c6-98b41c84a16e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-007b0d-7bcff96687-b8kvb\" (UID: \"f9454ca7-0414-4da3-a8c6-98b41c84a16e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-007b0d-7bcff96687-b8kvb" Nov 03 13:40:07.973991 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:07.973917 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjkh5\" (UniqueName: \"kubernetes.io/projected/f9454ca7-0414-4da3-a8c6-98b41c84a16e-kube-api-access-pjkh5\") pod \"payload-pre-processing-e2e-praxis-b-007b0d-7bcff96687-b8kvb\" (UID: \"f9454ca7-0414-4da3-a8c6-98b41c84a16e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-007b0d-7bcff96687-b8kvb" Nov 03 13:40:07.992680 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:07.992652 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-007b0d-7bcff96687-b8kvb" Nov 03 13:40:08.021973 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:08.021941 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p26ff\" (UniqueName: \"kubernetes.io/projected/84b314b4-86d3-4849-b0a1-35fcd77f2747-kube-api-access-p26ff\") pod \"payload-processing-e2e-praxis-b-007b0d-54546cd87f-5b5kz\" (UID: \"84b314b4-86d3-4849-b0a1-35fcd77f2747\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-007b0d-54546cd87f-5b5kz" Nov 03 13:40:08.022167 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:08.022048 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84b314b4-86d3-4849-b0a1-35fcd77f2747-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-007b0d-54546cd87f-5b5kz\" (UID: \"84b314b4-86d3-4849-b0a1-35fcd77f2747\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-007b0d-54546cd87f-5b5kz" Nov 03 13:40:08.023040 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:08.023010 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84b314b4-86d3-4849-b0a1-35fcd77f2747-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-007b0d-54546cd87f-5b5kz\" (UID: \"84b314b4-86d3-4849-b0a1-35fcd77f2747\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-007b0d-54546cd87f-5b5kz" Nov 03 13:40:08.075641 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:08.075607 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p26ff\" (UniqueName: \"kubernetes.io/projected/84b314b4-86d3-4849-b0a1-35fcd77f2747-kube-api-access-p26ff\") pod \"payload-processing-e2e-praxis-b-007b0d-54546cd87f-5b5kz\" (UID: \"84b314b4-86d3-4849-b0a1-35fcd77f2747\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-007b0d-54546cd87f-5b5kz" Nov 03 13:40:08.160532 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:08.160497 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-007b0d-54546cd87f-5b5kz" Nov 03 13:40:08.367936 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:08.367842 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-007b0d-7bcff96687-b8kvb"] Nov 03 13:40:08.374615 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:40:08.374585 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9454ca7_0414_4da3_a8c6_98b41c84a16e.slice/crio-11f9fad8bde64f07ed7477f7e2b794cba65c8cb8d52c10045eede1ae414bb1c7 WatchSource:0}: Error finding container 11f9fad8bde64f07ed7477f7e2b794cba65c8cb8d52c10045eede1ae414bb1c7: Status 404 returned error can't find the container with id 11f9fad8bde64f07ed7477f7e2b794cba65c8cb8d52c10045eede1ae414bb1c7 Nov 03 13:40:08.535957 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:08.535919 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-007b0d-54546cd87f-5b5kz"] Nov 03 13:40:08.543409 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:40:08.543381 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84b314b4_86d3_4849_b0a1_35fcd77f2747.slice/crio-7ee9da396297ebae1140af88d5621f8e34ea77dd3c3d2cfdd2839e04cadebced WatchSource:0}: Error finding container 7ee9da396297ebae1140af88d5621f8e34ea77dd3c3d2cfdd2839e04cadebced: Status 404 returned error can't find the container with id 7ee9da396297ebae1140af88d5621f8e34ea77dd3c3d2cfdd2839e04cadebced Nov 03 13:40:08.964714 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:08.964678 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-007b0d-7bcff96687-b8kvb" event={"ID":"f9454ca7-0414-4da3-a8c6-98b41c84a16e","Type":"ContainerStarted","Data":"c3847467b04e211e87ed0f60cb3f13643e922da7f5bb98093560fbbf02927b81"} Nov 03 13:40:08.964916 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:08.964741 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-007b0d-7bcff96687-b8kvb" Nov 03 13:40:08.964916 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:08.964765 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-007b0d-7bcff96687-b8kvb" event={"ID":"f9454ca7-0414-4da3-a8c6-98b41c84a16e","Type":"ContainerStarted","Data":"11f9fad8bde64f07ed7477f7e2b794cba65c8cb8d52c10045eede1ae414bb1c7"} Nov 03 13:40:08.966280 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:08.966253 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-007b0d-54546cd87f-5b5kz" event={"ID":"84b314b4-86d3-4849-b0a1-35fcd77f2747","Type":"ContainerStarted","Data":"9e565c475fcd1a23216357f97a87809ad8cd598fd07bfadc342c3ec448e56716"} Nov 03 13:40:08.966377 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:08.966297 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-007b0d-54546cd87f-5b5kz" event={"ID":"84b314b4-86d3-4849-b0a1-35fcd77f2747","Type":"ContainerStarted","Data":"7ee9da396297ebae1140af88d5621f8e34ea77dd3c3d2cfdd2839e04cadebced"} Nov 03 13:40:08.966377 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:08.966323 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-007b0d-54546cd87f-5b5kz" Nov 03 13:40:09.073730 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:09.073668 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-007b0d-7bcff96687-b8kvb" podStartSLOduration=2.073653023 podStartE2EDuration="2.073653023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:40:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:40:09.071022291 +0000 UTC m=+2091.081710532" watchObservedRunningTime="2026-11-03 13:40:09.073653023 +0000 UTC m=+2091.084341265" Nov 03 13:40:09.178557 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:09.178496 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-007b0d-54546cd87f-5b5kz" podStartSLOduration=2.178481661 podStartE2EDuration="2.178481661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:40:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:40:09.177417558 +0000 UTC m=+2091.188105801" watchObservedRunningTime="2026-11-03 13:40:09.178481661 +0000 UTC m=+2091.189169894" Nov 03 13:40:09.900337 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:09.900304 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6bd5c1-78bb66789f-pjv6n" Nov 03 13:40:10.491856 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:10.491818 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:40:10.906696 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:10.906670 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6bd5c1-fd8fdd4bf-2p55l" Nov 03 13:40:15.047546 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:15.047512 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba0d2a06-2b9dm"] Nov 03 13:40:15.093836 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:15.093795 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba0d2a06-2b9dm"] Nov 03 13:40:16.116818 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:16.116779 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6c1fb16-1f63-4044-b7cc-0fb72ac53b31" path="/var/lib/kubelet/pods/b6c1fb16-1f63-4044-b7cc-0fb72ac53b31/volumes" Nov 03 13:40:19.975537 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:19.975424 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-007b0d-7bcff96687-b8kvb" Nov 03 13:40:19.975537 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:19.975511 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-007b0d-54546cd87f-5b5kz" Nov 03 13:40:22.131597 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:22.131563 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mksft"] Nov 03 13:40:22.132130 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:22.131784 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mksft" podUID="42cb885d-1072-4057-ab7d-71fe9f166f7d" containerName="istio-proxy" containerID="cri-o://d6319f06d469cad4eeae33f3859af5dc985efb88b2f5b185684c96f3d55ab0d2" gracePeriod=30 Nov 03 13:40:25.799676 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:25.799631 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-mksft container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 13:40:25.800177 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:25.799708 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mksft" podUID="42cb885d-1072-4057-ab7d-71fe9f166f7d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 13:40:27.393737 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:27.393709 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mksft" Nov 03 13:40:27.586061 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:27.585973 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42cb885d-1072-4057-ab7d-71fe9f166f7d-istio-token\" (UniqueName: \"kubernetes.io/projected/42cb885d-1072-4057-ab7d-71fe9f166f7d-istio-token\") pod \"42cb885d-1072-4057-ab7d-71fe9f166f7d\" (UID: \"42cb885d-1072-4057-ab7d-71fe9f166f7d\") " Nov 03 13:40:27.586061 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:27.586014 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42cb885d-1072-4057-ab7d-71fe9f166f7d-kube-api-access-lcl6b\" (UniqueName: \"kubernetes.io/projected/42cb885d-1072-4057-ab7d-71fe9f166f7d-kube-api-access-lcl6b\") pod \"42cb885d-1072-4057-ab7d-71fe9f166f7d\" (UID: \"42cb885d-1072-4057-ab7d-71fe9f166f7d\") " Nov 03 13:40:27.586313 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:27.586073 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42cb885d-1072-4057-ab7d-71fe9f166f7d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42cb885d-1072-4057-ab7d-71fe9f166f7d-istio-envoy\") pod \"42cb885d-1072-4057-ab7d-71fe9f166f7d\" (UID: \"42cb885d-1072-4057-ab7d-71fe9f166f7d\") " Nov 03 13:40:27.586313 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:27.586112 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42cb885d-1072-4057-ab7d-71fe9f166f7d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42cb885d-1072-4057-ab7d-71fe9f166f7d-workload-socket\") pod \"42cb885d-1072-4057-ab7d-71fe9f166f7d\" (UID: \"42cb885d-1072-4057-ab7d-71fe9f166f7d\") " Nov 03 13:40:27.586313 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:27.586192 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42cb885d-1072-4057-ab7d-71fe9f166f7d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42cb885d-1072-4057-ab7d-71fe9f166f7d-workload-certs\") pod \"42cb885d-1072-4057-ab7d-71fe9f166f7d\" (UID: \"42cb885d-1072-4057-ab7d-71fe9f166f7d\") " Nov 03 13:40:27.586313 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:27.586220 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42cb885d-1072-4057-ab7d-71fe9f166f7d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42cb885d-1072-4057-ab7d-71fe9f166f7d-istio-data\") pod \"42cb885d-1072-4057-ab7d-71fe9f166f7d\" (UID: \"42cb885d-1072-4057-ab7d-71fe9f166f7d\") " Nov 03 13:40:27.586313 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:27.586283 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/42cb885d-1072-4057-ab7d-71fe9f166f7d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42cb885d-1072-4057-ab7d-71fe9f166f7d-istio-podinfo\") pod \"42cb885d-1072-4057-ab7d-71fe9f166f7d\" (UID: \"42cb885d-1072-4057-ab7d-71fe9f166f7d\") " Nov 03 13:40:27.586637 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:27.586344 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42cb885d-1072-4057-ab7d-71fe9f166f7d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42cb885d-1072-4057-ab7d-71fe9f166f7d-credential-socket\") pod \"42cb885d-1072-4057-ab7d-71fe9f166f7d\" (UID: \"42cb885d-1072-4057-ab7d-71fe9f166f7d\") " Nov 03 13:40:27.586637 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:27.586377 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42cb885d-1072-4057-ab7d-71fe9f166f7d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42cb885d-1072-4057-ab7d-71fe9f166f7d-istiod-ca-cert\") pod \"42cb885d-1072-4057-ab7d-71fe9f166f7d\" (UID: \"42cb885d-1072-4057-ab7d-71fe9f166f7d\") " Nov 03 13:40:27.586637 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:27.586546 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42cb885d-1072-4057-ab7d-71fe9f166f7d-workload-certs" pod "42cb885d-1072-4057-ab7d-71fe9f166f7d" (UID: "42cb885d-1072-4057-ab7d-71fe9f166f7d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:40:27.586811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:27.586764 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42cb885d-1072-4057-ab7d-71fe9f166f7d-workload-socket" pod "42cb885d-1072-4057-ab7d-71fe9f166f7d" (UID: "42cb885d-1072-4057-ab7d-71fe9f166f7d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:40:27.586867 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:27.586838 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42cb885d-1072-4057-ab7d-71fe9f166f7d-istio-data" pod "42cb885d-1072-4057-ab7d-71fe9f166f7d" (UID: "42cb885d-1072-4057-ab7d-71fe9f166f7d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:40:27.586982 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:27.586947 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42cb885d-1072-4057-ab7d-71fe9f166f7d-istiod-ca-cert" pod "42cb885d-1072-4057-ab7d-71fe9f166f7d" (UID: "42cb885d-1072-4057-ab7d-71fe9f166f7d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:40:27.587388 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:27.587185 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42cb885d-1072-4057-ab7d-71fe9f166f7d-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:40:27.587388 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:27.587211 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42cb885d-1072-4057-ab7d-71fe9f166f7d-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:40:27.587388 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:27.587226 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42cb885d-1072-4057-ab7d-71fe9f166f7d-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:40:27.587388 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:27.587240 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42cb885d-1072-4057-ab7d-71fe9f166f7d-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:40:27.587388 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:27.587358 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42cb885d-1072-4057-ab7d-71fe9f166f7d-credential-socket" pod "42cb885d-1072-4057-ab7d-71fe9f166f7d" (UID: "42cb885d-1072-4057-ab7d-71fe9f166f7d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:40:27.588993 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:27.588950 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42cb885d-1072-4057-ab7d-71fe9f166f7d-kube-api-access-lcl6b" pod "42cb885d-1072-4057-ab7d-71fe9f166f7d" (UID: "42cb885d-1072-4057-ab7d-71fe9f166f7d"). InnerVolumeSpecName "kube-api-access-lcl6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:40:27.589112 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:27.589014 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42cb885d-1072-4057-ab7d-71fe9f166f7d-istio-podinfo" pod "42cb885d-1072-4057-ab7d-71fe9f166f7d" (UID: "42cb885d-1072-4057-ab7d-71fe9f166f7d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:40:27.589112 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:27.589021 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42cb885d-1072-4057-ab7d-71fe9f166f7d-istio-token" pod "42cb885d-1072-4057-ab7d-71fe9f166f7d" (UID: "42cb885d-1072-4057-ab7d-71fe9f166f7d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:40:27.589112 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:27.589032 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42cb885d-1072-4057-ab7d-71fe9f166f7d-istio-envoy" pod "42cb885d-1072-4057-ab7d-71fe9f166f7d" (UID: "42cb885d-1072-4057-ab7d-71fe9f166f7d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:40:27.688706 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:27.688675 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42cb885d-1072-4057-ab7d-71fe9f166f7d-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:40:27.688706 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:27.688704 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42cb885d-1072-4057-ab7d-71fe9f166f7d-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:40:27.688919 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:27.688716 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42cb885d-1072-4057-ab7d-71fe9f166f7d-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:40:27.688919 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:27.688730 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcl6b\" (UniqueName: \"kubernetes.io/projected/42cb885d-1072-4057-ab7d-71fe9f166f7d-kube-api-access-lcl6b\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:40:27.688919 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:27.688745 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42cb885d-1072-4057-ab7d-71fe9f166f7d-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:40:28.100829 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:28.100788 5979 generic.go:356] "Generic (PLEG): container finished" podID="42cb885d-1072-4057-ab7d-71fe9f166f7d" containerID="d6319f06d469cad4eeae33f3859af5dc985efb88b2f5b185684c96f3d55ab0d2" exitCode=0 Nov 03 13:40:28.101029 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:28.100849 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mksft" Nov 03 13:40:28.101029 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:28.100873 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mksft" event={"ID":"42cb885d-1072-4057-ab7d-71fe9f166f7d","Type":"ContainerDied","Data":"d6319f06d469cad4eeae33f3859af5dc985efb88b2f5b185684c96f3d55ab0d2"} Nov 03 13:40:28.101029 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:28.100961 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mksft" event={"ID":"42cb885d-1072-4057-ab7d-71fe9f166f7d","Type":"ContainerDied","Data":"82d8c8a1fa4d6121a24d97cbdd4c87a5cba3ffe4335812576e169aabe93550b5"} Nov 03 13:40:28.101029 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:28.100990 5979 scope.go:122] "RemoveContainer" containerID="d6319f06d469cad4eeae33f3859af5dc985efb88b2f5b185684c96f3d55ab0d2" Nov 03 13:40:28.111833 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:28.111811 5979 scope.go:122] "RemoveContainer" containerID="d6319f06d469cad4eeae33f3859af5dc985efb88b2f5b185684c96f3d55ab0d2" Nov 03 13:40:28.112167 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:40:28.112142 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6319f06d469cad4eeae33f3859af5dc985efb88b2f5b185684c96f3d55ab0d2\": container with ID starting with d6319f06d469cad4eeae33f3859af5dc985efb88b2f5b185684c96f3d55ab0d2 not found: ID does not exist" containerID="d6319f06d469cad4eeae33f3859af5dc985efb88b2f5b185684c96f3d55ab0d2" Nov 03 13:40:28.112235 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:28.112176 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6319f06d469cad4eeae33f3859af5dc985efb88b2f5b185684c96f3d55ab0d2"} err="failed to get container status \"d6319f06d469cad4eeae33f3859af5dc985efb88b2f5b185684c96f3d55ab0d2\": rpc error: code = NotFound desc = could not find container \"d6319f06d469cad4eeae33f3859af5dc985efb88b2f5b185684c96f3d55ab0d2\": container with ID starting with d6319f06d469cad4eeae33f3859af5dc985efb88b2f5b185684c96f3d55ab0d2 not found: ID does not exist" Nov 03 13:40:28.306227 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:28.306193 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mksft"] Nov 03 13:40:28.352825 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:28.352739 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-mksft"] Nov 03 13:40:30.116340 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:30.116296 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42cb885d-1072-4057-ab7d-71fe9f166f7d" path="/var/lib/kubelet/pods/42cb885d-1072-4057-ab7d-71fe9f166f7d/volumes" Nov 03 13:40:33.923854 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:33.923818 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-32bec3-b5b998c44-7qx4z"] Nov 03 13:40:33.924755 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:33.924732 5979 cpu_manager.go:414] "containerMap: removing container" podUID="42cb885d-1072-4057-ab7d-71fe9f166f7d" containerName="istio-proxy" Nov 03 13:40:33.924864 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:33.924757 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42cb885d-1072-4057-ab7d-71fe9f166f7d" containerName="istio-proxy" Nov 03 13:40:33.924932 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:33.924921 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="42cb885d-1072-4057-ab7d-71fe9f166f7d" containerName="istio-proxy" Nov 03 13:40:33.929708 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:33.929689 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-32bec3-b5b998c44-7qx4z" Nov 03 13:40:33.968889 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:33.968851 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bddwx\" (UniqueName: \"kubernetes.io/projected/a47855e5-140b-478a-bd1e-33c43c14482e-kube-api-access-bddwx\") pod \"maas-api-e2e-ipp-b-32bec3-b5b998c44-7qx4z\" (UID: \"a47855e5-140b-478a-bd1e-33c43c14482e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-32bec3-b5b998c44-7qx4z" Nov 03 13:40:33.969070 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:33.968983 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a47855e5-140b-478a-bd1e-33c43c14482e-metrics-certs\") pod \"maas-api-e2e-ipp-b-32bec3-b5b998c44-7qx4z\" (UID: \"a47855e5-140b-478a-bd1e-33c43c14482e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-32bec3-b5b998c44-7qx4z" Nov 03 13:40:33.969070 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:33.969023 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a47855e5-140b-478a-bd1e-33c43c14482e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-32bec3-b5b998c44-7qx4z\" (UID: \"a47855e5-140b-478a-bd1e-33c43c14482e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-32bec3-b5b998c44-7qx4z" Nov 03 13:40:33.969070 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:33.969063 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a47855e5-140b-478a-bd1e-33c43c14482e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-32bec3-b5b998c44-7qx4z\" (UID: \"a47855e5-140b-478a-bd1e-33c43c14482e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-32bec3-b5b998c44-7qx4z" Nov 03 13:40:33.979735 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:33.979709 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-32bec3-serving-cert\"" Nov 03 13:40:34.029065 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:34.029022 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-32bec3-b5b998c44-7qx4z"] Nov 03 13:40:34.070881 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:34.070838 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bddwx\" (UniqueName: \"kubernetes.io/projected/a47855e5-140b-478a-bd1e-33c43c14482e-kube-api-access-bddwx\") pod \"maas-api-e2e-ipp-b-32bec3-b5b998c44-7qx4z\" (UID: \"a47855e5-140b-478a-bd1e-33c43c14482e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-32bec3-b5b998c44-7qx4z" Nov 03 13:40:34.071092 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:34.070989 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a47855e5-140b-478a-bd1e-33c43c14482e-metrics-certs\") pod \"maas-api-e2e-ipp-b-32bec3-b5b998c44-7qx4z\" (UID: \"a47855e5-140b-478a-bd1e-33c43c14482e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-32bec3-b5b998c44-7qx4z" Nov 03 13:40:34.071092 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:34.071033 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a47855e5-140b-478a-bd1e-33c43c14482e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-32bec3-b5b998c44-7qx4z\" (UID: \"a47855e5-140b-478a-bd1e-33c43c14482e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-32bec3-b5b998c44-7qx4z" Nov 03 13:40:34.071092 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:34.071089 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a47855e5-140b-478a-bd1e-33c43c14482e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-32bec3-b5b998c44-7qx4z\" (UID: \"a47855e5-140b-478a-bd1e-33c43c14482e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-32bec3-b5b998c44-7qx4z" Nov 03 13:40:34.071321 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:40:34.071297 5979 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-32bec3-serving-cert: secret "maas-api-e2e-ipp-b-32bec3-serving-cert" not found Nov 03 13:40:34.071381 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:40:34.071373 5979 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a47855e5-140b-478a-bd1e-33c43c14482e-maas-api-tls podName:a47855e5-140b-478a-bd1e-33c43c14482e nodeName:}" failed. No retries permitted until 2026-11-03 13:40:34.571353929 +0000 UTC m=+2116.582042152 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a47855e5-140b-478a-bd1e-33c43c14482e-maas-api-tls") pod "maas-api-e2e-ipp-b-32bec3-b5b998c44-7qx4z" (UID: "a47855e5-140b-478a-bd1e-33c43c14482e") : secret "maas-api-e2e-ipp-b-32bec3-serving-cert" not found Nov 03 13:40:34.071942 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:34.071914 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a47855e5-140b-478a-bd1e-33c43c14482e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-32bec3-b5b998c44-7qx4z\" (UID: \"a47855e5-140b-478a-bd1e-33c43c14482e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-32bec3-b5b998c44-7qx4z" Nov 03 13:40:34.074025 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:34.074001 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a47855e5-140b-478a-bd1e-33c43c14482e-metrics-certs\") pod \"maas-api-e2e-ipp-b-32bec3-b5b998c44-7qx4z\" (UID: \"a47855e5-140b-478a-bd1e-33c43c14482e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-32bec3-b5b998c44-7qx4z" Nov 03 13:40:34.127375 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:34.127341 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bddwx\" (UniqueName: \"kubernetes.io/projected/a47855e5-140b-478a-bd1e-33c43c14482e-kube-api-access-bddwx\") pod \"maas-api-e2e-ipp-b-32bec3-b5b998c44-7qx4z\" (UID: \"a47855e5-140b-478a-bd1e-33c43c14482e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-32bec3-b5b998c44-7qx4z" Nov 03 13:40:34.578109 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:34.578073 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a47855e5-140b-478a-bd1e-33c43c14482e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-32bec3-b5b998c44-7qx4z\" (UID: \"a47855e5-140b-478a-bd1e-33c43c14482e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-32bec3-b5b998c44-7qx4z" Nov 03 13:40:34.580462 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:34.580439 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a47855e5-140b-478a-bd1e-33c43c14482e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-32bec3-b5b998c44-7qx4z\" (UID: \"a47855e5-140b-478a-bd1e-33c43c14482e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-32bec3-b5b998c44-7qx4z" Nov 03 13:40:34.843121 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:34.843036 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-32bec3-b5b998c44-7qx4z" Nov 03 13:40:35.220311 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:35.220271 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-32bec3-b5b998c44-7qx4z"] Nov 03 13:40:35.227551 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:40:35.227456 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda47855e5_140b_478a_bd1e_33c43c14482e.slice/crio-b7ec1d39bfc4fbd04ae9060320a5014a22a04dfcb9b119a78b0d475a8e80c0c3 WatchSource:0}: Error finding container b7ec1d39bfc4fbd04ae9060320a5014a22a04dfcb9b119a78b0d475a8e80c0c3: Status 404 returned error can't find the container with id b7ec1d39bfc4fbd04ae9060320a5014a22a04dfcb9b119a78b0d475a8e80c0c3 Nov 03 13:40:36.160886 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:36.157413 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-32bec3-b5b998c44-7qx4z" event={"ID":"a47855e5-140b-478a-bd1e-33c43c14482e","Type":"ContainerStarted","Data":"b7ec1d39bfc4fbd04ae9060320a5014a22a04dfcb9b119a78b0d475a8e80c0c3"} Nov 03 13:40:37.165392 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:37.165350 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-32bec3-b5b998c44-7qx4z" event={"ID":"a47855e5-140b-478a-bd1e-33c43c14482e","Type":"ContainerStarted","Data":"193f2c4a7a239d17851d9b637744fb9040933a808938ff4ae37cb2be4f363f5e"} Nov 03 13:40:37.165827 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:37.165437 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-32bec3-b5b998c44-7qx4z" Nov 03 13:40:37.236351 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:40:37.236311 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:40:37.277218 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:37.277156 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-32bec3-b5b998c44-7qx4z" podStartSLOduration=3.365878603 podStartE2EDuration="4.277140307s" totalImagesPullingTime="911.261704ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:40:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:40:37.274263848 +0000 UTC m=+2119.284952091" watchObservedRunningTime="2026-11-03 13:40:37.277140307 +0000 UTC m=+2119.287828581" Nov 03 13:40:43.177107 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:43.177076 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-32bec3-b5b998c44-7qx4z" Nov 03 13:40:50.948090 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:50.948049 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5474b7-75c7f99fd7-lvxls"] Nov 03 13:40:50.948737 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:50.948229 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5474b7-75c7f99fd7-lvxls" podUID="0e90f590-e39e-4e48-9aa7-215be078347b" containerName="maas-api" containerID="cri-o://bb7ab0470c9fbe796aa14715f32f5915a213b185ba11286a599e9a5abab29583" gracePeriod=30 Nov 03 13:40:51.255339 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:51.255307 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5474b7-75c7f99fd7-lvxls" Nov 03 13:40:51.265715 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:51.265686 5979 generic.go:356] "Generic (PLEG): container finished" podID="0e90f590-e39e-4e48-9aa7-215be078347b" containerID="bb7ab0470c9fbe796aa14715f32f5915a213b185ba11286a599e9a5abab29583" exitCode=0 Nov 03 13:40:51.265854 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:51.265748 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5474b7-75c7f99fd7-lvxls" Nov 03 13:40:51.265854 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:51.265781 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5474b7-75c7f99fd7-lvxls" event={"ID":"0e90f590-e39e-4e48-9aa7-215be078347b","Type":"ContainerDied","Data":"bb7ab0470c9fbe796aa14715f32f5915a213b185ba11286a599e9a5abab29583"} Nov 03 13:40:51.265854 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:51.265836 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5474b7-75c7f99fd7-lvxls" event={"ID":"0e90f590-e39e-4e48-9aa7-215be078347b","Type":"ContainerDied","Data":"5bd44f6d9a0900c5d74f43d0f7142886b9aee3c482e5dea73e0ab34b2172e430"} Nov 03 13:40:51.265985 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:51.265864 5979 scope.go:122] "RemoveContainer" containerID="bb7ab0470c9fbe796aa14715f32f5915a213b185ba11286a599e9a5abab29583" Nov 03 13:40:51.277512 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:51.277487 5979 scope.go:122] "RemoveContainer" containerID="bb7ab0470c9fbe796aa14715f32f5915a213b185ba11286a599e9a5abab29583" Nov 03 13:40:51.277779 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:40:51.277758 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb7ab0470c9fbe796aa14715f32f5915a213b185ba11286a599e9a5abab29583\": container with ID starting with bb7ab0470c9fbe796aa14715f32f5915a213b185ba11286a599e9a5abab29583 not found: ID does not exist" containerID="bb7ab0470c9fbe796aa14715f32f5915a213b185ba11286a599e9a5abab29583" Nov 03 13:40:51.277841 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:51.277788 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb7ab0470c9fbe796aa14715f32f5915a213b185ba11286a599e9a5abab29583"} err="failed to get container status \"bb7ab0470c9fbe796aa14715f32f5915a213b185ba11286a599e9a5abab29583\": rpc error: code = NotFound desc = could not find container \"bb7ab0470c9fbe796aa14715f32f5915a213b185ba11286a599e9a5abab29583\": container with ID starting with bb7ab0470c9fbe796aa14715f32f5915a213b185ba11286a599e9a5abab29583 not found: ID does not exist" Nov 03 13:40:51.354060 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:51.349867 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e90f590-e39e-4e48-9aa7-215be078347b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e90f590-e39e-4e48-9aa7-215be078347b-openshift-service-ca\") pod \"0e90f590-e39e-4e48-9aa7-215be078347b\" (UID: \"0e90f590-e39e-4e48-9aa7-215be078347b\") " Nov 03 13:40:51.354060 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:51.349970 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e90f590-e39e-4e48-9aa7-215be078347b-kube-api-access-ts6wc\" (UniqueName: \"kubernetes.io/projected/0e90f590-e39e-4e48-9aa7-215be078347b-kube-api-access-ts6wc\") pod \"0e90f590-e39e-4e48-9aa7-215be078347b\" (UID: \"0e90f590-e39e-4e48-9aa7-215be078347b\") " Nov 03 13:40:51.354060 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:51.350046 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e90f590-e39e-4e48-9aa7-215be078347b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e90f590-e39e-4e48-9aa7-215be078347b-metrics-certs\") pod \"0e90f590-e39e-4e48-9aa7-215be078347b\" (UID: \"0e90f590-e39e-4e48-9aa7-215be078347b\") " Nov 03 13:40:51.354060 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:51.350229 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e90f590-e39e-4e48-9aa7-215be078347b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e90f590-e39e-4e48-9aa7-215be078347b-maas-api-tls\") pod \"0e90f590-e39e-4e48-9aa7-215be078347b\" (UID: \"0e90f590-e39e-4e48-9aa7-215be078347b\") " Nov 03 13:40:51.356701 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:51.356286 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e90f590-e39e-4e48-9aa7-215be078347b-openshift-service-ca" pod "0e90f590-e39e-4e48-9aa7-215be078347b" (UID: "0e90f590-e39e-4e48-9aa7-215be078347b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:40:51.364842 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:51.364799 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e90f590-e39e-4e48-9aa7-215be078347b-kube-api-access-ts6wc" pod "0e90f590-e39e-4e48-9aa7-215be078347b" (UID: "0e90f590-e39e-4e48-9aa7-215be078347b"). InnerVolumeSpecName "kube-api-access-ts6wc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:40:51.367358 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:51.366748 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e90f590-e39e-4e48-9aa7-215be078347b-maas-api-tls" pod "0e90f590-e39e-4e48-9aa7-215be078347b" (UID: "0e90f590-e39e-4e48-9aa7-215be078347b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:40:51.375683 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:51.375648 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e90f590-e39e-4e48-9aa7-215be078347b-metrics-certs" pod "0e90f590-e39e-4e48-9aa7-215be078347b" (UID: "0e90f590-e39e-4e48-9aa7-215be078347b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:40:51.420834 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:51.420797 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5474b7-cdb6b4598-t5szx"] Nov 03 13:40:51.421033 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:51.420997 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-5474b7-cdb6b4598-t5szx" podUID="4fbb116e-cbe7-4b76-b379-15537008e6a9" containerName="payload-processing" containerID="cri-o://68add378ec3313b3934426cf7bdc9676113195f5e8f816c4c47cffc8c4517ff5" gracePeriod=30 Nov 03 13:40:51.456165 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:51.456131 5979 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e90f590-e39e-4e48-9aa7-215be078347b-metrics-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:40:51.456165 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:51.456162 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e90f590-e39e-4e48-9aa7-215be078347b-maas-api-tls\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:40:51.456398 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:51.456172 5979 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e90f590-e39e-4e48-9aa7-215be078347b-openshift-service-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:40:51.456398 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:51.456183 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts6wc\" (UniqueName: \"kubernetes.io/projected/0e90f590-e39e-4e48-9aa7-215be078347b-kube-api-access-ts6wc\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:40:51.483681 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:51.483606 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5474b7-547b5c768c-kkfdb"] Nov 03 13:40:51.483836 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:51.483786 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5474b7-547b5c768c-kkfdb" podUID="3b504f5b-6fd2-47fd-9e95-ef445a0775ce" containerName="payload-pre-processing" containerID="cri-o://34a2b24e1e183e3ef397535bc7f6db743b4b6248b27174d2be0f41c217308d54" gracePeriod=30 Nov 03 13:40:51.755709 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:51.755558 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5474b7-75c7f99fd7-lvxls"] Nov 03 13:40:51.762517 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:51.760191 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5474b7-75c7f99fd7-lvxls"] Nov 03 13:40:51.775735 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:51.775711 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5474b7-cdb6b4598-t5szx" Nov 03 13:40:51.810487 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:51.810439 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5474b7-547b5c768c-kkfdb" Nov 03 13:40:51.859929 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:51.859891 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fbb116e-cbe7-4b76-b379-15537008e6a9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fbb116e-cbe7-4b76-b379-15537008e6a9-plugins-config-volume\") pod \"4fbb116e-cbe7-4b76-b379-15537008e6a9\" (UID: \"4fbb116e-cbe7-4b76-b379-15537008e6a9\") " Nov 03 13:40:51.860131 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:51.859981 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b504f5b-6fd2-47fd-9e95-ef445a0775ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b504f5b-6fd2-47fd-9e95-ef445a0775ce-plugins-config-volume\") pod \"3b504f5b-6fd2-47fd-9e95-ef445a0775ce\" (UID: \"3b504f5b-6fd2-47fd-9e95-ef445a0775ce\") " Nov 03 13:40:51.860131 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:51.860023 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fbb116e-cbe7-4b76-b379-15537008e6a9-kube-api-access-h5b6d\" (UniqueName: \"kubernetes.io/projected/4fbb116e-cbe7-4b76-b379-15537008e6a9-kube-api-access-h5b6d\") pod \"4fbb116e-cbe7-4b76-b379-15537008e6a9\" (UID: \"4fbb116e-cbe7-4b76-b379-15537008e6a9\") " Nov 03 13:40:51.860131 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:51.860073 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b504f5b-6fd2-47fd-9e95-ef445a0775ce-kube-api-access-csg96\" (UniqueName: \"kubernetes.io/projected/3b504f5b-6fd2-47fd-9e95-ef445a0775ce-kube-api-access-csg96\") pod \"3b504f5b-6fd2-47fd-9e95-ef445a0775ce\" (UID: \"3b504f5b-6fd2-47fd-9e95-ef445a0775ce\") " Nov 03 13:40:51.860324 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:51.860288 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fbb116e-cbe7-4b76-b379-15537008e6a9-plugins-config-volume" pod "4fbb116e-cbe7-4b76-b379-15537008e6a9" (UID: "4fbb116e-cbe7-4b76-b379-15537008e6a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:40:51.860391 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:51.860359 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b504f5b-6fd2-47fd-9e95-ef445a0775ce-plugins-config-volume" pod "3b504f5b-6fd2-47fd-9e95-ef445a0775ce" (UID: "3b504f5b-6fd2-47fd-9e95-ef445a0775ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:40:51.860575 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:51.860554 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fbb116e-cbe7-4b76-b379-15537008e6a9-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:40:51.860649 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:51.860577 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b504f5b-6fd2-47fd-9e95-ef445a0775ce-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:40:51.862193 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:51.862167 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fbb116e-cbe7-4b76-b379-15537008e6a9-kube-api-access-h5b6d" pod "4fbb116e-cbe7-4b76-b379-15537008e6a9" (UID: "4fbb116e-cbe7-4b76-b379-15537008e6a9"). InnerVolumeSpecName "kube-api-access-h5b6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:40:51.862533 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:51.862509 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b504f5b-6fd2-47fd-9e95-ef445a0775ce-kube-api-access-csg96" pod "3b504f5b-6fd2-47fd-9e95-ef445a0775ce" (UID: "3b504f5b-6fd2-47fd-9e95-ef445a0775ce"). InnerVolumeSpecName "kube-api-access-csg96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:40:51.962542 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:51.962203 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5b6d\" (UniqueName: \"kubernetes.io/projected/4fbb116e-cbe7-4b76-b379-15537008e6a9-kube-api-access-h5b6d\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:40:51.962542 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:51.962244 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csg96\" (UniqueName: \"kubernetes.io/projected/3b504f5b-6fd2-47fd-9e95-ef445a0775ce-kube-api-access-csg96\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:40:52.119891 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:52.119805 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e90f590-e39e-4e48-9aa7-215be078347b" path="/var/lib/kubelet/pods/0e90f590-e39e-4e48-9aa7-215be078347b/volumes" Nov 03 13:40:52.276052 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:52.276010 5979 generic.go:356] "Generic (PLEG): container finished" podID="3b504f5b-6fd2-47fd-9e95-ef445a0775ce" containerID="34a2b24e1e183e3ef397535bc7f6db743b4b6248b27174d2be0f41c217308d54" exitCode=0 Nov 03 13:40:52.276277 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:52.276066 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5474b7-547b5c768c-kkfdb" Nov 03 13:40:52.276277 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:52.276105 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5474b7-547b5c768c-kkfdb" event={"ID":"3b504f5b-6fd2-47fd-9e95-ef445a0775ce","Type":"ContainerDied","Data":"34a2b24e1e183e3ef397535bc7f6db743b4b6248b27174d2be0f41c217308d54"} Nov 03 13:40:52.276277 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:52.276157 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5474b7-547b5c768c-kkfdb" event={"ID":"3b504f5b-6fd2-47fd-9e95-ef445a0775ce","Type":"ContainerDied","Data":"77d4f0a5eb700ff9c47b742387a13a6996007742ebf0ccc82306f10c49fa08a4"} Nov 03 13:40:52.276277 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:52.276195 5979 scope.go:122] "RemoveContainer" containerID="34a2b24e1e183e3ef397535bc7f6db743b4b6248b27174d2be0f41c217308d54" Nov 03 13:40:52.282529 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:52.281227 5979 generic.go:356] "Generic (PLEG): container finished" podID="4fbb116e-cbe7-4b76-b379-15537008e6a9" containerID="68add378ec3313b3934426cf7bdc9676113195f5e8f816c4c47cffc8c4517ff5" exitCode=0 Nov 03 13:40:52.282529 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:52.281284 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5474b7-cdb6b4598-t5szx" event={"ID":"4fbb116e-cbe7-4b76-b379-15537008e6a9","Type":"ContainerDied","Data":"68add378ec3313b3934426cf7bdc9676113195f5e8f816c4c47cffc8c4517ff5"} Nov 03 13:40:52.282529 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:52.281323 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5474b7-cdb6b4598-t5szx" event={"ID":"4fbb116e-cbe7-4b76-b379-15537008e6a9","Type":"ContainerDied","Data":"d38a24c9068f10db43d3b4cbd19fb2d0ee449c5d6789f5ef69a00414ce2ff0b4"} Nov 03 13:40:52.282529 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:52.281394 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5474b7-cdb6b4598-t5szx" Nov 03 13:40:52.290046 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:52.290025 5979 scope.go:122] "RemoveContainer" containerID="34a2b24e1e183e3ef397535bc7f6db743b4b6248b27174d2be0f41c217308d54" Nov 03 13:40:52.290370 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:40:52.290340 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34a2b24e1e183e3ef397535bc7f6db743b4b6248b27174d2be0f41c217308d54\": container with ID starting with 34a2b24e1e183e3ef397535bc7f6db743b4b6248b27174d2be0f41c217308d54 not found: ID does not exist" containerID="34a2b24e1e183e3ef397535bc7f6db743b4b6248b27174d2be0f41c217308d54" Nov 03 13:40:52.290526 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:52.290373 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34a2b24e1e183e3ef397535bc7f6db743b4b6248b27174d2be0f41c217308d54"} err="failed to get container status \"34a2b24e1e183e3ef397535bc7f6db743b4b6248b27174d2be0f41c217308d54\": rpc error: code = NotFound desc = could not find container \"34a2b24e1e183e3ef397535bc7f6db743b4b6248b27174d2be0f41c217308d54\": container with ID starting with 34a2b24e1e183e3ef397535bc7f6db743b4b6248b27174d2be0f41c217308d54 not found: ID does not exist" Nov 03 13:40:52.290526 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:52.290397 5979 scope.go:122] "RemoveContainer" containerID="68add378ec3313b3934426cf7bdc9676113195f5e8f816c4c47cffc8c4517ff5" Nov 03 13:40:52.300974 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:52.300955 5979 scope.go:122] "RemoveContainer" containerID="68add378ec3313b3934426cf7bdc9676113195f5e8f816c4c47cffc8c4517ff5" Nov 03 13:40:52.301264 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:40:52.301245 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68add378ec3313b3934426cf7bdc9676113195f5e8f816c4c47cffc8c4517ff5\": container with ID starting with 68add378ec3313b3934426cf7bdc9676113195f5e8f816c4c47cffc8c4517ff5 not found: ID does not exist" containerID="68add378ec3313b3934426cf7bdc9676113195f5e8f816c4c47cffc8c4517ff5" Nov 03 13:40:52.301340 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:52.301271 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68add378ec3313b3934426cf7bdc9676113195f5e8f816c4c47cffc8c4517ff5"} err="failed to get container status \"68add378ec3313b3934426cf7bdc9676113195f5e8f816c4c47cffc8c4517ff5\": rpc error: code = NotFound desc = could not find container \"68add378ec3313b3934426cf7bdc9676113195f5e8f816c4c47cffc8c4517ff5\": container with ID starting with 68add378ec3313b3934426cf7bdc9676113195f5e8f816c4c47cffc8c4517ff5 not found: ID does not exist" Nov 03 13:40:52.432033 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:52.432002 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5474b7-547b5c768c-kkfdb"] Nov 03 13:40:52.439101 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:52.439070 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5474b7-547b5c768c-kkfdb"] Nov 03 13:40:52.541570 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:52.541536 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5474b7-cdb6b4598-t5szx"] Nov 03 13:40:52.546872 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:52.546842 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5474b7-cdb6b4598-t5szx"] Nov 03 13:40:54.117086 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:54.117045 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b504f5b-6fd2-47fd-9e95-ef445a0775ce" path="/var/lib/kubelet/pods/3b504f5b-6fd2-47fd-9e95-ef445a0775ce/volumes" Nov 03 13:40:54.117625 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:54.117426 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fbb116e-cbe7-4b76-b379-15537008e6a9" path="/var/lib/kubelet/pods/4fbb116e-cbe7-4b76-b379-15537008e6a9/volumes" Nov 03 13:40:54.737383 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:54.737351 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf"] Nov 03 13:40:54.737647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:54.737625 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf" podUID="efd2a382-b6ea-419f-b33a-9ebfdf0bd61b" containerName="istio-proxy" containerID="cri-o://89dc0798d357859d7d39d3dcae3278d83018164920f3d2ca4344f29b3fcb459d" gracePeriod=30 Nov 03 13:40:55.322667 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:55.322627 5979 generic.go:356] "Generic (PLEG): container finished" podID="87552227-fce6-474b-9b1b-fe167d1f6547" containerID="1d343e63cb3beedf3c9030d3ec7edaba1256dd00b507a492462edf1af793c031" exitCode=0 Nov 03 13:40:55.323096 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:55.322699 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686" event={"ID":"87552227-fce6-474b-9b1b-fe167d1f6547","Type":"ContainerDied","Data":"1d343e63cb3beedf3c9030d3ec7edaba1256dd00b507a492462edf1af793c031"} Nov 03 13:40:55.323143 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:55.323096 5979 scope.go:122] "RemoveContainer" containerID="1d343e63cb3beedf3c9030d3ec7edaba1256dd00b507a492462edf1af793c031" Nov 03 13:40:55.324493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:55.324449 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:40:55.324612 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:55.324541 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:40:55.324612 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:55.324590 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:40:56.330866 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:56.330833 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686" event={"ID":"87552227-fce6-474b-9b1b-fe167d1f6547","Type":"ContainerStarted","Data":"07dce1585d30ffbfde67bb61883dbef54c2fd155be42615a8a7a79b1ae8cbbcd"} Nov 03 13:40:56.530545 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:56.530507 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686" Nov 03 13:40:56.530757 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:56.530728 5979 patch_prober.go:29] interesting pod/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Nov 03 13:40:56.530845 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:56.530795 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686" podUID="87552227-fce6-474b-9b1b-fe167d1f6547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Nov 03 13:40:57.068663 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:57.068613 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq"] Nov 03 13:40:57.069645 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:57.069617 5979 cpu_manager.go:414] "containerMap: removing container" podUID="3b504f5b-6fd2-47fd-9e95-ef445a0775ce" containerName="payload-pre-processing" Nov 03 13:40:57.069645 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:57.069644 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b504f5b-6fd2-47fd-9e95-ef445a0775ce" containerName="payload-pre-processing" Nov 03 13:40:57.069873 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:57.069670 5979 cpu_manager.go:414] "containerMap: removing container" podUID="0e90f590-e39e-4e48-9aa7-215be078347b" containerName="maas-api" Nov 03 13:40:57.069873 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:57.069678 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e90f590-e39e-4e48-9aa7-215be078347b" containerName="maas-api" Nov 03 13:40:57.069873 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:57.069722 5979 cpu_manager.go:414] "containerMap: removing container" podUID="4fbb116e-cbe7-4b76-b379-15537008e6a9" containerName="payload-processing" Nov 03 13:40:57.069873 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:57.069730 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fbb116e-cbe7-4b76-b379-15537008e6a9" containerName="payload-processing" Nov 03 13:40:57.070065 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:57.069886 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b504f5b-6fd2-47fd-9e95-ef445a0775ce" containerName="payload-pre-processing" Nov 03 13:40:57.070065 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:57.069903 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e90f590-e39e-4e48-9aa7-215be078347b" containerName="maas-api" Nov 03 13:40:57.070065 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:57.069917 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fbb116e-cbe7-4b76-b379-15537008e6a9" containerName="payload-processing" Nov 03 13:40:57.074697 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:57.074670 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq" Nov 03 13:40:57.111014 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:57.110983 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq"] Nov 03 13:40:57.142518 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:57.142484 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/763f2c9e-811d-49dc-b971-4782cb8a0964-istio-data\") pod \"e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq\" (UID: \"763f2c9e-811d-49dc-b971-4782cb8a0964\") " pod="openshift-ingress/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq" Nov 03 13:40:57.142734 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:57.142528 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxjm9\" (UniqueName: \"kubernetes.io/projected/763f2c9e-811d-49dc-b971-4782cb8a0964-kube-api-access-nxjm9\") pod \"e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq\" (UID: \"763f2c9e-811d-49dc-b971-4782cb8a0964\") " pod="openshift-ingress/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq" Nov 03 13:40:57.142734 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:57.142701 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/763f2c9e-811d-49dc-b971-4782cb8a0964-credential-socket\") pod \"e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq\" (UID: \"763f2c9e-811d-49dc-b971-4782cb8a0964\") " pod="openshift-ingress/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq" Nov 03 13:40:57.142871 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:57.142750 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/763f2c9e-811d-49dc-b971-4782cb8a0964-istio-envoy\") pod \"e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq\" (UID: \"763f2c9e-811d-49dc-b971-4782cb8a0964\") " pod="openshift-ingress/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq" Nov 03 13:40:57.142944 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:57.142902 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/763f2c9e-811d-49dc-b971-4782cb8a0964-workload-socket\") pod \"e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq\" (UID: \"763f2c9e-811d-49dc-b971-4782cb8a0964\") " pod="openshift-ingress/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq" Nov 03 13:40:57.143158 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:57.143133 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/763f2c9e-811d-49dc-b971-4782cb8a0964-workload-certs\") pod \"e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq\" (UID: \"763f2c9e-811d-49dc-b971-4782cb8a0964\") " pod="openshift-ingress/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq" Nov 03 13:40:57.143264 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:57.143197 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/763f2c9e-811d-49dc-b971-4782cb8a0964-istio-podinfo\") pod \"e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq\" (UID: \"763f2c9e-811d-49dc-b971-4782cb8a0964\") " pod="openshift-ingress/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq" Nov 03 13:40:57.143419 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:57.143399 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/763f2c9e-811d-49dc-b971-4782cb8a0964-istiod-ca-cert\") pod \"e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq\" (UID: \"763f2c9e-811d-49dc-b971-4782cb8a0964\") " pod="openshift-ingress/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq" Nov 03 13:40:57.157599 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:57.143444 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/763f2c9e-811d-49dc-b971-4782cb8a0964-istio-token\") pod \"e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq\" (UID: \"763f2c9e-811d-49dc-b971-4782cb8a0964\") " pod="openshift-ingress/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq" Nov 03 13:40:57.200332 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:57.200204 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-4b9658-openshift-default-dockercfg-574h7\"" Nov 03 13:40:57.247499 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:57.246295 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/763f2c9e-811d-49dc-b971-4782cb8a0964-istiod-ca-cert\") pod \"e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq\" (UID: \"763f2c9e-811d-49dc-b971-4782cb8a0964\") " pod="openshift-ingress/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq" Nov 03 13:40:57.247499 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:57.246360 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/763f2c9e-811d-49dc-b971-4782cb8a0964-istio-token\") pod \"e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq\" (UID: \"763f2c9e-811d-49dc-b971-4782cb8a0964\") " pod="openshift-ingress/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq" Nov 03 13:40:57.247499 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:57.246425 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/763f2c9e-811d-49dc-b971-4782cb8a0964-istio-data\") pod \"e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq\" (UID: \"763f2c9e-811d-49dc-b971-4782cb8a0964\") " pod="openshift-ingress/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq" Nov 03 13:40:57.247499 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:57.246456 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxjm9\" (UniqueName: \"kubernetes.io/projected/763f2c9e-811d-49dc-b971-4782cb8a0964-kube-api-access-nxjm9\") pod \"e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq\" (UID: \"763f2c9e-811d-49dc-b971-4782cb8a0964\") " pod="openshift-ingress/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq" Nov 03 13:40:57.247499 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:57.246524 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/763f2c9e-811d-49dc-b971-4782cb8a0964-credential-socket\") pod \"e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq\" (UID: \"763f2c9e-811d-49dc-b971-4782cb8a0964\") " pod="openshift-ingress/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq" Nov 03 13:40:57.247499 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:57.246559 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/763f2c9e-811d-49dc-b971-4782cb8a0964-istio-envoy\") pod \"e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq\" (UID: \"763f2c9e-811d-49dc-b971-4782cb8a0964\") " pod="openshift-ingress/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq" Nov 03 13:40:57.247499 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:57.246620 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/763f2c9e-811d-49dc-b971-4782cb8a0964-workload-socket\") pod \"e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq\" (UID: \"763f2c9e-811d-49dc-b971-4782cb8a0964\") " pod="openshift-ingress/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq" Nov 03 13:40:57.247499 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:57.246651 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/763f2c9e-811d-49dc-b971-4782cb8a0964-workload-certs\") pod \"e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq\" (UID: \"763f2c9e-811d-49dc-b971-4782cb8a0964\") " pod="openshift-ingress/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq" Nov 03 13:40:57.247499 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:57.246701 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/763f2c9e-811d-49dc-b971-4782cb8a0964-istio-podinfo\") pod \"e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq\" (UID: \"763f2c9e-811d-49dc-b971-4782cb8a0964\") " pod="openshift-ingress/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq" Nov 03 13:40:57.272493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:57.248285 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/763f2c9e-811d-49dc-b971-4782cb8a0964-istiod-ca-cert\") pod \"e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq\" (UID: \"763f2c9e-811d-49dc-b971-4782cb8a0964\") " pod="openshift-ingress/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq" Nov 03 13:40:57.272493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:57.248848 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/763f2c9e-811d-49dc-b971-4782cb8a0964-istio-data\") pod \"e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq\" (UID: \"763f2c9e-811d-49dc-b971-4782cb8a0964\") " pod="openshift-ingress/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq" Nov 03 13:40:57.272493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:57.258941 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/763f2c9e-811d-49dc-b971-4782cb8a0964-credential-socket\") pod \"e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq\" (UID: \"763f2c9e-811d-49dc-b971-4782cb8a0964\") " pod="openshift-ingress/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq" Nov 03 13:40:57.272493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:57.259194 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/763f2c9e-811d-49dc-b971-4782cb8a0964-workload-socket\") pod \"e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq\" (UID: \"763f2c9e-811d-49dc-b971-4782cb8a0964\") " pod="openshift-ingress/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq" Nov 03 13:40:57.272493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:57.259456 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/763f2c9e-811d-49dc-b971-4782cb8a0964-workload-certs\") pod \"e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq\" (UID: \"763f2c9e-811d-49dc-b971-4782cb8a0964\") " pod="openshift-ingress/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq" Nov 03 13:40:57.275337 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:57.275308 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/763f2c9e-811d-49dc-b971-4782cb8a0964-istio-envoy\") pod \"e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq\" (UID: \"763f2c9e-811d-49dc-b971-4782cb8a0964\") " pod="openshift-ingress/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq" Nov 03 13:40:57.275744 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:57.275708 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/763f2c9e-811d-49dc-b971-4782cb8a0964-istio-podinfo\") pod \"e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq\" (UID: \"763f2c9e-811d-49dc-b971-4782cb8a0964\") " pod="openshift-ingress/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq" Nov 03 13:40:57.301684 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:57.301653 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/763f2c9e-811d-49dc-b971-4782cb8a0964-istio-token\") pod \"e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq\" (UID: \"763f2c9e-811d-49dc-b971-4782cb8a0964\") " pod="openshift-ingress/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq" Nov 03 13:40:57.301795 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:57.301770 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxjm9\" (UniqueName: \"kubernetes.io/projected/763f2c9e-811d-49dc-b971-4782cb8a0964-kube-api-access-nxjm9\") pod \"e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq\" (UID: \"763f2c9e-811d-49dc-b971-4782cb8a0964\") " pod="openshift-ingress/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq" Nov 03 13:40:57.392118 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:57.392080 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq" Nov 03 13:40:57.531256 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:57.531221 5979 patch_prober.go:29] interesting pod/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Nov 03 13:40:57.531453 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:57.531294 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686" podUID="87552227-fce6-474b-9b1b-fe167d1f6547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Nov 03 13:40:57.871229 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:57.871179 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq"] Nov 03 13:40:57.873002 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:40:57.872973 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod763f2c9e_811d_49dc_b971_4782cb8a0964.slice/crio-50a9d7ca7bdab0665b6a3ff0dd12689659520a0aeba5bc1cad02c9c43750e67d WatchSource:0}: Error finding container 50a9d7ca7bdab0665b6a3ff0dd12689659520a0aeba5bc1cad02c9c43750e67d: Status 404 returned error can't find the container with id 50a9d7ca7bdab0665b6a3ff0dd12689659520a0aeba5bc1cad02c9c43750e67d Nov 03 13:40:57.875752 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:57.875723 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:40:57.875824 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:57.875789 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:40:57.875824 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:57.875816 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:40:58.262497 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:58.261893 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6bd5c1-565d8c4b9-9shxg"] Nov 03 13:40:58.274426 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:58.274399 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6bd5c1-565d8c4b9-9shxg" Nov 03 13:40:58.318819 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:58.318785 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6bd5c1-565d8c4b9-9shxg"] Nov 03 13:40:58.323612 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:58.323582 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-6bd5c1-serving-cert\"" Nov 03 13:40:58.381545 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:58.379989 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5aa090e7-2473-4ecf-b8f9-d48182cefbd6-metrics-certs\") pod \"maas-api-e2e-ipp-b-6bd5c1-565d8c4b9-9shxg\" (UID: \"5aa090e7-2473-4ecf-b8f9-d48182cefbd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6bd5c1-565d8c4b9-9shxg" Nov 03 13:40:58.381545 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:58.380118 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5aa090e7-2473-4ecf-b8f9-d48182cefbd6-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6bd5c1-565d8c4b9-9shxg\" (UID: \"5aa090e7-2473-4ecf-b8f9-d48182cefbd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6bd5c1-565d8c4b9-9shxg" Nov 03 13:40:58.381545 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:58.380417 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqmnj\" (UniqueName: \"kubernetes.io/projected/5aa090e7-2473-4ecf-b8f9-d48182cefbd6-kube-api-access-hqmnj\") pod \"maas-api-e2e-ipp-b-6bd5c1-565d8c4b9-9shxg\" (UID: \"5aa090e7-2473-4ecf-b8f9-d48182cefbd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6bd5c1-565d8c4b9-9shxg" Nov 03 13:40:58.381545 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:58.380499 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5aa090e7-2473-4ecf-b8f9-d48182cefbd6-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-6bd5c1-565d8c4b9-9shxg\" (UID: \"5aa090e7-2473-4ecf-b8f9-d48182cefbd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6bd5c1-565d8c4b9-9shxg" Nov 03 13:40:58.381545 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:58.380573 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq" event={"ID":"763f2c9e-811d-49dc-b971-4782cb8a0964","Type":"ContainerStarted","Data":"f23261b12105120dd094972ca2c57abe63e04748f496e89d9653d7141b4088f1"} Nov 03 13:40:58.381545 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:58.380621 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq" event={"ID":"763f2c9e-811d-49dc-b971-4782cb8a0964","Type":"ContainerStarted","Data":"50a9d7ca7bdab0665b6a3ff0dd12689659520a0aeba5bc1cad02c9c43750e67d"} Nov 03 13:40:58.392612 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:58.392583 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq" Nov 03 13:40:58.394044 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:58.394019 5979 patch_prober.go:29] interesting pod/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body= Nov 03 13:40:58.394163 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:58.394068 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq" podUID="763f2c9e-811d-49dc-b971-4782cb8a0964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" Nov 03 13:40:58.481431 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:58.481391 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqmnj\" (UniqueName: \"kubernetes.io/projected/5aa090e7-2473-4ecf-b8f9-d48182cefbd6-kube-api-access-hqmnj\") pod \"maas-api-e2e-ipp-b-6bd5c1-565d8c4b9-9shxg\" (UID: \"5aa090e7-2473-4ecf-b8f9-d48182cefbd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6bd5c1-565d8c4b9-9shxg" Nov 03 13:40:58.481659 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:58.481529 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5aa090e7-2473-4ecf-b8f9-d48182cefbd6-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-6bd5c1-565d8c4b9-9shxg\" (UID: \"5aa090e7-2473-4ecf-b8f9-d48182cefbd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6bd5c1-565d8c4b9-9shxg" Nov 03 13:40:58.481659 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:58.481624 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5aa090e7-2473-4ecf-b8f9-d48182cefbd6-metrics-certs\") pod \"maas-api-e2e-ipp-b-6bd5c1-565d8c4b9-9shxg\" (UID: \"5aa090e7-2473-4ecf-b8f9-d48182cefbd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6bd5c1-565d8c4b9-9shxg" Nov 03 13:40:58.481800 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:58.481690 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5aa090e7-2473-4ecf-b8f9-d48182cefbd6-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6bd5c1-565d8c4b9-9shxg\" (UID: \"5aa090e7-2473-4ecf-b8f9-d48182cefbd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6bd5c1-565d8c4b9-9shxg" Nov 03 13:40:58.482135 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:40:58.482111 5979 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-6bd5c1-serving-cert: secret "maas-api-e2e-ipp-b-6bd5c1-serving-cert" not found Nov 03 13:40:58.482231 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:40:58.482185 5979 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5aa090e7-2473-4ecf-b8f9-d48182cefbd6-maas-api-tls podName:5aa090e7-2473-4ecf-b8f9-d48182cefbd6 nodeName:}" failed. No retries permitted until 2026-11-03 13:40:58.982164221 +0000 UTC m=+2140.992852446 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5aa090e7-2473-4ecf-b8f9-d48182cefbd6-maas-api-tls") pod "maas-api-e2e-ipp-b-6bd5c1-565d8c4b9-9shxg" (UID: "5aa090e7-2473-4ecf-b8f9-d48182cefbd6") : secret "maas-api-e2e-ipp-b-6bd5c1-serving-cert" not found Nov 03 13:40:58.482299 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:58.482247 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5aa090e7-2473-4ecf-b8f9-d48182cefbd6-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-6bd5c1-565d8c4b9-9shxg\" (UID: \"5aa090e7-2473-4ecf-b8f9-d48182cefbd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6bd5c1-565d8c4b9-9shxg" Nov 03 13:40:58.484439 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:58.484418 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5aa090e7-2473-4ecf-b8f9-d48182cefbd6-metrics-certs\") pod \"maas-api-e2e-ipp-b-6bd5c1-565d8c4b9-9shxg\" (UID: \"5aa090e7-2473-4ecf-b8f9-d48182cefbd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6bd5c1-565d8c4b9-9shxg" Nov 03 13:40:58.498330 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:58.498283 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq" podStartSLOduration=1.498271822 podStartE2EDuration="1.498271822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:40:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:40:58.495675769 +0000 UTC m=+2140.506363993" watchObservedRunningTime="2026-11-03 13:40:58.498271822 +0000 UTC m=+2140.508960064" Nov 03 13:40:58.530998 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:58.530924 5979 patch_prober.go:29] interesting pod/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Nov 03 13:40:58.530998 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:58.530980 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686" podUID="87552227-fce6-474b-9b1b-fe167d1f6547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Nov 03 13:40:58.564525 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:58.543811 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqmnj\" (UniqueName: \"kubernetes.io/projected/5aa090e7-2473-4ecf-b8f9-d48182cefbd6-kube-api-access-hqmnj\") pod \"maas-api-e2e-ipp-b-6bd5c1-565d8c4b9-9shxg\" (UID: \"5aa090e7-2473-4ecf-b8f9-d48182cefbd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6bd5c1-565d8c4b9-9shxg" Nov 03 13:40:58.997840 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:58.997804 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5aa090e7-2473-4ecf-b8f9-d48182cefbd6-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6bd5c1-565d8c4b9-9shxg\" (UID: \"5aa090e7-2473-4ecf-b8f9-d48182cefbd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6bd5c1-565d8c4b9-9shxg" Nov 03 13:40:59.000362 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:59.000337 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5aa090e7-2473-4ecf-b8f9-d48182cefbd6-maas-api-tls\") pod \"maas-api-e2e-ipp-b-6bd5c1-565d8c4b9-9shxg\" (UID: \"5aa090e7-2473-4ecf-b8f9-d48182cefbd6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6bd5c1-565d8c4b9-9shxg" Nov 03 13:40:59.192968 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:59.192932 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6bd5c1-565d8c4b9-9shxg" Nov 03 13:40:59.396881 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:59.396851 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq" Nov 03 13:40:59.532992 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:59.532859 5979 patch_prober.go:29] interesting pod/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Nov 03 13:40:59.532992 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:59.532923 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686" podUID="87552227-fce6-474b-9b1b-fe167d1f6547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Nov 03 13:40:59.670085 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:40:59.670041 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6bd5c1-565d8c4b9-9shxg"] Nov 03 13:40:59.674751 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:40:59.674724 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aa090e7_2473_4ecf_b8f9_d48182cefbd6.slice/crio-49e831f76c3247f5a3720f49480d7e43d73e67c00da33f5be66aa48b4e8cc9a1 WatchSource:0}: Error finding container 49e831f76c3247f5a3720f49480d7e43d73e67c00da33f5be66aa48b4e8cc9a1: Status 404 returned error can't find the container with id 49e831f76c3247f5a3720f49480d7e43d73e67c00da33f5be66aa48b4e8cc9a1 Nov 03 13:41:00.069876 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:00.069842 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf" Nov 03 13:41:00.110644 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:00.110557 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-workload-certs\") pod \"efd2a382-b6ea-419f-b33a-9ebfdf0bd61b\" (UID: \"efd2a382-b6ea-419f-b33a-9ebfdf0bd61b\") " Nov 03 13:41:00.110803 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:00.110643 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-istiod-ca-cert\") pod \"efd2a382-b6ea-419f-b33a-9ebfdf0bd61b\" (UID: \"efd2a382-b6ea-419f-b33a-9ebfdf0bd61b\") " Nov 03 13:41:00.110803 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:00.110670 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-kube-api-access-8gcmp\" (UniqueName: \"kubernetes.io/projected/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-kube-api-access-8gcmp\") pod \"efd2a382-b6ea-419f-b33a-9ebfdf0bd61b\" (UID: \"efd2a382-b6ea-419f-b33a-9ebfdf0bd61b\") " Nov 03 13:41:00.110803 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:00.110694 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-istio-podinfo\") pod \"efd2a382-b6ea-419f-b33a-9ebfdf0bd61b\" (UID: \"efd2a382-b6ea-419f-b33a-9ebfdf0bd61b\") " Nov 03 13:41:00.110959 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:00.110808 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-istio-envoy\") pod \"efd2a382-b6ea-419f-b33a-9ebfdf0bd61b\" (UID: \"efd2a382-b6ea-419f-b33a-9ebfdf0bd61b\") " Nov 03 13:41:00.110959 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:00.110860 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-workload-socket\") pod \"efd2a382-b6ea-419f-b33a-9ebfdf0bd61b\" (UID: \"efd2a382-b6ea-419f-b33a-9ebfdf0bd61b\") " Nov 03 13:41:00.110959 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:00.110873 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-workload-certs" pod "efd2a382-b6ea-419f-b33a-9ebfdf0bd61b" (UID: "efd2a382-b6ea-419f-b33a-9ebfdf0bd61b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:41:00.110959 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:00.110894 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-istio-data\") pod \"efd2a382-b6ea-419f-b33a-9ebfdf0bd61b\" (UID: \"efd2a382-b6ea-419f-b33a-9ebfdf0bd61b\") " Nov 03 13:41:00.110959 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:00.110935 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-credential-socket\") pod \"efd2a382-b6ea-419f-b33a-9ebfdf0bd61b\" (UID: \"efd2a382-b6ea-419f-b33a-9ebfdf0bd61b\") " Nov 03 13:41:00.111237 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:00.110997 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-istiod-ca-cert" pod "efd2a382-b6ea-419f-b33a-9ebfdf0bd61b" (UID: "efd2a382-b6ea-419f-b33a-9ebfdf0bd61b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:41:00.111237 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:00.111016 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-istio-token\" (UniqueName: \"kubernetes.io/projected/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-istio-token\") pod \"efd2a382-b6ea-419f-b33a-9ebfdf0bd61b\" (UID: \"efd2a382-b6ea-419f-b33a-9ebfdf0bd61b\") " Nov 03 13:41:00.111237 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:00.111189 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-workload-socket" pod "efd2a382-b6ea-419f-b33a-9ebfdf0bd61b" (UID: "efd2a382-b6ea-419f-b33a-9ebfdf0bd61b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:41:00.111417 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:00.111307 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-credential-socket" pod "efd2a382-b6ea-419f-b33a-9ebfdf0bd61b" (UID: "efd2a382-b6ea-419f-b33a-9ebfdf0bd61b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:41:00.111417 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:00.111323 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-istio-data" pod "efd2a382-b6ea-419f-b33a-9ebfdf0bd61b" (UID: "efd2a382-b6ea-419f-b33a-9ebfdf0bd61b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:41:00.111666 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:00.111644 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:00.111816 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:00.111668 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:00.111816 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:00.111685 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:00.111816 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:00.111698 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:00.111816 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:00.111711 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:00.113277 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:00.113250 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-istio-podinfo" pod "efd2a382-b6ea-419f-b33a-9ebfdf0bd61b" (UID: "efd2a382-b6ea-419f-b33a-9ebfdf0bd61b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:41:00.113381 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:00.113283 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-kube-api-access-8gcmp" pod "efd2a382-b6ea-419f-b33a-9ebfdf0bd61b" (UID: "efd2a382-b6ea-419f-b33a-9ebfdf0bd61b"). InnerVolumeSpecName "kube-api-access-8gcmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:41:00.113381 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:00.113304 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-istio-envoy" pod "efd2a382-b6ea-419f-b33a-9ebfdf0bd61b" (UID: "efd2a382-b6ea-419f-b33a-9ebfdf0bd61b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:41:00.113381 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:00.113335 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-istio-token" pod "efd2a382-b6ea-419f-b33a-9ebfdf0bd61b" (UID: "efd2a382-b6ea-419f-b33a-9ebfdf0bd61b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:41:00.213853 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:00.213591 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gcmp\" (UniqueName: \"kubernetes.io/projected/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-kube-api-access-8gcmp\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:00.213853 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:00.213762 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:00.213853 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:00.213789 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:00.213853 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:00.213817 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:41:00.395139 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:00.395100 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6bd5c1-565d8c4b9-9shxg" event={"ID":"5aa090e7-2473-4ecf-b8f9-d48182cefbd6","Type":"ContainerStarted","Data":"49e831f76c3247f5a3720f49480d7e43d73e67c00da33f5be66aa48b4e8cc9a1"} Nov 03 13:41:00.396786 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:00.396760 5979 generic.go:356] "Generic (PLEG): container finished" podID="efd2a382-b6ea-419f-b33a-9ebfdf0bd61b" containerID="89dc0798d357859d7d39d3dcae3278d83018164920f3d2ca4344f29b3fcb459d" exitCode=0 Nov 03 13:41:00.396903 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:00.396816 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf" Nov 03 13:41:00.396903 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:00.396841 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf" event={"ID":"efd2a382-b6ea-419f-b33a-9ebfdf0bd61b","Type":"ContainerDied","Data":"89dc0798d357859d7d39d3dcae3278d83018164920f3d2ca4344f29b3fcb459d"} Nov 03 13:41:00.396903 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:00.396892 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf" event={"ID":"efd2a382-b6ea-419f-b33a-9ebfdf0bd61b","Type":"ContainerDied","Data":"94b0e2cc756552a6364b0ab46f6e154f4ff635fffd5ee13bb81d9aa71bf66a0d"} Nov 03 13:41:00.397311 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:00.396924 5979 scope.go:122] "RemoveContainer" containerID="89dc0798d357859d7d39d3dcae3278d83018164920f3d2ca4344f29b3fcb459d" Nov 03 13:41:00.397311 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:00.397286 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq" Nov 03 13:41:00.398330 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:00.398315 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq" Nov 03 13:41:00.406683 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:00.406645 5979 scope.go:122] "RemoveContainer" containerID="d714840db823ea2dea8bdda3ef53b5e66a20c5bc6204dc3ca8b5e2bb2a4b2125" Nov 03 13:41:00.415783 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:00.415767 5979 scope.go:122] "RemoveContainer" containerID="89dc0798d357859d7d39d3dcae3278d83018164920f3d2ca4344f29b3fcb459d" Nov 03 13:41:00.416064 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:41:00.416047 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89dc0798d357859d7d39d3dcae3278d83018164920f3d2ca4344f29b3fcb459d\": container with ID starting with 89dc0798d357859d7d39d3dcae3278d83018164920f3d2ca4344f29b3fcb459d not found: ID does not exist" containerID="89dc0798d357859d7d39d3dcae3278d83018164920f3d2ca4344f29b3fcb459d" Nov 03 13:41:00.416128 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:00.416070 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89dc0798d357859d7d39d3dcae3278d83018164920f3d2ca4344f29b3fcb459d"} err="failed to get container status \"89dc0798d357859d7d39d3dcae3278d83018164920f3d2ca4344f29b3fcb459d\": rpc error: code = NotFound desc = could not find container \"89dc0798d357859d7d39d3dcae3278d83018164920f3d2ca4344f29b3fcb459d\": container with ID starting with 89dc0798d357859d7d39d3dcae3278d83018164920f3d2ca4344f29b3fcb459d not found: ID does not exist" Nov 03 13:41:00.416128 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:00.416092 5979 scope.go:122] "RemoveContainer" containerID="d714840db823ea2dea8bdda3ef53b5e66a20c5bc6204dc3ca8b5e2bb2a4b2125" Nov 03 13:41:00.416341 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:41:00.416324 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d714840db823ea2dea8bdda3ef53b5e66a20c5bc6204dc3ca8b5e2bb2a4b2125\": container with ID starting with d714840db823ea2dea8bdda3ef53b5e66a20c5bc6204dc3ca8b5e2bb2a4b2125 not found: ID does not exist" containerID="d714840db823ea2dea8bdda3ef53b5e66a20c5bc6204dc3ca8b5e2bb2a4b2125" Nov 03 13:41:00.416383 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:00.416348 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d714840db823ea2dea8bdda3ef53b5e66a20c5bc6204dc3ca8b5e2bb2a4b2125"} err="failed to get container status \"d714840db823ea2dea8bdda3ef53b5e66a20c5bc6204dc3ca8b5e2bb2a4b2125\": rpc error: code = NotFound desc = could not find container \"d714840db823ea2dea8bdda3ef53b5e66a20c5bc6204dc3ca8b5e2bb2a4b2125\": container with ID starting with d714840db823ea2dea8bdda3ef53b5e66a20c5bc6204dc3ca8b5e2bb2a4b2125 not found: ID does not exist" Nov 03 13:41:00.531384 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:00.531348 5979 patch_prober.go:29] interesting pod/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body= Nov 03 13:41:00.531574 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:00.531430 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686" podUID="87552227-fce6-474b-9b1b-fe167d1f6547" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" Nov 03 13:41:00.575268 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:00.573534 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf"] Nov 03 13:41:00.576691 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:00.576666 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-5474b7-openshift-default-66cc6664c9-c6cxf"] Nov 03 13:41:01.406077 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:01.406039 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6bd5c1-565d8c4b9-9shxg" event={"ID":"5aa090e7-2473-4ecf-b8f9-d48182cefbd6","Type":"ContainerStarted","Data":"dcc5f881440c295b18c0d52ff88aa73887e90de29ee1509c7cda4e4b1a9a91db"} Nov 03 13:41:01.406568 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:01.406530 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6bd5c1-565d8c4b9-9shxg" Nov 03 13:41:01.519037 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:01.518973 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6bd5c1-565d8c4b9-9shxg" podStartSLOduration=2.594778308 podStartE2EDuration="3.51895854s" totalImagesPullingTime="924.180232ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:40:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:41:01.515936074 +0000 UTC m=+2143.526624316" watchObservedRunningTime="2026-11-03 13:41:01.51895854 +0000 UTC m=+2143.529646829" Nov 03 13:41:01.544556 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:01.544518 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686" Nov 03 13:41:01.547402 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:01.545879 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686" Nov 03 13:41:01.547402 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:01.547258 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686" Nov 03 13:41:02.117100 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:02.117069 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efd2a382-b6ea-419f-b33a-9ebfdf0bd61b" path="/var/lib/kubelet/pods/efd2a382-b6ea-419f-b33a-9ebfdf0bd61b/volumes" Nov 03 13:41:02.828734 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:02.828682 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:41:05.212200 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:05.212159 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4b9658-7c5887bdf9-8nldv"] Nov 03 13:41:05.212833 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:05.212818 5979 cpu_manager.go:414] "containerMap: removing container" podUID="efd2a382-b6ea-419f-b33a-9ebfdf0bd61b" containerName="istio-proxy" Nov 03 13:41:05.212898 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:05.212833 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efd2a382-b6ea-419f-b33a-9ebfdf0bd61b" containerName="istio-proxy" Nov 03 13:41:05.212898 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:05.212851 5979 cpu_manager.go:414] "containerMap: removing container" podUID="efd2a382-b6ea-419f-b33a-9ebfdf0bd61b" containerName="istio-proxy" Nov 03 13:41:05.212898 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:05.212858 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efd2a382-b6ea-419f-b33a-9ebfdf0bd61b" containerName="istio-proxy" Nov 03 13:41:05.213004 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:05.212989 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="efd2a382-b6ea-419f-b33a-9ebfdf0bd61b" containerName="istio-proxy" Nov 03 13:41:05.213040 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:05.213003 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="efd2a382-b6ea-419f-b33a-9ebfdf0bd61b" containerName="istio-proxy" Nov 03 13:41:05.217300 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:05.217274 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4b9658-7c5887bdf9-8nldv" Nov 03 13:41:05.282111 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:05.282068 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5prg\" (UniqueName: \"kubernetes.io/projected/76ffa3a3-1a28-4468-aff0-b1e96f33d3e8-kube-api-access-h5prg\") pod \"payload-pre-processing-e2e-praxis-a-4b9658-7c5887bdf9-8nldv\" (UID: \"76ffa3a3-1a28-4468-aff0-b1e96f33d3e8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4b9658-7c5887bdf9-8nldv" Nov 03 13:41:05.282567 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:05.282532 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76ffa3a3-1a28-4468-aff0-b1e96f33d3e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-4b9658-7c5887bdf9-8nldv\" (UID: \"76ffa3a3-1a28-4468-aff0-b1e96f33d3e8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4b9658-7c5887bdf9-8nldv" Nov 03 13:41:05.320858 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:05.320746 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-4b9658\"" Nov 03 13:41:05.320858 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:05.320831 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-4b9658-dockercfg-9ksw8\"" Nov 03 13:41:05.325955 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:05.325926 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4b9658-7c5887bdf9-8nldv"] Nov 03 13:41:05.386512 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:05.386455 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5prg\" (UniqueName: \"kubernetes.io/projected/76ffa3a3-1a28-4468-aff0-b1e96f33d3e8-kube-api-access-h5prg\") pod \"payload-pre-processing-e2e-praxis-a-4b9658-7c5887bdf9-8nldv\" (UID: \"76ffa3a3-1a28-4468-aff0-b1e96f33d3e8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4b9658-7c5887bdf9-8nldv" Nov 03 13:41:05.386891 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:05.386861 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76ffa3a3-1a28-4468-aff0-b1e96f33d3e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-4b9658-7c5887bdf9-8nldv\" (UID: \"76ffa3a3-1a28-4468-aff0-b1e96f33d3e8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4b9658-7c5887bdf9-8nldv" Nov 03 13:41:05.387928 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:05.387908 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76ffa3a3-1a28-4468-aff0-b1e96f33d3e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-4b9658-7c5887bdf9-8nldv\" (UID: \"76ffa3a3-1a28-4468-aff0-b1e96f33d3e8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4b9658-7c5887bdf9-8nldv" Nov 03 13:41:05.388909 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:05.388889 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4b9658-999455b7f-l6wwv"] Nov 03 13:41:05.393365 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:05.393352 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-4b9658-999455b7f-l6wwv" Nov 03 13:41:05.430012 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:05.429981 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4b9658-999455b7f-l6wwv"] Nov 03 13:41:05.476839 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:05.476775 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5prg\" (UniqueName: \"kubernetes.io/projected/76ffa3a3-1a28-4468-aff0-b1e96f33d3e8-kube-api-access-h5prg\") pod \"payload-pre-processing-e2e-praxis-a-4b9658-7c5887bdf9-8nldv\" (UID: \"76ffa3a3-1a28-4468-aff0-b1e96f33d3e8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4b9658-7c5887bdf9-8nldv" Nov 03 13:41:05.531245 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:05.531216 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4b9658-7c5887bdf9-8nldv" Nov 03 13:41:05.589221 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:05.589185 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvh5m\" (UniqueName: \"kubernetes.io/projected/5f2c1f4c-f81b-46fc-9f29-22ad372565c9-kube-api-access-wvh5m\") pod \"payload-processing-e2e-praxis-a-4b9658-999455b7f-l6wwv\" (UID: \"5f2c1f4c-f81b-46fc-9f29-22ad372565c9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4b9658-999455b7f-l6wwv" Nov 03 13:41:05.589392 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:05.589323 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f2c1f4c-f81b-46fc-9f29-22ad372565c9-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-4b9658-999455b7f-l6wwv\" (UID: \"5f2c1f4c-f81b-46fc-9f29-22ad372565c9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4b9658-999455b7f-l6wwv" Nov 03 13:41:05.690234 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:05.690184 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvh5m\" (UniqueName: \"kubernetes.io/projected/5f2c1f4c-f81b-46fc-9f29-22ad372565c9-kube-api-access-wvh5m\") pod \"payload-processing-e2e-praxis-a-4b9658-999455b7f-l6wwv\" (UID: \"5f2c1f4c-f81b-46fc-9f29-22ad372565c9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4b9658-999455b7f-l6wwv" Nov 03 13:41:05.690415 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:05.690298 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f2c1f4c-f81b-46fc-9f29-22ad372565c9-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-4b9658-999455b7f-l6wwv\" (UID: \"5f2c1f4c-f81b-46fc-9f29-22ad372565c9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4b9658-999455b7f-l6wwv" Nov 03 13:41:05.691116 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:05.691083 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f2c1f4c-f81b-46fc-9f29-22ad372565c9-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-4b9658-999455b7f-l6wwv\" (UID: \"5f2c1f4c-f81b-46fc-9f29-22ad372565c9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4b9658-999455b7f-l6wwv" Nov 03 13:41:05.769434 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:05.769262 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvh5m\" (UniqueName: \"kubernetes.io/projected/5f2c1f4c-f81b-46fc-9f29-22ad372565c9-kube-api-access-wvh5m\") pod \"payload-processing-e2e-praxis-a-4b9658-999455b7f-l6wwv\" (UID: \"5f2c1f4c-f81b-46fc-9f29-22ad372565c9\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-4b9658-999455b7f-l6wwv" Nov 03 13:41:05.936694 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:05.936661 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4b9658-7c5887bdf9-8nldv"] Nov 03 13:41:06.010329 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:06.010302 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-4b9658-999455b7f-l6wwv" Nov 03 13:41:06.406215 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:06.406181 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4b9658-999455b7f-l6wwv"] Nov 03 13:41:06.470539 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:06.470503 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-4b9658-999455b7f-l6wwv" event={"ID":"5f2c1f4c-f81b-46fc-9f29-22ad372565c9","Type":"ContainerStarted","Data":"70f2d5c170665fa7027137ec6b9c719a5544dce5c9095a0f10e969101ba25d87"} Nov 03 13:41:06.472381 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:06.472355 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4b9658-7c5887bdf9-8nldv" event={"ID":"76ffa3a3-1a28-4468-aff0-b1e96f33d3e8","Type":"ContainerStarted","Data":"6252df7f84d7b9d27a8b06e80c4cff2f5eeec24816df2c3ea9fd4c1df7a8c1db"} Nov 03 13:41:06.472535 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:06.472403 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4b9658-7c5887bdf9-8nldv" event={"ID":"76ffa3a3-1a28-4468-aff0-b1e96f33d3e8","Type":"ContainerStarted","Data":"91f454194e3124f00d2c2cfccd6b39065c0ba9b512071a51a391bdd3c07180e5"} Nov 03 13:41:06.472535 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:06.472436 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4b9658-7c5887bdf9-8nldv" Nov 03 13:41:06.581419 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:06.581360 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4b9658-7c5887bdf9-8nldv" podStartSLOduration=1.581345628 podStartE2EDuration="1.581345628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:41:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:41:06.579846935 +0000 UTC m=+2148.590535177" watchObservedRunningTime="2026-11-03 13:41:06.581345628 +0000 UTC m=+2148.592033870" Nov 03 13:41:07.233968 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:41:07.233931 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:41:07.426055 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:07.426025 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6bd5c1-565d8c4b9-9shxg" Nov 03 13:41:07.482019 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:07.481977 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-4b9658-999455b7f-l6wwv" event={"ID":"5f2c1f4c-f81b-46fc-9f29-22ad372565c9","Type":"ContainerStarted","Data":"53c5ae4fc25865cec1716c0e7d3504fc7e1482a2b300f53e637bb50ee804d21c"} Nov 03 13:41:07.482172 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:07.482072 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-4b9658-999455b7f-l6wwv" Nov 03 13:41:07.673931 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:07.673854 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-4b9658-999455b7f-l6wwv" podStartSLOduration=2.673833883 podStartE2EDuration="2.673833883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:41:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:41:07.64323721 +0000 UTC m=+2149.653925452" watchObservedRunningTime="2026-11-03 13:41:07.673833883 +0000 UTC m=+2149.684522127" Nov 03 13:41:13.523664 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:13.523632 5979 generic.go:356] "Generic (PLEG): container finished" podID="f1705a6f-0727-437b-af88-8fb5448050ac" containerID="62d75487cbfbe1dd189530177ccc545a6ecbdeebe59d06f61632f1d1eee49d48" exitCode=0 Nov 03 13:41:13.524119 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:13.523701 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x" event={"ID":"f1705a6f-0727-437b-af88-8fb5448050ac","Type":"ContainerDied","Data":"62d75487cbfbe1dd189530177ccc545a6ecbdeebe59d06f61632f1d1eee49d48"} Nov 03 13:41:13.524186 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:13.524162 5979 scope.go:122] "RemoveContainer" containerID="62d75487cbfbe1dd189530177ccc545a6ecbdeebe59d06f61632f1d1eee49d48" Nov 03 13:41:13.525830 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:13.525803 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:41:13.525905 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:13.525879 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:41:13.525944 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:13.525910 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:41:14.534557 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:14.534517 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x" event={"ID":"f1705a6f-0727-437b-af88-8fb5448050ac","Type":"ContainerStarted","Data":"bb8d311f146f5d6d55d0016dd801c0dd1de9f9dfb9ee79fb9a49906dea9f64ab"} Nov 03 13:41:15.412379 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:15.412347 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x" Nov 03 13:41:15.412609 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:15.412577 5979 patch_prober.go:29] interesting pod/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Nov 03 13:41:15.412693 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:15.412618 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x" podUID="f1705a6f-0727-437b-af88-8fb5448050ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Nov 03 13:41:16.412824 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:16.412782 5979 patch_prober.go:29] interesting pod/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Nov 03 13:41:16.413217 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:16.412842 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x" podUID="f1705a6f-0727-437b-af88-8fb5448050ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Nov 03 13:41:17.412646 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:17.412603 5979 patch_prober.go:29] interesting pod/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Nov 03 13:41:17.412841 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:17.412677 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x" podUID="f1705a6f-0727-437b-af88-8fb5448050ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Nov 03 13:41:17.485845 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:17.485815 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4b9658-7c5887bdf9-8nldv" Nov 03 13:41:18.412089 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:18.412059 5979 patch_prober.go:29] interesting pod/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Nov 03 13:41:18.412288 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:18.412106 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x" podUID="f1705a6f-0727-437b-af88-8fb5448050ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Nov 03 13:41:18.500117 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:18.497808 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-4b9658-999455b7f-l6wwv" Nov 03 13:41:19.414391 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:19.412328 5979 patch_prober.go:29] interesting pod/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Nov 03 13:41:19.414391 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:19.412464 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x" podUID="f1705a6f-0727-437b-af88-8fb5448050ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Nov 03 13:41:19.675031 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:19.674926 5979 scope.go:122] "RemoveContainer" containerID="eefcfb9dc482406a86b0d8dbf0a1e90c62d8a2d9ed10e71af9c1bf1db3c11862" Nov 03 13:41:19.688001 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:19.687977 5979 scope.go:122] "RemoveContainer" containerID="7b23dfb888b62aa3a607b8e1e5fe8772593cdefe4360eb3876883a4c8ffab79f" Nov 03 13:41:20.416313 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:20.416283 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x" Nov 03 13:41:20.416743 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:20.416726 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x" Nov 03 13:41:20.417309 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:20.417293 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x" Nov 03 13:41:32.001711 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:32.001674 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:41:37.235509 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:41:37.235444 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:41:50.679715 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:50.679680 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq"] Nov 03 13:41:50.683329 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:50.683307 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq" Nov 03 13:41:50.743687 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:50.743641 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-84e22c-openshift-default-dockercfg-t62pr\"" Nov 03 13:41:50.804266 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:50.798075 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a8276f7-206c-402e-8e82-087316e9fc3d-istio-data\") pod \"e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq\" (UID: \"3a8276f7-206c-402e-8e82-087316e9fc3d\") " pod="openshift-ingress/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq" Nov 03 13:41:50.804266 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:50.798145 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a8276f7-206c-402e-8e82-087316e9fc3d-credential-socket\") pod \"e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq\" (UID: \"3a8276f7-206c-402e-8e82-087316e9fc3d\") " pod="openshift-ingress/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq" Nov 03 13:41:50.804266 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:50.798278 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a8276f7-206c-402e-8e82-087316e9fc3d-istio-podinfo\") pod \"e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq\" (UID: \"3a8276f7-206c-402e-8e82-087316e9fc3d\") " pod="openshift-ingress/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq" Nov 03 13:41:50.804266 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:50.798328 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a8276f7-206c-402e-8e82-087316e9fc3d-istiod-ca-cert\") pod \"e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq\" (UID: \"3a8276f7-206c-402e-8e82-087316e9fc3d\") " pod="openshift-ingress/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq" Nov 03 13:41:50.804266 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:50.798339 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq"] Nov 03 13:41:50.804266 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:50.798383 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a8276f7-206c-402e-8e82-087316e9fc3d-workload-certs\") pod \"e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq\" (UID: \"3a8276f7-206c-402e-8e82-087316e9fc3d\") " pod="openshift-ingress/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq" Nov 03 13:41:50.804266 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:50.798493 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a8276f7-206c-402e-8e82-087316e9fc3d-istio-token\") pod \"e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq\" (UID: \"3a8276f7-206c-402e-8e82-087316e9fc3d\") " pod="openshift-ingress/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq" Nov 03 13:41:50.804266 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:50.798661 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a8276f7-206c-402e-8e82-087316e9fc3d-istio-envoy\") pod \"e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq\" (UID: \"3a8276f7-206c-402e-8e82-087316e9fc3d\") " pod="openshift-ingress/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq" Nov 03 13:41:50.804266 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:50.798698 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc6l5\" (UniqueName: \"kubernetes.io/projected/3a8276f7-206c-402e-8e82-087316e9fc3d-kube-api-access-bc6l5\") pod \"e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq\" (UID: \"3a8276f7-206c-402e-8e82-087316e9fc3d\") " pod="openshift-ingress/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq" Nov 03 13:41:50.804266 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:50.798803 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a8276f7-206c-402e-8e82-087316e9fc3d-workload-socket\") pod \"e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq\" (UID: \"3a8276f7-206c-402e-8e82-087316e9fc3d\") " pod="openshift-ingress/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq" Nov 03 13:41:50.899899 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:50.899855 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a8276f7-206c-402e-8e82-087316e9fc3d-workload-socket\") pod \"e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq\" (UID: \"3a8276f7-206c-402e-8e82-087316e9fc3d\") " pod="openshift-ingress/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq" Nov 03 13:41:50.900082 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:50.899909 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a8276f7-206c-402e-8e82-087316e9fc3d-istio-data\") pod \"e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq\" (UID: \"3a8276f7-206c-402e-8e82-087316e9fc3d\") " pod="openshift-ingress/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq" Nov 03 13:41:50.900082 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:50.899942 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a8276f7-206c-402e-8e82-087316e9fc3d-credential-socket\") pod \"e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq\" (UID: \"3a8276f7-206c-402e-8e82-087316e9fc3d\") " pod="openshift-ingress/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq" Nov 03 13:41:50.900082 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:50.900023 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a8276f7-206c-402e-8e82-087316e9fc3d-istio-podinfo\") pod \"e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq\" (UID: \"3a8276f7-206c-402e-8e82-087316e9fc3d\") " pod="openshift-ingress/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq" Nov 03 13:41:50.900082 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:50.900049 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a8276f7-206c-402e-8e82-087316e9fc3d-istiod-ca-cert\") pod \"e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq\" (UID: \"3a8276f7-206c-402e-8e82-087316e9fc3d\") " pod="openshift-ingress/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq" Nov 03 13:41:50.900323 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:50.900088 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a8276f7-206c-402e-8e82-087316e9fc3d-workload-certs\") pod \"e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq\" (UID: \"3a8276f7-206c-402e-8e82-087316e9fc3d\") " pod="openshift-ingress/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq" Nov 03 13:41:50.900323 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:50.900122 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a8276f7-206c-402e-8e82-087316e9fc3d-istio-token\") pod \"e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq\" (UID: \"3a8276f7-206c-402e-8e82-087316e9fc3d\") " pod="openshift-ingress/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq" Nov 03 13:41:50.900323 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:50.900160 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a8276f7-206c-402e-8e82-087316e9fc3d-istio-envoy\") pod \"e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq\" (UID: \"3a8276f7-206c-402e-8e82-087316e9fc3d\") " pod="openshift-ingress/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq" Nov 03 13:41:50.900323 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:50.900190 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc6l5\" (UniqueName: \"kubernetes.io/projected/3a8276f7-206c-402e-8e82-087316e9fc3d-kube-api-access-bc6l5\") pod \"e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq\" (UID: \"3a8276f7-206c-402e-8e82-087316e9fc3d\") " pod="openshift-ingress/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq" Nov 03 13:41:50.900323 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:50.900313 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a8276f7-206c-402e-8e82-087316e9fc3d-workload-socket\") pod \"e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq\" (UID: \"3a8276f7-206c-402e-8e82-087316e9fc3d\") " pod="openshift-ingress/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq" Nov 03 13:41:50.900626 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:50.900538 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a8276f7-206c-402e-8e82-087316e9fc3d-credential-socket\") pod \"e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq\" (UID: \"3a8276f7-206c-402e-8e82-087316e9fc3d\") " pod="openshift-ingress/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq" Nov 03 13:41:50.900626 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:50.900556 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a8276f7-206c-402e-8e82-087316e9fc3d-istio-data\") pod \"e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq\" (UID: \"3a8276f7-206c-402e-8e82-087316e9fc3d\") " pod="openshift-ingress/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq" Nov 03 13:41:50.900707 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:50.900627 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a8276f7-206c-402e-8e82-087316e9fc3d-workload-certs\") pod \"e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq\" (UID: \"3a8276f7-206c-402e-8e82-087316e9fc3d\") " pod="openshift-ingress/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq" Nov 03 13:41:50.900908 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:50.900890 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a8276f7-206c-402e-8e82-087316e9fc3d-istiod-ca-cert\") pod \"e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq\" (UID: \"3a8276f7-206c-402e-8e82-087316e9fc3d\") " pod="openshift-ingress/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq" Nov 03 13:41:50.902445 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:50.902422 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a8276f7-206c-402e-8e82-087316e9fc3d-istio-envoy\") pod \"e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq\" (UID: \"3a8276f7-206c-402e-8e82-087316e9fc3d\") " pod="openshift-ingress/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq" Nov 03 13:41:50.902723 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:50.902702 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a8276f7-206c-402e-8e82-087316e9fc3d-istio-podinfo\") pod \"e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq\" (UID: \"3a8276f7-206c-402e-8e82-087316e9fc3d\") " pod="openshift-ingress/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq" Nov 03 13:41:50.958033 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:50.957961 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc6l5\" (UniqueName: \"kubernetes.io/projected/3a8276f7-206c-402e-8e82-087316e9fc3d-kube-api-access-bc6l5\") pod \"e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq\" (UID: \"3a8276f7-206c-402e-8e82-087316e9fc3d\") " pod="openshift-ingress/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq" Nov 03 13:41:50.958360 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:50.958096 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a8276f7-206c-402e-8e82-087316e9fc3d-istio-token\") pod \"e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq\" (UID: \"3a8276f7-206c-402e-8e82-087316e9fc3d\") " pod="openshift-ingress/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq" Nov 03 13:41:50.997128 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:50.997095 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq" Nov 03 13:41:51.428084 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:51.428050 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq"] Nov 03 13:41:51.435250 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:41:51.435220 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a8276f7_206c_402e_8e82_087316e9fc3d.slice/crio-40a7520e1d800afe42be7a679b3a41979a7273ef053570f1c3e6dc11a003637f WatchSource:0}: Error finding container 40a7520e1d800afe42be7a679b3a41979a7273ef053570f1c3e6dc11a003637f: Status 404 returned error can't find the container with id 40a7520e1d800afe42be7a679b3a41979a7273ef053570f1c3e6dc11a003637f Nov 03 13:41:51.437958 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:51.437926 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:41:51.438048 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:51.437996 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:41:51.438048 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:51.438030 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:41:51.825805 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:51.825714 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq" event={"ID":"3a8276f7-206c-402e-8e82-087316e9fc3d","Type":"ContainerStarted","Data":"c4131785842b46468d82a45668859d90988a152807f235d3aafd576451608e2c"} Nov 03 13:41:51.825805 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:51.825767 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq" event={"ID":"3a8276f7-206c-402e-8e82-087316e9fc3d","Type":"ContainerStarted","Data":"40a7520e1d800afe42be7a679b3a41979a7273ef053570f1c3e6dc11a003637f"} Nov 03 13:41:51.937671 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:51.937613 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq" podStartSLOduration=1.937596803 podStartE2EDuration="1.937596803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:41:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:41:51.936696721 +0000 UTC m=+2193.947384963" watchObservedRunningTime="2026-11-03 13:41:51.937596803 +0000 UTC m=+2193.948285048" Nov 03 13:41:51.997993 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:51.997953 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq" Nov 03 13:41:53.003257 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:53.003231 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq" Nov 03 13:41:53.838257 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:53.838220 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq" Nov 03 13:41:53.839232 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:41:53.839211 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq" Nov 03 13:42:01.165356 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:01.165319 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-32bec3-654f99d4f9-m9m9c"] Nov 03 13:42:01.270336 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:01.270304 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-32bec3-6c4596d787-tsnlp"] Nov 03 13:42:06.165884 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:06.165835 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-32bec3-654f99d4f9-m9m9c" podUID="247afc70-fb0c-4092-91f0-ecd7d4f3a03c" containerName="payload-processing" containerID="cri-o://37637c2fe572152a87f0cd0af692b003aac20f9d6fdc80a53880371b2d84844a" gracePeriod=25 Nov 03 13:42:06.271132 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:06.271093 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-32bec3-6c4596d787-tsnlp" podUID="c4239490-ae74-411a-8b11-237b853b70e1" containerName="payload-pre-processing" containerID="cri-o://4c963100dec989f0edb9c8168f668efb32b66f9d6d0520657022897143f2b3be" gracePeriod=25 Nov 03 13:42:06.423737 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:06.423675 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-32bec3-654f99d4f9-m9m9c" Nov 03 13:42:06.539853 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:06.539458 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/247afc70-fb0c-4092-91f0-ecd7d4f3a03c-kube-api-access-nk97s\" (UniqueName: \"kubernetes.io/projected/247afc70-fb0c-4092-91f0-ecd7d4f3a03c-kube-api-access-nk97s\") pod \"247afc70-fb0c-4092-91f0-ecd7d4f3a03c\" (UID: \"247afc70-fb0c-4092-91f0-ecd7d4f3a03c\") " Nov 03 13:42:06.539853 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:06.539694 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/247afc70-fb0c-4092-91f0-ecd7d4f3a03c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/247afc70-fb0c-4092-91f0-ecd7d4f3a03c-plugins-config-volume\") pod \"247afc70-fb0c-4092-91f0-ecd7d4f3a03c\" (UID: \"247afc70-fb0c-4092-91f0-ecd7d4f3a03c\") " Nov 03 13:42:06.540303 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:06.540273 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/247afc70-fb0c-4092-91f0-ecd7d4f3a03c-plugins-config-volume" pod "247afc70-fb0c-4092-91f0-ecd7d4f3a03c" (UID: "247afc70-fb0c-4092-91f0-ecd7d4f3a03c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:42:06.542293 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:06.542269 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/247afc70-fb0c-4092-91f0-ecd7d4f3a03c-kube-api-access-nk97s" pod "247afc70-fb0c-4092-91f0-ecd7d4f3a03c" (UID: "247afc70-fb0c-4092-91f0-ecd7d4f3a03c"). InnerVolumeSpecName "kube-api-access-nk97s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:42:06.552768 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:06.552745 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-32bec3-6c4596d787-tsnlp" Nov 03 13:42:06.640878 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:06.640844 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4239490-ae74-411a-8b11-237b853b70e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4239490-ae74-411a-8b11-237b853b70e1-plugins-config-volume\") pod \"c4239490-ae74-411a-8b11-237b853b70e1\" (UID: \"c4239490-ae74-411a-8b11-237b853b70e1\") " Nov 03 13:42:06.641056 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:06.640979 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4239490-ae74-411a-8b11-237b853b70e1-kube-api-access-6hgrr\" (UniqueName: \"kubernetes.io/projected/c4239490-ae74-411a-8b11-237b853b70e1-kube-api-access-6hgrr\") pod \"c4239490-ae74-411a-8b11-237b853b70e1\" (UID: \"c4239490-ae74-411a-8b11-237b853b70e1\") " Nov 03 13:42:06.641234 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:06.641212 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4239490-ae74-411a-8b11-237b853b70e1-plugins-config-volume" pod "c4239490-ae74-411a-8b11-237b853b70e1" (UID: "c4239490-ae74-411a-8b11-237b853b70e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:42:06.641291 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:06.641279 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/247afc70-fb0c-4092-91f0-ecd7d4f3a03c-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:06.641331 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:06.641294 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk97s\" (UniqueName: \"kubernetes.io/projected/247afc70-fb0c-4092-91f0-ecd7d4f3a03c-kube-api-access-nk97s\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:06.642996 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:06.642978 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4239490-ae74-411a-8b11-237b853b70e1-kube-api-access-6hgrr" pod "c4239490-ae74-411a-8b11-237b853b70e1" (UID: "c4239490-ae74-411a-8b11-237b853b70e1"). InnerVolumeSpecName "kube-api-access-6hgrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:42:06.743134 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:06.743045 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4239490-ae74-411a-8b11-237b853b70e1-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:06.743134 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:06.743079 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hgrr\" (UniqueName: \"kubernetes.io/projected/c4239490-ae74-411a-8b11-237b853b70e1-kube-api-access-6hgrr\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:06.926383 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:06.926350 5979 generic.go:356] "Generic (PLEG): container finished" podID="247afc70-fb0c-4092-91f0-ecd7d4f3a03c" containerID="37637c2fe572152a87f0cd0af692b003aac20f9d6fdc80a53880371b2d84844a" exitCode=0 Nov 03 13:42:06.926614 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:06.926393 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-32bec3-654f99d4f9-m9m9c" Nov 03 13:42:06.926614 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:06.926445 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-32bec3-654f99d4f9-m9m9c" event={"ID":"247afc70-fb0c-4092-91f0-ecd7d4f3a03c","Type":"ContainerDied","Data":"37637c2fe572152a87f0cd0af692b003aac20f9d6fdc80a53880371b2d84844a"} Nov 03 13:42:06.926614 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:06.926511 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-32bec3-654f99d4f9-m9m9c" event={"ID":"247afc70-fb0c-4092-91f0-ecd7d4f3a03c","Type":"ContainerDied","Data":"7ba275a5547112c178667234402c498da462af94e6e4824af08a569cde8414fa"} Nov 03 13:42:06.926614 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:06.926535 5979 scope.go:122] "RemoveContainer" containerID="37637c2fe572152a87f0cd0af692b003aac20f9d6fdc80a53880371b2d84844a" Nov 03 13:42:06.927939 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:06.927911 5979 generic.go:356] "Generic (PLEG): container finished" podID="c4239490-ae74-411a-8b11-237b853b70e1" containerID="4c963100dec989f0edb9c8168f668efb32b66f9d6d0520657022897143f2b3be" exitCode=0 Nov 03 13:42:06.928052 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:06.927954 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-32bec3-6c4596d787-tsnlp" Nov 03 13:42:06.928052 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:06.927990 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-32bec3-6c4596d787-tsnlp" event={"ID":"c4239490-ae74-411a-8b11-237b853b70e1","Type":"ContainerDied","Data":"4c963100dec989f0edb9c8168f668efb32b66f9d6d0520657022897143f2b3be"} Nov 03 13:42:06.928052 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:06.928036 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-32bec3-6c4596d787-tsnlp" event={"ID":"c4239490-ae74-411a-8b11-237b853b70e1","Type":"ContainerDied","Data":"3c1b0b02a98f5b20f30d3dd915eb352f1d27d8b35d02bb4ea96320da216274b7"} Nov 03 13:42:06.938134 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:06.938111 5979 scope.go:122] "RemoveContainer" containerID="37637c2fe572152a87f0cd0af692b003aac20f9d6fdc80a53880371b2d84844a" Nov 03 13:42:06.938399 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:42:06.938381 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37637c2fe572152a87f0cd0af692b003aac20f9d6fdc80a53880371b2d84844a\": container with ID starting with 37637c2fe572152a87f0cd0af692b003aac20f9d6fdc80a53880371b2d84844a not found: ID does not exist" containerID="37637c2fe572152a87f0cd0af692b003aac20f9d6fdc80a53880371b2d84844a" Nov 03 13:42:06.938512 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:06.938408 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37637c2fe572152a87f0cd0af692b003aac20f9d6fdc80a53880371b2d84844a"} err="failed to get container status \"37637c2fe572152a87f0cd0af692b003aac20f9d6fdc80a53880371b2d84844a\": rpc error: code = NotFound desc = could not find container \"37637c2fe572152a87f0cd0af692b003aac20f9d6fdc80a53880371b2d84844a\": container with ID starting with 37637c2fe572152a87f0cd0af692b003aac20f9d6fdc80a53880371b2d84844a not found: ID does not exist" Nov 03 13:42:06.938512 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:06.938426 5979 scope.go:122] "RemoveContainer" containerID="4c963100dec989f0edb9c8168f668efb32b66f9d6d0520657022897143f2b3be" Nov 03 13:42:06.947208 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:06.947190 5979 scope.go:122] "RemoveContainer" containerID="4c963100dec989f0edb9c8168f668efb32b66f9d6d0520657022897143f2b3be" Nov 03 13:42:06.947434 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:42:06.947417 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c963100dec989f0edb9c8168f668efb32b66f9d6d0520657022897143f2b3be\": container with ID starting with 4c963100dec989f0edb9c8168f668efb32b66f9d6d0520657022897143f2b3be not found: ID does not exist" containerID="4c963100dec989f0edb9c8168f668efb32b66f9d6d0520657022897143f2b3be" Nov 03 13:42:06.947495 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:06.947440 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c963100dec989f0edb9c8168f668efb32b66f9d6d0520657022897143f2b3be"} err="failed to get container status \"4c963100dec989f0edb9c8168f668efb32b66f9d6d0520657022897143f2b3be\": rpc error: code = NotFound desc = could not find container \"4c963100dec989f0edb9c8168f668efb32b66f9d6d0520657022897143f2b3be\": container with ID starting with 4c963100dec989f0edb9c8168f668efb32b66f9d6d0520657022897143f2b3be not found: ID does not exist" Nov 03 13:42:07.083283 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:07.083201 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-32bec3-6c4596d787-tsnlp"] Nov 03 13:42:07.129611 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:07.129583 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-32bec3-6c4596d787-tsnlp"] Nov 03 13:42:07.189922 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:07.189889 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-32bec3-654f99d4f9-m9m9c"] Nov 03 13:42:07.193253 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:07.193226 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-32bec3-654f99d4f9-m9m9c"] Nov 03 13:42:07.235109 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:42:07.235073 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:42:08.126493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:08.126084 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="247afc70-fb0c-4092-91f0-ecd7d4f3a03c" path="/var/lib/kubelet/pods/247afc70-fb0c-4092-91f0-ecd7d4f3a03c/volumes" Nov 03 13:42:08.126704 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:08.126652 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4239490-ae74-411a-8b11-237b853b70e1" path="/var/lib/kubelet/pods/c4239490-ae74-411a-8b11-237b853b70e1/volumes" Nov 03 13:42:17.332216 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:17.330791 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4b9658-86c4d8b589-5fdvq"] Nov 03 13:42:17.332789 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:17.332410 5979 cpu_manager.go:414] "containerMap: removing container" podUID="247afc70-fb0c-4092-91f0-ecd7d4f3a03c" containerName="payload-processing" Nov 03 13:42:17.332789 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:17.332438 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="247afc70-fb0c-4092-91f0-ecd7d4f3a03c" containerName="payload-processing" Nov 03 13:42:17.332789 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:17.332457 5979 cpu_manager.go:414] "containerMap: removing container" podUID="c4239490-ae74-411a-8b11-237b853b70e1" containerName="payload-pre-processing" Nov 03 13:42:17.332789 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:17.332465 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4239490-ae74-411a-8b11-237b853b70e1" containerName="payload-pre-processing" Nov 03 13:42:17.333008 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:17.332856 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="247afc70-fb0c-4092-91f0-ecd7d4f3a03c" containerName="payload-processing" Nov 03 13:42:17.333008 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:17.332875 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4239490-ae74-411a-8b11-237b853b70e1" containerName="payload-pre-processing" Nov 03 13:42:17.338101 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:17.338071 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4b9658-86c4d8b589-5fdvq" Nov 03 13:42:17.372754 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:17.372715 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkwv9\" (UniqueName: \"kubernetes.io/projected/f79a66f2-0c10-4260-ba7e-6cc21b2a52c4-kube-api-access-hkwv9\") pod \"maas-api-e2e-praxis-a-4b9658-86c4d8b589-5fdvq\" (UID: \"f79a66f2-0c10-4260-ba7e-6cc21b2a52c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4b9658-86c4d8b589-5fdvq" Nov 03 13:42:17.372913 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:17.372773 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f79a66f2-0c10-4260-ba7e-6cc21b2a52c4-metrics-certs\") pod \"maas-api-e2e-praxis-a-4b9658-86c4d8b589-5fdvq\" (UID: \"f79a66f2-0c10-4260-ba7e-6cc21b2a52c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4b9658-86c4d8b589-5fdvq" Nov 03 13:42:17.372913 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:17.372886 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f79a66f2-0c10-4260-ba7e-6cc21b2a52c4-maas-api-tls\") pod \"maas-api-e2e-praxis-a-4b9658-86c4d8b589-5fdvq\" (UID: \"f79a66f2-0c10-4260-ba7e-6cc21b2a52c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4b9658-86c4d8b589-5fdvq" Nov 03 13:42:17.372991 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:17.372919 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f79a66f2-0c10-4260-ba7e-6cc21b2a52c4-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-4b9658-86c4d8b589-5fdvq\" (UID: \"f79a66f2-0c10-4260-ba7e-6cc21b2a52c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4b9658-86c4d8b589-5fdvq" Nov 03 13:42:17.389805 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:17.389776 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4b9658-86c4d8b589-5fdvq"] Nov 03 13:42:17.432647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:17.432618 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-4b9658-serving-cert\"" Nov 03 13:42:17.474105 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:17.474069 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f79a66f2-0c10-4260-ba7e-6cc21b2a52c4-maas-api-tls\") pod \"maas-api-e2e-praxis-a-4b9658-86c4d8b589-5fdvq\" (UID: \"f79a66f2-0c10-4260-ba7e-6cc21b2a52c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4b9658-86c4d8b589-5fdvq" Nov 03 13:42:17.474326 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:17.474117 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f79a66f2-0c10-4260-ba7e-6cc21b2a52c4-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-4b9658-86c4d8b589-5fdvq\" (UID: \"f79a66f2-0c10-4260-ba7e-6cc21b2a52c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4b9658-86c4d8b589-5fdvq" Nov 03 13:42:17.474326 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:42:17.474237 5979 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-4b9658-serving-cert: secret "maas-api-e2e-praxis-a-4b9658-serving-cert" not found Nov 03 13:42:17.474326 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:42:17.474326 5979 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f79a66f2-0c10-4260-ba7e-6cc21b2a52c4-maas-api-tls podName:f79a66f2-0c10-4260-ba7e-6cc21b2a52c4 nodeName:}" failed. No retries permitted until 2026-11-03 13:42:17.974305784 +0000 UTC m=+2219.984994018 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f79a66f2-0c10-4260-ba7e-6cc21b2a52c4-maas-api-tls") pod "maas-api-e2e-praxis-a-4b9658-86c4d8b589-5fdvq" (UID: "f79a66f2-0c10-4260-ba7e-6cc21b2a52c4") : secret "maas-api-e2e-praxis-a-4b9658-serving-cert" not found Nov 03 13:42:17.474604 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:17.474531 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkwv9\" (UniqueName: \"kubernetes.io/projected/f79a66f2-0c10-4260-ba7e-6cc21b2a52c4-kube-api-access-hkwv9\") pod \"maas-api-e2e-praxis-a-4b9658-86c4d8b589-5fdvq\" (UID: \"f79a66f2-0c10-4260-ba7e-6cc21b2a52c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4b9658-86c4d8b589-5fdvq" Nov 03 13:42:17.474604 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:17.474590 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f79a66f2-0c10-4260-ba7e-6cc21b2a52c4-metrics-certs\") pod \"maas-api-e2e-praxis-a-4b9658-86c4d8b589-5fdvq\" (UID: \"f79a66f2-0c10-4260-ba7e-6cc21b2a52c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4b9658-86c4d8b589-5fdvq" Nov 03 13:42:17.475120 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:17.475086 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f79a66f2-0c10-4260-ba7e-6cc21b2a52c4-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-4b9658-86c4d8b589-5fdvq\" (UID: \"f79a66f2-0c10-4260-ba7e-6cc21b2a52c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4b9658-86c4d8b589-5fdvq" Nov 03 13:42:17.479950 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:17.479924 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f79a66f2-0c10-4260-ba7e-6cc21b2a52c4-metrics-certs\") pod \"maas-api-e2e-praxis-a-4b9658-86c4d8b589-5fdvq\" (UID: \"f79a66f2-0c10-4260-ba7e-6cc21b2a52c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4b9658-86c4d8b589-5fdvq" Nov 03 13:42:17.530238 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:17.530198 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkwv9\" (UniqueName: \"kubernetes.io/projected/f79a66f2-0c10-4260-ba7e-6cc21b2a52c4-kube-api-access-hkwv9\") pod \"maas-api-e2e-praxis-a-4b9658-86c4d8b589-5fdvq\" (UID: \"f79a66f2-0c10-4260-ba7e-6cc21b2a52c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4b9658-86c4d8b589-5fdvq" Nov 03 13:42:17.985131 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:17.985093 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f79a66f2-0c10-4260-ba7e-6cc21b2a52c4-maas-api-tls\") pod \"maas-api-e2e-praxis-a-4b9658-86c4d8b589-5fdvq\" (UID: \"f79a66f2-0c10-4260-ba7e-6cc21b2a52c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4b9658-86c4d8b589-5fdvq" Nov 03 13:42:17.987527 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:17.987500 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f79a66f2-0c10-4260-ba7e-6cc21b2a52c4-maas-api-tls\") pod \"maas-api-e2e-praxis-a-4b9658-86c4d8b589-5fdvq\" (UID: \"f79a66f2-0c10-4260-ba7e-6cc21b2a52c4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4b9658-86c4d8b589-5fdvq" Nov 03 13:42:18.257559 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:18.257454 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4b9658-86c4d8b589-5fdvq" Nov 03 13:42:18.647591 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:18.647561 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4b9658-86c4d8b589-5fdvq"] Nov 03 13:42:18.654425 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:42:18.654390 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf79a66f2_0c10_4260_ba7e_6cc21b2a52c4.slice/crio-eedfd1696856db2722071c4cf6d046dbc43ef34b02b0799c2c3422d5fb125469 WatchSource:0}: Error finding container eedfd1696856db2722071c4cf6d046dbc43ef34b02b0799c2c3422d5fb125469: Status 404 returned error can't find the container with id eedfd1696856db2722071c4cf6d046dbc43ef34b02b0799c2c3422d5fb125469 Nov 03 13:42:19.004530 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:19.004412 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4b9658-86c4d8b589-5fdvq" event={"ID":"f79a66f2-0c10-4260-ba7e-6cc21b2a52c4","Type":"ContainerStarted","Data":"eedfd1696856db2722071c4cf6d046dbc43ef34b02b0799c2c3422d5fb125469"} Nov 03 13:42:20.013230 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:20.013195 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4b9658-86c4d8b589-5fdvq" event={"ID":"f79a66f2-0c10-4260-ba7e-6cc21b2a52c4","Type":"ContainerStarted","Data":"e3727ad043e5cc405c1320025081fb513d2a43d0b312e4197751d5b07a965f60"} Nov 03 13:42:20.013664 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:20.013304 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4b9658-86c4d8b589-5fdvq" Nov 03 13:42:20.123492 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:20.123430 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4b9658-86c4d8b589-5fdvq" podStartSLOduration=2.214730162 podStartE2EDuration="3.123417137s" totalImagesPullingTime="908.686975ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:42:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:42:20.121215269 +0000 UTC m=+2222.131903488" watchObservedRunningTime="2026-11-03 13:42:20.123417137 +0000 UTC m=+2222.134105369" Nov 03 13:42:26.033423 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:26.033390 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4b9658-86c4d8b589-5fdvq" Nov 03 13:42:26.511515 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:26.511478 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6bd5c1-fd8fdd4bf-2p55l"] Nov 03 13:42:26.628122 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:26.628088 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6bd5c1-78bb66789f-pjv6n"] Nov 03 13:42:31.512588 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:31.512543 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-6bd5c1-fd8fdd4bf-2p55l" podUID="8ca552c2-8806-4fd9-b015-759d3aac38bf" containerName="payload-processing" containerID="cri-o://7028016cd60bca7175e8f5c20ed3e58963a5bc190ea2dc503eaee7ad9bf3f655" gracePeriod=25 Nov 03 13:42:31.629351 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:31.629294 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6bd5c1-78bb66789f-pjv6n" podUID="122981ad-160e-4bd3-a78d-de1649e61667" containerName="payload-pre-processing" containerID="cri-o://4328f08b2b2c1894d36e64ede7ec2925da2e3d0c17555361778762a74097fd8a" gracePeriod=25 Nov 03 13:42:31.801141 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:31.801111 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6bd5c1-fd8fdd4bf-2p55l" Nov 03 13:42:31.910163 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:31.910138 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6bd5c1-78bb66789f-pjv6n" Nov 03 13:42:31.962516 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:31.962406 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ca552c2-8806-4fd9-b015-759d3aac38bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ca552c2-8806-4fd9-b015-759d3aac38bf-plugins-config-volume\") pod \"8ca552c2-8806-4fd9-b015-759d3aac38bf\" (UID: \"8ca552c2-8806-4fd9-b015-759d3aac38bf\") " Nov 03 13:42:31.962727 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:31.962700 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ca552c2-8806-4fd9-b015-759d3aac38bf-kube-api-access-shnkj\" (UniqueName: \"kubernetes.io/projected/8ca552c2-8806-4fd9-b015-759d3aac38bf-kube-api-access-shnkj\") pod \"8ca552c2-8806-4fd9-b015-759d3aac38bf\" (UID: \"8ca552c2-8806-4fd9-b015-759d3aac38bf\") " Nov 03 13:42:31.963102 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:31.963076 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ca552c2-8806-4fd9-b015-759d3aac38bf-plugins-config-volume" pod "8ca552c2-8806-4fd9-b015-759d3aac38bf" (UID: "8ca552c2-8806-4fd9-b015-759d3aac38bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:42:31.963283 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:31.963250 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ca552c2-8806-4fd9-b015-759d3aac38bf-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:31.965889 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:31.965814 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ca552c2-8806-4fd9-b015-759d3aac38bf-kube-api-access-shnkj" pod "8ca552c2-8806-4fd9-b015-759d3aac38bf" (UID: "8ca552c2-8806-4fd9-b015-759d3aac38bf"). InnerVolumeSpecName "kube-api-access-shnkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:42:32.063977 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:32.063893 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/122981ad-160e-4bd3-a78d-de1649e61667-kube-api-access-qt8wt\" (UniqueName: \"kubernetes.io/projected/122981ad-160e-4bd3-a78d-de1649e61667-kube-api-access-qt8wt\") pod \"122981ad-160e-4bd3-a78d-de1649e61667\" (UID: \"122981ad-160e-4bd3-a78d-de1649e61667\") " Nov 03 13:42:32.064260 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:32.064007 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/122981ad-160e-4bd3-a78d-de1649e61667-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/122981ad-160e-4bd3-a78d-de1649e61667-plugins-config-volume\") pod \"122981ad-160e-4bd3-a78d-de1649e61667\" (UID: \"122981ad-160e-4bd3-a78d-de1649e61667\") " Nov 03 13:42:32.064395 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:32.064374 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/122981ad-160e-4bd3-a78d-de1649e61667-plugins-config-volume" pod "122981ad-160e-4bd3-a78d-de1649e61667" (UID: "122981ad-160e-4bd3-a78d-de1649e61667"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:42:32.064548 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:32.064532 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shnkj\" (UniqueName: \"kubernetes.io/projected/8ca552c2-8806-4fd9-b015-759d3aac38bf-kube-api-access-shnkj\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:32.064610 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:32.064550 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/122981ad-160e-4bd3-a78d-de1649e61667-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:32.066552 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:32.066523 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/122981ad-160e-4bd3-a78d-de1649e61667-kube-api-access-qt8wt" pod "122981ad-160e-4bd3-a78d-de1649e61667" (UID: "122981ad-160e-4bd3-a78d-de1649e61667"). InnerVolumeSpecName "kube-api-access-qt8wt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:42:32.120980 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:32.120938 5979 generic.go:356] "Generic (PLEG): container finished" podID="122981ad-160e-4bd3-a78d-de1649e61667" containerID="4328f08b2b2c1894d36e64ede7ec2925da2e3d0c17555361778762a74097fd8a" exitCode=0 Nov 03 13:42:32.121229 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:32.121003 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6bd5c1-78bb66789f-pjv6n" Nov 03 13:42:32.121229 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:32.121016 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6bd5c1-78bb66789f-pjv6n" event={"ID":"122981ad-160e-4bd3-a78d-de1649e61667","Type":"ContainerDied","Data":"4328f08b2b2c1894d36e64ede7ec2925da2e3d0c17555361778762a74097fd8a"} Nov 03 13:42:32.121229 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:32.121081 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6bd5c1-78bb66789f-pjv6n" event={"ID":"122981ad-160e-4bd3-a78d-de1649e61667","Type":"ContainerDied","Data":"bef40e603803eb06e0a7675e9e62cde25ef1250af1a29a51ae242bd8f5add58c"} Nov 03 13:42:32.121229 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:32.121116 5979 scope.go:122] "RemoveContainer" containerID="4328f08b2b2c1894d36e64ede7ec2925da2e3d0c17555361778762a74097fd8a" Nov 03 13:42:32.128035 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:32.127294 5979 generic.go:356] "Generic (PLEG): container finished" podID="8ca552c2-8806-4fd9-b015-759d3aac38bf" containerID="7028016cd60bca7175e8f5c20ed3e58963a5bc190ea2dc503eaee7ad9bf3f655" exitCode=0 Nov 03 13:42:32.128035 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:32.127342 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6bd5c1-fd8fdd4bf-2p55l" event={"ID":"8ca552c2-8806-4fd9-b015-759d3aac38bf","Type":"ContainerDied","Data":"7028016cd60bca7175e8f5c20ed3e58963a5bc190ea2dc503eaee7ad9bf3f655"} Nov 03 13:42:32.128035 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:32.127375 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6bd5c1-fd8fdd4bf-2p55l" event={"ID":"8ca552c2-8806-4fd9-b015-759d3aac38bf","Type":"ContainerDied","Data":"5ec21083f9ec3910fec7e65564b2284e9dbdea68c4da7e01b68ec1e336f16252"} Nov 03 13:42:32.128035 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:32.127437 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6bd5c1-fd8fdd4bf-2p55l" Nov 03 13:42:32.146076 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:32.146055 5979 scope.go:122] "RemoveContainer" containerID="4328f08b2b2c1894d36e64ede7ec2925da2e3d0c17555361778762a74097fd8a" Nov 03 13:42:32.146426 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:42:32.146400 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4328f08b2b2c1894d36e64ede7ec2925da2e3d0c17555361778762a74097fd8a\": container with ID starting with 4328f08b2b2c1894d36e64ede7ec2925da2e3d0c17555361778762a74097fd8a not found: ID does not exist" containerID="4328f08b2b2c1894d36e64ede7ec2925da2e3d0c17555361778762a74097fd8a" Nov 03 13:42:32.146535 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:32.146437 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4328f08b2b2c1894d36e64ede7ec2925da2e3d0c17555361778762a74097fd8a"} err="failed to get container status \"4328f08b2b2c1894d36e64ede7ec2925da2e3d0c17555361778762a74097fd8a\": rpc error: code = NotFound desc = could not find container \"4328f08b2b2c1894d36e64ede7ec2925da2e3d0c17555361778762a74097fd8a\": container with ID starting with 4328f08b2b2c1894d36e64ede7ec2925da2e3d0c17555361778762a74097fd8a not found: ID does not exist" Nov 03 13:42:32.146535 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:32.146460 5979 scope.go:122] "RemoveContainer" containerID="7028016cd60bca7175e8f5c20ed3e58963a5bc190ea2dc503eaee7ad9bf3f655" Nov 03 13:42:32.158063 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:32.158044 5979 scope.go:122] "RemoveContainer" containerID="7028016cd60bca7175e8f5c20ed3e58963a5bc190ea2dc503eaee7ad9bf3f655" Nov 03 13:42:32.158368 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:42:32.158352 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7028016cd60bca7175e8f5c20ed3e58963a5bc190ea2dc503eaee7ad9bf3f655\": container with ID starting with 7028016cd60bca7175e8f5c20ed3e58963a5bc190ea2dc503eaee7ad9bf3f655 not found: ID does not exist" containerID="7028016cd60bca7175e8f5c20ed3e58963a5bc190ea2dc503eaee7ad9bf3f655" Nov 03 13:42:32.158424 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:32.158375 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7028016cd60bca7175e8f5c20ed3e58963a5bc190ea2dc503eaee7ad9bf3f655"} err="failed to get container status \"7028016cd60bca7175e8f5c20ed3e58963a5bc190ea2dc503eaee7ad9bf3f655\": rpc error: code = NotFound desc = could not find container \"7028016cd60bca7175e8f5c20ed3e58963a5bc190ea2dc503eaee7ad9bf3f655\": container with ID starting with 7028016cd60bca7175e8f5c20ed3e58963a5bc190ea2dc503eaee7ad9bf3f655 not found: ID does not exist" Nov 03 13:42:32.166213 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:32.166193 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qt8wt\" (UniqueName: \"kubernetes.io/projected/122981ad-160e-4bd3-a78d-de1649e61667-kube-api-access-qt8wt\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:32.277080 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:32.277042 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6bd5c1-fd8fdd4bf-2p55l"] Nov 03 13:42:32.282305 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:32.282280 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6bd5c1-fd8fdd4bf-2p55l"] Nov 03 13:42:32.332517 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:32.332423 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:42:32.396734 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:32.396704 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6bd5c1-78bb66789f-pjv6n"] Nov 03 13:42:32.402063 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:32.402039 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6bd5c1-78bb66789f-pjv6n"] Nov 03 13:42:34.116785 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:34.116753 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="122981ad-160e-4bd3-a78d-de1649e61667" path="/var/lib/kubelet/pods/122981ad-160e-4bd3-a78d-de1649e61667/volumes" Nov 03 13:42:34.117216 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:34.117200 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ca552c2-8806-4fd9-b015-759d3aac38bf" path="/var/lib/kubelet/pods/8ca552c2-8806-4fd9-b015-759d3aac38bf/volumes" Nov 03 13:42:37.234664 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:42:37.234631 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:42:42.230677 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:42.230645 5979 generic.go:356] "Generic (PLEG): container finished" podID="763f2c9e-811d-49dc-b971-4782cb8a0964" containerID="f23261b12105120dd094972ca2c57abe63e04748f496e89d9653d7141b4088f1" exitCode=0 Nov 03 13:42:42.231260 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:42.230736 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq" event={"ID":"763f2c9e-811d-49dc-b971-4782cb8a0964","Type":"ContainerDied","Data":"f23261b12105120dd094972ca2c57abe63e04748f496e89d9653d7141b4088f1"} Nov 03 13:42:42.231260 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:42.231124 5979 scope.go:122] "RemoveContainer" containerID="f23261b12105120dd094972ca2c57abe63e04748f496e89d9653d7141b4088f1" Nov 03 13:42:42.232771 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:42.232746 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:42:42.232845 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:42.232831 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:42:42.232883 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:42.232864 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:42:43.238903 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:43.238872 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq" event={"ID":"763f2c9e-811d-49dc-b971-4782cb8a0964","Type":"ContainerStarted","Data":"15f9c496b01302ac6b60bb5c5af4e3fbfc26d47e7e01323191ac4bd585fcea48"} Nov 03 13:42:43.393239 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:43.393200 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq" Nov 03 13:42:43.394535 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:43.394505 5979 patch_prober.go:29] interesting pod/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body= Nov 03 13:42:43.394704 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:43.394553 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq" podUID="763f2c9e-811d-49dc-b971-4782cb8a0964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" Nov 03 13:42:44.393164 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:44.393126 5979 patch_prober.go:29] interesting pod/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body= Nov 03 13:42:44.393597 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:44.393175 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq" podUID="763f2c9e-811d-49dc-b971-4782cb8a0964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" Nov 03 13:42:45.393030 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:45.392993 5979 patch_prober.go:29] interesting pod/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body= Nov 03 13:42:45.393259 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:45.393077 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq" podUID="763f2c9e-811d-49dc-b971-4782cb8a0964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" Nov 03 13:42:45.398158 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:45.398129 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq" Nov 03 13:42:46.393484 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:46.393423 5979 patch_prober.go:29] interesting pod/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body= Nov 03 13:42:46.393988 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:46.393500 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq" podUID="763f2c9e-811d-49dc-b971-4782cb8a0964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" Nov 03 13:42:47.392999 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:47.392955 5979 patch_prober.go:29] interesting pod/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body= Nov 03 13:42:47.393207 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:47.393042 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq" podUID="763f2c9e-811d-49dc-b971-4782cb8a0964" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" Nov 03 13:42:48.397405 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:48.397374 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq" Nov 03 13:42:48.398188 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:48.398162 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq" Nov 03 13:42:50.875316 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:50.875273 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-21f111-5dc5cbd779-vkggd"] Nov 03 13:42:50.990889 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:50.990857 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-21f111-5f9499d695-qpxbk"] Nov 03 13:42:55.876258 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:55.876212 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-21f111-5dc5cbd779-vkggd" podUID="c355a440-18fd-4f9f-8016-8c679e8f1aa4" containerName="payload-processing" containerID="cri-o://7c60e6140cb78cb263e770bbd6b90670115a44aa54954f06a92a9438577ad364" gracePeriod=25 Nov 03 13:42:55.991530 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:55.991449 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-21f111-5f9499d695-qpxbk" podUID="c61c5a2c-8b5d-4c88-96d3-add6b701eb0f" containerName="payload-pre-processing" containerID="cri-o://28bd3d3542ce2a069072316e8e2923698d622d8c15fe3eb58aeb0967ea52b26f" gracePeriod=25 Nov 03 13:42:56.157588 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:56.157550 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-21f111-5dc5cbd779-vkggd" Nov 03 13:42:56.219031 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:56.218987 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c355a440-18fd-4f9f-8016-8c679e8f1aa4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c355a440-18fd-4f9f-8016-8c679e8f1aa4-plugins-config-volume\") pod \"c355a440-18fd-4f9f-8016-8c679e8f1aa4\" (UID: \"c355a440-18fd-4f9f-8016-8c679e8f1aa4\") " Nov 03 13:42:56.219213 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:56.219172 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c355a440-18fd-4f9f-8016-8c679e8f1aa4-kube-api-access-kvm68\" (UniqueName: \"kubernetes.io/projected/c355a440-18fd-4f9f-8016-8c679e8f1aa4-kube-api-access-kvm68\") pod \"c355a440-18fd-4f9f-8016-8c679e8f1aa4\" (UID: \"c355a440-18fd-4f9f-8016-8c679e8f1aa4\") " Nov 03 13:42:56.219514 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:56.219484 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c355a440-18fd-4f9f-8016-8c679e8f1aa4-plugins-config-volume" pod "c355a440-18fd-4f9f-8016-8c679e8f1aa4" (UID: "c355a440-18fd-4f9f-8016-8c679e8f1aa4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:42:56.219923 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:56.219702 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c355a440-18fd-4f9f-8016-8c679e8f1aa4-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:56.223033 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:56.222915 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c355a440-18fd-4f9f-8016-8c679e8f1aa4-kube-api-access-kvm68" pod "c355a440-18fd-4f9f-8016-8c679e8f1aa4" (UID: "c355a440-18fd-4f9f-8016-8c679e8f1aa4"). InnerVolumeSpecName "kube-api-access-kvm68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:42:56.263939 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:56.263915 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-21f111-5f9499d695-qpxbk" Nov 03 13:42:56.320205 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:56.320173 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c61c5a2c-8b5d-4c88-96d3-add6b701eb0f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c61c5a2c-8b5d-4c88-96d3-add6b701eb0f-plugins-config-volume\") pod \"c61c5a2c-8b5d-4c88-96d3-add6b701eb0f\" (UID: \"c61c5a2c-8b5d-4c88-96d3-add6b701eb0f\") " Nov 03 13:42:56.320385 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:56.320358 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c61c5a2c-8b5d-4c88-96d3-add6b701eb0f-kube-api-access-g6rpj\" (UniqueName: \"kubernetes.io/projected/c61c5a2c-8b5d-4c88-96d3-add6b701eb0f-kube-api-access-g6rpj\") pod \"c61c5a2c-8b5d-4c88-96d3-add6b701eb0f\" (UID: \"c61c5a2c-8b5d-4c88-96d3-add6b701eb0f\") " Nov 03 13:42:56.320607 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:56.320587 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c61c5a2c-8b5d-4c88-96d3-add6b701eb0f-plugins-config-volume" pod "c61c5a2c-8b5d-4c88-96d3-add6b701eb0f" (UID: "c61c5a2c-8b5d-4c88-96d3-add6b701eb0f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:42:56.320819 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:56.320795 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c61c5a2c-8b5d-4c88-96d3-add6b701eb0f-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:56.320819 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:56.320818 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvm68\" (UniqueName: \"kubernetes.io/projected/c355a440-18fd-4f9f-8016-8c679e8f1aa4-kube-api-access-kvm68\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:56.322420 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:56.322400 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c61c5a2c-8b5d-4c88-96d3-add6b701eb0f-kube-api-access-g6rpj" pod "c61c5a2c-8b5d-4c88-96d3-add6b701eb0f" (UID: "c61c5a2c-8b5d-4c88-96d3-add6b701eb0f"). InnerVolumeSpecName "kube-api-access-g6rpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:42:56.368071 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:56.368030 5979 generic.go:356] "Generic (PLEG): container finished" podID="c355a440-18fd-4f9f-8016-8c679e8f1aa4" containerID="7c60e6140cb78cb263e770bbd6b90670115a44aa54954f06a92a9438577ad364" exitCode=0 Nov 03 13:42:56.368255 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:56.368085 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-21f111-5dc5cbd779-vkggd" Nov 03 13:42:56.368255 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:56.368118 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-21f111-5dc5cbd779-vkggd" event={"ID":"c355a440-18fd-4f9f-8016-8c679e8f1aa4","Type":"ContainerDied","Data":"7c60e6140cb78cb263e770bbd6b90670115a44aa54954f06a92a9438577ad364"} Nov 03 13:42:56.368255 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:56.368177 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-21f111-5dc5cbd779-vkggd" event={"ID":"c355a440-18fd-4f9f-8016-8c679e8f1aa4","Type":"ContainerDied","Data":"52f577d5c68fef7ab312353be8d7da0ffa88e821faa0dda23301f8d7bfd22c1d"} Nov 03 13:42:56.368255 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:56.368200 5979 scope.go:122] "RemoveContainer" containerID="7c60e6140cb78cb263e770bbd6b90670115a44aa54954f06a92a9438577ad364" Nov 03 13:42:56.369495 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:56.369454 5979 generic.go:356] "Generic (PLEG): container finished" podID="c61c5a2c-8b5d-4c88-96d3-add6b701eb0f" containerID="28bd3d3542ce2a069072316e8e2923698d622d8c15fe3eb58aeb0967ea52b26f" exitCode=0 Nov 03 13:42:56.369612 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:56.369523 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-21f111-5f9499d695-qpxbk" Nov 03 13:42:56.369612 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:56.369525 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-21f111-5f9499d695-qpxbk" event={"ID":"c61c5a2c-8b5d-4c88-96d3-add6b701eb0f","Type":"ContainerDied","Data":"28bd3d3542ce2a069072316e8e2923698d622d8c15fe3eb58aeb0967ea52b26f"} Nov 03 13:42:56.369612 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:56.369567 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-21f111-5f9499d695-qpxbk" event={"ID":"c61c5a2c-8b5d-4c88-96d3-add6b701eb0f","Type":"ContainerDied","Data":"a5c02cf54d24abc9162822cd40e40c167eba2bbd10617a02a716a44b9a17dca3"} Nov 03 13:42:56.378867 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:56.378774 5979 scope.go:122] "RemoveContainer" containerID="7c60e6140cb78cb263e770bbd6b90670115a44aa54954f06a92a9438577ad364" Nov 03 13:42:56.379171 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:42:56.379144 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c60e6140cb78cb263e770bbd6b90670115a44aa54954f06a92a9438577ad364\": container with ID starting with 7c60e6140cb78cb263e770bbd6b90670115a44aa54954f06a92a9438577ad364 not found: ID does not exist" containerID="7c60e6140cb78cb263e770bbd6b90670115a44aa54954f06a92a9438577ad364" Nov 03 13:42:56.379256 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:56.379180 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c60e6140cb78cb263e770bbd6b90670115a44aa54954f06a92a9438577ad364"} err="failed to get container status \"7c60e6140cb78cb263e770bbd6b90670115a44aa54954f06a92a9438577ad364\": rpc error: code = NotFound desc = could not find container \"7c60e6140cb78cb263e770bbd6b90670115a44aa54954f06a92a9438577ad364\": container with ID starting with 7c60e6140cb78cb263e770bbd6b90670115a44aa54954f06a92a9438577ad364 not found: ID does not exist" Nov 03 13:42:56.379256 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:56.379201 5979 scope.go:122] "RemoveContainer" containerID="28bd3d3542ce2a069072316e8e2923698d622d8c15fe3eb58aeb0967ea52b26f" Nov 03 13:42:56.389924 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:56.389904 5979 scope.go:122] "RemoveContainer" containerID="28bd3d3542ce2a069072316e8e2923698d622d8c15fe3eb58aeb0967ea52b26f" Nov 03 13:42:56.390202 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:42:56.390183 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28bd3d3542ce2a069072316e8e2923698d622d8c15fe3eb58aeb0967ea52b26f\": container with ID starting with 28bd3d3542ce2a069072316e8e2923698d622d8c15fe3eb58aeb0967ea52b26f not found: ID does not exist" containerID="28bd3d3542ce2a069072316e8e2923698d622d8c15fe3eb58aeb0967ea52b26f" Nov 03 13:42:56.390272 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:56.390209 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28bd3d3542ce2a069072316e8e2923698d622d8c15fe3eb58aeb0967ea52b26f"} err="failed to get container status \"28bd3d3542ce2a069072316e8e2923698d622d8c15fe3eb58aeb0967ea52b26f\": rpc error: code = NotFound desc = could not find container \"28bd3d3542ce2a069072316e8e2923698d622d8c15fe3eb58aeb0967ea52b26f\": container with ID starting with 28bd3d3542ce2a069072316e8e2923698d622d8c15fe3eb58aeb0967ea52b26f not found: ID does not exist" Nov 03 13:42:56.422487 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:56.422450 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6rpj\" (UniqueName: \"kubernetes.io/projected/c61c5a2c-8b5d-4c88-96d3-add6b701eb0f-kube-api-access-g6rpj\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:42:56.581515 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:56.581453 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-21f111-5dc5cbd779-vkggd"] Nov 03 13:42:56.588457 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:56.588421 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-21f111-5dc5cbd779-vkggd"] Nov 03 13:42:56.687532 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:56.687433 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-21f111-5f9499d695-qpxbk"] Nov 03 13:42:56.693485 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:56.693447 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-21f111-5f9499d695-qpxbk"] Nov 03 13:42:58.115458 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:58.115423 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c355a440-18fd-4f9f-8016-8c679e8f1aa4" path="/var/lib/kubelet/pods/c355a440-18fd-4f9f-8016-8c679e8f1aa4/volumes" Nov 03 13:42:58.115948 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:42:58.115932 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c61c5a2c-8b5d-4c88-96d3-add6b701eb0f" path="/var/lib/kubelet/pods/c61c5a2c-8b5d-4c88-96d3-add6b701eb0f/volumes" Nov 03 13:43:07.234406 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:43:07.234324 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:43:14.888934 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:14.888900 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-32bec3-78c6df84dc-wr8qq"] Nov 03 13:43:14.889534 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:14.889517 5979 cpu_manager.go:414] "containerMap: removing container" podUID="c61c5a2c-8b5d-4c88-96d3-add6b701eb0f" containerName="payload-pre-processing" Nov 03 13:43:14.889589 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:14.889536 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c61c5a2c-8b5d-4c88-96d3-add6b701eb0f" containerName="payload-pre-processing" Nov 03 13:43:14.889589 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:14.889558 5979 cpu_manager.go:414] "containerMap: removing container" podUID="122981ad-160e-4bd3-a78d-de1649e61667" containerName="payload-pre-processing" Nov 03 13:43:14.889589 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:14.889563 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="122981ad-160e-4bd3-a78d-de1649e61667" containerName="payload-pre-processing" Nov 03 13:43:14.889589 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:14.889575 5979 cpu_manager.go:414] "containerMap: removing container" podUID="c355a440-18fd-4f9f-8016-8c679e8f1aa4" containerName="payload-processing" Nov 03 13:43:14.889589 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:14.889580 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c355a440-18fd-4f9f-8016-8c679e8f1aa4" containerName="payload-processing" Nov 03 13:43:14.889754 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:14.889608 5979 cpu_manager.go:414] "containerMap: removing container" podUID="8ca552c2-8806-4fd9-b015-759d3aac38bf" containerName="payload-processing" Nov 03 13:43:14.889754 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:14.889613 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ca552c2-8806-4fd9-b015-759d3aac38bf" containerName="payload-processing" Nov 03 13:43:14.889754 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:14.889713 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ca552c2-8806-4fd9-b015-759d3aac38bf" containerName="payload-processing" Nov 03 13:43:14.889754 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:14.889727 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="c61c5a2c-8b5d-4c88-96d3-add6b701eb0f" containerName="payload-pre-processing" Nov 03 13:43:14.889754 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:14.889733 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="122981ad-160e-4bd3-a78d-de1649e61667" containerName="payload-pre-processing" Nov 03 13:43:14.889754 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:14.889741 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="c355a440-18fd-4f9f-8016-8c679e8f1aa4" containerName="payload-processing" Nov 03 13:43:14.893238 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:14.893218 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-32bec3-78c6df84dc-wr8qq" Nov 03 13:43:14.964619 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:14.964584 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b23dfaf-6b15-4c7e-8515-019f404475de-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-32bec3-78c6df84dc-wr8qq\" (UID: \"2b23dfaf-6b15-4c7e-8515-019f404475de\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-32bec3-78c6df84dc-wr8qq" Nov 03 13:43:14.964619 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:14.964634 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzj8l\" (UniqueName: \"kubernetes.io/projected/2b23dfaf-6b15-4c7e-8515-019f404475de-kube-api-access-tzj8l\") pod \"payload-pre-processing-e2e-ipp-b-32bec3-78c6df84dc-wr8qq\" (UID: \"2b23dfaf-6b15-4c7e-8515-019f404475de\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-32bec3-78c6df84dc-wr8qq" Nov 03 13:43:14.972445 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:14.972407 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-32bec3-dockercfg-sc8xr\"" Nov 03 13:43:14.972445 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:14.972438 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-32bec3\"" Nov 03 13:43:14.976199 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:14.976168 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-32bec3-78c6df84dc-wr8qq"] Nov 03 13:43:15.066246 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:15.066213 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b23dfaf-6b15-4c7e-8515-019f404475de-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-32bec3-78c6df84dc-wr8qq\" (UID: \"2b23dfaf-6b15-4c7e-8515-019f404475de\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-32bec3-78c6df84dc-wr8qq" Nov 03 13:43:15.066440 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:15.066268 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzj8l\" (UniqueName: \"kubernetes.io/projected/2b23dfaf-6b15-4c7e-8515-019f404475de-kube-api-access-tzj8l\") pod \"payload-pre-processing-e2e-ipp-b-32bec3-78c6df84dc-wr8qq\" (UID: \"2b23dfaf-6b15-4c7e-8515-019f404475de\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-32bec3-78c6df84dc-wr8qq" Nov 03 13:43:15.066990 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:15.066970 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b23dfaf-6b15-4c7e-8515-019f404475de-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-32bec3-78c6df84dc-wr8qq\" (UID: \"2b23dfaf-6b15-4c7e-8515-019f404475de\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-32bec3-78c6df84dc-wr8qq" Nov 03 13:43:15.121200 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:15.121160 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzj8l\" (UniqueName: \"kubernetes.io/projected/2b23dfaf-6b15-4c7e-8515-019f404475de-kube-api-access-tzj8l\") pod \"payload-pre-processing-e2e-ipp-b-32bec3-78c6df84dc-wr8qq\" (UID: \"2b23dfaf-6b15-4c7e-8515-019f404475de\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-32bec3-78c6df84dc-wr8qq" Nov 03 13:43:15.207169 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:15.207069 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-32bec3-78c6df84dc-wr8qq" Nov 03 13:43:15.583689 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:15.583609 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-32bec3-78c6df84dc-wr8qq"] Nov 03 13:43:15.590673 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:43:15.590644 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b23dfaf_6b15_4c7e_8515_019f404475de.slice/crio-76ea25e7773097e46558e58dd4553bc263adc56b5bab54b259183cda55e5a4f4 WatchSource:0}: Error finding container 76ea25e7773097e46558e58dd4553bc263adc56b5bab54b259183cda55e5a4f4: Status 404 returned error can't find the container with id 76ea25e7773097e46558e58dd4553bc263adc56b5bab54b259183cda55e5a4f4 Nov 03 13:43:15.637669 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:15.637640 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-32bec3-678877f9d4-qjwmv"] Nov 03 13:43:15.643483 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:15.643449 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-32bec3-678877f9d4-qjwmv" Nov 03 13:43:15.673545 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:15.673516 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f6351e2-84a8-414e-a0d8-dd394abec66a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-32bec3-678877f9d4-qjwmv\" (UID: \"0f6351e2-84a8-414e-a0d8-dd394abec66a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-32bec3-678877f9d4-qjwmv" Nov 03 13:43:15.673692 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:15.673589 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xjxx\" (UniqueName: \"kubernetes.io/projected/0f6351e2-84a8-414e-a0d8-dd394abec66a-kube-api-access-4xjxx\") pod \"payload-processing-e2e-ipp-b-32bec3-678877f9d4-qjwmv\" (UID: \"0f6351e2-84a8-414e-a0d8-dd394abec66a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-32bec3-678877f9d4-qjwmv" Nov 03 13:43:15.700094 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:15.700065 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-32bec3-678877f9d4-qjwmv"] Nov 03 13:43:15.774962 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:15.774930 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f6351e2-84a8-414e-a0d8-dd394abec66a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-32bec3-678877f9d4-qjwmv\" (UID: \"0f6351e2-84a8-414e-a0d8-dd394abec66a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-32bec3-678877f9d4-qjwmv" Nov 03 13:43:15.775134 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:15.774989 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xjxx\" (UniqueName: \"kubernetes.io/projected/0f6351e2-84a8-414e-a0d8-dd394abec66a-kube-api-access-4xjxx\") pod \"payload-processing-e2e-ipp-b-32bec3-678877f9d4-qjwmv\" (UID: \"0f6351e2-84a8-414e-a0d8-dd394abec66a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-32bec3-678877f9d4-qjwmv" Nov 03 13:43:15.775626 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:15.775605 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f6351e2-84a8-414e-a0d8-dd394abec66a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-32bec3-678877f9d4-qjwmv\" (UID: \"0f6351e2-84a8-414e-a0d8-dd394abec66a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-32bec3-678877f9d4-qjwmv" Nov 03 13:43:15.829142 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:15.829114 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xjxx\" (UniqueName: \"kubernetes.io/projected/0f6351e2-84a8-414e-a0d8-dd394abec66a-kube-api-access-4xjxx\") pod \"payload-processing-e2e-ipp-b-32bec3-678877f9d4-qjwmv\" (UID: \"0f6351e2-84a8-414e-a0d8-dd394abec66a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-32bec3-678877f9d4-qjwmv" Nov 03 13:43:15.956630 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:15.956590 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-32bec3-678877f9d4-qjwmv" Nov 03 13:43:16.344020 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:16.343935 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-32bec3-678877f9d4-qjwmv"] Nov 03 13:43:16.351683 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:43:16.351648 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f6351e2_84a8_414e_a0d8_dd394abec66a.slice/crio-d41fae08d372065827e5db7023954e8a7537b61a65b7659f36ff5cfa4b8afede WatchSource:0}: Error finding container d41fae08d372065827e5db7023954e8a7537b61a65b7659f36ff5cfa4b8afede: Status 404 returned error can't find the container with id d41fae08d372065827e5db7023954e8a7537b61a65b7659f36ff5cfa4b8afede Nov 03 13:43:16.510922 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:16.510872 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-32bec3-78c6df84dc-wr8qq" event={"ID":"2b23dfaf-6b15-4c7e-8515-019f404475de","Type":"ContainerStarted","Data":"c9b7352f3a6f1a261ce0fed34cfcb3e619228a5c437fa76297aca47010704b5b"} Nov 03 13:43:16.511092 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:16.510948 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-32bec3-78c6df84dc-wr8qq" event={"ID":"2b23dfaf-6b15-4c7e-8515-019f404475de","Type":"ContainerStarted","Data":"76ea25e7773097e46558e58dd4553bc263adc56b5bab54b259183cda55e5a4f4"} Nov 03 13:43:16.512240 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:16.511247 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-32bec3-78c6df84dc-wr8qq" Nov 03 13:43:16.513730 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:16.513699 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-32bec3-678877f9d4-qjwmv" event={"ID":"0f6351e2-84a8-414e-a0d8-dd394abec66a","Type":"ContainerStarted","Data":"5c82f6ddfd174b221aa2b3176b14bbead8408213d0ec9f60ba10b33849cfa2fd"} Nov 03 13:43:16.513840 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:16.513757 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-32bec3-678877f9d4-qjwmv" event={"ID":"0f6351e2-84a8-414e-a0d8-dd394abec66a","Type":"ContainerStarted","Data":"d41fae08d372065827e5db7023954e8a7537b61a65b7659f36ff5cfa4b8afede"} Nov 03 13:43:16.514011 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:16.513990 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-32bec3-678877f9d4-qjwmv" Nov 03 13:43:16.632592 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:16.630876 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-32bec3-78c6df84dc-wr8qq" podStartSLOduration=2.630857003 podStartE2EDuration="2.630857003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:43:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:43:16.627929829 +0000 UTC m=+2278.638618072" watchObservedRunningTime="2026-11-03 13:43:16.630857003 +0000 UTC m=+2278.641545246" Nov 03 13:43:16.738136 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:16.738068 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-32bec3-678877f9d4-qjwmv" podStartSLOduration=1.738049223 podStartE2EDuration="1.738049223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:43:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:43:16.736607608 +0000 UTC m=+2278.747295856" watchObservedRunningTime="2026-11-03 13:43:16.738049223 +0000 UTC m=+2278.748737467" Nov 03 13:43:21.179264 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:21.179233 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-02f3a0-5b5967f877-c652k"] Nov 03 13:43:21.179693 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:21.179423 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-02f3a0-5b5967f877-c652k" podUID="a3cd2275-2289-4acc-8966-5b6515421385" containerName="maas-api" containerID="cri-o://ebc88bd82454bd54097a33883fa8cd26665a67bb02197764c2138761c00baf66" gracePeriod=30 Nov 03 13:43:21.483811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:21.483785 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-02f3a0-5b5967f877-c652k" Nov 03 13:43:21.580406 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:21.579021 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3cd2275-2289-4acc-8966-5b6515421385-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3cd2275-2289-4acc-8966-5b6515421385-maas-api-tls\") pod \"a3cd2275-2289-4acc-8966-5b6515421385\" (UID: \"a3cd2275-2289-4acc-8966-5b6515421385\") " Nov 03 13:43:21.580406 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:21.579101 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3cd2275-2289-4acc-8966-5b6515421385-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3cd2275-2289-4acc-8966-5b6515421385-metrics-certs\") pod \"a3cd2275-2289-4acc-8966-5b6515421385\" (UID: \"a3cd2275-2289-4acc-8966-5b6515421385\") " Nov 03 13:43:21.580406 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:21.579204 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3cd2275-2289-4acc-8966-5b6515421385-kube-api-access-vsl58\" (UniqueName: \"kubernetes.io/projected/a3cd2275-2289-4acc-8966-5b6515421385-kube-api-access-vsl58\") pod \"a3cd2275-2289-4acc-8966-5b6515421385\" (UID: \"a3cd2275-2289-4acc-8966-5b6515421385\") " Nov 03 13:43:21.580406 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:21.579264 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3cd2275-2289-4acc-8966-5b6515421385-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3cd2275-2289-4acc-8966-5b6515421385-openshift-service-ca\") pod \"a3cd2275-2289-4acc-8966-5b6515421385\" (UID: \"a3cd2275-2289-4acc-8966-5b6515421385\") " Nov 03 13:43:21.580982 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:21.580950 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3cd2275-2289-4acc-8966-5b6515421385-openshift-service-ca" pod "a3cd2275-2289-4acc-8966-5b6515421385" (UID: "a3cd2275-2289-4acc-8966-5b6515421385"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:43:21.606886 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:21.606820 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3cd2275-2289-4acc-8966-5b6515421385-maas-api-tls" pod "a3cd2275-2289-4acc-8966-5b6515421385" (UID: "a3cd2275-2289-4acc-8966-5b6515421385"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:43:21.611793 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:21.610715 5979 generic.go:356] "Generic (PLEG): container finished" podID="a3cd2275-2289-4acc-8966-5b6515421385" containerID="ebc88bd82454bd54097a33883fa8cd26665a67bb02197764c2138761c00baf66" exitCode=0 Nov 03 13:43:21.611793 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:21.610797 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-02f3a0-5b5967f877-c652k" event={"ID":"a3cd2275-2289-4acc-8966-5b6515421385","Type":"ContainerDied","Data":"ebc88bd82454bd54097a33883fa8cd26665a67bb02197764c2138761c00baf66"} Nov 03 13:43:21.611793 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:21.610848 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-02f3a0-5b5967f877-c652k" event={"ID":"a3cd2275-2289-4acc-8966-5b6515421385","Type":"ContainerDied","Data":"75fb7067d031071bb36500ed90fdee01420ab65ae5bbac6d47212bba6fd6b131"} Nov 03 13:43:21.611793 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:21.610960 5979 scope.go:122] "RemoveContainer" containerID="ebc88bd82454bd54097a33883fa8cd26665a67bb02197764c2138761c00baf66" Nov 03 13:43:21.611793 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:21.611178 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-02f3a0-5b5967f877-c652k" Nov 03 13:43:21.614758 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:21.614717 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3cd2275-2289-4acc-8966-5b6515421385-kube-api-access-vsl58" pod "a3cd2275-2289-4acc-8966-5b6515421385" (UID: "a3cd2275-2289-4acc-8966-5b6515421385"). InnerVolumeSpecName "kube-api-access-vsl58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:43:21.615539 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:21.615504 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3cd2275-2289-4acc-8966-5b6515421385-metrics-certs" pod "a3cd2275-2289-4acc-8966-5b6515421385" (UID: "a3cd2275-2289-4acc-8966-5b6515421385"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:43:21.638934 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:21.638905 5979 scope.go:122] "RemoveContainer" containerID="ebc88bd82454bd54097a33883fa8cd26665a67bb02197764c2138761c00baf66" Nov 03 13:43:21.639301 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:43:21.639265 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebc88bd82454bd54097a33883fa8cd26665a67bb02197764c2138761c00baf66\": container with ID starting with ebc88bd82454bd54097a33883fa8cd26665a67bb02197764c2138761c00baf66 not found: ID does not exist" containerID="ebc88bd82454bd54097a33883fa8cd26665a67bb02197764c2138761c00baf66" Nov 03 13:43:21.639403 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:21.639298 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebc88bd82454bd54097a33883fa8cd26665a67bb02197764c2138761c00baf66"} err="failed to get container status \"ebc88bd82454bd54097a33883fa8cd26665a67bb02197764c2138761c00baf66\": rpc error: code = NotFound desc = could not find container \"ebc88bd82454bd54097a33883fa8cd26665a67bb02197764c2138761c00baf66\": container with ID starting with ebc88bd82454bd54097a33883fa8cd26665a67bb02197764c2138761c00baf66 not found: ID does not exist" Nov 03 13:43:21.642613 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:21.642587 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-02f3a0-c66d764b6-qwl7q"] Nov 03 13:43:21.642833 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:21.642808 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-02f3a0-c66d764b6-qwl7q" podUID="5faad0ec-c254-4ed2-a88c-8f1012747b57" containerName="payload-processing" containerID="cri-o://b11aaf27837c2d79839bb790d39ad92003cba1659ba89ad25a3147ceb493c781" gracePeriod=30 Nov 03 13:43:21.681125 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:21.681096 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a3cd2275-2289-4acc-8966-5b6515421385-maas-api-tls\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:21.681125 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:21.681126 5979 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a3cd2275-2289-4acc-8966-5b6515421385-metrics-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:21.681295 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:21.681142 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsl58\" (UniqueName: \"kubernetes.io/projected/a3cd2275-2289-4acc-8966-5b6515421385-kube-api-access-vsl58\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:21.681295 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:21.681157 5979 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3cd2275-2289-4acc-8966-5b6515421385-openshift-service-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:21.698864 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:21.698826 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-02f3a0-68759d857f-9c2dn"] Nov 03 13:43:21.699745 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:21.699687 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02f3a0-68759d857f-9c2dn" podUID="fd43ea90-19f2-4109-85a7-07098fc3da7c" containerName="payload-pre-processing" containerID="cri-o://58dbd0850b4491b94e7bd4602d6ddf0f326f3f8a1b76f0b71f7d1a4f1415a10e" gracePeriod=30 Nov 03 13:43:21.961705 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:21.961632 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-02f3a0-c66d764b6-qwl7q" Nov 03 13:43:21.987544 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:21.987516 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02f3a0-68759d857f-9c2dn" Nov 03 13:43:22.030514 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:22.030485 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-02f3a0-5b5967f877-c652k"] Nov 03 13:43:22.035521 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:22.035497 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-02f3a0-5b5967f877-c652k"] Nov 03 13:43:22.088764 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:22.088737 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd43ea90-19f2-4109-85a7-07098fc3da7c-kube-api-access-2kzqq\" (UniqueName: \"kubernetes.io/projected/fd43ea90-19f2-4109-85a7-07098fc3da7c-kube-api-access-2kzqq\") pod \"fd43ea90-19f2-4109-85a7-07098fc3da7c\" (UID: \"fd43ea90-19f2-4109-85a7-07098fc3da7c\") " Nov 03 13:43:22.088938 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:22.088926 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5faad0ec-c254-4ed2-a88c-8f1012747b57-kube-api-access-lbdj6\" (UniqueName: \"kubernetes.io/projected/5faad0ec-c254-4ed2-a88c-8f1012747b57-kube-api-access-lbdj6\") pod \"5faad0ec-c254-4ed2-a88c-8f1012747b57\" (UID: \"5faad0ec-c254-4ed2-a88c-8f1012747b57\") " Nov 03 13:43:22.088999 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:22.088968 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5faad0ec-c254-4ed2-a88c-8f1012747b57-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5faad0ec-c254-4ed2-a88c-8f1012747b57-plugins-config-volume\") pod \"5faad0ec-c254-4ed2-a88c-8f1012747b57\" (UID: \"5faad0ec-c254-4ed2-a88c-8f1012747b57\") " Nov 03 13:43:22.089055 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:22.089010 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd43ea90-19f2-4109-85a7-07098fc3da7c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd43ea90-19f2-4109-85a7-07098fc3da7c-plugins-config-volume\") pod \"fd43ea90-19f2-4109-85a7-07098fc3da7c\" (UID: \"fd43ea90-19f2-4109-85a7-07098fc3da7c\") " Nov 03 13:43:22.089417 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:22.089390 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5faad0ec-c254-4ed2-a88c-8f1012747b57-plugins-config-volume" pod "5faad0ec-c254-4ed2-a88c-8f1012747b57" (UID: "5faad0ec-c254-4ed2-a88c-8f1012747b57"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:43:22.089573 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:22.089489 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd43ea90-19f2-4109-85a7-07098fc3da7c-plugins-config-volume" pod "fd43ea90-19f2-4109-85a7-07098fc3da7c" (UID: "fd43ea90-19f2-4109-85a7-07098fc3da7c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:43:22.090842 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:22.090816 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd43ea90-19f2-4109-85a7-07098fc3da7c-kube-api-access-2kzqq" pod "fd43ea90-19f2-4109-85a7-07098fc3da7c" (UID: "fd43ea90-19f2-4109-85a7-07098fc3da7c"). InnerVolumeSpecName "kube-api-access-2kzqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:43:22.090946 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:22.090896 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5faad0ec-c254-4ed2-a88c-8f1012747b57-kube-api-access-lbdj6" pod "5faad0ec-c254-4ed2-a88c-8f1012747b57" (UID: "5faad0ec-c254-4ed2-a88c-8f1012747b57"). InnerVolumeSpecName "kube-api-access-lbdj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:43:22.116877 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:22.116846 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3cd2275-2289-4acc-8966-5b6515421385" path="/var/lib/kubelet/pods/a3cd2275-2289-4acc-8966-5b6515421385/volumes" Nov 03 13:43:22.192948 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:22.191641 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd43ea90-19f2-4109-85a7-07098fc3da7c-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:22.192948 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:22.191679 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kzqq\" (UniqueName: \"kubernetes.io/projected/fd43ea90-19f2-4109-85a7-07098fc3da7c-kube-api-access-2kzqq\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:22.192948 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:22.191697 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbdj6\" (UniqueName: \"kubernetes.io/projected/5faad0ec-c254-4ed2-a88c-8f1012747b57-kube-api-access-lbdj6\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:22.192948 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:22.191715 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5faad0ec-c254-4ed2-a88c-8f1012747b57-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:22.619719 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:22.619683 5979 generic.go:356] "Generic (PLEG): container finished" podID="fd43ea90-19f2-4109-85a7-07098fc3da7c" containerID="58dbd0850b4491b94e7bd4602d6ddf0f326f3f8a1b76f0b71f7d1a4f1415a10e" exitCode=0 Nov 03 13:43:22.619932 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:22.619732 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02f3a0-68759d857f-9c2dn" Nov 03 13:43:22.619932 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:22.619753 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02f3a0-68759d857f-9c2dn" event={"ID":"fd43ea90-19f2-4109-85a7-07098fc3da7c","Type":"ContainerDied","Data":"58dbd0850b4491b94e7bd4602d6ddf0f326f3f8a1b76f0b71f7d1a4f1415a10e"} Nov 03 13:43:22.619932 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:22.619811 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-02f3a0-68759d857f-9c2dn" event={"ID":"fd43ea90-19f2-4109-85a7-07098fc3da7c","Type":"ContainerDied","Data":"b63d61b8bd623f918c527c5e0434bd967b842c4c469b647eee0d5b6ca40317c0"} Nov 03 13:43:22.619932 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:22.619844 5979 scope.go:122] "RemoveContainer" containerID="58dbd0850b4491b94e7bd4602d6ddf0f326f3f8a1b76f0b71f7d1a4f1415a10e" Nov 03 13:43:22.621801 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:22.621759 5979 generic.go:356] "Generic (PLEG): container finished" podID="5faad0ec-c254-4ed2-a88c-8f1012747b57" containerID="b11aaf27837c2d79839bb790d39ad92003cba1659ba89ad25a3147ceb493c781" exitCode=0 Nov 03 13:43:22.622078 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:22.622061 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-02f3a0-c66d764b6-qwl7q" Nov 03 13:43:22.622995 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:22.622972 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-02f3a0-c66d764b6-qwl7q" event={"ID":"5faad0ec-c254-4ed2-a88c-8f1012747b57","Type":"ContainerDied","Data":"b11aaf27837c2d79839bb790d39ad92003cba1659ba89ad25a3147ceb493c781"} Nov 03 13:43:22.623102 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:22.623018 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-02f3a0-c66d764b6-qwl7q" event={"ID":"5faad0ec-c254-4ed2-a88c-8f1012747b57","Type":"ContainerDied","Data":"078650ba1345da7fedf10f08f22c0d24131f552db9c489425bbafc5d3670aca2"} Nov 03 13:43:22.631853 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:22.631836 5979 scope.go:122] "RemoveContainer" containerID="58dbd0850b4491b94e7bd4602d6ddf0f326f3f8a1b76f0b71f7d1a4f1415a10e" Nov 03 13:43:22.632133 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:43:22.632112 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58dbd0850b4491b94e7bd4602d6ddf0f326f3f8a1b76f0b71f7d1a4f1415a10e\": container with ID starting with 58dbd0850b4491b94e7bd4602d6ddf0f326f3f8a1b76f0b71f7d1a4f1415a10e not found: ID does not exist" containerID="58dbd0850b4491b94e7bd4602d6ddf0f326f3f8a1b76f0b71f7d1a4f1415a10e" Nov 03 13:43:22.632185 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:22.632141 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58dbd0850b4491b94e7bd4602d6ddf0f326f3f8a1b76f0b71f7d1a4f1415a10e"} err="failed to get container status \"58dbd0850b4491b94e7bd4602d6ddf0f326f3f8a1b76f0b71f7d1a4f1415a10e\": rpc error: code = NotFound desc = could not find container \"58dbd0850b4491b94e7bd4602d6ddf0f326f3f8a1b76f0b71f7d1a4f1415a10e\": container with ID starting with 58dbd0850b4491b94e7bd4602d6ddf0f326f3f8a1b76f0b71f7d1a4f1415a10e not found: ID does not exist" Nov 03 13:43:22.632185 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:22.632157 5979 scope.go:122] "RemoveContainer" containerID="b11aaf27837c2d79839bb790d39ad92003cba1659ba89ad25a3147ceb493c781" Nov 03 13:43:22.640593 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:22.640567 5979 scope.go:122] "RemoveContainer" containerID="b11aaf27837c2d79839bb790d39ad92003cba1659ba89ad25a3147ceb493c781" Nov 03 13:43:22.640855 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:43:22.640834 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b11aaf27837c2d79839bb790d39ad92003cba1659ba89ad25a3147ceb493c781\": container with ID starting with b11aaf27837c2d79839bb790d39ad92003cba1659ba89ad25a3147ceb493c781 not found: ID does not exist" containerID="b11aaf27837c2d79839bb790d39ad92003cba1659ba89ad25a3147ceb493c781" Nov 03 13:43:22.640897 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:22.640863 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b11aaf27837c2d79839bb790d39ad92003cba1659ba89ad25a3147ceb493c781"} err="failed to get container status \"b11aaf27837c2d79839bb790d39ad92003cba1659ba89ad25a3147ceb493c781\": rpc error: code = NotFound desc = could not find container \"b11aaf27837c2d79839bb790d39ad92003cba1659ba89ad25a3147ceb493c781\": container with ID starting with b11aaf27837c2d79839bb790d39ad92003cba1659ba89ad25a3147ceb493c781 not found: ID does not exist" Nov 03 13:43:22.775226 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:22.775192 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-02f3a0-c66d764b6-qwl7q"] Nov 03 13:43:22.778188 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:22.778164 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-02f3a0-c66d764b6-qwl7q"] Nov 03 13:43:22.887741 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:22.887708 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-02f3a0-68759d857f-9c2dn"] Nov 03 13:43:22.901737 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:22.901698 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-02f3a0-68759d857f-9c2dn"] Nov 03 13:43:24.116933 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:24.116899 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5faad0ec-c254-4ed2-a88c-8f1012747b57" path="/var/lib/kubelet/pods/5faad0ec-c254-4ed2-a88c-8f1012747b57/volumes" Nov 03 13:43:24.117465 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:24.117443 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd43ea90-19f2-4109-85a7-07098fc3da7c" path="/var/lib/kubelet/pods/fd43ea90-19f2-4109-85a7-07098fc3da7c/volumes" Nov 03 13:43:27.523726 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:27.523696 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-32bec3-678877f9d4-qjwmv" Nov 03 13:43:27.524167 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:27.524105 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-32bec3-78c6df84dc-wr8qq" Nov 03 13:43:27.697942 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:27.692315 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-21f111-9f4fb987-jb2sv"] Nov 03 13:43:27.697942 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:27.692533 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-21f111-9f4fb987-jb2sv" podUID="66067047-9f8c-499e-af1c-1245b56b39f8" containerName="maas-api" containerID="cri-o://e3d439da8081444a1aa0e817d175aed7ca9501f87f98082bf4333ff524a49332" gracePeriod=30 Nov 03 13:43:27.983676 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:27.983651 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-21f111-9f4fb987-jb2sv" Nov 03 13:43:28.172991 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:28.172955 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66067047-9f8c-499e-af1c-1245b56b39f8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66067047-9f8c-499e-af1c-1245b56b39f8-openshift-service-ca\") pod \"66067047-9f8c-499e-af1c-1245b56b39f8\" (UID: \"66067047-9f8c-499e-af1c-1245b56b39f8\") " Nov 03 13:43:28.173196 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:28.173177 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66067047-9f8c-499e-af1c-1245b56b39f8-kube-api-access-h4jdf\" (UniqueName: \"kubernetes.io/projected/66067047-9f8c-499e-af1c-1245b56b39f8-kube-api-access-h4jdf\") pod \"66067047-9f8c-499e-af1c-1245b56b39f8\" (UID: \"66067047-9f8c-499e-af1c-1245b56b39f8\") " Nov 03 13:43:28.173258 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:28.173238 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66067047-9f8c-499e-af1c-1245b56b39f8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/66067047-9f8c-499e-af1c-1245b56b39f8-metrics-certs\") pod \"66067047-9f8c-499e-af1c-1245b56b39f8\" (UID: \"66067047-9f8c-499e-af1c-1245b56b39f8\") " Nov 03 13:43:28.173318 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:28.173295 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66067047-9f8c-499e-af1c-1245b56b39f8-openshift-service-ca" pod "66067047-9f8c-499e-af1c-1245b56b39f8" (UID: "66067047-9f8c-499e-af1c-1245b56b39f8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:43:28.173404 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:28.173370 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66067047-9f8c-499e-af1c-1245b56b39f8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66067047-9f8c-499e-af1c-1245b56b39f8-maas-api-tls\") pod \"66067047-9f8c-499e-af1c-1245b56b39f8\" (UID: \"66067047-9f8c-499e-af1c-1245b56b39f8\") " Nov 03 13:43:28.173905 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:28.173878 5979 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66067047-9f8c-499e-af1c-1245b56b39f8-openshift-service-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:28.175322 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:28.175299 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66067047-9f8c-499e-af1c-1245b56b39f8-maas-api-tls" pod "66067047-9f8c-499e-af1c-1245b56b39f8" (UID: "66067047-9f8c-499e-af1c-1245b56b39f8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:43:28.175498 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:28.175399 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66067047-9f8c-499e-af1c-1245b56b39f8-kube-api-access-h4jdf" pod "66067047-9f8c-499e-af1c-1245b56b39f8" (UID: "66067047-9f8c-499e-af1c-1245b56b39f8"). InnerVolumeSpecName "kube-api-access-h4jdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:43:28.175580 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:28.175526 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66067047-9f8c-499e-af1c-1245b56b39f8-metrics-certs" pod "66067047-9f8c-499e-af1c-1245b56b39f8" (UID: "66067047-9f8c-499e-af1c-1245b56b39f8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:43:28.274710 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:28.274681 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4jdf\" (UniqueName: \"kubernetes.io/projected/66067047-9f8c-499e-af1c-1245b56b39f8-kube-api-access-h4jdf\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:28.274710 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:28.274707 5979 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66067047-9f8c-499e-af1c-1245b56b39f8-metrics-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:28.274710 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:28.274716 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66067047-9f8c-499e-af1c-1245b56b39f8-maas-api-tls\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:28.672124 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:28.672085 5979 generic.go:356] "Generic (PLEG): container finished" podID="66067047-9f8c-499e-af1c-1245b56b39f8" containerID="e3d439da8081444a1aa0e817d175aed7ca9501f87f98082bf4333ff524a49332" exitCode=0 Nov 03 13:43:28.672669 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:28.672173 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-21f111-9f4fb987-jb2sv" event={"ID":"66067047-9f8c-499e-af1c-1245b56b39f8","Type":"ContainerDied","Data":"e3d439da8081444a1aa0e817d175aed7ca9501f87f98082bf4333ff524a49332"} Nov 03 13:43:28.672669 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:28.672224 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-21f111-9f4fb987-jb2sv" event={"ID":"66067047-9f8c-499e-af1c-1245b56b39f8","Type":"ContainerDied","Data":"05f4e76d52ebc1bda56dae424371081761a49b5ac21741bf89ef68be81496bb0"} Nov 03 13:43:28.672669 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:28.672254 5979 scope.go:122] "RemoveContainer" containerID="e3d439da8081444a1aa0e817d175aed7ca9501f87f98082bf4333ff524a49332" Nov 03 13:43:28.672669 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:28.672390 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-21f111-9f4fb987-jb2sv" Nov 03 13:43:28.716086 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:28.715934 5979 scope.go:122] "RemoveContainer" containerID="e3d439da8081444a1aa0e817d175aed7ca9501f87f98082bf4333ff524a49332" Nov 03 13:43:28.716321 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:43:28.716298 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3d439da8081444a1aa0e817d175aed7ca9501f87f98082bf4333ff524a49332\": container with ID starting with e3d439da8081444a1aa0e817d175aed7ca9501f87f98082bf4333ff524a49332 not found: ID does not exist" containerID="e3d439da8081444a1aa0e817d175aed7ca9501f87f98082bf4333ff524a49332" Nov 03 13:43:28.716403 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:28.716335 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3d439da8081444a1aa0e817d175aed7ca9501f87f98082bf4333ff524a49332"} err="failed to get container status \"e3d439da8081444a1aa0e817d175aed7ca9501f87f98082bf4333ff524a49332\": rpc error: code = NotFound desc = could not find container \"e3d439da8081444a1aa0e817d175aed7ca9501f87f98082bf4333ff524a49332\": container with ID starting with e3d439da8081444a1aa0e817d175aed7ca9501f87f98082bf4333ff524a49332 not found: ID does not exist" Nov 03 13:43:28.849095 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:28.849059 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-21f111-9f4fb987-jb2sv"] Nov 03 13:43:28.895084 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:28.895051 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-21f111-9f4fb987-jb2sv"] Nov 03 13:43:30.115991 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:30.115957 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66067047-9f8c-499e-af1c-1245b56b39f8" path="/var/lib/kubelet/pods/66067047-9f8c-499e-af1c-1245b56b39f8/volumes" Nov 03 13:43:35.592551 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:35.592513 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6bd5c1-786d9ddfc5-f8tmc"] Nov 03 13:43:35.593160 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:35.593144 5979 cpu_manager.go:414] "containerMap: removing container" podUID="a3cd2275-2289-4acc-8966-5b6515421385" containerName="maas-api" Nov 03 13:43:35.593207 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:35.593161 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3cd2275-2289-4acc-8966-5b6515421385" containerName="maas-api" Nov 03 13:43:35.593207 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:35.593174 5979 cpu_manager.go:414] "containerMap: removing container" podUID="fd43ea90-19f2-4109-85a7-07098fc3da7c" containerName="payload-pre-processing" Nov 03 13:43:35.593207 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:35.593179 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd43ea90-19f2-4109-85a7-07098fc3da7c" containerName="payload-pre-processing" Nov 03 13:43:35.593207 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:35.593205 5979 cpu_manager.go:414] "containerMap: removing container" podUID="5faad0ec-c254-4ed2-a88c-8f1012747b57" containerName="payload-processing" Nov 03 13:43:35.593731 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:35.593210 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5faad0ec-c254-4ed2-a88c-8f1012747b57" containerName="payload-processing" Nov 03 13:43:35.593731 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:35.593223 5979 cpu_manager.go:414] "containerMap: removing container" podUID="66067047-9f8c-499e-af1c-1245b56b39f8" containerName="maas-api" Nov 03 13:43:35.593731 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:35.593228 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66067047-9f8c-499e-af1c-1245b56b39f8" containerName="maas-api" Nov 03 13:43:35.593731 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:35.593321 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="5faad0ec-c254-4ed2-a88c-8f1012747b57" containerName="payload-processing" Nov 03 13:43:35.593731 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:35.593332 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="66067047-9f8c-499e-af1c-1245b56b39f8" containerName="maas-api" Nov 03 13:43:35.593731 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:35.593345 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3cd2275-2289-4acc-8966-5b6515421385" containerName="maas-api" Nov 03 13:43:35.593731 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:35.593352 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd43ea90-19f2-4109-85a7-07098fc3da7c" containerName="payload-pre-processing" Nov 03 13:43:35.598260 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:35.598237 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6bd5c1-786d9ddfc5-f8tmc" Nov 03 13:43:35.683786 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:35.683739 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07f7ad33-7cfb-484e-a6e2-d048b6f7b385-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6bd5c1-786d9ddfc5-f8tmc\" (UID: \"07f7ad33-7cfb-484e-a6e2-d048b6f7b385\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6bd5c1-786d9ddfc5-f8tmc" Nov 03 13:43:35.684002 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:35.683924 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t9nf\" (UniqueName: \"kubernetes.io/projected/07f7ad33-7cfb-484e-a6e2-d048b6f7b385-kube-api-access-9t9nf\") pod \"payload-pre-processing-e2e-ipp-b-6bd5c1-786d9ddfc5-f8tmc\" (UID: \"07f7ad33-7cfb-484e-a6e2-d048b6f7b385\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6bd5c1-786d9ddfc5-f8tmc" Nov 03 13:43:35.694962 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:35.694932 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-6bd5c1\"" Nov 03 13:43:35.695163 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:35.695059 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-6bd5c1-dockercfg-j6mr9\"" Nov 03 13:43:35.697406 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:35.697378 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6bd5c1-786d9ddfc5-f8tmc"] Nov 03 13:43:35.785751 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:35.785707 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07f7ad33-7cfb-484e-a6e2-d048b6f7b385-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6bd5c1-786d9ddfc5-f8tmc\" (UID: \"07f7ad33-7cfb-484e-a6e2-d048b6f7b385\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6bd5c1-786d9ddfc5-f8tmc" Nov 03 13:43:35.785965 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:35.785826 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t9nf\" (UniqueName: \"kubernetes.io/projected/07f7ad33-7cfb-484e-a6e2-d048b6f7b385-kube-api-access-9t9nf\") pod \"payload-pre-processing-e2e-ipp-b-6bd5c1-786d9ddfc5-f8tmc\" (UID: \"07f7ad33-7cfb-484e-a6e2-d048b6f7b385\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6bd5c1-786d9ddfc5-f8tmc" Nov 03 13:43:35.786412 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:35.786389 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07f7ad33-7cfb-484e-a6e2-d048b6f7b385-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-6bd5c1-786d9ddfc5-f8tmc\" (UID: \"07f7ad33-7cfb-484e-a6e2-d048b6f7b385\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6bd5c1-786d9ddfc5-f8tmc" Nov 03 13:43:35.841365 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:35.841328 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t9nf\" (UniqueName: \"kubernetes.io/projected/07f7ad33-7cfb-484e-a6e2-d048b6f7b385-kube-api-access-9t9nf\") pod \"payload-pre-processing-e2e-ipp-b-6bd5c1-786d9ddfc5-f8tmc\" (UID: \"07f7ad33-7cfb-484e-a6e2-d048b6f7b385\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6bd5c1-786d9ddfc5-f8tmc" Nov 03 13:43:35.911767 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:35.911730 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6bd5c1-786d9ddfc5-f8tmc" Nov 03 13:43:36.291590 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:36.291509 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6bd5c1-786d9ddfc5-f8tmc"] Nov 03 13:43:36.298671 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:43:36.298643 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07f7ad33_7cfb_484e_a6e2_d048b6f7b385.slice/crio-cf0569527e4b5edd32a48c97f550e71a37bf606a8edc4ff417858908947ad214 WatchSource:0}: Error finding container cf0569527e4b5edd32a48c97f550e71a37bf606a8edc4ff417858908947ad214: Status 404 returned error can't find the container with id cf0569527e4b5edd32a48c97f550e71a37bf606a8edc4ff417858908947ad214 Nov 03 13:43:36.361991 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:36.361964 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6bd5c1-6d9c48cdcd-bshhx"] Nov 03 13:43:36.366173 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:36.366146 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6bd5c1-6d9c48cdcd-bshhx" Nov 03 13:43:36.393037 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:36.393006 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq72d\" (UniqueName: \"kubernetes.io/projected/80c6a028-1d5c-4ba1-84b6-113c617c5cb7-kube-api-access-mq72d\") pod \"payload-processing-e2e-ipp-b-6bd5c1-6d9c48cdcd-bshhx\" (UID: \"80c6a028-1d5c-4ba1-84b6-113c617c5cb7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6bd5c1-6d9c48cdcd-bshhx" Nov 03 13:43:36.393204 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:36.393157 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80c6a028-1d5c-4ba1-84b6-113c617c5cb7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6bd5c1-6d9c48cdcd-bshhx\" (UID: \"80c6a028-1d5c-4ba1-84b6-113c617c5cb7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6bd5c1-6d9c48cdcd-bshhx" Nov 03 13:43:36.410441 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:36.410399 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6bd5c1-6d9c48cdcd-bshhx"] Nov 03 13:43:36.494813 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:36.494774 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq72d\" (UniqueName: \"kubernetes.io/projected/80c6a028-1d5c-4ba1-84b6-113c617c5cb7-kube-api-access-mq72d\") pod \"payload-processing-e2e-ipp-b-6bd5c1-6d9c48cdcd-bshhx\" (UID: \"80c6a028-1d5c-4ba1-84b6-113c617c5cb7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6bd5c1-6d9c48cdcd-bshhx" Nov 03 13:43:36.495012 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:36.494884 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80c6a028-1d5c-4ba1-84b6-113c617c5cb7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6bd5c1-6d9c48cdcd-bshhx\" (UID: \"80c6a028-1d5c-4ba1-84b6-113c617c5cb7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6bd5c1-6d9c48cdcd-bshhx" Nov 03 13:43:36.495688 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:36.495668 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80c6a028-1d5c-4ba1-84b6-113c617c5cb7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6bd5c1-6d9c48cdcd-bshhx\" (UID: \"80c6a028-1d5c-4ba1-84b6-113c617c5cb7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6bd5c1-6d9c48cdcd-bshhx" Nov 03 13:43:36.551042 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:36.550973 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq72d\" (UniqueName: \"kubernetes.io/projected/80c6a028-1d5c-4ba1-84b6-113c617c5cb7-kube-api-access-mq72d\") pod \"payload-processing-e2e-ipp-b-6bd5c1-6d9c48cdcd-bshhx\" (UID: \"80c6a028-1d5c-4ba1-84b6-113c617c5cb7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6bd5c1-6d9c48cdcd-bshhx" Nov 03 13:43:36.680866 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:36.680817 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6bd5c1-6d9c48cdcd-bshhx" Nov 03 13:43:36.723997 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:36.723963 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6bd5c1-786d9ddfc5-f8tmc" event={"ID":"07f7ad33-7cfb-484e-a6e2-d048b6f7b385","Type":"ContainerStarted","Data":"568a939c3c98fc958fb2817431b277a464220498a4c2d230b56e05a6a018bebb"} Nov 03 13:43:36.724172 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:36.724017 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6bd5c1-786d9ddfc5-f8tmc" event={"ID":"07f7ad33-7cfb-484e-a6e2-d048b6f7b385","Type":"ContainerStarted","Data":"cf0569527e4b5edd32a48c97f550e71a37bf606a8edc4ff417858908947ad214"} Nov 03 13:43:36.724172 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:36.724152 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6bd5c1-786d9ddfc5-f8tmc" Nov 03 13:43:36.836402 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:36.836279 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6bd5c1-786d9ddfc5-f8tmc" podStartSLOduration=1.836256871 podStartE2EDuration="1.836256871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:43:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:43:36.835394471 +0000 UTC m=+2298.846082731" watchObservedRunningTime="2026-11-03 13:43:36.836256871 +0000 UTC m=+2298.846945113" Nov 03 13:43:37.104571 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:37.104390 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6bd5c1-6d9c48cdcd-bshhx"] Nov 03 13:43:37.111215 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:43:37.111183 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80c6a028_1d5c_4ba1_84b6_113c617c5cb7.slice/crio-7e6372ffe004332b706c94074d6a197cd03ddb3a0191f3e6241935aaebcaa378 WatchSource:0}: Error finding container 7e6372ffe004332b706c94074d6a197cd03ddb3a0191f3e6241935aaebcaa378: Status 404 returned error can't find the container with id 7e6372ffe004332b706c94074d6a197cd03ddb3a0191f3e6241935aaebcaa378 Nov 03 13:43:37.235302 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:43:37.235268 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:43:37.736900 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:37.736784 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6bd5c1-6d9c48cdcd-bshhx" event={"ID":"80c6a028-1d5c-4ba1-84b6-113c617c5cb7","Type":"ContainerStarted","Data":"9f5c6212882eb9e93671539c67893708f6a73845b1c51ae4d665883beba4d441"} Nov 03 13:43:37.736900 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:37.736857 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6bd5c1-6d9c48cdcd-bshhx" event={"ID":"80c6a028-1d5c-4ba1-84b6-113c617c5cb7","Type":"ContainerStarted","Data":"7e6372ffe004332b706c94074d6a197cd03ddb3a0191f3e6241935aaebcaa378"} Nov 03 13:43:37.737527 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:37.736922 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6bd5c1-6d9c48cdcd-bshhx" Nov 03 13:43:37.849964 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:37.849907 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-6bd5c1-6d9c48cdcd-bshhx" podStartSLOduration=1.849891819 podStartE2EDuration="1.849891819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:43:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:43:37.848006964 +0000 UTC m=+2299.858695207" watchObservedRunningTime="2026-11-03 13:43:37.849891819 +0000 UTC m=+2299.860580061" Nov 03 13:43:45.244663 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:45.244628 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:43:47.409495 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:47.409106 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6"] Nov 03 13:43:47.426729 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:47.424559 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6" Nov 03 13:43:47.503984 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:47.503951 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6"] Nov 03 13:43:47.510656 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:47.510631 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0403729d-8803-40c2-9e5d-1cb2cc9914a1-istio-podinfo\") pod \"e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6\" (UID: \"0403729d-8803-40c2-9e5d-1cb2cc9914a1\") " pod="openshift-ingress/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6" Nov 03 13:43:47.510774 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:47.510665 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0403729d-8803-40c2-9e5d-1cb2cc9914a1-workload-certs\") pod \"e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6\" (UID: \"0403729d-8803-40c2-9e5d-1cb2cc9914a1\") " pod="openshift-ingress/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6" Nov 03 13:43:47.510774 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:47.510761 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0403729d-8803-40c2-9e5d-1cb2cc9914a1-workload-socket\") pod \"e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6\" (UID: \"0403729d-8803-40c2-9e5d-1cb2cc9914a1\") " pod="openshift-ingress/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6" Nov 03 13:43:47.510881 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:47.510867 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv96m\" (UniqueName: \"kubernetes.io/projected/0403729d-8803-40c2-9e5d-1cb2cc9914a1-kube-api-access-wv96m\") pod \"e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6\" (UID: \"0403729d-8803-40c2-9e5d-1cb2cc9914a1\") " pod="openshift-ingress/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6" Nov 03 13:43:47.510926 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:47.510898 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0403729d-8803-40c2-9e5d-1cb2cc9914a1-istio-data\") pod \"e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6\" (UID: \"0403729d-8803-40c2-9e5d-1cb2cc9914a1\") " pod="openshift-ingress/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6" Nov 03 13:43:47.510963 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:47.510925 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0403729d-8803-40c2-9e5d-1cb2cc9914a1-credential-socket\") pod \"e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6\" (UID: \"0403729d-8803-40c2-9e5d-1cb2cc9914a1\") " pod="openshift-ingress/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6" Nov 03 13:43:47.510999 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:47.510981 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0403729d-8803-40c2-9e5d-1cb2cc9914a1-istio-token\") pod \"e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6\" (UID: \"0403729d-8803-40c2-9e5d-1cb2cc9914a1\") " pod="openshift-ingress/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6" Nov 03 13:43:47.511036 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:47.511028 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0403729d-8803-40c2-9e5d-1cb2cc9914a1-istiod-ca-cert\") pod \"e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6\" (UID: \"0403729d-8803-40c2-9e5d-1cb2cc9914a1\") " pod="openshift-ingress/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6" Nov 03 13:43:47.511074 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:47.511057 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0403729d-8803-40c2-9e5d-1cb2cc9914a1-istio-envoy\") pod \"e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6\" (UID: \"0403729d-8803-40c2-9e5d-1cb2cc9914a1\") " pod="openshift-ingress/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6" Nov 03 13:43:47.523380 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:47.522626 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-ba6fbd-openshift-default-dockercfg-gpsdc\"" Nov 03 13:43:47.612127 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:47.612093 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0403729d-8803-40c2-9e5d-1cb2cc9914a1-istio-token\") pod \"e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6\" (UID: \"0403729d-8803-40c2-9e5d-1cb2cc9914a1\") " pod="openshift-ingress/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6" Nov 03 13:43:47.612314 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:47.612148 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0403729d-8803-40c2-9e5d-1cb2cc9914a1-istiod-ca-cert\") pod \"e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6\" (UID: \"0403729d-8803-40c2-9e5d-1cb2cc9914a1\") " pod="openshift-ingress/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6" Nov 03 13:43:47.612365 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:47.612309 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0403729d-8803-40c2-9e5d-1cb2cc9914a1-istio-envoy\") pod \"e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6\" (UID: \"0403729d-8803-40c2-9e5d-1cb2cc9914a1\") " pod="openshift-ingress/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6" Nov 03 13:43:47.612412 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:47.612373 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0403729d-8803-40c2-9e5d-1cb2cc9914a1-istio-podinfo\") pod \"e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6\" (UID: \"0403729d-8803-40c2-9e5d-1cb2cc9914a1\") " pod="openshift-ingress/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6" Nov 03 13:43:47.612463 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:47.612411 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0403729d-8803-40c2-9e5d-1cb2cc9914a1-workload-certs\") pod \"e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6\" (UID: \"0403729d-8803-40c2-9e5d-1cb2cc9914a1\") " pod="openshift-ingress/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6" Nov 03 13:43:47.612544 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:47.612462 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0403729d-8803-40c2-9e5d-1cb2cc9914a1-workload-socket\") pod \"e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6\" (UID: \"0403729d-8803-40c2-9e5d-1cb2cc9914a1\") " pod="openshift-ingress/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6" Nov 03 13:43:47.612604 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:47.612585 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv96m\" (UniqueName: \"kubernetes.io/projected/0403729d-8803-40c2-9e5d-1cb2cc9914a1-kube-api-access-wv96m\") pod \"e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6\" (UID: \"0403729d-8803-40c2-9e5d-1cb2cc9914a1\") " pod="openshift-ingress/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6" Nov 03 13:43:47.612648 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:47.612628 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0403729d-8803-40c2-9e5d-1cb2cc9914a1-istio-data\") pod \"e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6\" (UID: \"0403729d-8803-40c2-9e5d-1cb2cc9914a1\") " pod="openshift-ingress/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6" Nov 03 13:43:47.612689 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:47.612676 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0403729d-8803-40c2-9e5d-1cb2cc9914a1-credential-socket\") pod \"e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6\" (UID: \"0403729d-8803-40c2-9e5d-1cb2cc9914a1\") " pod="openshift-ingress/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6" Nov 03 13:43:47.612946 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:47.612905 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0403729d-8803-40c2-9e5d-1cb2cc9914a1-istiod-ca-cert\") pod \"e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6\" (UID: \"0403729d-8803-40c2-9e5d-1cb2cc9914a1\") " pod="openshift-ingress/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6" Nov 03 13:43:47.613111 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:47.613085 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0403729d-8803-40c2-9e5d-1cb2cc9914a1-workload-socket\") pod \"e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6\" (UID: \"0403729d-8803-40c2-9e5d-1cb2cc9914a1\") " pod="openshift-ingress/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6" Nov 03 13:43:47.613181 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:47.613117 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0403729d-8803-40c2-9e5d-1cb2cc9914a1-credential-socket\") pod \"e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6\" (UID: \"0403729d-8803-40c2-9e5d-1cb2cc9914a1\") " pod="openshift-ingress/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6" Nov 03 13:43:47.613411 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:47.613387 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0403729d-8803-40c2-9e5d-1cb2cc9914a1-istio-data\") pod \"e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6\" (UID: \"0403729d-8803-40c2-9e5d-1cb2cc9914a1\") " pod="openshift-ingress/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6" Nov 03 13:43:47.613553 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:47.613536 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0403729d-8803-40c2-9e5d-1cb2cc9914a1-workload-certs\") pod \"e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6\" (UID: \"0403729d-8803-40c2-9e5d-1cb2cc9914a1\") " pod="openshift-ingress/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6" Nov 03 13:43:47.615740 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:47.615696 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0403729d-8803-40c2-9e5d-1cb2cc9914a1-istio-envoy\") pod \"e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6\" (UID: \"0403729d-8803-40c2-9e5d-1cb2cc9914a1\") " pod="openshift-ingress/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6" Nov 03 13:43:47.616277 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:47.616251 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0403729d-8803-40c2-9e5d-1cb2cc9914a1-istio-podinfo\") pod \"e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6\" (UID: \"0403729d-8803-40c2-9e5d-1cb2cc9914a1\") " pod="openshift-ingress/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6" Nov 03 13:43:47.673535 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:47.673004 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv96m\" (UniqueName: \"kubernetes.io/projected/0403729d-8803-40c2-9e5d-1cb2cc9914a1-kube-api-access-wv96m\") pod \"e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6\" (UID: \"0403729d-8803-40c2-9e5d-1cb2cc9914a1\") " pod="openshift-ingress/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6" Nov 03 13:43:47.674493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:47.673797 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0403729d-8803-40c2-9e5d-1cb2cc9914a1-istio-token\") pod \"e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6\" (UID: \"0403729d-8803-40c2-9e5d-1cb2cc9914a1\") " pod="openshift-ingress/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6" Nov 03 13:43:47.738902 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:47.738874 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6bd5c1-786d9ddfc5-f8tmc" Nov 03 13:43:47.813546 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:47.813516 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6" Nov 03 13:43:48.291286 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:48.291248 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6"] Nov 03 13:43:48.298121 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:43:48.298097 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0403729d_8803_40c2_9e5d_1cb2cc9914a1.slice/crio-adc778446fb7672951f3a07ad9a691642bb1bb3033c192f934e73715d0005f1c WatchSource:0}: Error finding container adc778446fb7672951f3a07ad9a691642bb1bb3033c192f934e73715d0005f1c: Status 404 returned error can't find the container with id adc778446fb7672951f3a07ad9a691642bb1bb3033c192f934e73715d0005f1c Nov 03 13:43:48.300694 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:48.300664 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:43:48.300787 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:48.300727 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:43:48.300787 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:48.300756 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:43:48.767495 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:48.763613 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6bd5c1-6d9c48cdcd-bshhx" Nov 03 13:43:48.809894 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:48.809850 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6" event={"ID":"0403729d-8803-40c2-9e5d-1cb2cc9914a1","Type":"ContainerStarted","Data":"2d13cb7796b4dd992f24aa0a404f34946439c117c567f652df7148d34bf9bbd0"} Nov 03 13:43:48.810097 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:48.809923 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6" event={"ID":"0403729d-8803-40c2-9e5d-1cb2cc9914a1","Type":"ContainerStarted","Data":"adc778446fb7672951f3a07ad9a691642bb1bb3033c192f934e73715d0005f1c"} Nov 03 13:43:48.813761 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:48.813736 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6" Nov 03 13:43:49.013415 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:49.013284 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6" podStartSLOduration=2.013237141 podStartE2EDuration="2.013237141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:43:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:43:49.007792615 +0000 UTC m=+2311.018480858" watchObservedRunningTime="2026-11-03 13:43:49.013237141 +0000 UTC m=+2311.023925384" Nov 03 13:43:49.092529 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:49.092431 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-007b0d-54546cd87f-5b5kz"] Nov 03 13:43:49.204421 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:49.204380 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-007b0d-7bcff96687-b8kvb"] Nov 03 13:43:49.820315 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:49.820283 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6" Nov 03 13:43:50.821901 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:50.821876 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6" Nov 03 13:43:50.822873 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:50.822857 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6" Nov 03 13:43:54.093727 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:54.093680 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-007b0d-54546cd87f-5b5kz" podUID="84b314b4-86d3-4849-b0a1-35fcd77f2747" containerName="payload-processing" containerID="cri-o://9e565c475fcd1a23216357f97a87809ad8cd598fd07bfadc342c3ec448e56716" gracePeriod=25 Nov 03 13:43:54.205402 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:54.205324 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-007b0d-7bcff96687-b8kvb" podUID="f9454ca7-0414-4da3-a8c6-98b41c84a16e" containerName="payload-pre-processing" containerID="cri-o://c3847467b04e211e87ed0f60cb3f13643e922da7f5bb98093560fbbf02927b81" gracePeriod=25 Nov 03 13:43:54.389729 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:54.388809 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-007b0d-54546cd87f-5b5kz" Nov 03 13:43:54.428910 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:54.428879 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84b314b4-86d3-4849-b0a1-35fcd77f2747-kube-api-access-p26ff\" (UniqueName: \"kubernetes.io/projected/84b314b4-86d3-4849-b0a1-35fcd77f2747-kube-api-access-p26ff\") pod \"84b314b4-86d3-4849-b0a1-35fcd77f2747\" (UID: \"84b314b4-86d3-4849-b0a1-35fcd77f2747\") " Nov 03 13:43:54.429101 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:54.428973 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84b314b4-86d3-4849-b0a1-35fcd77f2747-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84b314b4-86d3-4849-b0a1-35fcd77f2747-plugins-config-volume\") pod \"84b314b4-86d3-4849-b0a1-35fcd77f2747\" (UID: \"84b314b4-86d3-4849-b0a1-35fcd77f2747\") " Nov 03 13:43:54.429698 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:54.429670 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84b314b4-86d3-4849-b0a1-35fcd77f2747-plugins-config-volume" pod "84b314b4-86d3-4849-b0a1-35fcd77f2747" (UID: "84b314b4-86d3-4849-b0a1-35fcd77f2747"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:43:54.431559 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:54.431532 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84b314b4-86d3-4849-b0a1-35fcd77f2747-kube-api-access-p26ff" pod "84b314b4-86d3-4849-b0a1-35fcd77f2747" (UID: "84b314b4-86d3-4849-b0a1-35fcd77f2747"). InnerVolumeSpecName "kube-api-access-p26ff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:43:54.468551 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:54.468527 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-007b0d-7bcff96687-b8kvb" Nov 03 13:43:54.535968 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:54.535903 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9454ca7-0414-4da3-a8c6-98b41c84a16e-kube-api-access-pjkh5\" (UniqueName: \"kubernetes.io/projected/f9454ca7-0414-4da3-a8c6-98b41c84a16e-kube-api-access-pjkh5\") pod \"f9454ca7-0414-4da3-a8c6-98b41c84a16e\" (UID: \"f9454ca7-0414-4da3-a8c6-98b41c84a16e\") " Nov 03 13:43:54.537148 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:54.536829 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9454ca7-0414-4da3-a8c6-98b41c84a16e-plugins-config-volume" pod "f9454ca7-0414-4da3-a8c6-98b41c84a16e" (UID: "f9454ca7-0414-4da3-a8c6-98b41c84a16e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:43:54.537359 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:54.537334 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9454ca7-0414-4da3-a8c6-98b41c84a16e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9454ca7-0414-4da3-a8c6-98b41c84a16e-plugins-config-volume\") pod \"f9454ca7-0414-4da3-a8c6-98b41c84a16e\" (UID: \"f9454ca7-0414-4da3-a8c6-98b41c84a16e\") " Nov 03 13:43:54.538050 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:54.538007 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84b314b4-86d3-4849-b0a1-35fcd77f2747-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:54.538207 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:54.538182 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9454ca7-0414-4da3-a8c6-98b41c84a16e-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:54.538325 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:54.538299 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p26ff\" (UniqueName: \"kubernetes.io/projected/84b314b4-86d3-4849-b0a1-35fcd77f2747-kube-api-access-p26ff\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:54.538778 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:54.538755 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9454ca7-0414-4da3-a8c6-98b41c84a16e-kube-api-access-pjkh5" pod "f9454ca7-0414-4da3-a8c6-98b41c84a16e" (UID: "f9454ca7-0414-4da3-a8c6-98b41c84a16e"). InnerVolumeSpecName "kube-api-access-pjkh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:43:54.639612 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:54.639568 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjkh5\" (UniqueName: \"kubernetes.io/projected/f9454ca7-0414-4da3-a8c6-98b41c84a16e-kube-api-access-pjkh5\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:43:54.851013 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:54.850980 5979 generic.go:356] "Generic (PLEG): container finished" podID="f9454ca7-0414-4da3-a8c6-98b41c84a16e" containerID="c3847467b04e211e87ed0f60cb3f13643e922da7f5bb98093560fbbf02927b81" exitCode=0 Nov 03 13:43:54.851227 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:54.851045 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-007b0d-7bcff96687-b8kvb" Nov 03 13:43:54.851227 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:54.851059 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-007b0d-7bcff96687-b8kvb" event={"ID":"f9454ca7-0414-4da3-a8c6-98b41c84a16e","Type":"ContainerDied","Data":"c3847467b04e211e87ed0f60cb3f13643e922da7f5bb98093560fbbf02927b81"} Nov 03 13:43:54.851227 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:54.851150 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-007b0d-7bcff96687-b8kvb" event={"ID":"f9454ca7-0414-4da3-a8c6-98b41c84a16e","Type":"ContainerDied","Data":"11f9fad8bde64f07ed7477f7e2b794cba65c8cb8d52c10045eede1ae414bb1c7"} Nov 03 13:43:54.851227 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:54.851184 5979 scope.go:122] "RemoveContainer" containerID="c3847467b04e211e87ed0f60cb3f13643e922da7f5bb98093560fbbf02927b81" Nov 03 13:43:54.853156 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:54.853121 5979 generic.go:356] "Generic (PLEG): container finished" podID="84b314b4-86d3-4849-b0a1-35fcd77f2747" containerID="9e565c475fcd1a23216357f97a87809ad8cd598fd07bfadc342c3ec448e56716" exitCode=0 Nov 03 13:43:54.853290 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:54.853179 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-007b0d-54546cd87f-5b5kz" Nov 03 13:43:54.853364 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:54.853296 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-007b0d-54546cd87f-5b5kz" event={"ID":"84b314b4-86d3-4849-b0a1-35fcd77f2747","Type":"ContainerDied","Data":"9e565c475fcd1a23216357f97a87809ad8cd598fd07bfadc342c3ec448e56716"} Nov 03 13:43:54.853364 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:54.853343 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-007b0d-54546cd87f-5b5kz" event={"ID":"84b314b4-86d3-4849-b0a1-35fcd77f2747","Type":"ContainerDied","Data":"7ee9da396297ebae1140af88d5621f8e34ea77dd3c3d2cfdd2839e04cadebced"} Nov 03 13:43:54.865258 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:54.865182 5979 scope.go:122] "RemoveContainer" containerID="c3847467b04e211e87ed0f60cb3f13643e922da7f5bb98093560fbbf02927b81" Nov 03 13:43:54.865564 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:43:54.865528 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3847467b04e211e87ed0f60cb3f13643e922da7f5bb98093560fbbf02927b81\": container with ID starting with c3847467b04e211e87ed0f60cb3f13643e922da7f5bb98093560fbbf02927b81 not found: ID does not exist" containerID="c3847467b04e211e87ed0f60cb3f13643e922da7f5bb98093560fbbf02927b81" Nov 03 13:43:54.865684 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:54.865565 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3847467b04e211e87ed0f60cb3f13643e922da7f5bb98093560fbbf02927b81"} err="failed to get container status \"c3847467b04e211e87ed0f60cb3f13643e922da7f5bb98093560fbbf02927b81\": rpc error: code = NotFound desc = could not find container \"c3847467b04e211e87ed0f60cb3f13643e922da7f5bb98093560fbbf02927b81\": container with ID starting with c3847467b04e211e87ed0f60cb3f13643e922da7f5bb98093560fbbf02927b81 not found: ID does not exist" Nov 03 13:43:54.865684 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:54.865588 5979 scope.go:122] "RemoveContainer" containerID="9e565c475fcd1a23216357f97a87809ad8cd598fd07bfadc342c3ec448e56716" Nov 03 13:43:54.886406 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:54.880605 5979 patch_prober.go:29] interesting pod/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:43:54.886406 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:54.880662 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq" podUID="3a8276f7-206c-402e-8e82-087316e9fc3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:43:54.886825 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:54.886805 5979 scope.go:122] "RemoveContainer" containerID="9e565c475fcd1a23216357f97a87809ad8cd598fd07bfadc342c3ec448e56716" Nov 03 13:43:54.887244 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:43:54.887220 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e565c475fcd1a23216357f97a87809ad8cd598fd07bfadc342c3ec448e56716\": container with ID starting with 9e565c475fcd1a23216357f97a87809ad8cd598fd07bfadc342c3ec448e56716 not found: ID does not exist" containerID="9e565c475fcd1a23216357f97a87809ad8cd598fd07bfadc342c3ec448e56716" Nov 03 13:43:54.887338 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:54.887255 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e565c475fcd1a23216357f97a87809ad8cd598fd07bfadc342c3ec448e56716"} err="failed to get container status \"9e565c475fcd1a23216357f97a87809ad8cd598fd07bfadc342c3ec448e56716\": rpc error: code = NotFound desc = could not find container \"9e565c475fcd1a23216357f97a87809ad8cd598fd07bfadc342c3ec448e56716\": container with ID starting with 9e565c475fcd1a23216357f97a87809ad8cd598fd07bfadc342c3ec448e56716 not found: ID does not exist" Nov 03 13:43:55.022264 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:55.022169 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-007b0d-7bcff96687-b8kvb"] Nov 03 13:43:55.026312 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:55.026271 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-007b0d-7bcff96687-b8kvb"] Nov 03 13:43:55.137882 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:55.137844 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-007b0d-54546cd87f-5b5kz"] Nov 03 13:43:55.149863 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:55.149827 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-007b0d-54546cd87f-5b5kz"] Nov 03 13:43:55.861606 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:55.861524 5979 generic.go:356] "Generic (PLEG): container finished" podID="3a8276f7-206c-402e-8e82-087316e9fc3d" containerID="c4131785842b46468d82a45668859d90988a152807f235d3aafd576451608e2c" exitCode=0 Nov 03 13:43:55.861811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:55.861597 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq" event={"ID":"3a8276f7-206c-402e-8e82-087316e9fc3d","Type":"ContainerDied","Data":"c4131785842b46468d82a45668859d90988a152807f235d3aafd576451608e2c"} Nov 03 13:43:55.862019 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:55.862003 5979 scope.go:122] "RemoveContainer" containerID="c4131785842b46468d82a45668859d90988a152807f235d3aafd576451608e2c" Nov 03 13:43:55.863607 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:55.863579 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:43:55.863726 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:55.863659 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:43:55.863726 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:55.863694 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:43:56.120140 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:56.120106 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84b314b4-86d3-4849-b0a1-35fcd77f2747" path="/var/lib/kubelet/pods/84b314b4-86d3-4849-b0a1-35fcd77f2747/volumes" Nov 03 13:43:56.120734 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:56.120717 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9454ca7-0414-4da3-a8c6-98b41c84a16e" path="/var/lib/kubelet/pods/f9454ca7-0414-4da3-a8c6-98b41c84a16e/volumes" Nov 03 13:43:56.313091 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:56.312996 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ba6fbd-7856699cbc-gpnbs"] Nov 03 13:43:56.314023 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:56.313987 5979 cpu_manager.go:414] "containerMap: removing container" podUID="f9454ca7-0414-4da3-a8c6-98b41c84a16e" containerName="payload-pre-processing" Nov 03 13:43:56.314023 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:56.314018 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9454ca7-0414-4da3-a8c6-98b41c84a16e" containerName="payload-pre-processing" Nov 03 13:43:56.314281 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:56.314076 5979 cpu_manager.go:414] "containerMap: removing container" podUID="84b314b4-86d3-4849-b0a1-35fcd77f2747" containerName="payload-processing" Nov 03 13:43:56.314281 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:56.314089 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84b314b4-86d3-4849-b0a1-35fcd77f2747" containerName="payload-processing" Nov 03 13:43:56.314281 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:56.314221 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="84b314b4-86d3-4849-b0a1-35fcd77f2747" containerName="payload-processing" Nov 03 13:43:56.314281 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:56.314244 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9454ca7-0414-4da3-a8c6-98b41c84a16e" containerName="payload-pre-processing" Nov 03 13:43:56.339781 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:56.338698 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ba6fbd-7856699cbc-gpnbs" Nov 03 13:43:56.366643 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:56.366610 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ba6fbd-7856699cbc-gpnbs"] Nov 03 13:43:56.375754 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:56.374692 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34740dc2-773c-4e06-b540-16752c227ba1-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-ba6fbd-7856699cbc-gpnbs\" (UID: \"34740dc2-773c-4e06-b540-16752c227ba1\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ba6fbd-7856699cbc-gpnbs" Nov 03 13:43:56.376848 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:56.376782 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cc4z\" (UniqueName: \"kubernetes.io/projected/34740dc2-773c-4e06-b540-16752c227ba1-kube-api-access-9cc4z\") pod \"payload-processing-e2e-praxis-b-ba6fbd-7856699cbc-gpnbs\" (UID: \"34740dc2-773c-4e06-b540-16752c227ba1\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ba6fbd-7856699cbc-gpnbs" Nov 03 13:43:56.392524 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:56.390821 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-ba6fbd\"" Nov 03 13:43:56.392524 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:56.391074 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-ba6fbd-dockercfg-fnn58\"" Nov 03 13:43:56.456623 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:56.456588 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ba6fbd-5f664c6899-7p4bb"] Nov 03 13:43:56.460511 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:56.460029 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ba6fbd-5f664c6899-7p4bb" Nov 03 13:43:56.480291 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:56.480235 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34740dc2-773c-4e06-b540-16752c227ba1-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-ba6fbd-7856699cbc-gpnbs\" (UID: \"34740dc2-773c-4e06-b540-16752c227ba1\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ba6fbd-7856699cbc-gpnbs" Nov 03 13:43:56.480565 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:56.480533 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cc4z\" (UniqueName: \"kubernetes.io/projected/34740dc2-773c-4e06-b540-16752c227ba1-kube-api-access-9cc4z\") pod \"payload-processing-e2e-praxis-b-ba6fbd-7856699cbc-gpnbs\" (UID: \"34740dc2-773c-4e06-b540-16752c227ba1\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ba6fbd-7856699cbc-gpnbs" Nov 03 13:43:56.481587 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:56.481555 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34740dc2-773c-4e06-b540-16752c227ba1-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-ba6fbd-7856699cbc-gpnbs\" (UID: \"34740dc2-773c-4e06-b540-16752c227ba1\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ba6fbd-7856699cbc-gpnbs" Nov 03 13:43:56.525753 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:56.525712 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ba6fbd-5f664c6899-7p4bb"] Nov 03 13:43:56.536033 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:56.535990 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cc4z\" (UniqueName: \"kubernetes.io/projected/34740dc2-773c-4e06-b540-16752c227ba1-kube-api-access-9cc4z\") pod \"payload-processing-e2e-praxis-b-ba6fbd-7856699cbc-gpnbs\" (UID: \"34740dc2-773c-4e06-b540-16752c227ba1\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ba6fbd-7856699cbc-gpnbs" Nov 03 13:43:56.582395 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:56.582054 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99mns\" (UniqueName: \"kubernetes.io/projected/9e256eb2-38e4-4f44-b796-4c47201cfe6d-kube-api-access-99mns\") pod \"payload-pre-processing-e2e-praxis-b-ba6fbd-5f664c6899-7p4bb\" (UID: \"9e256eb2-38e4-4f44-b796-4c47201cfe6d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ba6fbd-5f664c6899-7p4bb" Nov 03 13:43:56.582395 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:56.582196 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e256eb2-38e4-4f44-b796-4c47201cfe6d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-ba6fbd-5f664c6899-7p4bb\" (UID: \"9e256eb2-38e4-4f44-b796-4c47201cfe6d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ba6fbd-5f664c6899-7p4bb" Nov 03 13:43:56.677428 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:56.677390 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ba6fbd-7856699cbc-gpnbs" Nov 03 13:43:56.683982 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:56.683956 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99mns\" (UniqueName: \"kubernetes.io/projected/9e256eb2-38e4-4f44-b796-4c47201cfe6d-kube-api-access-99mns\") pod \"payload-pre-processing-e2e-praxis-b-ba6fbd-5f664c6899-7p4bb\" (UID: \"9e256eb2-38e4-4f44-b796-4c47201cfe6d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ba6fbd-5f664c6899-7p4bb" Nov 03 13:43:56.684111 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:56.684071 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e256eb2-38e4-4f44-b796-4c47201cfe6d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-ba6fbd-5f664c6899-7p4bb\" (UID: \"9e256eb2-38e4-4f44-b796-4c47201cfe6d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ba6fbd-5f664c6899-7p4bb" Nov 03 13:43:56.684769 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:56.684749 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e256eb2-38e4-4f44-b796-4c47201cfe6d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-ba6fbd-5f664c6899-7p4bb\" (UID: \"9e256eb2-38e4-4f44-b796-4c47201cfe6d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ba6fbd-5f664c6899-7p4bb" Nov 03 13:43:56.740554 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:56.740522 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99mns\" (UniqueName: \"kubernetes.io/projected/9e256eb2-38e4-4f44-b796-4c47201cfe6d-kube-api-access-99mns\") pod \"payload-pre-processing-e2e-praxis-b-ba6fbd-5f664c6899-7p4bb\" (UID: \"9e256eb2-38e4-4f44-b796-4c47201cfe6d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ba6fbd-5f664c6899-7p4bb" Nov 03 13:43:56.777713 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:56.777672 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ba6fbd-5f664c6899-7p4bb" Nov 03 13:43:56.882783 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:56.882744 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq" event={"ID":"3a8276f7-206c-402e-8e82-087316e9fc3d","Type":"ContainerStarted","Data":"a363512eebfb6d0d498fe495407394c03d37a8119a9ae67361bfa781fdcac1fc"} Nov 03 13:43:56.997640 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:56.997608 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq" Nov 03 13:43:56.997872 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:56.997850 5979 patch_prober.go:29] interesting pod/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Nov 03 13:43:56.997919 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:56.997896 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq" podUID="3a8276f7-206c-402e-8e82-087316e9fc3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Nov 03 13:43:57.079000 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:57.078963 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ba6fbd-7856699cbc-gpnbs"] Nov 03 13:43:57.103378 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:43:57.103337 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34740dc2_773c_4e06_b540_16752c227ba1.slice/crio-afb1ef657d0f4624a59123f4c2c004e29adcc28d02d00bfe8c43d86548a3ded5 WatchSource:0}: Error finding container afb1ef657d0f4624a59123f4c2c004e29adcc28d02d00bfe8c43d86548a3ded5: Status 404 returned error can't find the container with id afb1ef657d0f4624a59123f4c2c004e29adcc28d02d00bfe8c43d86548a3ded5 Nov 03 13:43:57.185656 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:57.185620 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ba6fbd-5f664c6899-7p4bb"] Nov 03 13:43:57.191109 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:43:57.191083 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e256eb2_38e4_4f44_b796_4c47201cfe6d.slice/crio-ea0dd2a634e5fc62197dedb0a092039f11b7395111919fba6b7e81710978cd0b WatchSource:0}: Error finding container ea0dd2a634e5fc62197dedb0a092039f11b7395111919fba6b7e81710978cd0b: Status 404 returned error can't find the container with id ea0dd2a634e5fc62197dedb0a092039f11b7395111919fba6b7e81710978cd0b Nov 03 13:43:57.899642 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:57.899594 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ba6fbd-7856699cbc-gpnbs" event={"ID":"34740dc2-773c-4e06-b540-16752c227ba1","Type":"ContainerStarted","Data":"c33486f9ff5339fc81c3d2593782c8b670010ca46858abc0b4a4badd63e0accb"} Nov 03 13:43:57.900173 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:57.899812 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-ba6fbd-7856699cbc-gpnbs" Nov 03 13:43:57.900173 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:57.899858 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ba6fbd-7856699cbc-gpnbs" event={"ID":"34740dc2-773c-4e06-b540-16752c227ba1","Type":"ContainerStarted","Data":"afb1ef657d0f4624a59123f4c2c004e29adcc28d02d00bfe8c43d86548a3ded5"} Nov 03 13:43:57.901642 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:57.901613 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ba6fbd-5f664c6899-7p4bb" event={"ID":"9e256eb2-38e4-4f44-b796-4c47201cfe6d","Type":"ContainerStarted","Data":"2e4b81d6c17e69597f9c7b6165c6ac61163e6a25b370e4415fc8727ddbbcaa6c"} Nov 03 13:43:57.901786 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:57.901669 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ba6fbd-5f664c6899-7p4bb" event={"ID":"9e256eb2-38e4-4f44-b796-4c47201cfe6d","Type":"ContainerStarted","Data":"ea0dd2a634e5fc62197dedb0a092039f11b7395111919fba6b7e81710978cd0b"} Nov 03 13:43:57.997975 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:57.997939 5979 patch_prober.go:29] interesting pod/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Nov 03 13:43:57.998167 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:57.997996 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq" podUID="3a8276f7-206c-402e-8e82-087316e9fc3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Nov 03 13:43:58.069840 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:58.069761 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-ba6fbd-7856699cbc-gpnbs" podStartSLOduration=2.069740611 podStartE2EDuration="2.069740611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:43:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:43:58.063870822 +0000 UTC m=+2320.074559065" watchObservedRunningTime="2026-11-03 13:43:58.069740611 +0000 UTC m=+2320.080428858" Nov 03 13:43:58.192720 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:58.192583 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ba6fbd-5f664c6899-7p4bb" podStartSLOduration=2.192559211 podStartE2EDuration="2.192559211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:43:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:43:58.187712689 +0000 UTC m=+2320.198400953" watchObservedRunningTime="2026-11-03 13:43:58.192559211 +0000 UTC m=+2320.203247469" Nov 03 13:43:58.909362 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:58.909321 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ba6fbd-5f664c6899-7p4bb" Nov 03 13:43:58.998423 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:58.998378 5979 patch_prober.go:29] interesting pod/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Nov 03 13:43:58.998663 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:58.998447 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq" podUID="3a8276f7-206c-402e-8e82-087316e9fc3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Nov 03 13:43:59.997997 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:59.997959 5979 patch_prober.go:29] interesting pod/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Nov 03 13:43:59.998409 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:43:59.998058 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq" podUID="3a8276f7-206c-402e-8e82-087316e9fc3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Nov 03 13:44:00.998055 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:00.998019 5979 patch_prober.go:29] interesting pod/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Nov 03 13:44:00.998459 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:00.998084 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq" podUID="3a8276f7-206c-402e-8e82-087316e9fc3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Nov 03 13:44:01.998330 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:01.998294 5979 patch_prober.go:29] interesting pod/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Nov 03 13:44:01.998817 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:01.998353 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq" podUID="3a8276f7-206c-402e-8e82-087316e9fc3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Nov 03 13:44:03.001595 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:03.001561 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq" Nov 03 13:44:03.002107 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:03.001998 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq" Nov 03 13:44:03.002713 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:03.002691 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq" Nov 03 13:44:07.234548 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:44:07.234507 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:44:08.913043 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:08.913011 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-ba6fbd-7856699cbc-gpnbs" Nov 03 13:44:09.918370 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:09.918338 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ba6fbd-5f664c6899-7p4bb" Nov 03 13:44:14.851807 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:14.851772 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65aae740-8thwr"] Nov 03 13:44:14.854870 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:14.854849 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65aae740-8thwr" Nov 03 13:44:14.905978 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:14.905950 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:44:14.937965 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:14.937933 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46aeee7c-881c-4bb5-8f0c-99cda4c0f2d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65aae740-8thwr\" (UID: \"46aeee7c-881c-4bb5-8f0c-99cda4c0f2d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65aae740-8thwr" Nov 03 13:44:14.938186 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:14.938161 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf4kp\" (UniqueName: \"kubernetes.io/projected/46aeee7c-881c-4bb5-8f0c-99cda4c0f2d9-kube-api-access-zf4kp\") pod \"maas-api-revoke-sub-65aae740-8thwr\" (UID: \"46aeee7c-881c-4bb5-8f0c-99cda4c0f2d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65aae740-8thwr" Nov 03 13:44:14.938316 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:14.938225 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46aeee7c-881c-4bb5-8f0c-99cda4c0f2d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65aae740-8thwr\" (UID: \"46aeee7c-881c-4bb5-8f0c-99cda4c0f2d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65aae740-8thwr" Nov 03 13:44:14.951768 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:14.951742 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:44:14.954378 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:14.954356 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65aae740-8thwr"] Nov 03 13:44:15.039412 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:15.039376 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46aeee7c-881c-4bb5-8f0c-99cda4c0f2d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65aae740-8thwr\" (UID: \"46aeee7c-881c-4bb5-8f0c-99cda4c0f2d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65aae740-8thwr" Nov 03 13:44:15.039606 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:15.039438 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf4kp\" (UniqueName: \"kubernetes.io/projected/46aeee7c-881c-4bb5-8f0c-99cda4c0f2d9-kube-api-access-zf4kp\") pod \"maas-api-revoke-sub-65aae740-8thwr\" (UID: \"46aeee7c-881c-4bb5-8f0c-99cda4c0f2d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65aae740-8thwr" Nov 03 13:44:15.039606 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:15.039508 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46aeee7c-881c-4bb5-8f0c-99cda4c0f2d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65aae740-8thwr\" (UID: \"46aeee7c-881c-4bb5-8f0c-99cda4c0f2d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65aae740-8thwr" Nov 03 13:44:15.040028 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:15.040004 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46aeee7c-881c-4bb5-8f0c-99cda4c0f2d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65aae740-8thwr\" (UID: \"46aeee7c-881c-4bb5-8f0c-99cda4c0f2d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65aae740-8thwr" Nov 03 13:44:15.041799 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:15.041775 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46aeee7c-881c-4bb5-8f0c-99cda4c0f2d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65aae740-8thwr\" (UID: \"46aeee7c-881c-4bb5-8f0c-99cda4c0f2d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65aae740-8thwr" Nov 03 13:44:15.093654 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:15.093628 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf4kp\" (UniqueName: \"kubernetes.io/projected/46aeee7c-881c-4bb5-8f0c-99cda4c0f2d9-kube-api-access-zf4kp\") pod \"maas-api-revoke-sub-65aae740-8thwr\" (UID: \"46aeee7c-881c-4bb5-8f0c-99cda4c0f2d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65aae740-8thwr" Nov 03 13:44:15.167602 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:15.167567 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65aae740-8thwr" Nov 03 13:44:15.541210 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:15.541110 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65aae740-8thwr"] Nov 03 13:44:15.549338 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:44:15.549308 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46aeee7c_881c_4bb5_8f0c_99cda4c0f2d9.slice/crio-5614bdd7844bb5232d3eb13175ee1f95cedfae41aa8e081e2774c31b76fcf3ce WatchSource:0}: Error finding container 5614bdd7844bb5232d3eb13175ee1f95cedfae41aa8e081e2774c31b76fcf3ce: Status 404 returned error can't find the container with id 5614bdd7844bb5232d3eb13175ee1f95cedfae41aa8e081e2774c31b76fcf3ce Nov 03 13:44:16.019552 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:16.019499 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65aae740-8thwr" event={"ID":"46aeee7c-881c-4bb5-8f0c-99cda4c0f2d9","Type":"ContainerStarted","Data":"93d89ac1509414e580d842e91d4271f2e51452aa1e7e3a54ed5911241e1529f2"} Nov 03 13:44:16.019988 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:16.019567 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65aae740-8thwr" event={"ID":"46aeee7c-881c-4bb5-8f0c-99cda4c0f2d9","Type":"ContainerStarted","Data":"5614bdd7844bb5232d3eb13175ee1f95cedfae41aa8e081e2774c31b76fcf3ce"} Nov 03 13:44:16.156386 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:16.156336 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65aae740-8thwr" podStartSLOduration=2.156323005 podStartE2EDuration="2.156323005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:44:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:44:16.155769004 +0000 UTC m=+2338.166457246" watchObservedRunningTime="2026-11-03 13:44:16.156323005 +0000 UTC m=+2338.167011246" Nov 03 13:44:17.027560 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:17.027526 5979 generic.go:356] "Generic (PLEG): container finished" podID="46aeee7c-881c-4bb5-8f0c-99cda4c0f2d9" containerID="93d89ac1509414e580d842e91d4271f2e51452aa1e7e3a54ed5911241e1529f2" exitCode=0 Nov 03 13:44:17.027956 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:17.027563 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65aae740-8thwr" event={"ID":"46aeee7c-881c-4bb5-8f0c-99cda4c0f2d9","Type":"ContainerDied","Data":"93d89ac1509414e580d842e91d4271f2e51452aa1e7e3a54ed5911241e1529f2"} Nov 03 13:44:18.175165 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:18.175138 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65aae740-8thwr" Nov 03 13:44:18.184764 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:18.184738 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46aeee7c-881c-4bb5-8f0c-99cda4c0f2d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46aeee7c-881c-4bb5-8f0c-99cda4c0f2d9-maas-api-openshift-ca\") pod \"46aeee7c-881c-4bb5-8f0c-99cda4c0f2d9\" (UID: \"46aeee7c-881c-4bb5-8f0c-99cda4c0f2d9\") " Nov 03 13:44:18.184894 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:18.184862 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46aeee7c-881c-4bb5-8f0c-99cda4c0f2d9-kube-api-access-zf4kp\" (UniqueName: \"kubernetes.io/projected/46aeee7c-881c-4bb5-8f0c-99cda4c0f2d9-kube-api-access-zf4kp\") pod \"46aeee7c-881c-4bb5-8f0c-99cda4c0f2d9\" (UID: \"46aeee7c-881c-4bb5-8f0c-99cda4c0f2d9\") " Nov 03 13:44:18.184962 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:18.184941 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46aeee7c-881c-4bb5-8f0c-99cda4c0f2d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46aeee7c-881c-4bb5-8f0c-99cda4c0f2d9-maas-api-xks-ca\") pod \"46aeee7c-881c-4bb5-8f0c-99cda4c0f2d9\" (UID: \"46aeee7c-881c-4bb5-8f0c-99cda4c0f2d9\") " Nov 03 13:44:18.185114 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:18.185091 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46aeee7c-881c-4bb5-8f0c-99cda4c0f2d9-maas-api-openshift-ca" pod "46aeee7c-881c-4bb5-8f0c-99cda4c0f2d9" (UID: "46aeee7c-881c-4bb5-8f0c-99cda4c0f2d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:44:18.185356 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:18.185340 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46aeee7c-881c-4bb5-8f0c-99cda4c0f2d9-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:18.186894 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:18.186869 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46aeee7c-881c-4bb5-8f0c-99cda4c0f2d9-kube-api-access-zf4kp" pod "46aeee7c-881c-4bb5-8f0c-99cda4c0f2d9" (UID: "46aeee7c-881c-4bb5-8f0c-99cda4c0f2d9"). InnerVolumeSpecName "kube-api-access-zf4kp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:44:18.186978 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:18.186906 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46aeee7c-881c-4bb5-8f0c-99cda4c0f2d9-maas-api-xks-ca" pod "46aeee7c-881c-4bb5-8f0c-99cda4c0f2d9" (UID: "46aeee7c-881c-4bb5-8f0c-99cda4c0f2d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:44:18.286430 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:18.286336 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46aeee7c-881c-4bb5-8f0c-99cda4c0f2d9-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:18.286430 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:18.286381 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zf4kp\" (UniqueName: \"kubernetes.io/projected/46aeee7c-881c-4bb5-8f0c-99cda4c0f2d9-kube-api-access-zf4kp\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:19.041631 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:19.041597 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65aae740-8thwr" event={"ID":"46aeee7c-881c-4bb5-8f0c-99cda4c0f2d9","Type":"ContainerDied","Data":"5614bdd7844bb5232d3eb13175ee1f95cedfae41aa8e081e2774c31b76fcf3ce"} Nov 03 13:44:19.041905 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:19.041652 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5614bdd7844bb5232d3eb13175ee1f95cedfae41aa8e081e2774c31b76fcf3ce" Nov 03 13:44:19.041905 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:19.041610 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65aae740-8thwr" Nov 03 13:44:25.293070 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:25.293039 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-a770b9-774d768db-9zksn_5a9531dd-b724-4801-ac0d-71d42b7b3868/payload-processing/0.log" Nov 03 13:44:26.231215 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:26.231182 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-32bec3-678877f9d4-qjwmv_0f6351e2-84a8-414e-a0d8-dd394abec66a/payload-processing/0.log" Nov 03 13:44:26.480878 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:26.480847 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-32bec3-678877f9d4-qjwmv_0f6351e2-84a8-414e-a0d8-dd394abec66a/payload-processing/0.log" Nov 03 13:44:33.052908 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:33.052858 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1e4e03c-dpjrz"] Nov 03 13:44:33.055909 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:33.055876 5979 cpu_manager.go:414] "containerMap: removing container" podUID="46aeee7c-881c-4bb5-8f0c-99cda4c0f2d9" containerName="revoke-keys" Nov 03 13:44:33.056138 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:33.056123 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46aeee7c-881c-4bb5-8f0c-99cda4c0f2d9" containerName="revoke-keys" Nov 03 13:44:33.056534 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:33.056517 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="46aeee7c-881c-4bb5-8f0c-99cda4c0f2d9" containerName="revoke-keys" Nov 03 13:44:33.065184 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:33.065156 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1e4e03c-dpjrz" Nov 03 13:44:33.113554 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:33.113525 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1e4e03c-dpjrz"] Nov 03 13:44:33.156959 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:33.156929 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:44:33.157144 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:33.157041 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:44:33.176552 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:33.176451 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nlc5\" (UniqueName: \"kubernetes.io/projected/2fa25a94-fb23-4745-b999-e77d4c54a035-kube-api-access-6nlc5\") pod \"maas-api-revoke-sub-a1e4e03c-dpjrz\" (UID: \"2fa25a94-fb23-4745-b999-e77d4c54a035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1e4e03c-dpjrz" Nov 03 13:44:33.176873 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:33.176848 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fa25a94-fb23-4745-b999-e77d4c54a035-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1e4e03c-dpjrz\" (UID: \"2fa25a94-fb23-4745-b999-e77d4c54a035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1e4e03c-dpjrz" Nov 03 13:44:33.177031 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:33.177016 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fa25a94-fb23-4745-b999-e77d4c54a035-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1e4e03c-dpjrz\" (UID: \"2fa25a94-fb23-4745-b999-e77d4c54a035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1e4e03c-dpjrz" Nov 03 13:44:33.278579 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:33.278544 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fa25a94-fb23-4745-b999-e77d4c54a035-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1e4e03c-dpjrz\" (UID: \"2fa25a94-fb23-4745-b999-e77d4c54a035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1e4e03c-dpjrz" Nov 03 13:44:33.278818 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:33.278796 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fa25a94-fb23-4745-b999-e77d4c54a035-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1e4e03c-dpjrz\" (UID: \"2fa25a94-fb23-4745-b999-e77d4c54a035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1e4e03c-dpjrz" Nov 03 13:44:33.278923 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:33.278906 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nlc5\" (UniqueName: \"kubernetes.io/projected/2fa25a94-fb23-4745-b999-e77d4c54a035-kube-api-access-6nlc5\") pod \"maas-api-revoke-sub-a1e4e03c-dpjrz\" (UID: \"2fa25a94-fb23-4745-b999-e77d4c54a035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1e4e03c-dpjrz" Nov 03 13:44:33.279580 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:33.279552 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fa25a94-fb23-4745-b999-e77d4c54a035-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1e4e03c-dpjrz\" (UID: \"2fa25a94-fb23-4745-b999-e77d4c54a035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1e4e03c-dpjrz" Nov 03 13:44:33.280996 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:33.280967 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fa25a94-fb23-4745-b999-e77d4c54a035-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1e4e03c-dpjrz\" (UID: \"2fa25a94-fb23-4745-b999-e77d4c54a035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1e4e03c-dpjrz" Nov 03 13:44:33.333185 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:33.333114 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nlc5\" (UniqueName: \"kubernetes.io/projected/2fa25a94-fb23-4745-b999-e77d4c54a035-kube-api-access-6nlc5\") pod \"maas-api-revoke-sub-a1e4e03c-dpjrz\" (UID: \"2fa25a94-fb23-4745-b999-e77d4c54a035\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1e4e03c-dpjrz" Nov 03 13:44:33.380197 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:33.380161 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1e4e03c-dpjrz" Nov 03 13:44:33.761729 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:33.761696 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1e4e03c-dpjrz"] Nov 03 13:44:33.768226 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:44:33.768199 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fa25a94_fb23_4745_b999_e77d4c54a035.slice/crio-4f14ec212616a3ac1a15050d6bec2d0b487bebf4d60a8e01c649cb3f6ca097ee WatchSource:0}: Error finding container 4f14ec212616a3ac1a15050d6bec2d0b487bebf4d60a8e01c649cb3f6ca097ee: Status 404 returned error can't find the container with id 4f14ec212616a3ac1a15050d6bec2d0b487bebf4d60a8e01c649cb3f6ca097ee Nov 03 13:44:34.139401 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:34.139367 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1e4e03c-dpjrz" event={"ID":"2fa25a94-fb23-4745-b999-e77d4c54a035","Type":"ContainerStarted","Data":"1dd465655c278c52f3518ff064040817b4a706ec664c3200dac644078b84044f"} Nov 03 13:44:34.139802 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:34.139417 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1e4e03c-dpjrz" event={"ID":"2fa25a94-fb23-4745-b999-e77d4c54a035","Type":"ContainerStarted","Data":"4f14ec212616a3ac1a15050d6bec2d0b487bebf4d60a8e01c649cb3f6ca097ee"} Nov 03 13:44:35.149813 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:35.149783 5979 generic.go:356] "Generic (PLEG): container finished" podID="2fa25a94-fb23-4745-b999-e77d4c54a035" containerID="1dd465655c278c52f3518ff064040817b4a706ec664c3200dac644078b84044f" exitCode=0 Nov 03 13:44:35.150233 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:35.149867 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1e4e03c-dpjrz" event={"ID":"2fa25a94-fb23-4745-b999-e77d4c54a035","Type":"ContainerDied","Data":"1dd465655c278c52f3518ff064040817b4a706ec664c3200dac644078b84044f"} Nov 03 13:44:36.304361 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:36.304334 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1e4e03c-dpjrz" Nov 03 13:44:36.417072 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:36.416860 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fa25a94-fb23-4745-b999-e77d4c54a035-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fa25a94-fb23-4745-b999-e77d4c54a035-maas-api-openshift-ca\") pod \"2fa25a94-fb23-4745-b999-e77d4c54a035\" (UID: \"2fa25a94-fb23-4745-b999-e77d4c54a035\") " Nov 03 13:44:36.417072 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:36.416979 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fa25a94-fb23-4745-b999-e77d4c54a035-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fa25a94-fb23-4745-b999-e77d4c54a035-maas-api-xks-ca\") pod \"2fa25a94-fb23-4745-b999-e77d4c54a035\" (UID: \"2fa25a94-fb23-4745-b999-e77d4c54a035\") " Nov 03 13:44:36.418069 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:36.417113 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fa25a94-fb23-4745-b999-e77d4c54a035-kube-api-access-6nlc5\" (UniqueName: \"kubernetes.io/projected/2fa25a94-fb23-4745-b999-e77d4c54a035-kube-api-access-6nlc5\") pod \"2fa25a94-fb23-4745-b999-e77d4c54a035\" (UID: \"2fa25a94-fb23-4745-b999-e77d4c54a035\") " Nov 03 13:44:36.418069 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:36.417608 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fa25a94-fb23-4745-b999-e77d4c54a035-maas-api-openshift-ca" pod "2fa25a94-fb23-4745-b999-e77d4c54a035" (UID: "2fa25a94-fb23-4745-b999-e77d4c54a035"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:44:36.418242 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:36.418217 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fa25a94-fb23-4745-b999-e77d4c54a035-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:36.419924 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:36.419897 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fa25a94-fb23-4745-b999-e77d4c54a035-maas-api-xks-ca" pod "2fa25a94-fb23-4745-b999-e77d4c54a035" (UID: "2fa25a94-fb23-4745-b999-e77d4c54a035"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:44:36.420867 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:36.420825 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fa25a94-fb23-4745-b999-e77d4c54a035-kube-api-access-6nlc5" pod "2fa25a94-fb23-4745-b999-e77d4c54a035" (UID: "2fa25a94-fb23-4745-b999-e77d4c54a035"). InnerVolumeSpecName "kube-api-access-6nlc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:44:36.519963 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:36.519922 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fa25a94-fb23-4745-b999-e77d4c54a035-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:36.519963 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:36.519962 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nlc5\" (UniqueName: \"kubernetes.io/projected/2fa25a94-fb23-4745-b999-e77d4c54a035-kube-api-access-6nlc5\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:37.162568 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:37.162535 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1e4e03c-dpjrz" Nov 03 13:44:37.162768 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:37.162572 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1e4e03c-dpjrz" event={"ID":"2fa25a94-fb23-4745-b999-e77d4c54a035","Type":"ContainerDied","Data":"4f14ec212616a3ac1a15050d6bec2d0b487bebf4d60a8e01c649cb3f6ca097ee"} Nov 03 13:44:37.162768 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:37.162618 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f14ec212616a3ac1a15050d6bec2d0b487bebf4d60a8e01c649cb3f6ca097ee" Nov 03 13:44:37.234852 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:44:37.234813 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:44:40.518622 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:40.518589 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:44:42.897941 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:42.897906 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6bd5c1-565d8c4b9-9shxg"] Nov 03 13:44:42.898337 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:42.898113 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6bd5c1-565d8c4b9-9shxg" podUID="5aa090e7-2473-4ecf-b8f9-d48182cefbd6" containerName="maas-api" containerID="cri-o://dcc5f881440c295b18c0d52ff88aa73887e90de29ee1509c7cda4e4b1a9a91db" gracePeriod=30 Nov 03 13:44:43.202765 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:43.202737 5979 generic.go:356] "Generic (PLEG): container finished" podID="5aa090e7-2473-4ecf-b8f9-d48182cefbd6" containerID="dcc5f881440c295b18c0d52ff88aa73887e90de29ee1509c7cda4e4b1a9a91db" exitCode=0 Nov 03 13:44:43.202903 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:43.202814 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6bd5c1-565d8c4b9-9shxg" event={"ID":"5aa090e7-2473-4ecf-b8f9-d48182cefbd6","Type":"ContainerDied","Data":"dcc5f881440c295b18c0d52ff88aa73887e90de29ee1509c7cda4e4b1a9a91db"} Nov 03 13:44:43.202903 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:43.202874 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6bd5c1-565d8c4b9-9shxg" event={"ID":"5aa090e7-2473-4ecf-b8f9-d48182cefbd6","Type":"ContainerDied","Data":"49e831f76c3247f5a3720f49480d7e43d73e67c00da33f5be66aa48b4e8cc9a1"} Nov 03 13:44:43.202903 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:43.202897 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49e831f76c3247f5a3720f49480d7e43d73e67c00da33f5be66aa48b4e8cc9a1" Nov 03 13:44:43.215113 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:43.215087 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6bd5c1-565d8c4b9-9shxg" Nov 03 13:44:43.300046 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:43.300015 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5aa090e7-2473-4ecf-b8f9-d48182cefbd6-kube-api-access-hqmnj\" (UniqueName: \"kubernetes.io/projected/5aa090e7-2473-4ecf-b8f9-d48182cefbd6-kube-api-access-hqmnj\") pod \"5aa090e7-2473-4ecf-b8f9-d48182cefbd6\" (UID: \"5aa090e7-2473-4ecf-b8f9-d48182cefbd6\") " Nov 03 13:44:43.300240 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:43.300121 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5aa090e7-2473-4ecf-b8f9-d48182cefbd6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5aa090e7-2473-4ecf-b8f9-d48182cefbd6-metrics-certs\") pod \"5aa090e7-2473-4ecf-b8f9-d48182cefbd6\" (UID: \"5aa090e7-2473-4ecf-b8f9-d48182cefbd6\") " Nov 03 13:44:43.300240 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:43.300227 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5aa090e7-2473-4ecf-b8f9-d48182cefbd6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5aa090e7-2473-4ecf-b8f9-d48182cefbd6-maas-api-tls\") pod \"5aa090e7-2473-4ecf-b8f9-d48182cefbd6\" (UID: \"5aa090e7-2473-4ecf-b8f9-d48182cefbd6\") " Nov 03 13:44:43.300371 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:43.300305 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5aa090e7-2473-4ecf-b8f9-d48182cefbd6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5aa090e7-2473-4ecf-b8f9-d48182cefbd6-openshift-service-ca\") pod \"5aa090e7-2473-4ecf-b8f9-d48182cefbd6\" (UID: \"5aa090e7-2473-4ecf-b8f9-d48182cefbd6\") " Nov 03 13:44:43.300686 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:43.300657 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5aa090e7-2473-4ecf-b8f9-d48182cefbd6-openshift-service-ca" pod "5aa090e7-2473-4ecf-b8f9-d48182cefbd6" (UID: "5aa090e7-2473-4ecf-b8f9-d48182cefbd6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:44:43.300834 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:43.300790 5979 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5aa090e7-2473-4ecf-b8f9-d48182cefbd6-openshift-service-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:43.302260 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:43.302231 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5aa090e7-2473-4ecf-b8f9-d48182cefbd6-maas-api-tls" pod "5aa090e7-2473-4ecf-b8f9-d48182cefbd6" (UID: "5aa090e7-2473-4ecf-b8f9-d48182cefbd6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:44:43.302360 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:43.302252 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5aa090e7-2473-4ecf-b8f9-d48182cefbd6-metrics-certs" pod "5aa090e7-2473-4ecf-b8f9-d48182cefbd6" (UID: "5aa090e7-2473-4ecf-b8f9-d48182cefbd6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:44:43.302360 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:43.302299 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5aa090e7-2473-4ecf-b8f9-d48182cefbd6-kube-api-access-hqmnj" pod "5aa090e7-2473-4ecf-b8f9-d48182cefbd6" (UID: "5aa090e7-2473-4ecf-b8f9-d48182cefbd6"). InnerVolumeSpecName "kube-api-access-hqmnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:44:43.402364 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:43.402332 5979 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5aa090e7-2473-4ecf-b8f9-d48182cefbd6-metrics-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:43.402364 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:43.402361 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5aa090e7-2473-4ecf-b8f9-d48182cefbd6-maas-api-tls\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:43.402364 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:43.402372 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqmnj\" (UniqueName: \"kubernetes.io/projected/5aa090e7-2473-4ecf-b8f9-d48182cefbd6-kube-api-access-hqmnj\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:43.417674 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:43.417646 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6bd5c1-6d9c48cdcd-bshhx"] Nov 03 13:44:43.417883 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:43.417858 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-6bd5c1-6d9c48cdcd-bshhx" podUID="80c6a028-1d5c-4ba1-84b6-113c617c5cb7" containerName="payload-processing" containerID="cri-o://9f5c6212882eb9e93671539c67893708f6a73845b1c51ae4d665883beba4d441" gracePeriod=30 Nov 03 13:44:43.492939 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:43.492868 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6bd5c1-786d9ddfc5-f8tmc"] Nov 03 13:44:43.493088 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:43.493062 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6bd5c1-786d9ddfc5-f8tmc" podUID="07f7ad33-7cfb-484e-a6e2-d048b6f7b385" containerName="payload-pre-processing" containerID="cri-o://568a939c3c98fc958fb2817431b277a464220498a4c2d230b56e05a6a018bebb" gracePeriod=30 Nov 03 13:44:43.747288 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:43.747199 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6bd5c1-6d9c48cdcd-bshhx" Nov 03 13:44:43.779252 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:43.779227 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6bd5c1-786d9ddfc5-f8tmc" Nov 03 13:44:43.808729 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:43.808693 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80c6a028-1d5c-4ba1-84b6-113c617c5cb7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80c6a028-1d5c-4ba1-84b6-113c617c5cb7-plugins-config-volume\") pod \"80c6a028-1d5c-4ba1-84b6-113c617c5cb7\" (UID: \"80c6a028-1d5c-4ba1-84b6-113c617c5cb7\") " Nov 03 13:44:43.808866 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:43.808755 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80c6a028-1d5c-4ba1-84b6-113c617c5cb7-kube-api-access-mq72d\" (UniqueName: \"kubernetes.io/projected/80c6a028-1d5c-4ba1-84b6-113c617c5cb7-kube-api-access-mq72d\") pod \"80c6a028-1d5c-4ba1-84b6-113c617c5cb7\" (UID: \"80c6a028-1d5c-4ba1-84b6-113c617c5cb7\") " Nov 03 13:44:43.808920 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:43.808883 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07f7ad33-7cfb-484e-a6e2-d048b6f7b385-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07f7ad33-7cfb-484e-a6e2-d048b6f7b385-plugins-config-volume\") pod \"07f7ad33-7cfb-484e-a6e2-d048b6f7b385\" (UID: \"07f7ad33-7cfb-484e-a6e2-d048b6f7b385\") " Nov 03 13:44:43.808964 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:43.808935 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07f7ad33-7cfb-484e-a6e2-d048b6f7b385-kube-api-access-9t9nf\" (UniqueName: \"kubernetes.io/projected/07f7ad33-7cfb-484e-a6e2-d048b6f7b385-kube-api-access-9t9nf\") pod \"07f7ad33-7cfb-484e-a6e2-d048b6f7b385\" (UID: \"07f7ad33-7cfb-484e-a6e2-d048b6f7b385\") " Nov 03 13:44:43.809236 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:43.809198 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80c6a028-1d5c-4ba1-84b6-113c617c5cb7-plugins-config-volume" pod "80c6a028-1d5c-4ba1-84b6-113c617c5cb7" (UID: "80c6a028-1d5c-4ba1-84b6-113c617c5cb7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:44:43.809364 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:43.809282 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07f7ad33-7cfb-484e-a6e2-d048b6f7b385-plugins-config-volume" pod "07f7ad33-7cfb-484e-a6e2-d048b6f7b385" (UID: "07f7ad33-7cfb-484e-a6e2-d048b6f7b385"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:44:43.810959 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:43.810940 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07f7ad33-7cfb-484e-a6e2-d048b6f7b385-kube-api-access-9t9nf" pod "07f7ad33-7cfb-484e-a6e2-d048b6f7b385" (UID: "07f7ad33-7cfb-484e-a6e2-d048b6f7b385"). InnerVolumeSpecName "kube-api-access-9t9nf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:44:43.811021 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:43.810962 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80c6a028-1d5c-4ba1-84b6-113c617c5cb7-kube-api-access-mq72d" pod "80c6a028-1d5c-4ba1-84b6-113c617c5cb7" (UID: "80c6a028-1d5c-4ba1-84b6-113c617c5cb7"). InnerVolumeSpecName "kube-api-access-mq72d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:44:43.910992 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:43.910948 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80c6a028-1d5c-4ba1-84b6-113c617c5cb7-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:43.910992 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:43.910978 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq72d\" (UniqueName: \"kubernetes.io/projected/80c6a028-1d5c-4ba1-84b6-113c617c5cb7-kube-api-access-mq72d\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:43.910992 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:43.910988 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07f7ad33-7cfb-484e-a6e2-d048b6f7b385-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:43.910992 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:43.910998 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t9nf\" (UniqueName: \"kubernetes.io/projected/07f7ad33-7cfb-484e-a6e2-d048b6f7b385-kube-api-access-9t9nf\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:44.210539 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:44.210466 5979 generic.go:356] "Generic (PLEG): container finished" podID="80c6a028-1d5c-4ba1-84b6-113c617c5cb7" containerID="9f5c6212882eb9e93671539c67893708f6a73845b1c51ae4d665883beba4d441" exitCode=0 Nov 03 13:44:44.210539 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:44.210517 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6bd5c1-6d9c48cdcd-bshhx" event={"ID":"80c6a028-1d5c-4ba1-84b6-113c617c5cb7","Type":"ContainerDied","Data":"9f5c6212882eb9e93671539c67893708f6a73845b1c51ae4d665883beba4d441"} Nov 03 13:44:44.210797 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:44.210547 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6bd5c1-6d9c48cdcd-bshhx" Nov 03 13:44:44.210797 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:44.210576 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6bd5c1-6d9c48cdcd-bshhx" event={"ID":"80c6a028-1d5c-4ba1-84b6-113c617c5cb7","Type":"ContainerDied","Data":"7e6372ffe004332b706c94074d6a197cd03ddb3a0191f3e6241935aaebcaa378"} Nov 03 13:44:44.210797 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:44.210599 5979 scope.go:122] "RemoveContainer" containerID="9f5c6212882eb9e93671539c67893708f6a73845b1c51ae4d665883beba4d441" Nov 03 13:44:44.212140 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:44.212112 5979 generic.go:356] "Generic (PLEG): container finished" podID="07f7ad33-7cfb-484e-a6e2-d048b6f7b385" containerID="568a939c3c98fc958fb2817431b277a464220498a4c2d230b56e05a6a018bebb" exitCode=0 Nov 03 13:44:44.212250 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:44.212144 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6bd5c1-786d9ddfc5-f8tmc" Nov 03 13:44:44.212250 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:44.212193 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6bd5c1-786d9ddfc5-f8tmc" event={"ID":"07f7ad33-7cfb-484e-a6e2-d048b6f7b385","Type":"ContainerDied","Data":"568a939c3c98fc958fb2817431b277a464220498a4c2d230b56e05a6a018bebb"} Nov 03 13:44:44.212250 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:44.212236 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-6bd5c1-786d9ddfc5-f8tmc" event={"ID":"07f7ad33-7cfb-484e-a6e2-d048b6f7b385","Type":"ContainerDied","Data":"cf0569527e4b5edd32a48c97f550e71a37bf606a8edc4ff417858908947ad214"} Nov 03 13:44:44.212409 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:44.212347 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-6bd5c1-565d8c4b9-9shxg" Nov 03 13:44:44.220989 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:44.220956 5979 scope.go:122] "RemoveContainer" containerID="9f5c6212882eb9e93671539c67893708f6a73845b1c51ae4d665883beba4d441" Nov 03 13:44:44.221281 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:44:44.221257 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f5c6212882eb9e93671539c67893708f6a73845b1c51ae4d665883beba4d441\": container with ID starting with 9f5c6212882eb9e93671539c67893708f6a73845b1c51ae4d665883beba4d441 not found: ID does not exist" containerID="9f5c6212882eb9e93671539c67893708f6a73845b1c51ae4d665883beba4d441" Nov 03 13:44:44.221389 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:44.221290 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f5c6212882eb9e93671539c67893708f6a73845b1c51ae4d665883beba4d441"} err="failed to get container status \"9f5c6212882eb9e93671539c67893708f6a73845b1c51ae4d665883beba4d441\": rpc error: code = NotFound desc = could not find container \"9f5c6212882eb9e93671539c67893708f6a73845b1c51ae4d665883beba4d441\": container with ID starting with 9f5c6212882eb9e93671539c67893708f6a73845b1c51ae4d665883beba4d441 not found: ID does not exist" Nov 03 13:44:44.221389 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:44.221307 5979 scope.go:122] "RemoveContainer" containerID="568a939c3c98fc958fb2817431b277a464220498a4c2d230b56e05a6a018bebb" Nov 03 13:44:44.230349 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:44.230326 5979 scope.go:122] "RemoveContainer" containerID="568a939c3c98fc958fb2817431b277a464220498a4c2d230b56e05a6a018bebb" Nov 03 13:44:44.230623 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:44:44.230605 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"568a939c3c98fc958fb2817431b277a464220498a4c2d230b56e05a6a018bebb\": container with ID starting with 568a939c3c98fc958fb2817431b277a464220498a4c2d230b56e05a6a018bebb not found: ID does not exist" containerID="568a939c3c98fc958fb2817431b277a464220498a4c2d230b56e05a6a018bebb" Nov 03 13:44:44.230687 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:44.230629 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"568a939c3c98fc958fb2817431b277a464220498a4c2d230b56e05a6a018bebb"} err="failed to get container status \"568a939c3c98fc958fb2817431b277a464220498a4c2d230b56e05a6a018bebb\": rpc error: code = NotFound desc = could not find container \"568a939c3c98fc958fb2817431b277a464220498a4c2d230b56e05a6a018bebb\": container with ID starting with 568a939c3c98fc958fb2817431b277a464220498a4c2d230b56e05a6a018bebb not found: ID does not exist" Nov 03 13:44:44.370297 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:44.370264 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6bd5c1-786d9ddfc5-f8tmc"] Nov 03 13:44:44.372813 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:44.372785 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-6bd5c1-786d9ddfc5-f8tmc"] Nov 03 13:44:44.473413 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:44.473330 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6bd5c1-6d9c48cdcd-bshhx"] Nov 03 13:44:44.479492 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:44.479454 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6bd5c1-6d9c48cdcd-bshhx"] Nov 03 13:44:44.588011 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:44.587981 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6bd5c1-565d8c4b9-9shxg"] Nov 03 13:44:44.598812 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:44.598784 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-6bd5c1-565d8c4b9-9shxg"] Nov 03 13:44:46.116309 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:46.116270 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07f7ad33-7cfb-484e-a6e2-d048b6f7b385" path="/var/lib/kubelet/pods/07f7ad33-7cfb-484e-a6e2-d048b6f7b385/volumes" Nov 03 13:44:46.117068 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:46.117048 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5aa090e7-2473-4ecf-b8f9-d48182cefbd6" path="/var/lib/kubelet/pods/5aa090e7-2473-4ecf-b8f9-d48182cefbd6/volumes" Nov 03 13:44:46.117508 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:46.117489 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80c6a028-1d5c-4ba1-84b6-113c617c5cb7" path="/var/lib/kubelet/pods/80c6a028-1d5c-4ba1-84b6-113c617c5cb7/volumes" Nov 03 13:44:49.590820 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:49.590777 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28b9e6c0-s9dc6"] Nov 03 13:44:49.591662 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:49.591639 5979 cpu_manager.go:414] "containerMap: removing container" podUID="07f7ad33-7cfb-484e-a6e2-d048b6f7b385" containerName="payload-pre-processing" Nov 03 13:44:49.591662 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:49.591662 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07f7ad33-7cfb-484e-a6e2-d048b6f7b385" containerName="payload-pre-processing" Nov 03 13:44:49.591841 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:49.591690 5979 cpu_manager.go:414] "containerMap: removing container" podUID="2fa25a94-fb23-4745-b999-e77d4c54a035" containerName="revoke-keys" Nov 03 13:44:49.591841 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:49.591698 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fa25a94-fb23-4745-b999-e77d4c54a035" containerName="revoke-keys" Nov 03 13:44:49.591841 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:49.591742 5979 cpu_manager.go:414] "containerMap: removing container" podUID="80c6a028-1d5c-4ba1-84b6-113c617c5cb7" containerName="payload-processing" Nov 03 13:44:49.591841 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:49.591749 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80c6a028-1d5c-4ba1-84b6-113c617c5cb7" containerName="payload-processing" Nov 03 13:44:49.591841 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:49.591763 5979 cpu_manager.go:414] "containerMap: removing container" podUID="5aa090e7-2473-4ecf-b8f9-d48182cefbd6" containerName="maas-api" Nov 03 13:44:49.591841 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:49.591770 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5aa090e7-2473-4ecf-b8f9-d48182cefbd6" containerName="maas-api" Nov 03 13:44:49.592168 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:49.591948 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="07f7ad33-7cfb-484e-a6e2-d048b6f7b385" containerName="payload-pre-processing" Nov 03 13:44:49.592168 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:49.591973 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fa25a94-fb23-4745-b999-e77d4c54a035" containerName="revoke-keys" Nov 03 13:44:49.592168 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:49.591991 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="80c6a028-1d5c-4ba1-84b6-113c617c5cb7" containerName="payload-processing" Nov 03 13:44:49.592168 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:49.592001 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="5aa090e7-2473-4ecf-b8f9-d48182cefbd6" containerName="maas-api" Nov 03 13:44:49.596662 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:49.596637 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b9e6c0-s9dc6" Nov 03 13:44:49.650848 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:49.650813 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:44:49.651022 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:49.650888 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:44:49.689384 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:49.689336 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5swx5\" (UniqueName: \"kubernetes.io/projected/bab0d665-2857-40ea-a07f-7c4b535d84e4-kube-api-access-5swx5\") pod \"maas-api-revoke-sub-28b9e6c0-s9dc6\" (UID: \"bab0d665-2857-40ea-a07f-7c4b535d84e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b9e6c0-s9dc6" Nov 03 13:44:49.689631 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:49.689599 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bab0d665-2857-40ea-a07f-7c4b535d84e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28b9e6c0-s9dc6\" (UID: \"bab0d665-2857-40ea-a07f-7c4b535d84e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b9e6c0-s9dc6" Nov 03 13:44:49.689763 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:49.689744 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bab0d665-2857-40ea-a07f-7c4b535d84e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28b9e6c0-s9dc6\" (UID: \"bab0d665-2857-40ea-a07f-7c4b535d84e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b9e6c0-s9dc6" Nov 03 13:44:49.700015 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:49.699983 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28b9e6c0-s9dc6"] Nov 03 13:44:49.791069 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:49.791031 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bab0d665-2857-40ea-a07f-7c4b535d84e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28b9e6c0-s9dc6\" (UID: \"bab0d665-2857-40ea-a07f-7c4b535d84e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b9e6c0-s9dc6" Nov 03 13:44:49.791263 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:49.791096 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5swx5\" (UniqueName: \"kubernetes.io/projected/bab0d665-2857-40ea-a07f-7c4b535d84e4-kube-api-access-5swx5\") pod \"maas-api-revoke-sub-28b9e6c0-s9dc6\" (UID: \"bab0d665-2857-40ea-a07f-7c4b535d84e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b9e6c0-s9dc6" Nov 03 13:44:49.791263 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:49.791247 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bab0d665-2857-40ea-a07f-7c4b535d84e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28b9e6c0-s9dc6\" (UID: \"bab0d665-2857-40ea-a07f-7c4b535d84e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b9e6c0-s9dc6" Nov 03 13:44:49.791912 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:49.791893 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bab0d665-2857-40ea-a07f-7c4b535d84e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28b9e6c0-s9dc6\" (UID: \"bab0d665-2857-40ea-a07f-7c4b535d84e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b9e6c0-s9dc6" Nov 03 13:44:49.793442 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:49.793423 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bab0d665-2857-40ea-a07f-7c4b535d84e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28b9e6c0-s9dc6\" (UID: \"bab0d665-2857-40ea-a07f-7c4b535d84e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b9e6c0-s9dc6" Nov 03 13:44:49.851901 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:49.851829 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5swx5\" (UniqueName: \"kubernetes.io/projected/bab0d665-2857-40ea-a07f-7c4b535d84e4-kube-api-access-5swx5\") pod \"maas-api-revoke-sub-28b9e6c0-s9dc6\" (UID: \"bab0d665-2857-40ea-a07f-7c4b535d84e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b9e6c0-s9dc6" Nov 03 13:44:49.910611 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:49.910569 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b9e6c0-s9dc6" Nov 03 13:44:50.117138 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:50.117062 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9"] Nov 03 13:44:50.121372 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:50.121353 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9" Nov 03 13:44:50.175360 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:50.175333 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-cdb532-openshift-default-dockercfg-q2fdz\"" Nov 03 13:44:50.196007 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:50.195978 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77989bc1-5b8d-4939-9fd9-383865b9484d-istio-data\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9\" (UID: \"77989bc1-5b8d-4939-9fd9-383865b9484d\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9" Nov 03 13:44:50.196128 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:50.196010 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77989bc1-5b8d-4939-9fd9-383865b9484d-istio-podinfo\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9\" (UID: \"77989bc1-5b8d-4939-9fd9-383865b9484d\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9" Nov 03 13:44:50.196200 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:50.196177 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77989bc1-5b8d-4939-9fd9-383865b9484d-credential-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9\" (UID: \"77989bc1-5b8d-4939-9fd9-383865b9484d\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9" Nov 03 13:44:50.196242 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:50.196217 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmwl4\" (UniqueName: \"kubernetes.io/projected/77989bc1-5b8d-4939-9fd9-383865b9484d-kube-api-access-hmwl4\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9\" (UID: \"77989bc1-5b8d-4939-9fd9-383865b9484d\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9" Nov 03 13:44:50.196316 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:50.196304 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77989bc1-5b8d-4939-9fd9-383865b9484d-workload-certs\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9\" (UID: \"77989bc1-5b8d-4939-9fd9-383865b9484d\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9" Nov 03 13:44:50.196358 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:50.196324 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77989bc1-5b8d-4939-9fd9-383865b9484d-istio-envoy\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9\" (UID: \"77989bc1-5b8d-4939-9fd9-383865b9484d\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9" Nov 03 13:44:50.196422 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:50.196410 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77989bc1-5b8d-4939-9fd9-383865b9484d-workload-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9\" (UID: \"77989bc1-5b8d-4939-9fd9-383865b9484d\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9" Nov 03 13:44:50.196464 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:50.196436 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77989bc1-5b8d-4939-9fd9-383865b9484d-istiod-ca-cert\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9\" (UID: \"77989bc1-5b8d-4939-9fd9-383865b9484d\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9" Nov 03 13:44:50.196464 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:50.196454 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77989bc1-5b8d-4939-9fd9-383865b9484d-istio-token\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9\" (UID: \"77989bc1-5b8d-4939-9fd9-383865b9484d\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9" Nov 03 13:44:50.224999 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:50.224965 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9"] Nov 03 13:44:50.290868 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:50.290835 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28b9e6c0-s9dc6"] Nov 03 13:44:50.297736 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:50.297708 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77989bc1-5b8d-4939-9fd9-383865b9484d-workload-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9\" (UID: \"77989bc1-5b8d-4939-9fd9-383865b9484d\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9" Nov 03 13:44:50.297879 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:50.297760 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77989bc1-5b8d-4939-9fd9-383865b9484d-istiod-ca-cert\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9\" (UID: \"77989bc1-5b8d-4939-9fd9-383865b9484d\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9" Nov 03 13:44:50.297879 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:50.297786 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77989bc1-5b8d-4939-9fd9-383865b9484d-istio-token\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9\" (UID: \"77989bc1-5b8d-4939-9fd9-383865b9484d\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9" Nov 03 13:44:50.297879 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:50.297854 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77989bc1-5b8d-4939-9fd9-383865b9484d-istio-data\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9\" (UID: \"77989bc1-5b8d-4939-9fd9-383865b9484d\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9" Nov 03 13:44:50.298043 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:50.297878 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77989bc1-5b8d-4939-9fd9-383865b9484d-istio-podinfo\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9\" (UID: \"77989bc1-5b8d-4939-9fd9-383865b9484d\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9" Nov 03 13:44:50.298102 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:50.298047 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77989bc1-5b8d-4939-9fd9-383865b9484d-credential-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9\" (UID: \"77989bc1-5b8d-4939-9fd9-383865b9484d\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9" Nov 03 13:44:50.298102 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:50.298084 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmwl4\" (UniqueName: \"kubernetes.io/projected/77989bc1-5b8d-4939-9fd9-383865b9484d-kube-api-access-hmwl4\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9\" (UID: \"77989bc1-5b8d-4939-9fd9-383865b9484d\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9" Nov 03 13:44:50.298215 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:50.298139 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77989bc1-5b8d-4939-9fd9-383865b9484d-workload-certs\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9\" (UID: \"77989bc1-5b8d-4939-9fd9-383865b9484d\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9" Nov 03 13:44:50.298215 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:50.298168 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77989bc1-5b8d-4939-9fd9-383865b9484d-istio-envoy\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9\" (UID: \"77989bc1-5b8d-4939-9fd9-383865b9484d\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9" Nov 03 13:44:50.298447 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:44:50.298421 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbab0d665_2857_40ea_a07f_7c4b535d84e4.slice/crio-bb0695566c0015d2f37ed4d4e4d8face5a4fcd2fe377d4700621773f804383d2 WatchSource:0}: Error finding container bb0695566c0015d2f37ed4d4e4d8face5a4fcd2fe377d4700621773f804383d2: Status 404 returned error can't find the container with id bb0695566c0015d2f37ed4d4e4d8face5a4fcd2fe377d4700621773f804383d2 Nov 03 13:44:50.298578 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:50.298545 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77989bc1-5b8d-4939-9fd9-383865b9484d-workload-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9\" (UID: \"77989bc1-5b8d-4939-9fd9-383865b9484d\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9" Nov 03 13:44:50.298747 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:50.298720 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77989bc1-5b8d-4939-9fd9-383865b9484d-workload-certs\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9\" (UID: \"77989bc1-5b8d-4939-9fd9-383865b9484d\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9" Nov 03 13:44:50.299244 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:50.299196 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77989bc1-5b8d-4939-9fd9-383865b9484d-istio-data\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9\" (UID: \"77989bc1-5b8d-4939-9fd9-383865b9484d\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9" Nov 03 13:44:50.299244 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:50.299224 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77989bc1-5b8d-4939-9fd9-383865b9484d-credential-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9\" (UID: \"77989bc1-5b8d-4939-9fd9-383865b9484d\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9" Nov 03 13:44:50.299859 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:50.299831 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77989bc1-5b8d-4939-9fd9-383865b9484d-istiod-ca-cert\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9\" (UID: \"77989bc1-5b8d-4939-9fd9-383865b9484d\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9" Nov 03 13:44:50.301626 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:50.300844 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77989bc1-5b8d-4939-9fd9-383865b9484d-istio-envoy\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9\" (UID: \"77989bc1-5b8d-4939-9fd9-383865b9484d\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9" Nov 03 13:44:50.301626 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:50.301513 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77989bc1-5b8d-4939-9fd9-383865b9484d-istio-podinfo\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9\" (UID: \"77989bc1-5b8d-4939-9fd9-383865b9484d\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9" Nov 03 13:44:50.352357 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:50.352330 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmwl4\" (UniqueName: \"kubernetes.io/projected/77989bc1-5b8d-4939-9fd9-383865b9484d-kube-api-access-hmwl4\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9\" (UID: \"77989bc1-5b8d-4939-9fd9-383865b9484d\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9" Nov 03 13:44:50.352644 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:50.352613 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77989bc1-5b8d-4939-9fd9-383865b9484d-istio-token\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9\" (UID: \"77989bc1-5b8d-4939-9fd9-383865b9484d\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9" Nov 03 13:44:50.435388 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:50.435357 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9" Nov 03 13:44:50.877647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:50.877558 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9"] Nov 03 13:44:50.884367 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:44:50.884334 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77989bc1_5b8d_4939_9fd9_383865b9484d.slice/crio-c8d8aea498a6a74d203b369a495501d76d69d1dd03894df563003fa81e8a1287 WatchSource:0}: Error finding container c8d8aea498a6a74d203b369a495501d76d69d1dd03894df563003fa81e8a1287: Status 404 returned error can't find the container with id c8d8aea498a6a74d203b369a495501d76d69d1dd03894df563003fa81e8a1287 Nov 03 13:44:50.887753 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:50.887712 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:44:50.887980 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:50.887939 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:44:50.888076 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:50.888007 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:44:51.274633 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:51.271798 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b9e6c0-s9dc6" event={"ID":"bab0d665-2857-40ea-a07f-7c4b535d84e4","Type":"ContainerStarted","Data":"97cfb540abb72c1eb43176ff0db9e33f741735c57c56e0cd5931731cc7abfc7a"} Nov 03 13:44:51.274633 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:51.271865 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b9e6c0-s9dc6" event={"ID":"bab0d665-2857-40ea-a07f-7c4b535d84e4","Type":"ContainerStarted","Data":"bb0695566c0015d2f37ed4d4e4d8face5a4fcd2fe377d4700621773f804383d2"} Nov 03 13:44:51.275035 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:51.274997 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9" event={"ID":"77989bc1-5b8d-4939-9fd9-383865b9484d","Type":"ContainerStarted","Data":"ceeaee78fe0d459f4a1496aa067c3406667e6139f71287a7654613a676672289"} Nov 03 13:44:51.275146 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:51.275055 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9" event={"ID":"77989bc1-5b8d-4939-9fd9-383865b9484d","Type":"ContainerStarted","Data":"c8d8aea498a6a74d203b369a495501d76d69d1dd03894df563003fa81e8a1287"} Nov 03 13:44:51.384836 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:51.384781 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b9e6c0-s9dc6" podStartSLOduration=2.384767187 podStartE2EDuration="2.384767187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:44:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:44:51.38460131 +0000 UTC m=+2373.395289551" watchObservedRunningTime="2026-11-03 13:44:51.384767187 +0000 UTC m=+2373.395455428" Nov 03 13:44:51.435991 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:51.435952 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9" Nov 03 13:44:51.437279 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:51.437261 5979 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Nov 03 13:44:51.437346 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:51.437302 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9" podUID="77989bc1-5b8d-4939-9fd9-383865b9484d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Nov 03 13:44:51.506193 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:51.503538 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9" podStartSLOduration=1.503518489 podStartE2EDuration="1.503518489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:44:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:44:51.503272604 +0000 UTC m=+2373.513960848" watchObservedRunningTime="2026-11-03 13:44:51.503518489 +0000 UTC m=+2373.514206733" Nov 03 13:44:52.282737 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:52.282699 5979 generic.go:356] "Generic (PLEG): container finished" podID="bab0d665-2857-40ea-a07f-7c4b535d84e4" containerID="97cfb540abb72c1eb43176ff0db9e33f741735c57c56e0cd5931731cc7abfc7a" exitCode=0 Nov 03 13:44:52.283241 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:52.282766 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b9e6c0-s9dc6" event={"ID":"bab0d665-2857-40ea-a07f-7c4b535d84e4","Type":"ContainerDied","Data":"97cfb540abb72c1eb43176ff0db9e33f741735c57c56e0cd5931731cc7abfc7a"} Nov 03 13:44:52.435861 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:52.435816 5979 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Nov 03 13:44:52.436069 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:52.435870 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9" podUID="77989bc1-5b8d-4939-9fd9-383865b9484d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Nov 03 13:44:53.440633 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:53.440596 5979 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Nov 03 13:44:53.444663 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:53.444625 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9" podUID="77989bc1-5b8d-4939-9fd9-383865b9484d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Nov 03 13:44:53.608615 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:53.608546 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b9e6c0-s9dc6" Nov 03 13:44:53.669047 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:53.668990 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bab0d665-2857-40ea-a07f-7c4b535d84e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bab0d665-2857-40ea-a07f-7c4b535d84e4-maas-api-xks-ca\") pod \"bab0d665-2857-40ea-a07f-7c4b535d84e4\" (UID: \"bab0d665-2857-40ea-a07f-7c4b535d84e4\") " Nov 03 13:44:53.669230 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:53.669119 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bab0d665-2857-40ea-a07f-7c4b535d84e4-kube-api-access-5swx5\" (UniqueName: \"kubernetes.io/projected/bab0d665-2857-40ea-a07f-7c4b535d84e4-kube-api-access-5swx5\") pod \"bab0d665-2857-40ea-a07f-7c4b535d84e4\" (UID: \"bab0d665-2857-40ea-a07f-7c4b535d84e4\") " Nov 03 13:44:53.669230 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:53.669154 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bab0d665-2857-40ea-a07f-7c4b535d84e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bab0d665-2857-40ea-a07f-7c4b535d84e4-maas-api-openshift-ca\") pod \"bab0d665-2857-40ea-a07f-7c4b535d84e4\" (UID: \"bab0d665-2857-40ea-a07f-7c4b535d84e4\") " Nov 03 13:44:53.669999 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:53.669967 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bab0d665-2857-40ea-a07f-7c4b535d84e4-maas-api-openshift-ca" pod "bab0d665-2857-40ea-a07f-7c4b535d84e4" (UID: "bab0d665-2857-40ea-a07f-7c4b535d84e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:44:53.670192 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:53.670170 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bab0d665-2857-40ea-a07f-7c4b535d84e4-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:53.671756 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:53.671728 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bab0d665-2857-40ea-a07f-7c4b535d84e4-kube-api-access-5swx5" pod "bab0d665-2857-40ea-a07f-7c4b535d84e4" (UID: "bab0d665-2857-40ea-a07f-7c4b535d84e4"). InnerVolumeSpecName "kube-api-access-5swx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:44:53.672140 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:53.672114 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bab0d665-2857-40ea-a07f-7c4b535d84e4-maas-api-xks-ca" pod "bab0d665-2857-40ea-a07f-7c4b535d84e4" (UID: "bab0d665-2857-40ea-a07f-7c4b535d84e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:44:53.796572 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:53.796521 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bab0d665-2857-40ea-a07f-7c4b535d84e4-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:53.796572 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:53.796564 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5swx5\" (UniqueName: \"kubernetes.io/projected/bab0d665-2857-40ea-a07f-7c4b535d84e4-kube-api-access-5swx5\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:54.297896 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:54.297863 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq"] Nov 03 13:44:54.298095 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:54.298065 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq" podUID="ad7c3ee7-2be7-4124-9a89-1be00bbc1b21" containerName="istio-proxy" containerID="cri-o://10149c7f0295a4527780a4427491469fccef3771d539f615733ae686a7328110" gracePeriod=30 Nov 03 13:44:54.352252 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:54.351373 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b9e6c0-s9dc6" event={"ID":"bab0d665-2857-40ea-a07f-7c4b535d84e4","Type":"ContainerDied","Data":"bb0695566c0015d2f37ed4d4e4d8face5a4fcd2fe377d4700621773f804383d2"} Nov 03 13:44:54.352252 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:54.351436 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb0695566c0015d2f37ed4d4e4d8face5a4fcd2fe377d4700621773f804383d2" Nov 03 13:44:54.352252 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:54.351527 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b9e6c0-s9dc6" Nov 03 13:44:54.435670 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:54.435634 5979 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Nov 03 13:44:54.435875 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:54.435690 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9" podUID="77989bc1-5b8d-4939-9fd9-383865b9484d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Nov 03 13:44:54.631955 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:54.631923 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-8d8af2-6778d74d77-f5xjw"] Nov 03 13:44:54.775054 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:54.774958 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-8d8af2-6979795ddc-f4x64"] Nov 03 13:44:55.232024 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:55.231990 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 13:44:55.232223 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:55.232052 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq" podUID="ad7c3ee7-2be7-4124-9a89-1be00bbc1b21" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 13:44:55.436678 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:55.436646 5979 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Nov 03 13:44:55.436837 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:55.436708 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9" podUID="77989bc1-5b8d-4939-9fd9-383865b9484d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Nov 03 13:44:56.436740 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:56.436672 5979 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Nov 03 13:44:56.437202 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:56.436735 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9" podUID="77989bc1-5b8d-4939-9fd9-383865b9484d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Nov 03 13:44:57.436316 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:57.436274 5979 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Nov 03 13:44:57.436507 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:57.436344 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9" podUID="77989bc1-5b8d-4939-9fd9-383865b9484d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Nov 03 13:44:58.476842 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:58.476809 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9" Nov 03 13:44:58.479401 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:58.479370 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9" Nov 03 13:44:58.480195 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:58.480169 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9" Nov 03 13:44:59.583719 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:59.583655 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq" Nov 03 13:44:59.632217 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:59.632178 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-8d8af2-6778d74d77-f5xjw" podUID="f5c4766a-e455-497c-ba06-716c55b6b8aa" containerName="payload-processing" containerID="cri-o://5d6ea38e37ce9cfefdd5912bc6d2809296b66f809c29e1c9bbadaa535a14757c" gracePeriod=25 Nov 03 13:44:59.706086 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:59.706053 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-istio-podinfo\") pod \"ad7c3ee7-2be7-4124-9a89-1be00bbc1b21\" (UID: \"ad7c3ee7-2be7-4124-9a89-1be00bbc1b21\") " Nov 03 13:44:59.706254 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:59.706099 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-workload-socket\") pod \"ad7c3ee7-2be7-4124-9a89-1be00bbc1b21\" (UID: \"ad7c3ee7-2be7-4124-9a89-1be00bbc1b21\") " Nov 03 13:44:59.706254 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:59.706183 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-kube-api-access-qp9vl\" (UniqueName: \"kubernetes.io/projected/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-kube-api-access-qp9vl\") pod \"ad7c3ee7-2be7-4124-9a89-1be00bbc1b21\" (UID: \"ad7c3ee7-2be7-4124-9a89-1be00bbc1b21\") " Nov 03 13:44:59.706254 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:59.706209 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-workload-certs\") pod \"ad7c3ee7-2be7-4124-9a89-1be00bbc1b21\" (UID: \"ad7c3ee7-2be7-4124-9a89-1be00bbc1b21\") " Nov 03 13:44:59.706254 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:59.706242 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-istio-token\" (UniqueName: \"kubernetes.io/projected/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-istio-token\") pod \"ad7c3ee7-2be7-4124-9a89-1be00bbc1b21\" (UID: \"ad7c3ee7-2be7-4124-9a89-1be00bbc1b21\") " Nov 03 13:44:59.706512 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:59.706277 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-credential-socket\") pod \"ad7c3ee7-2be7-4124-9a89-1be00bbc1b21\" (UID: \"ad7c3ee7-2be7-4124-9a89-1be00bbc1b21\") " Nov 03 13:44:59.706512 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:59.706380 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-istio-data\") pod \"ad7c3ee7-2be7-4124-9a89-1be00bbc1b21\" (UID: \"ad7c3ee7-2be7-4124-9a89-1be00bbc1b21\") " Nov 03 13:44:59.706512 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:59.706414 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-istiod-ca-cert\") pod \"ad7c3ee7-2be7-4124-9a89-1be00bbc1b21\" (UID: \"ad7c3ee7-2be7-4124-9a89-1be00bbc1b21\") " Nov 03 13:44:59.706512 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:59.706454 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-istio-envoy\") pod \"ad7c3ee7-2be7-4124-9a89-1be00bbc1b21\" (UID: \"ad7c3ee7-2be7-4124-9a89-1be00bbc1b21\") " Nov 03 13:44:59.706751 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:59.706526 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-workload-socket" pod "ad7c3ee7-2be7-4124-9a89-1be00bbc1b21" (UID: "ad7c3ee7-2be7-4124-9a89-1be00bbc1b21"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:59.706751 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:59.706715 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-credential-socket" pod "ad7c3ee7-2be7-4124-9a89-1be00bbc1b21" (UID: "ad7c3ee7-2be7-4124-9a89-1be00bbc1b21"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:59.706751 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:59.706733 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-istio-data" pod "ad7c3ee7-2be7-4124-9a89-1be00bbc1b21" (UID: "ad7c3ee7-2be7-4124-9a89-1be00bbc1b21"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:59.707158 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:59.707013 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-workload-certs" pod "ad7c3ee7-2be7-4124-9a89-1be00bbc1b21" (UID: "ad7c3ee7-2be7-4124-9a89-1be00bbc1b21"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:59.707158 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:59.707130 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-istiod-ca-cert" pod "ad7c3ee7-2be7-4124-9a89-1be00bbc1b21" (UID: "ad7c3ee7-2be7-4124-9a89-1be00bbc1b21"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:44:59.707309 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:59.707156 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:59.707309 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:59.707175 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:59.707309 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:59.707187 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:59.707309 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:59.707202 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:59.708834 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:59.708806 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-istio-podinfo" pod "ad7c3ee7-2be7-4124-9a89-1be00bbc1b21" (UID: "ad7c3ee7-2be7-4124-9a89-1be00bbc1b21"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:44:59.708952 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:59.708812 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-kube-api-access-qp9vl" pod "ad7c3ee7-2be7-4124-9a89-1be00bbc1b21" (UID: "ad7c3ee7-2be7-4124-9a89-1be00bbc1b21"). InnerVolumeSpecName "kube-api-access-qp9vl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:44:59.709123 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:59.709104 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-istio-envoy" pod "ad7c3ee7-2be7-4124-9a89-1be00bbc1b21" (UID: "ad7c3ee7-2be7-4124-9a89-1be00bbc1b21"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:44:59.709186 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:59.709138 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-istio-token" pod "ad7c3ee7-2be7-4124-9a89-1be00bbc1b21" (UID: "ad7c3ee7-2be7-4124-9a89-1be00bbc1b21"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:44:59.775787 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:59.775702 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8d8af2-6979795ddc-f4x64" podUID="b92a7bfb-f886-4b87-9c00-e07d8fb43732" containerName="payload-pre-processing" containerID="cri-o://e3f0f7227cd6991da98ed826d98930eae2bce202c676a8e0b81b14f0e7fcde3f" gracePeriod=25 Nov 03 13:44:59.808500 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:59.808433 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp9vl\" (UniqueName: \"kubernetes.io/projected/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-kube-api-access-qp9vl\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:59.808500 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:59.808467 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:59.808500 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:59.808500 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:59.808809 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:59.808513 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:59.808809 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:59.808526 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:44:59.905179 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:44:59.905151 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-8d8af2-6778d74d77-f5xjw" Nov 03 13:45:00.010857 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.010822 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5c4766a-e455-497c-ba06-716c55b6b8aa-kube-api-access-q9kv2\" (UniqueName: \"kubernetes.io/projected/f5c4766a-e455-497c-ba06-716c55b6b8aa-kube-api-access-q9kv2\") pod \"f5c4766a-e455-497c-ba06-716c55b6b8aa\" (UID: \"f5c4766a-e455-497c-ba06-716c55b6b8aa\") " Nov 03 13:45:00.011015 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.010953 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5c4766a-e455-497c-ba06-716c55b6b8aa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5c4766a-e455-497c-ba06-716c55b6b8aa-plugins-config-volume\") pod \"f5c4766a-e455-497c-ba06-716c55b6b8aa\" (UID: \"f5c4766a-e455-497c-ba06-716c55b6b8aa\") " Nov 03 13:45:00.011439 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.011405 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5c4766a-e455-497c-ba06-716c55b6b8aa-plugins-config-volume" pod "f5c4766a-e455-497c-ba06-716c55b6b8aa" (UID: "f5c4766a-e455-497c-ba06-716c55b6b8aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:45:00.011707 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.011680 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5c4766a-e455-497c-ba06-716c55b6b8aa-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:00.013321 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.013297 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5c4766a-e455-497c-ba06-716c55b6b8aa-kube-api-access-q9kv2" pod "f5c4766a-e455-497c-ba06-716c55b6b8aa" (UID: "f5c4766a-e455-497c-ba06-716c55b6b8aa"). InnerVolumeSpecName "kube-api-access-q9kv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:45:00.018367 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.018341 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8d8af2-6979795ddc-f4x64" Nov 03 13:45:00.112216 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.112186 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b92a7bfb-f886-4b87-9c00-e07d8fb43732-kube-api-access-9jcnl\" (UniqueName: \"kubernetes.io/projected/b92a7bfb-f886-4b87-9c00-e07d8fb43732-kube-api-access-9jcnl\") pod \"b92a7bfb-f886-4b87-9c00-e07d8fb43732\" (UID: \"b92a7bfb-f886-4b87-9c00-e07d8fb43732\") " Nov 03 13:45:00.112404 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.112235 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b92a7bfb-f886-4b87-9c00-e07d8fb43732-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b92a7bfb-f886-4b87-9c00-e07d8fb43732-plugins-config-volume\") pod \"b92a7bfb-f886-4b87-9c00-e07d8fb43732\" (UID: \"b92a7bfb-f886-4b87-9c00-e07d8fb43732\") " Nov 03 13:45:00.112738 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.112718 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9kv2\" (UniqueName: \"kubernetes.io/projected/f5c4766a-e455-497c-ba06-716c55b6b8aa-kube-api-access-q9kv2\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:00.112738 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.112724 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b92a7bfb-f886-4b87-9c00-e07d8fb43732-plugins-config-volume" pod "b92a7bfb-f886-4b87-9c00-e07d8fb43732" (UID: "b92a7bfb-f886-4b87-9c00-e07d8fb43732"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:45:00.114345 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.114320 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b92a7bfb-f886-4b87-9c00-e07d8fb43732-kube-api-access-9jcnl" pod "b92a7bfb-f886-4b87-9c00-e07d8fb43732" (UID: "b92a7bfb-f886-4b87-9c00-e07d8fb43732"). InnerVolumeSpecName "kube-api-access-9jcnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:45:00.193673 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.193584 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a770b9-29895225-gvxrk"] Nov 03 13:45:00.194344 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.194320 5979 cpu_manager.go:414] "containerMap: removing container" podUID="ad7c3ee7-2be7-4124-9a89-1be00bbc1b21" containerName="istio-proxy" Nov 03 13:45:00.194483 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.194344 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad7c3ee7-2be7-4124-9a89-1be00bbc1b21" containerName="istio-proxy" Nov 03 13:45:00.194483 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.194364 5979 cpu_manager.go:414] "containerMap: removing container" podUID="f5c4766a-e455-497c-ba06-716c55b6b8aa" containerName="payload-processing" Nov 03 13:45:00.194483 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.194371 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5c4766a-e455-497c-ba06-716c55b6b8aa" containerName="payload-processing" Nov 03 13:45:00.194483 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.194419 5979 cpu_manager.go:414] "containerMap: removing container" podUID="bab0d665-2857-40ea-a07f-7c4b535d84e4" containerName="revoke-keys" Nov 03 13:45:00.194483 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.194427 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bab0d665-2857-40ea-a07f-7c4b535d84e4" containerName="revoke-keys" Nov 03 13:45:00.194483 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.194443 5979 cpu_manager.go:414] "containerMap: removing container" podUID="b92a7bfb-f886-4b87-9c00-e07d8fb43732" containerName="payload-pre-processing" Nov 03 13:45:00.194483 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.194452 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b92a7bfb-f886-4b87-9c00-e07d8fb43732" containerName="payload-pre-processing" Nov 03 13:45:00.194986 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.194599 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad7c3ee7-2be7-4124-9a89-1be00bbc1b21" containerName="istio-proxy" Nov 03 13:45:00.194986 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.194616 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5c4766a-e455-497c-ba06-716c55b6b8aa" containerName="payload-processing" Nov 03 13:45:00.194986 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.194632 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="bab0d665-2857-40ea-a07f-7c4b535d84e4" containerName="revoke-keys" Nov 03 13:45:00.194986 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.194644 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="b92a7bfb-f886-4b87-9c00-e07d8fb43732" containerName="payload-pre-processing" Nov 03 13:45:00.198430 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.198409 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a770b9-29895225-gvxrk" Nov 03 13:45:00.214843 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.214814 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jcnl\" (UniqueName: \"kubernetes.io/projected/b92a7bfb-f886-4b87-9c00-e07d8fb43732-kube-api-access-9jcnl\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:00.214843 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.214837 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b92a7bfb-f886-4b87-9c00-e07d8fb43732-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:00.227711 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.227679 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895225-ctv9x"] Nov 03 13:45:00.232147 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.232123 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-84e22c-29895225-mvwgc"] Nov 03 13:45:00.232299 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.232246 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895225-ctv9x" Nov 03 13:45:00.237010 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.236988 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4b9658-29895225-mjbc4"] Nov 03 13:45:00.237119 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.237102 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-84e22c-29895225-mvwgc" Nov 03 13:45:00.241051 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.241035 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4b9658-29895225-mjbc4" Nov 03 13:45:00.252105 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.252075 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:45:00.268813 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.268784 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a770b9-29895225-gvxrk"] Nov 03 13:45:00.287874 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.287840 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895225-568vs"] Nov 03 13:45:00.292460 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.292432 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895225-568vs" Nov 03 13:45:00.301992 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.301063 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895225-tllnw"] Nov 03 13:45:00.306791 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.306765 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895225-tllnw" Nov 03 13:45:00.308486 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.308454 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895225-ctv9x"] Nov 03 13:45:00.312293 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.312268 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4b9658-29895225-mjbc4"] Nov 03 13:45:00.316001 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.315972 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcd1cac1-4f49-4c3a-a46d-726a8a83798c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-a770b9-29895225-gvxrk\" (UID: \"bcd1cac1-4f49-4c3a-a46d-726a8a83798c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a770b9-29895225-gvxrk" Nov 03 13:45:00.316140 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.316020 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f54baec-8801-4ebf-86a8-e4dee3b46888-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-84e22c-29895225-mvwgc\" (UID: \"3f54baec-8801-4ebf-86a8-e4dee3b46888\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-84e22c-29895225-mvwgc" Nov 03 13:45:00.316140 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.316063 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lrdw\" (UniqueName: \"kubernetes.io/projected/9c8f3d30-d7ac-4c71-a738-003ebc263255-kube-api-access-7lrdw\") pod \"maas-api-key-cleanup-e2e-praxis-a-4b9658-29895225-mjbc4\" (UID: \"9c8f3d30-d7ac-4c71-a738-003ebc263255\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4b9658-29895225-mjbc4" Nov 03 13:45:00.316256 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.316145 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc2b1123-5771-4e33-bc6f-1654fec22a26-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-42c707-29895225-ctv9x\" (UID: \"bc2b1123-5771-4e33-bc6f-1654fec22a26\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895225-ctv9x" Nov 03 13:45:00.316256 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.316185 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-867kr\" (UniqueName: \"kubernetes.io/projected/bc2b1123-5771-4e33-bc6f-1654fec22a26-kube-api-access-867kr\") pod \"maas-api-key-cleanup-e2e-worker-w0-42c707-29895225-ctv9x\" (UID: \"bc2b1123-5771-4e33-bc6f-1654fec22a26\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895225-ctv9x" Nov 03 13:45:00.316360 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.316287 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdwgw\" (UniqueName: \"kubernetes.io/projected/bcd1cac1-4f49-4c3a-a46d-726a8a83798c-kube-api-access-sdwgw\") pod \"maas-api-key-cleanup-e2e-ipp-a-a770b9-29895225-gvxrk\" (UID: \"bcd1cac1-4f49-4c3a-a46d-726a8a83798c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a770b9-29895225-gvxrk" Nov 03 13:45:00.316360 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.316317 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c8f3d30-d7ac-4c71-a738-003ebc263255-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-4b9658-29895225-mjbc4\" (UID: \"9c8f3d30-d7ac-4c71-a738-003ebc263255\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4b9658-29895225-mjbc4" Nov 03 13:45:00.316360 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.316346 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg5v7\" (UniqueName: \"kubernetes.io/projected/3f54baec-8801-4ebf-86a8-e4dee3b46888-kube-api-access-pg5v7\") pod \"maas-api-key-cleanup-e2e-praxis-b-84e22c-29895225-mvwgc\" (UID: \"3f54baec-8801-4ebf-86a8-e4dee3b46888\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-84e22c-29895225-mvwgc" Nov 03 13:45:00.337974 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.337942 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-84e22c-29895225-mvwgc"] Nov 03 13:45:00.352999 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.352399 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895225-568vs"] Nov 03 13:45:00.360653 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.360616 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895225-tllnw"] Nov 03 13:45:00.418174 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.418119 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdwgw\" (UniqueName: \"kubernetes.io/projected/bcd1cac1-4f49-4c3a-a46d-726a8a83798c-kube-api-access-sdwgw\") pod \"maas-api-key-cleanup-e2e-ipp-a-a770b9-29895225-gvxrk\" (UID: \"bcd1cac1-4f49-4c3a-a46d-726a8a83798c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a770b9-29895225-gvxrk" Nov 03 13:45:00.418174 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.418170 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c8f3d30-d7ac-4c71-a738-003ebc263255-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-4b9658-29895225-mjbc4\" (UID: \"9c8f3d30-d7ac-4c71-a738-003ebc263255\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4b9658-29895225-mjbc4" Nov 03 13:45:00.418493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.418223 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg5v7\" (UniqueName: \"kubernetes.io/projected/3f54baec-8801-4ebf-86a8-e4dee3b46888-kube-api-access-pg5v7\") pod \"maas-api-key-cleanup-e2e-praxis-b-84e22c-29895225-mvwgc\" (UID: \"3f54baec-8801-4ebf-86a8-e4dee3b46888\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-84e22c-29895225-mvwgc" Nov 03 13:45:00.418493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.418259 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e40ff1a-b3d1-4aee-a76d-57e1373945ee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5bb463-29895225-tllnw\" (UID: \"3e40ff1a-b3d1-4aee-a76d-57e1373945ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895225-tllnw" Nov 03 13:45:00.418493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.418314 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcd1cac1-4f49-4c3a-a46d-726a8a83798c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-a770b9-29895225-gvxrk\" (UID: \"bcd1cac1-4f49-4c3a-a46d-726a8a83798c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a770b9-29895225-gvxrk" Nov 03 13:45:00.418493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.418343 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk9hd\" (UniqueName: \"kubernetes.io/projected/3e40ff1a-b3d1-4aee-a76d-57e1373945ee-kube-api-access-tk9hd\") pod \"maas-api-key-cleanup-e2e-worker-w3-5bb463-29895225-tllnw\" (UID: \"3e40ff1a-b3d1-4aee-a76d-57e1373945ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895225-tllnw" Nov 03 13:45:00.418493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.418371 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f54baec-8801-4ebf-86a8-e4dee3b46888-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-84e22c-29895225-mvwgc\" (UID: \"3f54baec-8801-4ebf-86a8-e4dee3b46888\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-84e22c-29895225-mvwgc" Nov 03 13:45:00.418493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.418409 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lrdw\" (UniqueName: \"kubernetes.io/projected/9c8f3d30-d7ac-4c71-a738-003ebc263255-kube-api-access-7lrdw\") pod \"maas-api-key-cleanup-e2e-praxis-a-4b9658-29895225-mjbc4\" (UID: \"9c8f3d30-d7ac-4c71-a738-003ebc263255\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4b9658-29895225-mjbc4" Nov 03 13:45:00.418808 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.418518 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc2b1123-5771-4e33-bc6f-1654fec22a26-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-42c707-29895225-ctv9x\" (UID: \"bc2b1123-5771-4e33-bc6f-1654fec22a26\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895225-ctv9x" Nov 03 13:45:00.418808 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.418566 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-867kr\" (UniqueName: \"kubernetes.io/projected/bc2b1123-5771-4e33-bc6f-1654fec22a26-kube-api-access-867kr\") pod \"maas-api-key-cleanup-e2e-worker-w0-42c707-29895225-ctv9x\" (UID: \"bc2b1123-5771-4e33-bc6f-1654fec22a26\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895225-ctv9x" Nov 03 13:45:00.418808 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.418684 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5nqr\" (UniqueName: \"kubernetes.io/projected/e0df41fd-c6ea-44af-a973-05243f6ddaae-kube-api-access-n5nqr\") pod \"maas-api-key-cleanup-e2e-worker-w1-34b13c-29895225-568vs\" (UID: \"e0df41fd-c6ea-44af-a973-05243f6ddaae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895225-568vs" Nov 03 13:45:00.418808 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.418738 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0df41fd-c6ea-44af-a973-05243f6ddaae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-34b13c-29895225-568vs\" (UID: \"e0df41fd-c6ea-44af-a973-05243f6ddaae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895225-568vs" Nov 03 13:45:00.419286 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.419260 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c8f3d30-d7ac-4c71-a738-003ebc263255-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-4b9658-29895225-mjbc4\" (UID: \"9c8f3d30-d7ac-4c71-a738-003ebc263255\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4b9658-29895225-mjbc4" Nov 03 13:45:00.419888 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.419863 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f54baec-8801-4ebf-86a8-e4dee3b46888-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-84e22c-29895225-mvwgc\" (UID: \"3f54baec-8801-4ebf-86a8-e4dee3b46888\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-84e22c-29895225-mvwgc" Nov 03 13:45:00.419991 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.419933 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc2b1123-5771-4e33-bc6f-1654fec22a26-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-42c707-29895225-ctv9x\" (UID: \"bc2b1123-5771-4e33-bc6f-1654fec22a26\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895225-ctv9x" Nov 03 13:45:00.420050 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.419998 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcd1cac1-4f49-4c3a-a46d-726a8a83798c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-a770b9-29895225-gvxrk\" (UID: \"bcd1cac1-4f49-4c3a-a46d-726a8a83798c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a770b9-29895225-gvxrk" Nov 03 13:45:00.427310 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.427276 5979 generic.go:356] "Generic (PLEG): container finished" podID="ad7c3ee7-2be7-4124-9a89-1be00bbc1b21" containerID="10149c7f0295a4527780a4427491469fccef3771d539f615733ae686a7328110" exitCode=0 Nov 03 13:45:00.427502 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.427317 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq" event={"ID":"ad7c3ee7-2be7-4124-9a89-1be00bbc1b21","Type":"ContainerDied","Data":"10149c7f0295a4527780a4427491469fccef3771d539f615733ae686a7328110"} Nov 03 13:45:00.427502 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.427382 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq" event={"ID":"ad7c3ee7-2be7-4124-9a89-1be00bbc1b21","Type":"ContainerDied","Data":"e555f8164b9b3d9d3d58b8a3d9ef16949b8df5cb7046bddfbc82366ff452696d"} Nov 03 13:45:00.427502 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.427325 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq" Nov 03 13:45:00.427770 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.427411 5979 scope.go:122] "RemoveContainer" containerID="10149c7f0295a4527780a4427491469fccef3771d539f615733ae686a7328110" Nov 03 13:45:00.429798 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.429775 5979 generic.go:356] "Generic (PLEG): container finished" podID="b92a7bfb-f886-4b87-9c00-e07d8fb43732" containerID="e3f0f7227cd6991da98ed826d98930eae2bce202c676a8e0b81b14f0e7fcde3f" exitCode=0 Nov 03 13:45:00.432269 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.429873 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8d8af2-6979795ddc-f4x64" Nov 03 13:45:00.432269 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.429889 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8d8af2-6979795ddc-f4x64" event={"ID":"b92a7bfb-f886-4b87-9c00-e07d8fb43732","Type":"ContainerDied","Data":"e3f0f7227cd6991da98ed826d98930eae2bce202c676a8e0b81b14f0e7fcde3f"} Nov 03 13:45:00.432269 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.430391 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-8d8af2-6979795ddc-f4x64" event={"ID":"b92a7bfb-f886-4b87-9c00-e07d8fb43732","Type":"ContainerDied","Data":"017959fb8bc481a27a72afef2c31b971c7627c6268819fb12de9531855cc5091"} Nov 03 13:45:00.434314 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.434269 5979 generic.go:356] "Generic (PLEG): container finished" podID="f5c4766a-e455-497c-ba06-716c55b6b8aa" containerID="5d6ea38e37ce9cfefdd5912bc6d2809296b66f809c29e1c9bbadaa535a14757c" exitCode=0 Nov 03 13:45:00.434420 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.434309 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-8d8af2-6778d74d77-f5xjw" event={"ID":"f5c4766a-e455-497c-ba06-716c55b6b8aa","Type":"ContainerDied","Data":"5d6ea38e37ce9cfefdd5912bc6d2809296b66f809c29e1c9bbadaa535a14757c"} Nov 03 13:45:00.434420 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.434321 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-8d8af2-6778d74d77-f5xjw" Nov 03 13:45:00.434420 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.434376 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-8d8af2-6778d74d77-f5xjw" event={"ID":"f5c4766a-e455-497c-ba06-716c55b6b8aa","Type":"ContainerDied","Data":"cf22dff6df37aa9f1ebc0e4268920e9e9705101bbd65d97dcf3dbdde1cead22f"} Nov 03 13:45:00.446757 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.446735 5979 scope.go:122] "RemoveContainer" containerID="10149c7f0295a4527780a4427491469fccef3771d539f615733ae686a7328110" Nov 03 13:45:00.447090 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:45:00.447065 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10149c7f0295a4527780a4427491469fccef3771d539f615733ae686a7328110\": container with ID starting with 10149c7f0295a4527780a4427491469fccef3771d539f615733ae686a7328110 not found: ID does not exist" containerID="10149c7f0295a4527780a4427491469fccef3771d539f615733ae686a7328110" Nov 03 13:45:00.447155 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.447102 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10149c7f0295a4527780a4427491469fccef3771d539f615733ae686a7328110"} err="failed to get container status \"10149c7f0295a4527780a4427491469fccef3771d539f615733ae686a7328110\": rpc error: code = NotFound desc = could not find container \"10149c7f0295a4527780a4427491469fccef3771d539f615733ae686a7328110\": container with ID starting with 10149c7f0295a4527780a4427491469fccef3771d539f615733ae686a7328110 not found: ID does not exist" Nov 03 13:45:00.447155 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.447126 5979 scope.go:122] "RemoveContainer" containerID="e3f0f7227cd6991da98ed826d98930eae2bce202c676a8e0b81b14f0e7fcde3f" Nov 03 13:45:00.456369 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.456345 5979 scope.go:122] "RemoveContainer" containerID="e3f0f7227cd6991da98ed826d98930eae2bce202c676a8e0b81b14f0e7fcde3f" Nov 03 13:45:00.456761 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:45:00.456742 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3f0f7227cd6991da98ed826d98930eae2bce202c676a8e0b81b14f0e7fcde3f\": container with ID starting with e3f0f7227cd6991da98ed826d98930eae2bce202c676a8e0b81b14f0e7fcde3f not found: ID does not exist" containerID="e3f0f7227cd6991da98ed826d98930eae2bce202c676a8e0b81b14f0e7fcde3f" Nov 03 13:45:00.456841 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.456771 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3f0f7227cd6991da98ed826d98930eae2bce202c676a8e0b81b14f0e7fcde3f"} err="failed to get container status \"e3f0f7227cd6991da98ed826d98930eae2bce202c676a8e0b81b14f0e7fcde3f\": rpc error: code = NotFound desc = could not find container \"e3f0f7227cd6991da98ed826d98930eae2bce202c676a8e0b81b14f0e7fcde3f\": container with ID starting with e3f0f7227cd6991da98ed826d98930eae2bce202c676a8e0b81b14f0e7fcde3f not found: ID does not exist" Nov 03 13:45:00.456841 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.456792 5979 scope.go:122] "RemoveContainer" containerID="5d6ea38e37ce9cfefdd5912bc6d2809296b66f809c29e1c9bbadaa535a14757c" Nov 03 13:45:00.466523 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.466464 5979 scope.go:122] "RemoveContainer" containerID="5d6ea38e37ce9cfefdd5912bc6d2809296b66f809c29e1c9bbadaa535a14757c" Nov 03 13:45:00.467052 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:45:00.467029 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d6ea38e37ce9cfefdd5912bc6d2809296b66f809c29e1c9bbadaa535a14757c\": container with ID starting with 5d6ea38e37ce9cfefdd5912bc6d2809296b66f809c29e1c9bbadaa535a14757c not found: ID does not exist" containerID="5d6ea38e37ce9cfefdd5912bc6d2809296b66f809c29e1c9bbadaa535a14757c" Nov 03 13:45:00.467112 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.467060 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d6ea38e37ce9cfefdd5912bc6d2809296b66f809c29e1c9bbadaa535a14757c"} err="failed to get container status \"5d6ea38e37ce9cfefdd5912bc6d2809296b66f809c29e1c9bbadaa535a14757c\": rpc error: code = NotFound desc = could not find container \"5d6ea38e37ce9cfefdd5912bc6d2809296b66f809c29e1c9bbadaa535a14757c\": container with ID starting with 5d6ea38e37ce9cfefdd5912bc6d2809296b66f809c29e1c9bbadaa535a14757c not found: ID does not exist" Nov 03 13:45:00.476949 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.476920 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-867kr\" (UniqueName: \"kubernetes.io/projected/bc2b1123-5771-4e33-bc6f-1654fec22a26-kube-api-access-867kr\") pod \"maas-api-key-cleanup-e2e-worker-w0-42c707-29895225-ctv9x\" (UID: \"bc2b1123-5771-4e33-bc6f-1654fec22a26\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895225-ctv9x" Nov 03 13:45:00.477115 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.476948 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg5v7\" (UniqueName: \"kubernetes.io/projected/3f54baec-8801-4ebf-86a8-e4dee3b46888-kube-api-access-pg5v7\") pod \"maas-api-key-cleanup-e2e-praxis-b-84e22c-29895225-mvwgc\" (UID: \"3f54baec-8801-4ebf-86a8-e4dee3b46888\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-84e22c-29895225-mvwgc" Nov 03 13:45:00.477115 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.476949 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdwgw\" (UniqueName: \"kubernetes.io/projected/bcd1cac1-4f49-4c3a-a46d-726a8a83798c-kube-api-access-sdwgw\") pod \"maas-api-key-cleanup-e2e-ipp-a-a770b9-29895225-gvxrk\" (UID: \"bcd1cac1-4f49-4c3a-a46d-726a8a83798c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a770b9-29895225-gvxrk" Nov 03 13:45:00.477268 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.477247 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lrdw\" (UniqueName: \"kubernetes.io/projected/9c8f3d30-d7ac-4c71-a738-003ebc263255-kube-api-access-7lrdw\") pod \"maas-api-key-cleanup-e2e-praxis-a-4b9658-29895225-mjbc4\" (UID: \"9c8f3d30-d7ac-4c71-a738-003ebc263255\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4b9658-29895225-mjbc4" Nov 03 13:45:00.512393 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.512358 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a770b9-29895225-gvxrk" Nov 03 13:45:00.519600 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.519569 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5nqr\" (UniqueName: \"kubernetes.io/projected/e0df41fd-c6ea-44af-a973-05243f6ddaae-kube-api-access-n5nqr\") pod \"maas-api-key-cleanup-e2e-worker-w1-34b13c-29895225-568vs\" (UID: \"e0df41fd-c6ea-44af-a973-05243f6ddaae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895225-568vs" Nov 03 13:45:00.519730 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.519687 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0df41fd-c6ea-44af-a973-05243f6ddaae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-34b13c-29895225-568vs\" (UID: \"e0df41fd-c6ea-44af-a973-05243f6ddaae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895225-568vs" Nov 03 13:45:00.519805 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.519776 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e40ff1a-b3d1-4aee-a76d-57e1373945ee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5bb463-29895225-tllnw\" (UID: \"3e40ff1a-b3d1-4aee-a76d-57e1373945ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895225-tllnw" Nov 03 13:45:00.520048 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.520015 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk9hd\" (UniqueName: \"kubernetes.io/projected/3e40ff1a-b3d1-4aee-a76d-57e1373945ee-kube-api-access-tk9hd\") pod \"maas-api-key-cleanup-e2e-worker-w3-5bb463-29895225-tllnw\" (UID: \"3e40ff1a-b3d1-4aee-a76d-57e1373945ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895225-tllnw" Nov 03 13:45:00.520512 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.520463 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e40ff1a-b3d1-4aee-a76d-57e1373945ee-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5bb463-29895225-tllnw\" (UID: \"3e40ff1a-b3d1-4aee-a76d-57e1373945ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895225-tllnw" Nov 03 13:45:00.520620 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.520565 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0df41fd-c6ea-44af-a973-05243f6ddaae-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-34b13c-29895225-568vs\" (UID: \"e0df41fd-c6ea-44af-a973-05243f6ddaae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895225-568vs" Nov 03 13:45:00.550513 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.548136 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895225-ctv9x" Nov 03 13:45:00.555616 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.555589 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-84e22c-29895225-mvwgc" Nov 03 13:45:00.562419 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.562390 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4b9658-29895225-mjbc4" Nov 03 13:45:00.584007 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.583973 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk9hd\" (UniqueName: \"kubernetes.io/projected/3e40ff1a-b3d1-4aee-a76d-57e1373945ee-kube-api-access-tk9hd\") pod \"maas-api-key-cleanup-e2e-worker-w3-5bb463-29895225-tllnw\" (UID: \"3e40ff1a-b3d1-4aee-a76d-57e1373945ee\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895225-tllnw" Nov 03 13:45:00.594367 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.594321 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5nqr\" (UniqueName: \"kubernetes.io/projected/e0df41fd-c6ea-44af-a973-05243f6ddaae-kube-api-access-n5nqr\") pod \"maas-api-key-cleanup-e2e-worker-w1-34b13c-29895225-568vs\" (UID: \"e0df41fd-c6ea-44af-a973-05243f6ddaae\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895225-568vs" Nov 03 13:45:00.619650 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.619616 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895225-568vs" Nov 03 13:45:00.655825 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.655792 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895225-tllnw" Nov 03 13:45:00.904548 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.904517 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a770b9-29895225-gvxrk"] Nov 03 13:45:00.911368 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:45:00.911337 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcd1cac1_4f49_4c3a_a46d_726a8a83798c.slice/crio-2eeb203e9398b1da3044446374f043116f8ebe02b1e3f74bc9392d21d9617ab0 WatchSource:0}: Error finding container 2eeb203e9398b1da3044446374f043116f8ebe02b1e3f74bc9392d21d9617ab0: Status 404 returned error can't find the container with id 2eeb203e9398b1da3044446374f043116f8ebe02b1e3f74bc9392d21d9617ab0 Nov 03 13:45:00.969694 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:00.969617 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895225-ctv9x"] Nov 03 13:45:00.974171 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:45:00.974144 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc2b1123_5771_4e33_bc6f_1654fec22a26.slice/crio-d5daeca6fc030d0b9e9cfcc911900617a8a7392bfa11adb730c513a8061346a6 WatchSource:0}: Error finding container d5daeca6fc030d0b9e9cfcc911900617a8a7392bfa11adb730c513a8061346a6: Status 404 returned error can't find the container with id d5daeca6fc030d0b9e9cfcc911900617a8a7392bfa11adb730c513a8061346a6 Nov 03 13:45:01.004265 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:01.004238 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4b9658-29895225-mjbc4"] Nov 03 13:45:01.011803 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:45:01.011776 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c8f3d30_d7ac_4c71_a738_003ebc263255.slice/crio-6068f0e09705eebccbeb38815f984fd2a135dc10496543066e3074dde11c63d5 WatchSource:0}: Error finding container 6068f0e09705eebccbeb38815f984fd2a135dc10496543066e3074dde11c63d5: Status 404 returned error can't find the container with id 6068f0e09705eebccbeb38815f984fd2a135dc10496543066e3074dde11c63d5 Nov 03 13:45:01.013340 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:01.013314 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq"] Nov 03 13:45:01.025562 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:01.025528 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-84e22c-29895225-mvwgc"] Nov 03 13:45:01.027547 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:01.027527 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j9rbq"] Nov 03 13:45:01.030064 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:45:01.030040 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f54baec_8801_4ebf_86a8_e4dee3b46888.slice/crio-38333336f7158971f3db8fd298e3d45774d5ff17476edd02365754ce97cff1e5 WatchSource:0}: Error finding container 38333336f7158971f3db8fd298e3d45774d5ff17476edd02365754ce97cff1e5: Status 404 returned error can't find the container with id 38333336f7158971f3db8fd298e3d45774d5ff17476edd02365754ce97cff1e5 Nov 03 13:45:01.093290 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:01.093236 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895225-568vs"] Nov 03 13:45:01.100356 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:45:01.100322 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0df41fd_c6ea_44af_a973_05243f6ddaae.slice/crio-596e7ce59572d38b6601f7d75b6b8374fb151fce8ba9632e199a56fcc8342c00 WatchSource:0}: Error finding container 596e7ce59572d38b6601f7d75b6b8374fb151fce8ba9632e199a56fcc8342c00: Status 404 returned error can't find the container with id 596e7ce59572d38b6601f7d75b6b8374fb151fce8ba9632e199a56fcc8342c00 Nov 03 13:45:01.101614 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:01.101592 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895225-tllnw"] Nov 03 13:45:01.117597 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:45:01.117563 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e40ff1a_b3d1_4aee_a76d_57e1373945ee.slice/crio-411c8be54c46d49864eccacad1cf382ccd8ae772dae3b471499e427101777fc3 WatchSource:0}: Error finding container 411c8be54c46d49864eccacad1cf382ccd8ae772dae3b471499e427101777fc3: Status 404 returned error can't find the container with id 411c8be54c46d49864eccacad1cf382ccd8ae772dae3b471499e427101777fc3 Nov 03 13:45:01.132629 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:01.132598 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-8d8af2-6778d74d77-f5xjw"] Nov 03 13:45:01.145653 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:01.144610 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-8d8af2-6778d74d77-f5xjw"] Nov 03 13:45:01.256883 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:01.256858 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-8d8af2-6979795ddc-f4x64"] Nov 03 13:45:01.266674 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:01.266643 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-8d8af2-6979795ddc-f4x64"] Nov 03 13:45:01.467634 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:01.467138 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-84e22c-29895225-mvwgc" event={"ID":"3f54baec-8801-4ebf-86a8-e4dee3b46888","Type":"ContainerStarted","Data":"38333336f7158971f3db8fd298e3d45774d5ff17476edd02365754ce97cff1e5"} Nov 03 13:45:01.471638 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:01.470912 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895225-568vs" event={"ID":"e0df41fd-c6ea-44af-a973-05243f6ddaae","Type":"ContainerStarted","Data":"596e7ce59572d38b6601f7d75b6b8374fb151fce8ba9632e199a56fcc8342c00"} Nov 03 13:45:01.473832 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:01.472893 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a770b9-29895225-gvxrk" event={"ID":"bcd1cac1-4f49-4c3a-a46d-726a8a83798c","Type":"ContainerStarted","Data":"2f3a0d2dd0483af21dc1c0de87b0908de07d617f951e57df75d530b976ce83c8"} Nov 03 13:45:01.473832 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:01.472943 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a770b9-29895225-gvxrk" event={"ID":"bcd1cac1-4f49-4c3a-a46d-726a8a83798c","Type":"ContainerStarted","Data":"2eeb203e9398b1da3044446374f043116f8ebe02b1e3f74bc9392d21d9617ab0"} Nov 03 13:45:01.474524 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:01.474494 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895225-ctv9x" event={"ID":"bc2b1123-5771-4e33-bc6f-1654fec22a26","Type":"ContainerStarted","Data":"d5daeca6fc030d0b9e9cfcc911900617a8a7392bfa11adb730c513a8061346a6"} Nov 03 13:45:01.477261 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:01.477231 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895225-tllnw" event={"ID":"3e40ff1a-b3d1-4aee-a76d-57e1373945ee","Type":"ContainerStarted","Data":"411c8be54c46d49864eccacad1cf382ccd8ae772dae3b471499e427101777fc3"} Nov 03 13:45:01.478920 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:01.478897 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4b9658-29895225-mjbc4" event={"ID":"9c8f3d30-d7ac-4c71-a738-003ebc263255","Type":"ContainerStarted","Data":"6068f0e09705eebccbeb38815f984fd2a135dc10496543066e3074dde11c63d5"} Nov 03 13:45:01.596642 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:01.596575 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a770b9-29895225-gvxrk" podStartSLOduration=1.596552739 podStartE2EDuration="1.596552739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:45:01.593190938 +0000 UTC m=+2383.603879182" watchObservedRunningTime="2026-11-03 13:45:01.596552739 +0000 UTC m=+2383.607240982" Nov 03 13:45:01.707051 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:01.706979 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4b9658-29895225-mjbc4" podStartSLOduration=1.706960359 podStartE2EDuration="1.706960359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:45:01.705556689 +0000 UTC m=+2383.716244929" watchObservedRunningTime="2026-11-03 13:45:01.706960359 +0000 UTC m=+2383.717648605" Nov 03 13:45:01.777588 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:01.777555 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-977f32-576d49df6c-8khqq"] Nov 03 13:45:01.781916 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:01.781892 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-977f32-576d49df6c-8khqq" Nov 03 13:45:01.849506 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:01.849397 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-977f32-dockercfg-jxw82\"" Nov 03 13:45:01.849506 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:01.849437 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-977f32\"" Nov 03 13:45:01.852168 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:01.852142 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-977f32-576d49df6c-8khqq"] Nov 03 13:45:01.854652 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:01.854628 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-977f32-7bcd65dd7f-cjkdb"] Nov 03 13:45:01.859282 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:01.859260 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-977f32-7bcd65dd7f-cjkdb" Nov 03 13:45:01.914014 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:01.913967 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-977f32-7bcd65dd7f-cjkdb"] Nov 03 13:45:01.958932 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:01.949279 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd8dc463-bd1d-4451-afba-337d5a6a74ba-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-977f32-7bcd65dd7f-cjkdb\" (UID: \"dd8dc463-bd1d-4451-afba-337d5a6a74ba\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-977f32-7bcd65dd7f-cjkdb" Nov 03 13:45:01.958932 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:01.949406 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsd9z\" (UniqueName: \"kubernetes.io/projected/dd8dc463-bd1d-4451-afba-337d5a6a74ba-kube-api-access-zsd9z\") pod \"payload-processing-e2e-praxis-a-977f32-7bcd65dd7f-cjkdb\" (UID: \"dd8dc463-bd1d-4451-afba-337d5a6a74ba\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-977f32-7bcd65dd7f-cjkdb" Nov 03 13:45:01.958932 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:01.949509 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xpwq\" (UniqueName: \"kubernetes.io/projected/5dc7b3ba-e302-41a5-8ef5-555da33e6222-kube-api-access-4xpwq\") pod \"payload-pre-processing-e2e-praxis-a-977f32-576d49df6c-8khqq\" (UID: \"5dc7b3ba-e302-41a5-8ef5-555da33e6222\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-977f32-576d49df6c-8khqq" Nov 03 13:45:01.958932 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:01.949564 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dc7b3ba-e302-41a5-8ef5-555da33e6222-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-977f32-576d49df6c-8khqq\" (UID: \"5dc7b3ba-e302-41a5-8ef5-555da33e6222\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-977f32-576d49df6c-8khqq" Nov 03 13:45:02.050948 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:02.050425 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xpwq\" (UniqueName: \"kubernetes.io/projected/5dc7b3ba-e302-41a5-8ef5-555da33e6222-kube-api-access-4xpwq\") pod \"payload-pre-processing-e2e-praxis-a-977f32-576d49df6c-8khqq\" (UID: \"5dc7b3ba-e302-41a5-8ef5-555da33e6222\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-977f32-576d49df6c-8khqq" Nov 03 13:45:02.050948 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:02.050595 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dc7b3ba-e302-41a5-8ef5-555da33e6222-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-977f32-576d49df6c-8khqq\" (UID: \"5dc7b3ba-e302-41a5-8ef5-555da33e6222\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-977f32-576d49df6c-8khqq" Nov 03 13:45:02.050948 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:02.050756 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd8dc463-bd1d-4451-afba-337d5a6a74ba-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-977f32-7bcd65dd7f-cjkdb\" (UID: \"dd8dc463-bd1d-4451-afba-337d5a6a74ba\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-977f32-7bcd65dd7f-cjkdb" Nov 03 13:45:02.050948 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:02.050856 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsd9z\" (UniqueName: \"kubernetes.io/projected/dd8dc463-bd1d-4451-afba-337d5a6a74ba-kube-api-access-zsd9z\") pod \"payload-processing-e2e-praxis-a-977f32-7bcd65dd7f-cjkdb\" (UID: \"dd8dc463-bd1d-4451-afba-337d5a6a74ba\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-977f32-7bcd65dd7f-cjkdb" Nov 03 13:45:02.052204 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:02.052175 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dc7b3ba-e302-41a5-8ef5-555da33e6222-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-977f32-576d49df6c-8khqq\" (UID: \"5dc7b3ba-e302-41a5-8ef5-555da33e6222\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-977f32-576d49df6c-8khqq" Nov 03 13:45:02.053519 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:02.053481 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd8dc463-bd1d-4451-afba-337d5a6a74ba-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-977f32-7bcd65dd7f-cjkdb\" (UID: \"dd8dc463-bd1d-4451-afba-337d5a6a74ba\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-977f32-7bcd65dd7f-cjkdb" Nov 03 13:45:02.116000 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:02.115914 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xpwq\" (UniqueName: \"kubernetes.io/projected/5dc7b3ba-e302-41a5-8ef5-555da33e6222-kube-api-access-4xpwq\") pod \"payload-pre-processing-e2e-praxis-a-977f32-576d49df6c-8khqq\" (UID: \"5dc7b3ba-e302-41a5-8ef5-555da33e6222\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-977f32-576d49df6c-8khqq" Nov 03 13:45:02.116374 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:02.116342 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsd9z\" (UniqueName: \"kubernetes.io/projected/dd8dc463-bd1d-4451-afba-337d5a6a74ba-kube-api-access-zsd9z\") pod \"payload-processing-e2e-praxis-a-977f32-7bcd65dd7f-cjkdb\" (UID: \"dd8dc463-bd1d-4451-afba-337d5a6a74ba\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-977f32-7bcd65dd7f-cjkdb" Nov 03 13:45:02.120809 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:02.120786 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad7c3ee7-2be7-4124-9a89-1be00bbc1b21" path="/var/lib/kubelet/pods/ad7c3ee7-2be7-4124-9a89-1be00bbc1b21/volumes" Nov 03 13:45:02.121217 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:02.121204 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b92a7bfb-f886-4b87-9c00-e07d8fb43732" path="/var/lib/kubelet/pods/b92a7bfb-f886-4b87-9c00-e07d8fb43732/volumes" Nov 03 13:45:02.121639 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:02.121625 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5c4766a-e455-497c-ba06-716c55b6b8aa" path="/var/lib/kubelet/pods/f5c4766a-e455-497c-ba06-716c55b6b8aa/volumes" Nov 03 13:45:02.175753 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:02.175705 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-977f32-7bcd65dd7f-cjkdb" Nov 03 13:45:02.397032 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:02.396995 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-977f32-576d49df6c-8khqq" Nov 03 13:45:02.498313 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:02.497517 5979 generic.go:356] "Generic (PLEG): container finished" podID="3e40ff1a-b3d1-4aee-a76d-57e1373945ee" containerID="34d5bc9521960c50f54830b9f5b6cfd152cff912d15518d915c5504d11ad9d79" exitCode=60 Nov 03 13:45:02.498313 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:02.497597 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895225-tllnw" event={"ID":"3e40ff1a-b3d1-4aee-a76d-57e1373945ee","Type":"ContainerDied","Data":"34d5bc9521960c50f54830b9f5b6cfd152cff912d15518d915c5504d11ad9d79"} Nov 03 13:45:02.498313 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:02.497958 5979 scope.go:122] "RemoveContainer" containerID="34d5bc9521960c50f54830b9f5b6cfd152cff912d15518d915c5504d11ad9d79" Nov 03 13:45:02.502213 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:02.502192 5979 generic.go:356] "Generic (PLEG): container finished" podID="9c8f3d30-d7ac-4c71-a738-003ebc263255" containerID="5f55283ba68151de9973677c7813513aea34f1216cad78e4b2fe7eae9ba1752d" exitCode=60 Nov 03 13:45:02.502305 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:02.502278 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4b9658-29895225-mjbc4" event={"ID":"9c8f3d30-d7ac-4c71-a738-003ebc263255","Type":"ContainerDied","Data":"5f55283ba68151de9973677c7813513aea34f1216cad78e4b2fe7eae9ba1752d"} Nov 03 13:45:02.502607 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:02.502586 5979 scope.go:122] "RemoveContainer" containerID="5f55283ba68151de9973677c7813513aea34f1216cad78e4b2fe7eae9ba1752d" Nov 03 13:45:02.503908 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:02.503875 5979 generic.go:356] "Generic (PLEG): container finished" podID="3f54baec-8801-4ebf-86a8-e4dee3b46888" containerID="77c75e9d0a00c478ad6948f7956f0d4e9ef4fb240948d9f632f6893fb21e7380" exitCode=60 Nov 03 13:45:02.504052 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:02.504027 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-84e22c-29895225-mvwgc" event={"ID":"3f54baec-8801-4ebf-86a8-e4dee3b46888","Type":"ContainerDied","Data":"77c75e9d0a00c478ad6948f7956f0d4e9ef4fb240948d9f632f6893fb21e7380"} Nov 03 13:45:02.504242 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:02.504225 5979 scope.go:122] "RemoveContainer" containerID="77c75e9d0a00c478ad6948f7956f0d4e9ef4fb240948d9f632f6893fb21e7380" Nov 03 13:45:02.505799 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:02.505776 5979 generic.go:356] "Generic (PLEG): container finished" podID="e0df41fd-c6ea-44af-a973-05243f6ddaae" containerID="63a244cf5f49cbffb8a9195eca33deaf061b531377f3c8262a5dd9dcbf0b5585" exitCode=60 Nov 03 13:45:02.505938 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:02.505845 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895225-568vs" event={"ID":"e0df41fd-c6ea-44af-a973-05243f6ddaae","Type":"ContainerDied","Data":"63a244cf5f49cbffb8a9195eca33deaf061b531377f3c8262a5dd9dcbf0b5585"} Nov 03 13:45:02.506060 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:02.506038 5979 scope.go:122] "RemoveContainer" containerID="63a244cf5f49cbffb8a9195eca33deaf061b531377f3c8262a5dd9dcbf0b5585" Nov 03 13:45:02.510497 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:02.509338 5979 generic.go:356] "Generic (PLEG): container finished" podID="bcd1cac1-4f49-4c3a-a46d-726a8a83798c" containerID="2f3a0d2dd0483af21dc1c0de87b0908de07d617f951e57df75d530b976ce83c8" exitCode=60 Nov 03 13:45:02.510497 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:02.509427 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a770b9-29895225-gvxrk" event={"ID":"bcd1cac1-4f49-4c3a-a46d-726a8a83798c","Type":"ContainerDied","Data":"2f3a0d2dd0483af21dc1c0de87b0908de07d617f951e57df75d530b976ce83c8"} Nov 03 13:45:02.510497 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:02.509741 5979 scope.go:122] "RemoveContainer" containerID="2f3a0d2dd0483af21dc1c0de87b0908de07d617f951e57df75d530b976ce83c8" Nov 03 13:45:02.522972 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:02.520081 5979 generic.go:356] "Generic (PLEG): container finished" podID="bc2b1123-5771-4e33-bc6f-1654fec22a26" containerID="90251580eb9ece6dbde88d1d5c0ab12b612aef893a63b88e6fe8ba64d7350f06" exitCode=60 Nov 03 13:45:02.522972 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:02.520137 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895225-ctv9x" event={"ID":"bc2b1123-5771-4e33-bc6f-1654fec22a26","Type":"ContainerDied","Data":"90251580eb9ece6dbde88d1d5c0ab12b612aef893a63b88e6fe8ba64d7350f06"} Nov 03 13:45:02.527781 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:02.527754 5979 scope.go:122] "RemoveContainer" containerID="90251580eb9ece6dbde88d1d5c0ab12b612aef893a63b88e6fe8ba64d7350f06" Nov 03 13:45:02.573780 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:02.573733 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-977f32-7bcd65dd7f-cjkdb"] Nov 03 13:45:02.582095 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:45:02.582068 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd8dc463_bd1d_4451_afba_337d5a6a74ba.slice/crio-9759a07aa9c88f8b1aba18331ea20c6969e2761b178ca90137dcea2418de55c7 WatchSource:0}: Error finding container 9759a07aa9c88f8b1aba18331ea20c6969e2761b178ca90137dcea2418de55c7: Status 404 returned error can't find the container with id 9759a07aa9c88f8b1aba18331ea20c6969e2761b178ca90137dcea2418de55c7 Nov 03 13:45:02.797076 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:02.797038 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-977f32-576d49df6c-8khqq"] Nov 03 13:45:02.803759 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:45:02.803729 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dc7b3ba_e302_41a5_8ef5_555da33e6222.slice/crio-61ca111b33aeadc4d410371fc37c4ae8229e24a01fd3aa37ff850720f5e1375e WatchSource:0}: Error finding container 61ca111b33aeadc4d410371fc37c4ae8229e24a01fd3aa37ff850720f5e1375e: Status 404 returned error can't find the container with id 61ca111b33aeadc4d410371fc37c4ae8229e24a01fd3aa37ff850720f5e1375e Nov 03 13:45:03.530532 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:03.530462 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895225-568vs" event={"ID":"e0df41fd-c6ea-44af-a973-05243f6ddaae","Type":"ContainerStarted","Data":"d0c56ffec33ec34541a4744f6da630b8105fa0e354e0a980a98edf7d07bfe3e4"} Nov 03 13:45:03.532498 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:03.532446 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a770b9-29895225-gvxrk" event={"ID":"bcd1cac1-4f49-4c3a-a46d-726a8a83798c","Type":"ContainerStarted","Data":"3e00ca2be1baf03b518466df14f21ef486d83cd25d1917f2d109bcb10a1866a9"} Nov 03 13:45:03.534124 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:03.534098 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-977f32-576d49df6c-8khqq" event={"ID":"5dc7b3ba-e302-41a5-8ef5-555da33e6222","Type":"ContainerStarted","Data":"f697b95b7979a2d2a29fd81240a7e51820f24930950fd03163b2404089dd582b"} Nov 03 13:45:03.534234 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:03.534142 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-977f32-576d49df6c-8khqq" event={"ID":"5dc7b3ba-e302-41a5-8ef5-555da33e6222","Type":"ContainerStarted","Data":"61ca111b33aeadc4d410371fc37c4ae8229e24a01fd3aa37ff850720f5e1375e"} Nov 03 13:45:03.534234 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:03.534201 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-977f32-576d49df6c-8khqq" Nov 03 13:45:03.536546 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:03.536237 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895225-ctv9x" event={"ID":"bc2b1123-5771-4e33-bc6f-1654fec22a26","Type":"ContainerStarted","Data":"47b07e34b19cdd9cea7ce6941de58ae4866becd27f39a3042d00f9c70a47ed5a"} Nov 03 13:45:03.538094 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:03.538066 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-977f32-7bcd65dd7f-cjkdb" event={"ID":"dd8dc463-bd1d-4451-afba-337d5a6a74ba","Type":"ContainerStarted","Data":"0495da0f0dc82e18f272f20a05249c9fdbbbf7a55e35fd6e8b4f87b509532521"} Nov 03 13:45:03.538232 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:03.538126 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-977f32-7bcd65dd7f-cjkdb" event={"ID":"dd8dc463-bd1d-4451-afba-337d5a6a74ba","Type":"ContainerStarted","Data":"9759a07aa9c88f8b1aba18331ea20c6969e2761b178ca90137dcea2418de55c7"} Nov 03 13:45:03.538232 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:03.538214 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-977f32-7bcd65dd7f-cjkdb" Nov 03 13:45:03.540260 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:03.540239 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895225-tllnw" event={"ID":"3e40ff1a-b3d1-4aee-a76d-57e1373945ee","Type":"ContainerStarted","Data":"0ccd2b5852656c06e4fee212e71f0b4ca17e9141b8545985c523ff796b17c8ef"} Nov 03 13:45:03.542578 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:03.542558 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4b9658-29895225-mjbc4" event={"ID":"9c8f3d30-d7ac-4c71-a738-003ebc263255","Type":"ContainerStarted","Data":"4345bfeed345c217a8d753e627ac682c54255d4677df4049347fc42b531f2c1e"} Nov 03 13:45:03.545310 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:03.545285 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-84e22c-29895225-mvwgc" event={"ID":"3f54baec-8801-4ebf-86a8-e4dee3b46888","Type":"ContainerStarted","Data":"a2ff2dbd6d0c37128590015da3d34a6243300e95dac998bc7f8c41398f27d39b"} Nov 03 13:45:03.665007 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:03.664958 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895225-568vs" podStartSLOduration=3.6649438 podStartE2EDuration="3.6649438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:45:03.663762485 +0000 UTC m=+2385.674450728" watchObservedRunningTime="2026-11-03 13:45:03.6649438 +0000 UTC m=+2385.675632073" Nov 03 13:45:03.786994 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:03.786898 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895225-tllnw" podStartSLOduration=3.7868843119999998 podStartE2EDuration="3.786884312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:45:03.785453656 +0000 UTC m=+2385.796141916" watchObservedRunningTime="2026-11-03 13:45:03.786884312 +0000 UTC m=+2385.797572554" Nov 03 13:45:04.036242 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:04.036181 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-977f32-7bcd65dd7f-cjkdb" podStartSLOduration=3.036162531 podStartE2EDuration="3.036162531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:45:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:45:04.033940277 +0000 UTC m=+2386.044628520" watchObservedRunningTime="2026-11-03 13:45:04.036162531 +0000 UTC m=+2386.046850766" Nov 03 13:45:04.169412 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:04.169344 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895225-ctv9x" podStartSLOduration=4.169323908 podStartE2EDuration="4.169323908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:45:04.164312718 +0000 UTC m=+2386.175000961" watchObservedRunningTime="2026-11-03 13:45:04.169323908 +0000 UTC m=+2386.180012151" Nov 03 13:45:04.281013 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:04.280951 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-977f32-576d49df6c-8khqq" podStartSLOduration=3.280932906 podStartE2EDuration="3.280932906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:45:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:45:04.278377901 +0000 UTC m=+2386.289066143" watchObservedRunningTime="2026-11-03 13:45:04.280932906 +0000 UTC m=+2386.291621150" Nov 03 13:45:04.515984 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:04.515862 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-84e22c-29895225-mvwgc" podStartSLOduration=4.515841928 podStartE2EDuration="4.515841928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:45:04.513347029 +0000 UTC m=+2386.524035273" watchObservedRunningTime="2026-11-03 13:45:04.515841928 +0000 UTC m=+2386.526530171" Nov 03 13:45:04.555587 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:04.555551 5979 generic.go:356] "Generic (PLEG): container finished" podID="3e40ff1a-b3d1-4aee-a76d-57e1373945ee" containerID="0ccd2b5852656c06e4fee212e71f0b4ca17e9141b8545985c523ff796b17c8ef" exitCode=60 Nov 03 13:45:04.555794 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:04.555643 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895225-tllnw" event={"ID":"3e40ff1a-b3d1-4aee-a76d-57e1373945ee","Type":"ContainerDied","Data":"0ccd2b5852656c06e4fee212e71f0b4ca17e9141b8545985c523ff796b17c8ef"} Nov 03 13:45:04.555794 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:04.555707 5979 scope.go:122] "RemoveContainer" containerID="34d5bc9521960c50f54830b9f5b6cfd152cff912d15518d915c5504d11ad9d79" Nov 03 13:45:04.555991 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:04.555960 5979 scope.go:122] "RemoveContainer" containerID="0ccd2b5852656c06e4fee212e71f0b4ca17e9141b8545985c523ff796b17c8ef" Nov 03 13:45:04.556314 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:45:04.556289 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5bb463-29895225-tllnw_odh-ai-gateway-infra(3e40ff1a-b3d1-4aee-a76d-57e1373945ee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895225-tllnw" podUID="3e40ff1a-b3d1-4aee-a76d-57e1373945ee" Nov 03 13:45:04.557908 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:04.557882 5979 generic.go:356] "Generic (PLEG): container finished" podID="9c8f3d30-d7ac-4c71-a738-003ebc263255" containerID="4345bfeed345c217a8d753e627ac682c54255d4677df4049347fc42b531f2c1e" exitCode=60 Nov 03 13:45:04.558003 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:04.557922 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4b9658-29895225-mjbc4" event={"ID":"9c8f3d30-d7ac-4c71-a738-003ebc263255","Type":"ContainerDied","Data":"4345bfeed345c217a8d753e627ac682c54255d4677df4049347fc42b531f2c1e"} Nov 03 13:45:04.558229 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:04.558212 5979 scope.go:122] "RemoveContainer" containerID="4345bfeed345c217a8d753e627ac682c54255d4677df4049347fc42b531f2c1e" Nov 03 13:45:04.558574 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:45:04.558551 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-4b9658-29895225-mjbc4_odh-ai-gateway-infra(9c8f3d30-d7ac-4c71-a738-003ebc263255)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4b9658-29895225-mjbc4" podUID="9c8f3d30-d7ac-4c71-a738-003ebc263255" Nov 03 13:45:04.559740 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:04.559720 5979 generic.go:356] "Generic (PLEG): container finished" podID="3f54baec-8801-4ebf-86a8-e4dee3b46888" containerID="a2ff2dbd6d0c37128590015da3d34a6243300e95dac998bc7f8c41398f27d39b" exitCode=60 Nov 03 13:45:04.559849 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:04.559830 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-84e22c-29895225-mvwgc" event={"ID":"3f54baec-8801-4ebf-86a8-e4dee3b46888","Type":"ContainerDied","Data":"a2ff2dbd6d0c37128590015da3d34a6243300e95dac998bc7f8c41398f27d39b"} Nov 03 13:45:04.560078 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:04.560062 5979 scope.go:122] "RemoveContainer" containerID="a2ff2dbd6d0c37128590015da3d34a6243300e95dac998bc7f8c41398f27d39b" Nov 03 13:45:04.560344 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:45:04.560326 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-84e22c-29895225-mvwgc_odh-ai-gateway-infra(3f54baec-8801-4ebf-86a8-e4dee3b46888)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-84e22c-29895225-mvwgc" podUID="3f54baec-8801-4ebf-86a8-e4dee3b46888" Nov 03 13:45:04.561752 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:04.561730 5979 generic.go:356] "Generic (PLEG): container finished" podID="e0df41fd-c6ea-44af-a973-05243f6ddaae" containerID="d0c56ffec33ec34541a4744f6da630b8105fa0e354e0a980a98edf7d07bfe3e4" exitCode=60 Nov 03 13:45:04.561832 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:04.561805 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895225-568vs" event={"ID":"e0df41fd-c6ea-44af-a973-05243f6ddaae","Type":"ContainerDied","Data":"d0c56ffec33ec34541a4744f6da630b8105fa0e354e0a980a98edf7d07bfe3e4"} Nov 03 13:45:04.561999 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:04.561983 5979 scope.go:122] "RemoveContainer" containerID="d0c56ffec33ec34541a4744f6da630b8105fa0e354e0a980a98edf7d07bfe3e4" Nov 03 13:45:04.562238 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:45:04.562216 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-34b13c-29895225-568vs_odh-ai-gateway-infra(e0df41fd-c6ea-44af-a973-05243f6ddaae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895225-568vs" podUID="e0df41fd-c6ea-44af-a973-05243f6ddaae" Nov 03 13:45:04.563672 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:04.563649 5979 generic.go:356] "Generic (PLEG): container finished" podID="bcd1cac1-4f49-4c3a-a46d-726a8a83798c" containerID="3e00ca2be1baf03b518466df14f21ef486d83cd25d1917f2d109bcb10a1866a9" exitCode=60 Nov 03 13:45:04.563764 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:04.563724 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a770b9-29895225-gvxrk" event={"ID":"bcd1cac1-4f49-4c3a-a46d-726a8a83798c","Type":"ContainerDied","Data":"3e00ca2be1baf03b518466df14f21ef486d83cd25d1917f2d109bcb10a1866a9"} Nov 03 13:45:04.563938 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:04.563924 5979 scope.go:122] "RemoveContainer" containerID="3e00ca2be1baf03b518466df14f21ef486d83cd25d1917f2d109bcb10a1866a9" Nov 03 13:45:04.564188 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:45:04.564154 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-a770b9-29895225-gvxrk_odh-ai-gateway-infra(bcd1cac1-4f49-4c3a-a46d-726a8a83798c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a770b9-29895225-gvxrk" podUID="bcd1cac1-4f49-4c3a-a46d-726a8a83798c" Nov 03 13:45:04.565582 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:04.565562 5979 generic.go:356] "Generic (PLEG): container finished" podID="bc2b1123-5771-4e33-bc6f-1654fec22a26" containerID="47b07e34b19cdd9cea7ce6941de58ae4866becd27f39a3042d00f9c70a47ed5a" exitCode=60 Nov 03 13:45:04.565674 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:04.565591 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895225-ctv9x" event={"ID":"bc2b1123-5771-4e33-bc6f-1654fec22a26","Type":"ContainerDied","Data":"47b07e34b19cdd9cea7ce6941de58ae4866becd27f39a3042d00f9c70a47ed5a"} Nov 03 13:45:04.565854 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:04.565832 5979 scope.go:122] "RemoveContainer" containerID="47b07e34b19cdd9cea7ce6941de58ae4866becd27f39a3042d00f9c70a47ed5a" Nov 03 13:45:04.566105 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:45:04.566085 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-42c707-29895225-ctv9x_odh-ai-gateway-infra(bc2b1123-5771-4e33-bc6f-1654fec22a26)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895225-ctv9x" podUID="bc2b1123-5771-4e33-bc6f-1654fec22a26" Nov 03 13:45:04.569942 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:04.569925 5979 scope.go:122] "RemoveContainer" containerID="5f55283ba68151de9973677c7813513aea34f1216cad78e4b2fe7eae9ba1752d" Nov 03 13:45:04.582906 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:04.582886 5979 scope.go:122] "RemoveContainer" containerID="77c75e9d0a00c478ad6948f7956f0d4e9ef4fb240948d9f632f6893fb21e7380" Nov 03 13:45:04.595503 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:04.595483 5979 scope.go:122] "RemoveContainer" containerID="63a244cf5f49cbffb8a9195eca33deaf061b531377f3c8262a5dd9dcbf0b5585" Nov 03 13:45:04.608616 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:04.608592 5979 scope.go:122] "RemoveContainer" containerID="2f3a0d2dd0483af21dc1c0de87b0908de07d617f951e57df75d530b976ce83c8" Nov 03 13:45:04.624202 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:04.624184 5979 scope.go:122] "RemoveContainer" containerID="90251580eb9ece6dbde88d1d5c0ab12b612aef893a63b88e6fe8ba64d7350f06" Nov 03 13:45:05.574555 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:05.574518 5979 scope.go:122] "RemoveContainer" containerID="d0c56ffec33ec34541a4744f6da630b8105fa0e354e0a980a98edf7d07bfe3e4" Nov 03 13:45:05.575035 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:45:05.574896 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-34b13c-29895225-568vs_odh-ai-gateway-infra(e0df41fd-c6ea-44af-a973-05243f6ddaae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895225-568vs" podUID="e0df41fd-c6ea-44af-a973-05243f6ddaae" Nov 03 13:45:05.576119 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:05.576094 5979 scope.go:122] "RemoveContainer" containerID="3e00ca2be1baf03b518466df14f21ef486d83cd25d1917f2d109bcb10a1866a9" Nov 03 13:45:05.576393 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:45:05.576374 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-a770b9-29895225-gvxrk_odh-ai-gateway-infra(bcd1cac1-4f49-4c3a-a46d-726a8a83798c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a770b9-29895225-gvxrk" podUID="bcd1cac1-4f49-4c3a-a46d-726a8a83798c" Nov 03 13:45:05.577742 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:05.577720 5979 scope.go:122] "RemoveContainer" containerID="47b07e34b19cdd9cea7ce6941de58ae4866becd27f39a3042d00f9c70a47ed5a" Nov 03 13:45:05.578003 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:45:05.577979 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-42c707-29895225-ctv9x_odh-ai-gateway-infra(bc2b1123-5771-4e33-bc6f-1654fec22a26)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895225-ctv9x" podUID="bc2b1123-5771-4e33-bc6f-1654fec22a26" Nov 03 13:45:05.579397 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:05.579377 5979 scope.go:122] "RemoveContainer" containerID="0ccd2b5852656c06e4fee212e71f0b4ca17e9141b8545985c523ff796b17c8ef" Nov 03 13:45:05.579648 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:45:05.579624 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5bb463-29895225-tllnw_odh-ai-gateway-infra(3e40ff1a-b3d1-4aee-a76d-57e1373945ee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895225-tllnw" podUID="3e40ff1a-b3d1-4aee-a76d-57e1373945ee" Nov 03 13:45:05.580902 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:05.580882 5979 scope.go:122] "RemoveContainer" containerID="4345bfeed345c217a8d753e627ac682c54255d4677df4049347fc42b531f2c1e" Nov 03 13:45:05.581114 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:45:05.581092 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-4b9658-29895225-mjbc4_odh-ai-gateway-infra(9c8f3d30-d7ac-4c71-a738-003ebc263255)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4b9658-29895225-mjbc4" podUID="9c8f3d30-d7ac-4c71-a738-003ebc263255" Nov 03 13:45:05.582314 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:05.582297 5979 scope.go:122] "RemoveContainer" containerID="a2ff2dbd6d0c37128590015da3d34a6243300e95dac998bc7f8c41398f27d39b" Nov 03 13:45:05.582509 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:45:05.582490 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-84e22c-29895225-mvwgc_odh-ai-gateway-infra(3f54baec-8801-4ebf-86a8-e4dee3b46888)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-84e22c-29895225-mvwgc" podUID="3f54baec-8801-4ebf-86a8-e4dee3b46888" Nov 03 13:45:07.235359 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:45:07.235316 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:45:14.569818 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:14.569782 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-977f32-7bcd65dd7f-cjkdb" Nov 03 13:45:14.570231 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:14.569969 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-977f32-576d49df6c-8khqq" Nov 03 13:45:15.108146 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:15.108114 5979 scope.go:122] "RemoveContainer" containerID="4345bfeed345c217a8d753e627ac682c54255d4677df4049347fc42b531f2c1e" Nov 03 13:45:15.108386 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:15.108354 5979 scope.go:122] "RemoveContainer" containerID="d0c56ffec33ec34541a4744f6da630b8105fa0e354e0a980a98edf7d07bfe3e4" Nov 03 13:45:15.108586 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:15.108408 5979 scope.go:122] "RemoveContainer" containerID="a2ff2dbd6d0c37128590015da3d34a6243300e95dac998bc7f8c41398f27d39b" Nov 03 13:45:15.108586 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:15.108557 5979 scope.go:122] "RemoveContainer" containerID="47b07e34b19cdd9cea7ce6941de58ae4866becd27f39a3042d00f9c70a47ed5a" Nov 03 13:45:15.655275 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:15.655240 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4b9658-29895225-mjbc4" event={"ID":"9c8f3d30-d7ac-4c71-a738-003ebc263255","Type":"ContainerStarted","Data":"134bf8ecd75fbe25a7efbd7f850bf16e1db6c6fbc36fdf0fbc031dc27df8b432"} Nov 03 13:45:15.657265 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:15.657237 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-84e22c-29895225-mvwgc" event={"ID":"3f54baec-8801-4ebf-86a8-e4dee3b46888","Type":"ContainerStarted","Data":"8723b070a72b9be08c1fd76fd3d4e35652ce8fd3f19fd61d276cc8904dad2f30"} Nov 03 13:45:15.659058 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:15.659035 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895225-568vs" event={"ID":"e0df41fd-c6ea-44af-a973-05243f6ddaae","Type":"ContainerStarted","Data":"8812abd5019b01e426822bebaf5430a9cd179da55c8063ee3ba5aa231ffeb301"} Nov 03 13:45:15.660876 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:15.660855 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895225-ctv9x" event={"ID":"bc2b1123-5771-4e33-bc6f-1654fec22a26","Type":"ContainerStarted","Data":"044c481d00a8d1a5e0a229c391ab8c945004210bf2a68f8628037fde0652f5fe"} Nov 03 13:45:16.108115 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:16.108037 5979 scope.go:122] "RemoveContainer" containerID="0ccd2b5852656c06e4fee212e71f0b4ca17e9141b8545985c523ff796b17c8ef" Nov 03 13:45:16.677091 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:16.676991 5979 generic.go:356] "Generic (PLEG): container finished" podID="3f54baec-8801-4ebf-86a8-e4dee3b46888" containerID="8723b070a72b9be08c1fd76fd3d4e35652ce8fd3f19fd61d276cc8904dad2f30" exitCode=60 Nov 03 13:45:16.677570 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:16.677192 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-84e22c-29895225-mvwgc" event={"ID":"3f54baec-8801-4ebf-86a8-e4dee3b46888","Type":"ContainerDied","Data":"8723b070a72b9be08c1fd76fd3d4e35652ce8fd3f19fd61d276cc8904dad2f30"} Nov 03 13:45:16.677570 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:16.677499 5979 scope.go:122] "RemoveContainer" containerID="a2ff2dbd6d0c37128590015da3d34a6243300e95dac998bc7f8c41398f27d39b" Nov 03 13:45:16.677878 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:16.677856 5979 scope.go:122] "RemoveContainer" containerID="8723b070a72b9be08c1fd76fd3d4e35652ce8fd3f19fd61d276cc8904dad2f30" Nov 03 13:45:16.678211 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:45:16.678185 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-84e22c-29895225-mvwgc_odh-ai-gateway-infra(3f54baec-8801-4ebf-86a8-e4dee3b46888)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-84e22c-29895225-mvwgc" podUID="3f54baec-8801-4ebf-86a8-e4dee3b46888" Nov 03 13:45:16.679548 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:16.679527 5979 generic.go:356] "Generic (PLEG): container finished" podID="e0df41fd-c6ea-44af-a973-05243f6ddaae" containerID="8812abd5019b01e426822bebaf5430a9cd179da55c8063ee3ba5aa231ffeb301" exitCode=60 Nov 03 13:45:16.679669 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:16.679603 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895225-568vs" event={"ID":"e0df41fd-c6ea-44af-a973-05243f6ddaae","Type":"ContainerDied","Data":"8812abd5019b01e426822bebaf5430a9cd179da55c8063ee3ba5aa231ffeb301"} Nov 03 13:45:16.679957 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:16.679941 5979 scope.go:122] "RemoveContainer" containerID="8812abd5019b01e426822bebaf5430a9cd179da55c8063ee3ba5aa231ffeb301" Nov 03 13:45:16.680207 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:45:16.680185 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-34b13c-29895225-568vs_odh-ai-gateway-infra(e0df41fd-c6ea-44af-a973-05243f6ddaae)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895225-568vs" podUID="e0df41fd-c6ea-44af-a973-05243f6ddaae" Nov 03 13:45:16.681556 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:16.681535 5979 generic.go:356] "Generic (PLEG): container finished" podID="bc2b1123-5771-4e33-bc6f-1654fec22a26" containerID="044c481d00a8d1a5e0a229c391ab8c945004210bf2a68f8628037fde0652f5fe" exitCode=60 Nov 03 13:45:16.681661 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:16.681606 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895225-ctv9x" event={"ID":"bc2b1123-5771-4e33-bc6f-1654fec22a26","Type":"ContainerDied","Data":"044c481d00a8d1a5e0a229c391ab8c945004210bf2a68f8628037fde0652f5fe"} Nov 03 13:45:16.681946 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:16.681931 5979 scope.go:122] "RemoveContainer" containerID="044c481d00a8d1a5e0a229c391ab8c945004210bf2a68f8628037fde0652f5fe" Nov 03 13:45:16.682186 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:45:16.682167 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-42c707-29895225-ctv9x_odh-ai-gateway-infra(bc2b1123-5771-4e33-bc6f-1654fec22a26)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895225-ctv9x" podUID="bc2b1123-5771-4e33-bc6f-1654fec22a26" Nov 03 13:45:16.683692 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:16.683666 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895225-tllnw" event={"ID":"3e40ff1a-b3d1-4aee-a76d-57e1373945ee","Type":"ContainerStarted","Data":"5d7828ef0cc301245e791fe96da455c5d68e6c0e46cfbefc958f331f68fe665e"} Nov 03 13:45:16.685547 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:16.685524 5979 generic.go:356] "Generic (PLEG): container finished" podID="9c8f3d30-d7ac-4c71-a738-003ebc263255" containerID="134bf8ecd75fbe25a7efbd7f850bf16e1db6c6fbc36fdf0fbc031dc27df8b432" exitCode=60 Nov 03 13:45:16.685651 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:16.685559 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4b9658-29895225-mjbc4" event={"ID":"9c8f3d30-d7ac-4c71-a738-003ebc263255","Type":"ContainerDied","Data":"134bf8ecd75fbe25a7efbd7f850bf16e1db6c6fbc36fdf0fbc031dc27df8b432"} Nov 03 13:45:16.685867 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:16.685852 5979 scope.go:122] "RemoveContainer" containerID="134bf8ecd75fbe25a7efbd7f850bf16e1db6c6fbc36fdf0fbc031dc27df8b432" Nov 03 13:45:16.686090 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:45:16.686073 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-4b9658-29895225-mjbc4_odh-ai-gateway-infra(9c8f3d30-d7ac-4c71-a738-003ebc263255)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4b9658-29895225-mjbc4" podUID="9c8f3d30-d7ac-4c71-a738-003ebc263255" Nov 03 13:45:16.691778 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:16.691755 5979 scope.go:122] "RemoveContainer" containerID="d0c56ffec33ec34541a4744f6da630b8105fa0e354e0a980a98edf7d07bfe3e4" Nov 03 13:45:16.704925 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:16.704907 5979 scope.go:122] "RemoveContainer" containerID="47b07e34b19cdd9cea7ce6941de58ae4866becd27f39a3042d00f9c70a47ed5a" Nov 03 13:45:16.716881 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:16.716860 5979 scope.go:122] "RemoveContainer" containerID="4345bfeed345c217a8d753e627ac682c54255d4677df4049347fc42b531f2c1e" Nov 03 13:45:16.780409 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:16.780376 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4b9658-29895225-mjbc4"] Nov 03 13:45:16.895484 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:16.895443 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895225-ctv9x"] Nov 03 13:45:17.016867 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:17.016824 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895225-568vs"] Nov 03 13:45:17.108671 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:17.108630 5979 scope.go:122] "RemoveContainer" containerID="3e00ca2be1baf03b518466df14f21ef486d83cd25d1917f2d109bcb10a1866a9" Nov 03 13:45:17.124981 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:17.124953 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-84e22c-29895225-mvwgc"] Nov 03 13:45:17.698095 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:17.698063 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a770b9-29895225-gvxrk" event={"ID":"bcd1cac1-4f49-4c3a-a46d-726a8a83798c","Type":"ContainerStarted","Data":"4cd1977d3d05fbaaf8ed90de91a8eec8ab7bd8cae59a0c03d564883a025b0066"} Nov 03 13:45:17.701908 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:17.701882 5979 generic.go:356] "Generic (PLEG): container finished" podID="3e40ff1a-b3d1-4aee-a76d-57e1373945ee" containerID="5d7828ef0cc301245e791fe96da455c5d68e6c0e46cfbefc958f331f68fe665e" exitCode=60 Nov 03 13:45:17.702032 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:17.701950 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895225-tllnw" event={"ID":"3e40ff1a-b3d1-4aee-a76d-57e1373945ee","Type":"ContainerDied","Data":"5d7828ef0cc301245e791fe96da455c5d68e6c0e46cfbefc958f331f68fe665e"} Nov 03 13:45:17.702032 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:17.702021 5979 scope.go:122] "RemoveContainer" containerID="0ccd2b5852656c06e4fee212e71f0b4ca17e9141b8545985c523ff796b17c8ef" Nov 03 13:45:17.702441 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:17.702422 5979 scope.go:122] "RemoveContainer" containerID="5d7828ef0cc301245e791fe96da455c5d68e6c0e46cfbefc958f331f68fe665e" Nov 03 13:45:17.702913 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:45:17.702862 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5bb463-29895225-tllnw_odh-ai-gateway-infra(3e40ff1a-b3d1-4aee-a76d-57e1373945ee)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895225-tllnw" podUID="3e40ff1a-b3d1-4aee-a76d-57e1373945ee" Nov 03 13:45:17.893062 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:17.893029 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895225-568vs" Nov 03 13:45:17.934848 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:17.933609 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895225-tllnw"] Nov 03 13:45:17.966622 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:17.966534 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4b9658-29895225-mjbc4" Nov 03 13:45:17.992290 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:17.983950 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0df41fd-c6ea-44af-a973-05243f6ddaae-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0df41fd-c6ea-44af-a973-05243f6ddaae-maas-api-service-ca\") pod \"e0df41fd-c6ea-44af-a973-05243f6ddaae\" (UID: \"e0df41fd-c6ea-44af-a973-05243f6ddaae\") " Nov 03 13:45:17.992290 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:17.984061 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0df41fd-c6ea-44af-a973-05243f6ddaae-kube-api-access-n5nqr\" (UniqueName: \"kubernetes.io/projected/e0df41fd-c6ea-44af-a973-05243f6ddaae-kube-api-access-n5nqr\") pod \"e0df41fd-c6ea-44af-a973-05243f6ddaae\" (UID: \"e0df41fd-c6ea-44af-a973-05243f6ddaae\") " Nov 03 13:45:17.993681 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:17.993654 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0df41fd-c6ea-44af-a973-05243f6ddaae-maas-api-service-ca" pod "e0df41fd-c6ea-44af-a973-05243f6ddaae" (UID: "e0df41fd-c6ea-44af-a973-05243f6ddaae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:45:18.044409 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:18.043926 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0df41fd-c6ea-44af-a973-05243f6ddaae-kube-api-access-n5nqr" pod "e0df41fd-c6ea-44af-a973-05243f6ddaae" (UID: "e0df41fd-c6ea-44af-a973-05243f6ddaae"). InnerVolumeSpecName "kube-api-access-n5nqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:45:18.085442 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:18.085404 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c8f3d30-d7ac-4c71-a738-003ebc263255-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c8f3d30-d7ac-4c71-a738-003ebc263255-maas-api-service-ca\") pod \"9c8f3d30-d7ac-4c71-a738-003ebc263255\" (UID: \"9c8f3d30-d7ac-4c71-a738-003ebc263255\") " Nov 03 13:45:18.085652 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:18.085636 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c8f3d30-d7ac-4c71-a738-003ebc263255-kube-api-access-7lrdw\" (UniqueName: \"kubernetes.io/projected/9c8f3d30-d7ac-4c71-a738-003ebc263255-kube-api-access-7lrdw\") pod \"9c8f3d30-d7ac-4c71-a738-003ebc263255\" (UID: \"9c8f3d30-d7ac-4c71-a738-003ebc263255\") " Nov 03 13:45:18.085885 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:18.085859 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c8f3d30-d7ac-4c71-a738-003ebc263255-maas-api-service-ca" pod "9c8f3d30-d7ac-4c71-a738-003ebc263255" (UID: "9c8f3d30-d7ac-4c71-a738-003ebc263255"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:45:18.086154 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:18.086129 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5nqr\" (UniqueName: \"kubernetes.io/projected/e0df41fd-c6ea-44af-a973-05243f6ddaae-kube-api-access-n5nqr\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:18.086154 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:18.086147 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c8f3d30-d7ac-4c71-a738-003ebc263255-maas-api-service-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:18.086154 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:18.086157 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0df41fd-c6ea-44af-a973-05243f6ddaae-maas-api-service-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:18.087762 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:18.087740 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c8f3d30-d7ac-4c71-a738-003ebc263255-kube-api-access-7lrdw" pod "9c8f3d30-d7ac-4c71-a738-003ebc263255" (UID: "9c8f3d30-d7ac-4c71-a738-003ebc263255"). InnerVolumeSpecName "kube-api-access-7lrdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:45:18.109466 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:18.109440 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895225-ctv9x" Nov 03 13:45:18.114395 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:18.114369 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-84e22c-29895225-mvwgc" Nov 03 13:45:18.186542 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:18.186510 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc2b1123-5771-4e33-bc6f-1654fec22a26-kube-api-access-867kr\" (UniqueName: \"kubernetes.io/projected/bc2b1123-5771-4e33-bc6f-1654fec22a26-kube-api-access-867kr\") pod \"bc2b1123-5771-4e33-bc6f-1654fec22a26\" (UID: \"bc2b1123-5771-4e33-bc6f-1654fec22a26\") " Nov 03 13:45:18.186723 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:18.186549 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f54baec-8801-4ebf-86a8-e4dee3b46888-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f54baec-8801-4ebf-86a8-e4dee3b46888-maas-api-service-ca\") pod \"3f54baec-8801-4ebf-86a8-e4dee3b46888\" (UID: \"3f54baec-8801-4ebf-86a8-e4dee3b46888\") " Nov 03 13:45:18.186723 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:18.186657 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f54baec-8801-4ebf-86a8-e4dee3b46888-kube-api-access-pg5v7\" (UniqueName: \"kubernetes.io/projected/3f54baec-8801-4ebf-86a8-e4dee3b46888-kube-api-access-pg5v7\") pod \"3f54baec-8801-4ebf-86a8-e4dee3b46888\" (UID: \"3f54baec-8801-4ebf-86a8-e4dee3b46888\") " Nov 03 13:45:18.186832 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:18.186814 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc2b1123-5771-4e33-bc6f-1654fec22a26-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc2b1123-5771-4e33-bc6f-1654fec22a26-maas-api-service-ca\") pod \"bc2b1123-5771-4e33-bc6f-1654fec22a26\" (UID: \"bc2b1123-5771-4e33-bc6f-1654fec22a26\") " Nov 03 13:45:18.187094 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:18.187066 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f54baec-8801-4ebf-86a8-e4dee3b46888-maas-api-service-ca" pod "3f54baec-8801-4ebf-86a8-e4dee3b46888" (UID: "3f54baec-8801-4ebf-86a8-e4dee3b46888"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:45:18.187223 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:18.187205 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc2b1123-5771-4e33-bc6f-1654fec22a26-maas-api-service-ca" pod "bc2b1123-5771-4e33-bc6f-1654fec22a26" (UID: "bc2b1123-5771-4e33-bc6f-1654fec22a26"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:45:18.187787 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:18.187755 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f54baec-8801-4ebf-86a8-e4dee3b46888-maas-api-service-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:18.188263 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:18.188240 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lrdw\" (UniqueName: \"kubernetes.io/projected/9c8f3d30-d7ac-4c71-a738-003ebc263255-kube-api-access-7lrdw\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:18.188362 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:18.188264 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc2b1123-5771-4e33-bc6f-1654fec22a26-maas-api-service-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:18.189059 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:18.189037 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc2b1123-5771-4e33-bc6f-1654fec22a26-kube-api-access-867kr" pod "bc2b1123-5771-4e33-bc6f-1654fec22a26" (UID: "bc2b1123-5771-4e33-bc6f-1654fec22a26"). InnerVolumeSpecName "kube-api-access-867kr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:45:18.189146 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:18.189084 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f54baec-8801-4ebf-86a8-e4dee3b46888-kube-api-access-pg5v7" pod "3f54baec-8801-4ebf-86a8-e4dee3b46888" (UID: "3f54baec-8801-4ebf-86a8-e4dee3b46888"). InnerVolumeSpecName "kube-api-access-pg5v7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:45:18.289935 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:18.289857 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-867kr\" (UniqueName: \"kubernetes.io/projected/bc2b1123-5771-4e33-bc6f-1654fec22a26-kube-api-access-867kr\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:18.289935 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:18.289886 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg5v7\" (UniqueName: \"kubernetes.io/projected/3f54baec-8801-4ebf-86a8-e4dee3b46888-kube-api-access-pg5v7\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:18.715977 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:18.715939 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4b9658-29895225-mjbc4" Nov 03 13:45:18.715977 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:18.715947 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4b9658-29895225-mjbc4" event={"ID":"9c8f3d30-d7ac-4c71-a738-003ebc263255","Type":"ContainerDied","Data":"6068f0e09705eebccbeb38815f984fd2a135dc10496543066e3074dde11c63d5"} Nov 03 13:45:18.716416 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:18.716045 5979 scope.go:122] "RemoveContainer" containerID="134bf8ecd75fbe25a7efbd7f850bf16e1db6c6fbc36fdf0fbc031dc27df8b432" Nov 03 13:45:18.717362 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:18.717340 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-84e22c-29895225-mvwgc" Nov 03 13:45:18.717535 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:18.717377 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-84e22c-29895225-mvwgc" event={"ID":"3f54baec-8801-4ebf-86a8-e4dee3b46888","Type":"ContainerDied","Data":"38333336f7158971f3db8fd298e3d45774d5ff17476edd02365754ce97cff1e5"} Nov 03 13:45:18.719172 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:18.719143 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895225-568vs" event={"ID":"e0df41fd-c6ea-44af-a973-05243f6ddaae","Type":"ContainerDied","Data":"596e7ce59572d38b6601f7d75b6b8374fb151fce8ba9632e199a56fcc8342c00"} Nov 03 13:45:18.719265 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:18.719174 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895225-568vs" Nov 03 13:45:18.726640 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:18.726618 5979 generic.go:356] "Generic (PLEG): container finished" podID="bcd1cac1-4f49-4c3a-a46d-726a8a83798c" containerID="4cd1977d3d05fbaaf8ed90de91a8eec8ab7bd8cae59a0c03d564883a025b0066" exitCode=60 Nov 03 13:45:18.726752 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:18.726699 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a770b9-29895225-gvxrk" event={"ID":"bcd1cac1-4f49-4c3a-a46d-726a8a83798c","Type":"ContainerDied","Data":"4cd1977d3d05fbaaf8ed90de91a8eec8ab7bd8cae59a0c03d564883a025b0066"} Nov 03 13:45:18.727110 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:18.727091 5979 scope.go:122] "RemoveContainer" containerID="4cd1977d3d05fbaaf8ed90de91a8eec8ab7bd8cae59a0c03d564883a025b0066" Nov 03 13:45:18.727387 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:45:18.727364 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-a770b9-29895225-gvxrk_odh-ai-gateway-infra(bcd1cac1-4f49-4c3a-a46d-726a8a83798c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a770b9-29895225-gvxrk" podUID="bcd1cac1-4f49-4c3a-a46d-726a8a83798c" Nov 03 13:45:18.728917 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:18.728894 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895225-ctv9x" event={"ID":"bc2b1123-5771-4e33-bc6f-1654fec22a26","Type":"ContainerDied","Data":"d5daeca6fc030d0b9e9cfcc911900617a8a7392bfa11adb730c513a8061346a6"} Nov 03 13:45:18.728917 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:18.728909 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895225-ctv9x" Nov 03 13:45:18.729710 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:18.729691 5979 scope.go:122] "RemoveContainer" containerID="8723b070a72b9be08c1fd76fd3d4e35652ce8fd3f19fd61d276cc8904dad2f30" Nov 03 13:45:18.741690 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:18.739996 5979 scope.go:122] "RemoveContainer" containerID="8812abd5019b01e426822bebaf5430a9cd179da55c8063ee3ba5aa231ffeb301" Nov 03 13:45:18.751327 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:18.751302 5979 scope.go:122] "RemoveContainer" containerID="3e00ca2be1baf03b518466df14f21ef486d83cd25d1917f2d109bcb10a1866a9" Nov 03 13:45:18.787369 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:18.784626 5979 scope.go:122] "RemoveContainer" containerID="044c481d00a8d1a5e0a229c391ab8c945004210bf2a68f8628037fde0652f5fe" Nov 03 13:45:18.947209 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:18.947184 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895225-tllnw" Nov 03 13:45:18.987233 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:18.987154 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895225-ctv9x"] Nov 03 13:45:18.993090 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:18.993063 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895225-ctv9x"] Nov 03 13:45:19.068199 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:19.068167 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a770b9-29895225-gvxrk"] Nov 03 13:45:19.109405 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:19.109275 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e40ff1a-b3d1-4aee-a76d-57e1373945ee-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e40ff1a-b3d1-4aee-a76d-57e1373945ee-maas-api-service-ca\") pod \"3e40ff1a-b3d1-4aee-a76d-57e1373945ee\" (UID: \"3e40ff1a-b3d1-4aee-a76d-57e1373945ee\") " Nov 03 13:45:19.109405 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:19.109348 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e40ff1a-b3d1-4aee-a76d-57e1373945ee-kube-api-access-tk9hd\" (UniqueName: \"kubernetes.io/projected/3e40ff1a-b3d1-4aee-a76d-57e1373945ee-kube-api-access-tk9hd\") pod \"3e40ff1a-b3d1-4aee-a76d-57e1373945ee\" (UID: \"3e40ff1a-b3d1-4aee-a76d-57e1373945ee\") " Nov 03 13:45:19.109758 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:19.109735 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e40ff1a-b3d1-4aee-a76d-57e1373945ee-maas-api-service-ca" pod "3e40ff1a-b3d1-4aee-a76d-57e1373945ee" (UID: "3e40ff1a-b3d1-4aee-a76d-57e1373945ee"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:45:19.110061 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:19.110038 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e40ff1a-b3d1-4aee-a76d-57e1373945ee-maas-api-service-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:19.111876 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:19.111850 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e40ff1a-b3d1-4aee-a76d-57e1373945ee-kube-api-access-tk9hd" pod "3e40ff1a-b3d1-4aee-a76d-57e1373945ee" (UID: "3e40ff1a-b3d1-4aee-a76d-57e1373945ee"). InnerVolumeSpecName "kube-api-access-tk9hd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:45:19.112391 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:19.112368 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895225-568vs"] Nov 03 13:45:19.117354 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:19.117332 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895225-568vs"] Nov 03 13:45:19.211193 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:19.211159 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tk9hd\" (UniqueName: \"kubernetes.io/projected/3e40ff1a-b3d1-4aee-a76d-57e1373945ee-kube-api-access-tk9hd\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:19.329254 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:19.329170 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-84e22c-29895225-mvwgc"] Nov 03 13:45:19.333301 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:19.333274 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-84e22c-29895225-mvwgc"] Nov 03 13:45:19.551137 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:19.551104 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4b9658-29895225-mjbc4"] Nov 03 13:45:19.554314 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:19.554289 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-4b9658-29895225-mjbc4"] Nov 03 13:45:19.742217 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:19.742185 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895225-tllnw" Nov 03 13:45:19.742694 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:19.742186 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895225-tllnw" event={"ID":"3e40ff1a-b3d1-4aee-a76d-57e1373945ee","Type":"ContainerDied","Data":"411c8be54c46d49864eccacad1cf382ccd8ae772dae3b471499e427101777fc3"} Nov 03 13:45:19.742694 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:19.742327 5979 scope.go:122] "RemoveContainer" containerID="5d7828ef0cc301245e791fe96da455c5d68e6c0e46cfbefc958f331f68fe665e" Nov 03 13:45:19.938919 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:19.938895 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a770b9-29895225-gvxrk" Nov 03 13:45:19.968459 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:19.968429 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895225-tllnw"] Nov 03 13:45:19.973449 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:19.973422 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5bb463-29895225-tllnw"] Nov 03 13:45:20.022328 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:20.022252 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcd1cac1-4f49-4c3a-a46d-726a8a83798c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcd1cac1-4f49-4c3a-a46d-726a8a83798c-maas-api-service-ca\") pod \"bcd1cac1-4f49-4c3a-a46d-726a8a83798c\" (UID: \"bcd1cac1-4f49-4c3a-a46d-726a8a83798c\") " Nov 03 13:45:20.022499 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:20.022348 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcd1cac1-4f49-4c3a-a46d-726a8a83798c-kube-api-access-sdwgw\" (UniqueName: \"kubernetes.io/projected/bcd1cac1-4f49-4c3a-a46d-726a8a83798c-kube-api-access-sdwgw\") pod \"bcd1cac1-4f49-4c3a-a46d-726a8a83798c\" (UID: \"bcd1cac1-4f49-4c3a-a46d-726a8a83798c\") " Nov 03 13:45:20.022635 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:20.022614 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcd1cac1-4f49-4c3a-a46d-726a8a83798c-maas-api-service-ca" pod "bcd1cac1-4f49-4c3a-a46d-726a8a83798c" (UID: "bcd1cac1-4f49-4c3a-a46d-726a8a83798c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:45:20.022754 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:20.022731 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcd1cac1-4f49-4c3a-a46d-726a8a83798c-maas-api-service-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:20.024458 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:20.024437 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcd1cac1-4f49-4c3a-a46d-726a8a83798c-kube-api-access-sdwgw" pod "bcd1cac1-4f49-4c3a-a46d-726a8a83798c" (UID: "bcd1cac1-4f49-4c3a-a46d-726a8a83798c"). InnerVolumeSpecName "kube-api-access-sdwgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:45:20.115549 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:20.115512 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e40ff1a-b3d1-4aee-a76d-57e1373945ee" path="/var/lib/kubelet/pods/3e40ff1a-b3d1-4aee-a76d-57e1373945ee/volumes" Nov 03 13:45:20.116005 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:20.115990 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f54baec-8801-4ebf-86a8-e4dee3b46888" path="/var/lib/kubelet/pods/3f54baec-8801-4ebf-86a8-e4dee3b46888/volumes" Nov 03 13:45:20.116433 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:20.116419 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c8f3d30-d7ac-4c71-a738-003ebc263255" path="/var/lib/kubelet/pods/9c8f3d30-d7ac-4c71-a738-003ebc263255/volumes" Nov 03 13:45:20.117171 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:20.117154 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc2b1123-5771-4e33-bc6f-1654fec22a26" path="/var/lib/kubelet/pods/bc2b1123-5771-4e33-bc6f-1654fec22a26/volumes" Nov 03 13:45:20.117588 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:20.117572 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0df41fd-c6ea-44af-a973-05243f6ddaae" path="/var/lib/kubelet/pods/e0df41fd-c6ea-44af-a973-05243f6ddaae/volumes" Nov 03 13:45:20.123868 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:20.123840 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdwgw\" (UniqueName: \"kubernetes.io/projected/bcd1cac1-4f49-4c3a-a46d-726a8a83798c-kube-api-access-sdwgw\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:20.750169 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:20.750139 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a770b9-29895225-gvxrk" Nov 03 13:45:20.750169 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:20.750149 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a770b9-29895225-gvxrk" event={"ID":"bcd1cac1-4f49-4c3a-a46d-726a8a83798c","Type":"ContainerDied","Data":"2eeb203e9398b1da3044446374f043116f8ebe02b1e3f74bc9392d21d9617ab0"} Nov 03 13:45:20.750785 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:20.750210 5979 scope.go:122] "RemoveContainer" containerID="4cd1977d3d05fbaaf8ed90de91a8eec8ab7bd8cae59a0c03d564883a025b0066" Nov 03 13:45:20.910105 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:20.910058 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a770b9-29895225-gvxrk"] Nov 03 13:45:20.952902 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:20.952865 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a770b9-29895225-gvxrk"] Nov 03 13:45:21.340292 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.340255 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2862039b-jpttz"] Nov 03 13:45:21.341043 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.341025 5979 cpu_manager.go:414] "containerMap: removing container" podUID="3f54baec-8801-4ebf-86a8-e4dee3b46888" containerName="cleanup" Nov 03 13:45:21.341103 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.341045 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f54baec-8801-4ebf-86a8-e4dee3b46888" containerName="cleanup" Nov 03 13:45:21.341103 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.341056 5979 cpu_manager.go:414] "containerMap: removing container" podUID="bc2b1123-5771-4e33-bc6f-1654fec22a26" containerName="cleanup" Nov 03 13:45:21.341103 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.341062 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc2b1123-5771-4e33-bc6f-1654fec22a26" containerName="cleanup" Nov 03 13:45:21.341103 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.341081 5979 cpu_manager.go:414] "containerMap: removing container" podUID="bc2b1123-5771-4e33-bc6f-1654fec22a26" containerName="cleanup" Nov 03 13:45:21.341103 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.341087 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc2b1123-5771-4e33-bc6f-1654fec22a26" containerName="cleanup" Nov 03 13:45:21.341103 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.341093 5979 cpu_manager.go:414] "containerMap: removing container" podUID="3e40ff1a-b3d1-4aee-a76d-57e1373945ee" containerName="cleanup" Nov 03 13:45:21.341103 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.341097 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e40ff1a-b3d1-4aee-a76d-57e1373945ee" containerName="cleanup" Nov 03 13:45:21.341103 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.341104 5979 cpu_manager.go:414] "containerMap: removing container" podUID="bc2b1123-5771-4e33-bc6f-1654fec22a26" containerName="cleanup" Nov 03 13:45:21.341378 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.341109 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc2b1123-5771-4e33-bc6f-1654fec22a26" containerName="cleanup" Nov 03 13:45:21.341378 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.341121 5979 cpu_manager.go:414] "containerMap: removing container" podUID="e0df41fd-c6ea-44af-a973-05243f6ddaae" containerName="cleanup" Nov 03 13:45:21.341378 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.341126 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0df41fd-c6ea-44af-a973-05243f6ddaae" containerName="cleanup" Nov 03 13:45:21.341378 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.341143 5979 cpu_manager.go:414] "containerMap: removing container" podUID="3f54baec-8801-4ebf-86a8-e4dee3b46888" containerName="cleanup" Nov 03 13:45:21.341378 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.341148 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f54baec-8801-4ebf-86a8-e4dee3b46888" containerName="cleanup" Nov 03 13:45:21.341378 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.341157 5979 cpu_manager.go:414] "containerMap: removing container" podUID="e0df41fd-c6ea-44af-a973-05243f6ddaae" containerName="cleanup" Nov 03 13:45:21.341378 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.341161 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0df41fd-c6ea-44af-a973-05243f6ddaae" containerName="cleanup" Nov 03 13:45:21.341378 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.341167 5979 cpu_manager.go:414] "containerMap: removing container" podUID="e0df41fd-c6ea-44af-a973-05243f6ddaae" containerName="cleanup" Nov 03 13:45:21.341378 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.341171 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0df41fd-c6ea-44af-a973-05243f6ddaae" containerName="cleanup" Nov 03 13:45:21.341378 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.341178 5979 cpu_manager.go:414] "containerMap: removing container" podUID="9c8f3d30-d7ac-4c71-a738-003ebc263255" containerName="cleanup" Nov 03 13:45:21.341378 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.341183 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c8f3d30-d7ac-4c71-a738-003ebc263255" containerName="cleanup" Nov 03 13:45:21.341378 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.341189 5979 cpu_manager.go:414] "containerMap: removing container" podUID="3e40ff1a-b3d1-4aee-a76d-57e1373945ee" containerName="cleanup" Nov 03 13:45:21.341378 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.341228 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e40ff1a-b3d1-4aee-a76d-57e1373945ee" containerName="cleanup" Nov 03 13:45:21.341378 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.341236 5979 cpu_manager.go:414] "containerMap: removing container" podUID="bcd1cac1-4f49-4c3a-a46d-726a8a83798c" containerName="cleanup" Nov 03 13:45:21.341378 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.341242 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcd1cac1-4f49-4c3a-a46d-726a8a83798c" containerName="cleanup" Nov 03 13:45:21.341378 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.341256 5979 cpu_manager.go:414] "containerMap: removing container" podUID="9c8f3d30-d7ac-4c71-a738-003ebc263255" containerName="cleanup" Nov 03 13:45:21.341378 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.341260 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c8f3d30-d7ac-4c71-a738-003ebc263255" containerName="cleanup" Nov 03 13:45:21.341378 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.341267 5979 cpu_manager.go:414] "containerMap: removing container" podUID="bcd1cac1-4f49-4c3a-a46d-726a8a83798c" containerName="cleanup" Nov 03 13:45:21.341378 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.341271 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcd1cac1-4f49-4c3a-a46d-726a8a83798c" containerName="cleanup" Nov 03 13:45:21.341378 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.341279 5979 cpu_manager.go:414] "containerMap: removing container" podUID="bcd1cac1-4f49-4c3a-a46d-726a8a83798c" containerName="cleanup" Nov 03 13:45:21.341378 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.341283 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcd1cac1-4f49-4c3a-a46d-726a8a83798c" containerName="cleanup" Nov 03 13:45:21.341378 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.341290 5979 cpu_manager.go:414] "containerMap: removing container" podUID="9c8f3d30-d7ac-4c71-a738-003ebc263255" containerName="cleanup" Nov 03 13:45:21.341378 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.341294 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c8f3d30-d7ac-4c71-a738-003ebc263255" containerName="cleanup" Nov 03 13:45:21.341378 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.341380 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e40ff1a-b3d1-4aee-a76d-57e1373945ee" containerName="cleanup" Nov 03 13:45:21.341378 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.341389 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc2b1123-5771-4e33-bc6f-1654fec22a26" containerName="cleanup" Nov 03 13:45:21.342275 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.341395 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f54baec-8801-4ebf-86a8-e4dee3b46888" containerName="cleanup" Nov 03 13:45:21.342275 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.341401 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0df41fd-c6ea-44af-a973-05243f6ddaae" containerName="cleanup" Nov 03 13:45:21.342275 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.341408 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e40ff1a-b3d1-4aee-a76d-57e1373945ee" containerName="cleanup" Nov 03 13:45:21.342275 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.341416 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0df41fd-c6ea-44af-a973-05243f6ddaae" containerName="cleanup" Nov 03 13:45:21.342275 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.341422 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0df41fd-c6ea-44af-a973-05243f6ddaae" containerName="cleanup" Nov 03 13:45:21.342275 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.341428 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c8f3d30-d7ac-4c71-a738-003ebc263255" containerName="cleanup" Nov 03 13:45:21.342275 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.341433 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e40ff1a-b3d1-4aee-a76d-57e1373945ee" containerName="cleanup" Nov 03 13:45:21.342275 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.341439 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcd1cac1-4f49-4c3a-a46d-726a8a83798c" containerName="cleanup" Nov 03 13:45:21.342275 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.341445 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c8f3d30-d7ac-4c71-a738-003ebc263255" containerName="cleanup" Nov 03 13:45:21.342275 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.341451 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcd1cac1-4f49-4c3a-a46d-726a8a83798c" containerName="cleanup" Nov 03 13:45:21.342275 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.341458 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcd1cac1-4f49-4c3a-a46d-726a8a83798c" containerName="cleanup" Nov 03 13:45:21.342275 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.341463 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c8f3d30-d7ac-4c71-a738-003ebc263255" containerName="cleanup" Nov 03 13:45:21.342275 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.341488 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f54baec-8801-4ebf-86a8-e4dee3b46888" containerName="cleanup" Nov 03 13:45:21.342275 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.341493 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc2b1123-5771-4e33-bc6f-1654fec22a26" containerName="cleanup" Nov 03 13:45:21.346553 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.346534 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2862039b-jpttz" Nov 03 13:45:21.404814 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.404670 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:45:21.404814 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.404756 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:45:21.407333 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.407304 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2862039b-jpttz"] Nov 03 13:45:21.439391 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.439354 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9911db5-2c92-4007-ad7e-68e14df3b6dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2862039b-jpttz\" (UID: \"d9911db5-2c92-4007-ad7e-68e14df3b6dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2862039b-jpttz" Nov 03 13:45:21.439605 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.439519 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9911db5-2c92-4007-ad7e-68e14df3b6dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2862039b-jpttz\" (UID: \"d9911db5-2c92-4007-ad7e-68e14df3b6dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2862039b-jpttz" Nov 03 13:45:21.439695 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.439636 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhd6w\" (UniqueName: \"kubernetes.io/projected/d9911db5-2c92-4007-ad7e-68e14df3b6dc-kube-api-access-dhd6w\") pod \"maas-api-revoke-sub-2862039b-jpttz\" (UID: \"d9911db5-2c92-4007-ad7e-68e14df3b6dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2862039b-jpttz" Nov 03 13:45:21.540872 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.540833 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9911db5-2c92-4007-ad7e-68e14df3b6dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2862039b-jpttz\" (UID: \"d9911db5-2c92-4007-ad7e-68e14df3b6dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2862039b-jpttz" Nov 03 13:45:21.541063 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.540949 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhd6w\" (UniqueName: \"kubernetes.io/projected/d9911db5-2c92-4007-ad7e-68e14df3b6dc-kube-api-access-dhd6w\") pod \"maas-api-revoke-sub-2862039b-jpttz\" (UID: \"d9911db5-2c92-4007-ad7e-68e14df3b6dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2862039b-jpttz" Nov 03 13:45:21.541167 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.541060 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9911db5-2c92-4007-ad7e-68e14df3b6dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2862039b-jpttz\" (UID: \"d9911db5-2c92-4007-ad7e-68e14df3b6dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2862039b-jpttz" Nov 03 13:45:21.541822 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.541798 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9911db5-2c92-4007-ad7e-68e14df3b6dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2862039b-jpttz\" (UID: \"d9911db5-2c92-4007-ad7e-68e14df3b6dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2862039b-jpttz" Nov 03 13:45:21.543318 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.543290 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9911db5-2c92-4007-ad7e-68e14df3b6dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2862039b-jpttz\" (UID: \"d9911db5-2c92-4007-ad7e-68e14df3b6dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2862039b-jpttz" Nov 03 13:45:21.595703 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.595622 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhd6w\" (UniqueName: \"kubernetes.io/projected/d9911db5-2c92-4007-ad7e-68e14df3b6dc-kube-api-access-dhd6w\") pod \"maas-api-revoke-sub-2862039b-jpttz\" (UID: \"d9911db5-2c92-4007-ad7e-68e14df3b6dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2862039b-jpttz" Nov 03 13:45:21.660392 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:21.660361 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2862039b-jpttz" Nov 03 13:45:22.044563 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:22.044531 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2862039b-jpttz"] Nov 03 13:45:22.050788 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:45:22.050760 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9911db5_2c92_4007_ad7e_68e14df3b6dc.slice/crio-3cf5038e64969926fb2f25516eafef1c5494ee6c187ea1b51072384eba51fdbe WatchSource:0}: Error finding container 3cf5038e64969926fb2f25516eafef1c5494ee6c187ea1b51072384eba51fdbe: Status 404 returned error can't find the container with id 3cf5038e64969926fb2f25516eafef1c5494ee6c187ea1b51072384eba51fdbe Nov 03 13:45:22.116393 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:22.116354 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcd1cac1-4f49-4c3a-a46d-726a8a83798c" path="/var/lib/kubelet/pods/bcd1cac1-4f49-4c3a-a46d-726a8a83798c/volumes" Nov 03 13:45:22.139770 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:22.139740 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e083cf9-lrpg7"] Nov 03 13:45:22.140454 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:22.140430 5979 cpu_manager.go:414] "containerMap: removing container" podUID="3f54baec-8801-4ebf-86a8-e4dee3b46888" containerName="cleanup" Nov 03 13:45:22.140565 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:22.140455 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f54baec-8801-4ebf-86a8-e4dee3b46888" containerName="cleanup" Nov 03 13:45:22.140565 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:22.140538 5979 cpu_manager.go:414] "containerMap: removing container" podUID="3e40ff1a-b3d1-4aee-a76d-57e1373945ee" containerName="cleanup" Nov 03 13:45:22.140565 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:22.140548 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e40ff1a-b3d1-4aee-a76d-57e1373945ee" containerName="cleanup" Nov 03 13:45:22.140697 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:22.140685 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc2b1123-5771-4e33-bc6f-1654fec22a26" containerName="cleanup" Nov 03 13:45:22.140738 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:22.140699 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f54baec-8801-4ebf-86a8-e4dee3b46888" containerName="cleanup" Nov 03 13:45:22.145956 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:22.145935 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e083cf9-lrpg7" Nov 03 13:45:22.203972 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:22.203930 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e083cf9-lrpg7"] Nov 03 13:45:22.250003 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:22.249976 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3ff6305-7ae8-4569-acec-f5360482bcd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e083cf9-lrpg7\" (UID: \"e3ff6305-7ae8-4569-acec-f5360482bcd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e083cf9-lrpg7" Nov 03 13:45:22.250181 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:22.250021 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqkvw\" (UniqueName: \"kubernetes.io/projected/e3ff6305-7ae8-4569-acec-f5360482bcd4-kube-api-access-nqkvw\") pod \"maas-api-revoke-sub-2e083cf9-lrpg7\" (UID: \"e3ff6305-7ae8-4569-acec-f5360482bcd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e083cf9-lrpg7" Nov 03 13:45:22.250181 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:22.250132 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3ff6305-7ae8-4569-acec-f5360482bcd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e083cf9-lrpg7\" (UID: \"e3ff6305-7ae8-4569-acec-f5360482bcd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e083cf9-lrpg7" Nov 03 13:45:22.351627 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:22.351598 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3ff6305-7ae8-4569-acec-f5360482bcd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e083cf9-lrpg7\" (UID: \"e3ff6305-7ae8-4569-acec-f5360482bcd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e083cf9-lrpg7" Nov 03 13:45:22.351781 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:22.351648 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqkvw\" (UniqueName: \"kubernetes.io/projected/e3ff6305-7ae8-4569-acec-f5360482bcd4-kube-api-access-nqkvw\") pod \"maas-api-revoke-sub-2e083cf9-lrpg7\" (UID: \"e3ff6305-7ae8-4569-acec-f5360482bcd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e083cf9-lrpg7" Nov 03 13:45:22.351781 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:22.351706 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3ff6305-7ae8-4569-acec-f5360482bcd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e083cf9-lrpg7\" (UID: \"e3ff6305-7ae8-4569-acec-f5360482bcd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e083cf9-lrpg7" Nov 03 13:45:22.352555 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:22.352511 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3ff6305-7ae8-4569-acec-f5360482bcd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e083cf9-lrpg7\" (UID: \"e3ff6305-7ae8-4569-acec-f5360482bcd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e083cf9-lrpg7" Nov 03 13:45:22.353934 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:22.353915 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3ff6305-7ae8-4569-acec-f5360482bcd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e083cf9-lrpg7\" (UID: \"e3ff6305-7ae8-4569-acec-f5360482bcd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e083cf9-lrpg7" Nov 03 13:45:22.406238 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:22.406210 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqkvw\" (UniqueName: \"kubernetes.io/projected/e3ff6305-7ae8-4569-acec-f5360482bcd4-kube-api-access-nqkvw\") pod \"maas-api-revoke-sub-2e083cf9-lrpg7\" (UID: \"e3ff6305-7ae8-4569-acec-f5360482bcd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e083cf9-lrpg7" Nov 03 13:45:22.460016 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:22.459985 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e083cf9-lrpg7" Nov 03 13:45:22.771158 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:22.771122 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2862039b-jpttz" event={"ID":"d9911db5-2c92-4007-ad7e-68e14df3b6dc","Type":"ContainerStarted","Data":"d0b90f0abe0d317f0b2afc11ad78963bdd4e47c1d24d010a4b9cb5fbfe8bd149"} Nov 03 13:45:22.771318 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:22.771172 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2862039b-jpttz" event={"ID":"d9911db5-2c92-4007-ad7e-68e14df3b6dc","Type":"ContainerStarted","Data":"3cf5038e64969926fb2f25516eafef1c5494ee6c187ea1b51072384eba51fdbe"} Nov 03 13:45:22.835190 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:22.835159 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e083cf9-lrpg7"] Nov 03 13:45:22.842692 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:45:22.842667 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3ff6305_7ae8_4569_acec_f5360482bcd4.slice/crio-16e4837d07ce86ed6616992c88cf479807cbbe719270fe7da48f02e18fdb56d6 WatchSource:0}: Error finding container 16e4837d07ce86ed6616992c88cf479807cbbe719270fe7da48f02e18fdb56d6: Status 404 returned error can't find the container with id 16e4837d07ce86ed6616992c88cf479807cbbe719270fe7da48f02e18fdb56d6 Nov 03 13:45:22.884735 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:22.884683 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2862039b-jpttz" podStartSLOduration=1.884668069 podStartE2EDuration="1.884668069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:45:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:45:22.882585436 +0000 UTC m=+2404.893273690" watchObservedRunningTime="2026-11-03 13:45:22.884668069 +0000 UTC m=+2404.895356311" Nov 03 13:45:23.779719 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:23.779689 5979 generic.go:356] "Generic (PLEG): container finished" podID="e3ff6305-7ae8-4569-acec-f5360482bcd4" containerID="ac8d23199a6d4a6e218ace7c9dacea099f609d11bba9f98df1be885c06265b4f" exitCode=0 Nov 03 13:45:23.780146 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:23.779767 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e083cf9-lrpg7" event={"ID":"e3ff6305-7ae8-4569-acec-f5360482bcd4","Type":"ContainerDied","Data":"ac8d23199a6d4a6e218ace7c9dacea099f609d11bba9f98df1be885c06265b4f"} Nov 03 13:45:23.780146 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:23.779822 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e083cf9-lrpg7" event={"ID":"e3ff6305-7ae8-4569-acec-f5360482bcd4","Type":"ContainerStarted","Data":"16e4837d07ce86ed6616992c88cf479807cbbe719270fe7da48f02e18fdb56d6"} Nov 03 13:45:23.781359 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:23.781335 5979 generic.go:356] "Generic (PLEG): container finished" podID="d9911db5-2c92-4007-ad7e-68e14df3b6dc" containerID="d0b90f0abe0d317f0b2afc11ad78963bdd4e47c1d24d010a4b9cb5fbfe8bd149" exitCode=0 Nov 03 13:45:23.781507 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:23.781362 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2862039b-jpttz" event={"ID":"d9911db5-2c92-4007-ad7e-68e14df3b6dc","Type":"ContainerDied","Data":"d0b90f0abe0d317f0b2afc11ad78963bdd4e47c1d24d010a4b9cb5fbfe8bd149"} Nov 03 13:45:24.946553 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:24.946527 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e083cf9-lrpg7" Nov 03 13:45:24.990992 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:24.990964 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2862039b-jpttz" Nov 03 13:45:25.086971 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:25.086892 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9911db5-2c92-4007-ad7e-68e14df3b6dc-kube-api-access-dhd6w\" (UniqueName: \"kubernetes.io/projected/d9911db5-2c92-4007-ad7e-68e14df3b6dc-kube-api-access-dhd6w\") pod \"d9911db5-2c92-4007-ad7e-68e14df3b6dc\" (UID: \"d9911db5-2c92-4007-ad7e-68e14df3b6dc\") " Nov 03 13:45:25.087147 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:25.087002 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9911db5-2c92-4007-ad7e-68e14df3b6dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9911db5-2c92-4007-ad7e-68e14df3b6dc-maas-api-openshift-ca\") pod \"d9911db5-2c92-4007-ad7e-68e14df3b6dc\" (UID: \"d9911db5-2c92-4007-ad7e-68e14df3b6dc\") " Nov 03 13:45:25.087147 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:25.087123 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9911db5-2c92-4007-ad7e-68e14df3b6dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9911db5-2c92-4007-ad7e-68e14df3b6dc-maas-api-xks-ca\") pod \"d9911db5-2c92-4007-ad7e-68e14df3b6dc\" (UID: \"d9911db5-2c92-4007-ad7e-68e14df3b6dc\") " Nov 03 13:45:25.087284 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:25.087209 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3ff6305-7ae8-4569-acec-f5360482bcd4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3ff6305-7ae8-4569-acec-f5360482bcd4-maas-api-openshift-ca\") pod \"e3ff6305-7ae8-4569-acec-f5360482bcd4\" (UID: \"e3ff6305-7ae8-4569-acec-f5360482bcd4\") " Nov 03 13:45:25.087284 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:25.087241 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3ff6305-7ae8-4569-acec-f5360482bcd4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3ff6305-7ae8-4569-acec-f5360482bcd4-maas-api-xks-ca\") pod \"e3ff6305-7ae8-4569-acec-f5360482bcd4\" (UID: \"e3ff6305-7ae8-4569-acec-f5360482bcd4\") " Nov 03 13:45:25.087402 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:25.087305 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3ff6305-7ae8-4569-acec-f5360482bcd4-kube-api-access-nqkvw\" (UniqueName: \"kubernetes.io/projected/e3ff6305-7ae8-4569-acec-f5360482bcd4-kube-api-access-nqkvw\") pod \"e3ff6305-7ae8-4569-acec-f5360482bcd4\" (UID: \"e3ff6305-7ae8-4569-acec-f5360482bcd4\") " Nov 03 13:45:25.087463 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:25.087399 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9911db5-2c92-4007-ad7e-68e14df3b6dc-maas-api-openshift-ca" pod "d9911db5-2c92-4007-ad7e-68e14df3b6dc" (UID: "d9911db5-2c92-4007-ad7e-68e14df3b6dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:45:25.087636 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:25.087608 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3ff6305-7ae8-4569-acec-f5360482bcd4-maas-api-openshift-ca" pod "e3ff6305-7ae8-4569-acec-f5360482bcd4" (UID: "e3ff6305-7ae8-4569-acec-f5360482bcd4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:45:25.087860 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:25.087841 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3ff6305-7ae8-4569-acec-f5360482bcd4-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:25.087930 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:25.087865 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9911db5-2c92-4007-ad7e-68e14df3b6dc-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:25.089971 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:25.089946 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3ff6305-7ae8-4569-acec-f5360482bcd4-maas-api-xks-ca" pod "e3ff6305-7ae8-4569-acec-f5360482bcd4" (UID: "e3ff6305-7ae8-4569-acec-f5360482bcd4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:45:25.090063 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:25.089986 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3ff6305-7ae8-4569-acec-f5360482bcd4-kube-api-access-nqkvw" pod "e3ff6305-7ae8-4569-acec-f5360482bcd4" (UID: "e3ff6305-7ae8-4569-acec-f5360482bcd4"). InnerVolumeSpecName "kube-api-access-nqkvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:45:25.090176 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:25.090160 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9911db5-2c92-4007-ad7e-68e14df3b6dc-kube-api-access-dhd6w" pod "d9911db5-2c92-4007-ad7e-68e14df3b6dc" (UID: "d9911db5-2c92-4007-ad7e-68e14df3b6dc"). InnerVolumeSpecName "kube-api-access-dhd6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:45:25.091623 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:25.091598 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9911db5-2c92-4007-ad7e-68e14df3b6dc-maas-api-xks-ca" pod "d9911db5-2c92-4007-ad7e-68e14df3b6dc" (UID: "d9911db5-2c92-4007-ad7e-68e14df3b6dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:45:25.189080 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:25.189039 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3ff6305-7ae8-4569-acec-f5360482bcd4-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:25.189080 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:25.189071 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqkvw\" (UniqueName: \"kubernetes.io/projected/e3ff6305-7ae8-4569-acec-f5360482bcd4-kube-api-access-nqkvw\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:25.189080 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:25.189082 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhd6w\" (UniqueName: \"kubernetes.io/projected/d9911db5-2c92-4007-ad7e-68e14df3b6dc-kube-api-access-dhd6w\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:25.189409 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:25.189094 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9911db5-2c92-4007-ad7e-68e14df3b6dc-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:25.796234 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:25.796197 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e083cf9-lrpg7" Nov 03 13:45:25.796399 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:25.796234 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e083cf9-lrpg7" event={"ID":"e3ff6305-7ae8-4569-acec-f5360482bcd4","Type":"ContainerDied","Data":"16e4837d07ce86ed6616992c88cf479807cbbe719270fe7da48f02e18fdb56d6"} Nov 03 13:45:25.796399 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:25.796290 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16e4837d07ce86ed6616992c88cf479807cbbe719270fe7da48f02e18fdb56d6" Nov 03 13:45:25.797690 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:25.797668 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2862039b-jpttz" Nov 03 13:45:25.797802 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:25.797705 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2862039b-jpttz" event={"ID":"d9911db5-2c92-4007-ad7e-68e14df3b6dc","Type":"ContainerDied","Data":"3cf5038e64969926fb2f25516eafef1c5494ee6c187ea1b51072384eba51fdbe"} Nov 03 13:45:25.797802 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:25.797742 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3cf5038e64969926fb2f25516eafef1c5494ee6c187ea1b51072384eba51fdbe" Nov 03 13:45:30.161519 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:30.161480 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:45:34.587777 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:34.587747 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686"] Nov 03 13:45:34.588242 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:34.587983 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686" podUID="87552227-fce6-474b-9b1b-fe167d1f6547" containerName="istio-proxy" containerID="cri-o://07dce1585d30ffbfde67bb61883dbef54c2fd155be42615a8a7a79b1ae8cbbcd" gracePeriod=30 Nov 03 13:45:37.235533 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:45:37.235494 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:45:38.274558 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:38.274522 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80c5861a-c9qg5"] Nov 03 13:45:38.275490 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:38.275440 5979 cpu_manager.go:414] "containerMap: removing container" podUID="e3ff6305-7ae8-4569-acec-f5360482bcd4" containerName="revoke-keys" Nov 03 13:45:38.275490 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:38.275481 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3ff6305-7ae8-4569-acec-f5360482bcd4" containerName="revoke-keys" Nov 03 13:45:38.275702 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:38.275541 5979 cpu_manager.go:414] "containerMap: removing container" podUID="d9911db5-2c92-4007-ad7e-68e14df3b6dc" containerName="revoke-keys" Nov 03 13:45:38.275702 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:38.275550 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9911db5-2c92-4007-ad7e-68e14df3b6dc" containerName="revoke-keys" Nov 03 13:45:38.275702 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:38.275681 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3ff6305-7ae8-4569-acec-f5360482bcd4" containerName="revoke-keys" Nov 03 13:45:38.275702 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:38.275701 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9911db5-2c92-4007-ad7e-68e14df3b6dc" containerName="revoke-keys" Nov 03 13:45:38.281218 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:38.281197 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c5861a-c9qg5" Nov 03 13:45:38.334448 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:38.334419 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80c5861a-c9qg5"] Nov 03 13:45:38.377419 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:38.377390 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:45:38.377592 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:38.377495 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:45:38.411546 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:38.411510 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1cfb1a0-cef9-4bf9-8392-e671178ef46e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80c5861a-c9qg5\" (UID: \"c1cfb1a0-cef9-4bf9-8392-e671178ef46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c5861a-c9qg5" Nov 03 13:45:38.411712 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:38.411595 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1cfb1a0-cef9-4bf9-8392-e671178ef46e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80c5861a-c9qg5\" (UID: \"c1cfb1a0-cef9-4bf9-8392-e671178ef46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c5861a-c9qg5" Nov 03 13:45:38.411788 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:38.411762 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gncwv\" (UniqueName: \"kubernetes.io/projected/c1cfb1a0-cef9-4bf9-8392-e671178ef46e-kube-api-access-gncwv\") pod \"maas-api-revoke-sub-80c5861a-c9qg5\" (UID: \"c1cfb1a0-cef9-4bf9-8392-e671178ef46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c5861a-c9qg5" Nov 03 13:45:38.512595 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:38.512557 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1cfb1a0-cef9-4bf9-8392-e671178ef46e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80c5861a-c9qg5\" (UID: \"c1cfb1a0-cef9-4bf9-8392-e671178ef46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c5861a-c9qg5" Nov 03 13:45:38.512764 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:38.512685 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gncwv\" (UniqueName: \"kubernetes.io/projected/c1cfb1a0-cef9-4bf9-8392-e671178ef46e-kube-api-access-gncwv\") pod \"maas-api-revoke-sub-80c5861a-c9qg5\" (UID: \"c1cfb1a0-cef9-4bf9-8392-e671178ef46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c5861a-c9qg5" Nov 03 13:45:38.512823 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:38.512808 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1cfb1a0-cef9-4bf9-8392-e671178ef46e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80c5861a-c9qg5\" (UID: \"c1cfb1a0-cef9-4bf9-8392-e671178ef46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c5861a-c9qg5" Nov 03 13:45:38.513400 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:38.513382 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1cfb1a0-cef9-4bf9-8392-e671178ef46e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80c5861a-c9qg5\" (UID: \"c1cfb1a0-cef9-4bf9-8392-e671178ef46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c5861a-c9qg5" Nov 03 13:45:38.514963 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:38.514943 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1cfb1a0-cef9-4bf9-8392-e671178ef46e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80c5861a-c9qg5\" (UID: \"c1cfb1a0-cef9-4bf9-8392-e671178ef46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c5861a-c9qg5" Nov 03 13:45:38.570967 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:38.570903 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gncwv\" (UniqueName: \"kubernetes.io/projected/c1cfb1a0-cef9-4bf9-8392-e671178ef46e-kube-api-access-gncwv\") pod \"maas-api-revoke-sub-80c5861a-c9qg5\" (UID: \"c1cfb1a0-cef9-4bf9-8392-e671178ef46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c5861a-c9qg5" Nov 03 13:45:38.596016 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:38.595989 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c5861a-c9qg5" Nov 03 13:45:38.982090 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:38.982055 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80c5861a-c9qg5"] Nov 03 13:45:38.988389 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:45:38.988362 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1cfb1a0_cef9_4bf9_8392_e671178ef46e.slice/crio-3142b0d1026c74a97a5a9312ebc86366c1a7a30582fbbf141e2bcdf44c1833a2 WatchSource:0}: Error finding container 3142b0d1026c74a97a5a9312ebc86366c1a7a30582fbbf141e2bcdf44c1833a2: Status 404 returned error can't find the container with id 3142b0d1026c74a97a5a9312ebc86366c1a7a30582fbbf141e2bcdf44c1833a2 Nov 03 13:45:39.875578 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:39.875513 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686" Nov 03 13:45:39.917003 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:39.916978 5979 generic.go:356] "Generic (PLEG): container finished" podID="87552227-fce6-474b-9b1b-fe167d1f6547" containerID="07dce1585d30ffbfde67bb61883dbef54c2fd155be42615a8a7a79b1ae8cbbcd" exitCode=0 Nov 03 13:45:39.917145 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:39.917028 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686" Nov 03 13:45:39.917145 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:39.917060 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686" event={"ID":"87552227-fce6-474b-9b1b-fe167d1f6547","Type":"ContainerDied","Data":"07dce1585d30ffbfde67bb61883dbef54c2fd155be42615a8a7a79b1ae8cbbcd"} Nov 03 13:45:39.917145 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:39.917116 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686" event={"ID":"87552227-fce6-474b-9b1b-fe167d1f6547","Type":"ContainerDied","Data":"717b25c78539d81f95df98d2710b46a0bae20cedb3b26a507d1fd99a9bd22f9e"} Nov 03 13:45:39.917145 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:39.917141 5979 scope.go:122] "RemoveContainer" containerID="07dce1585d30ffbfde67bb61883dbef54c2fd155be42615a8a7a79b1ae8cbbcd" Nov 03 13:45:39.918980 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:39.918955 5979 generic.go:356] "Generic (PLEG): container finished" podID="c1cfb1a0-cef9-4bf9-8392-e671178ef46e" containerID="7beb8632bdc9719249d00780113ce44c483d1eb4cb2720ea1404e37ddea59be9" exitCode=0 Nov 03 13:45:39.919113 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:39.919058 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c5861a-c9qg5" event={"ID":"c1cfb1a0-cef9-4bf9-8392-e671178ef46e","Type":"ContainerDied","Data":"7beb8632bdc9719249d00780113ce44c483d1eb4cb2720ea1404e37ddea59be9"} Nov 03 13:45:39.919113 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:39.919100 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c5861a-c9qg5" event={"ID":"c1cfb1a0-cef9-4bf9-8392-e671178ef46e","Type":"ContainerStarted","Data":"3142b0d1026c74a97a5a9312ebc86366c1a7a30582fbbf141e2bcdf44c1833a2"} Nov 03 13:45:39.962149 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:39.962120 5979 scope.go:122] "RemoveContainer" containerID="1d343e63cb3beedf3c9030d3ec7edaba1256dd00b507a492462edf1af793c031" Nov 03 13:45:39.972013 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:39.971987 5979 scope.go:122] "RemoveContainer" containerID="07dce1585d30ffbfde67bb61883dbef54c2fd155be42615a8a7a79b1ae8cbbcd" Nov 03 13:45:39.972350 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:45:39.972324 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07dce1585d30ffbfde67bb61883dbef54c2fd155be42615a8a7a79b1ae8cbbcd\": container with ID starting with 07dce1585d30ffbfde67bb61883dbef54c2fd155be42615a8a7a79b1ae8cbbcd not found: ID does not exist" containerID="07dce1585d30ffbfde67bb61883dbef54c2fd155be42615a8a7a79b1ae8cbbcd" Nov 03 13:45:39.972465 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:39.972360 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07dce1585d30ffbfde67bb61883dbef54c2fd155be42615a8a7a79b1ae8cbbcd"} err="failed to get container status \"07dce1585d30ffbfde67bb61883dbef54c2fd155be42615a8a7a79b1ae8cbbcd\": rpc error: code = NotFound desc = could not find container \"07dce1585d30ffbfde67bb61883dbef54c2fd155be42615a8a7a79b1ae8cbbcd\": container with ID starting with 07dce1585d30ffbfde67bb61883dbef54c2fd155be42615a8a7a79b1ae8cbbcd not found: ID does not exist" Nov 03 13:45:39.972465 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:39.972383 5979 scope.go:122] "RemoveContainer" containerID="1d343e63cb3beedf3c9030d3ec7edaba1256dd00b507a492462edf1af793c031" Nov 03 13:45:39.972736 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:45:39.972709 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d343e63cb3beedf3c9030d3ec7edaba1256dd00b507a492462edf1af793c031\": container with ID starting with 1d343e63cb3beedf3c9030d3ec7edaba1256dd00b507a492462edf1af793c031 not found: ID does not exist" containerID="1d343e63cb3beedf3c9030d3ec7edaba1256dd00b507a492462edf1af793c031" Nov 03 13:45:39.972800 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:39.972747 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d343e63cb3beedf3c9030d3ec7edaba1256dd00b507a492462edf1af793c031"} err="failed to get container status \"1d343e63cb3beedf3c9030d3ec7edaba1256dd00b507a492462edf1af793c031\": rpc error: code = NotFound desc = could not find container \"1d343e63cb3beedf3c9030d3ec7edaba1256dd00b507a492462edf1af793c031\": container with ID starting with 1d343e63cb3beedf3c9030d3ec7edaba1256dd00b507a492462edf1af793c031 not found: ID does not exist" Nov 03 13:45:40.031362 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:40.031332 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87552227-fce6-474b-9b1b-fe167d1f6547-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87552227-fce6-474b-9b1b-fe167d1f6547-workload-socket\") pod \"87552227-fce6-474b-9b1b-fe167d1f6547\" (UID: \"87552227-fce6-474b-9b1b-fe167d1f6547\") " Nov 03 13:45:40.031550 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:40.031390 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87552227-fce6-474b-9b1b-fe167d1f6547-kube-api-access-8hdb2\" (UniqueName: \"kubernetes.io/projected/87552227-fce6-474b-9b1b-fe167d1f6547-kube-api-access-8hdb2\") pod \"87552227-fce6-474b-9b1b-fe167d1f6547\" (UID: \"87552227-fce6-474b-9b1b-fe167d1f6547\") " Nov 03 13:45:40.031550 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:40.031413 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87552227-fce6-474b-9b1b-fe167d1f6547-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87552227-fce6-474b-9b1b-fe167d1f6547-credential-socket\") pod \"87552227-fce6-474b-9b1b-fe167d1f6547\" (UID: \"87552227-fce6-474b-9b1b-fe167d1f6547\") " Nov 03 13:45:40.031550 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:40.031448 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/87552227-fce6-474b-9b1b-fe167d1f6547-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87552227-fce6-474b-9b1b-fe167d1f6547-istio-podinfo\") pod \"87552227-fce6-474b-9b1b-fe167d1f6547\" (UID: \"87552227-fce6-474b-9b1b-fe167d1f6547\") " Nov 03 13:45:40.031550 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:40.031521 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87552227-fce6-474b-9b1b-fe167d1f6547-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87552227-fce6-474b-9b1b-fe167d1f6547-istio-envoy\") pod \"87552227-fce6-474b-9b1b-fe167d1f6547\" (UID: \"87552227-fce6-474b-9b1b-fe167d1f6547\") " Nov 03 13:45:40.031806 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:40.031588 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87552227-fce6-474b-9b1b-fe167d1f6547-istio-token\" (UniqueName: \"kubernetes.io/projected/87552227-fce6-474b-9b1b-fe167d1f6547-istio-token\") pod \"87552227-fce6-474b-9b1b-fe167d1f6547\" (UID: \"87552227-fce6-474b-9b1b-fe167d1f6547\") " Nov 03 13:45:40.031806 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:40.031668 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87552227-fce6-474b-9b1b-fe167d1f6547-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87552227-fce6-474b-9b1b-fe167d1f6547-istiod-ca-cert\") pod \"87552227-fce6-474b-9b1b-fe167d1f6547\" (UID: \"87552227-fce6-474b-9b1b-fe167d1f6547\") " Nov 03 13:45:40.031806 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:40.031708 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87552227-fce6-474b-9b1b-fe167d1f6547-workload-socket" pod "87552227-fce6-474b-9b1b-fe167d1f6547" (UID: "87552227-fce6-474b-9b1b-fe167d1f6547"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:45:40.031806 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:40.031721 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87552227-fce6-474b-9b1b-fe167d1f6547-credential-socket" pod "87552227-fce6-474b-9b1b-fe167d1f6547" (UID: "87552227-fce6-474b-9b1b-fe167d1f6547"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:45:40.031806 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:40.031713 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87552227-fce6-474b-9b1b-fe167d1f6547-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87552227-fce6-474b-9b1b-fe167d1f6547-workload-certs\") pod \"87552227-fce6-474b-9b1b-fe167d1f6547\" (UID: \"87552227-fce6-474b-9b1b-fe167d1f6547\") " Nov 03 13:45:40.032068 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:40.031947 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87552227-fce6-474b-9b1b-fe167d1f6547-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87552227-fce6-474b-9b1b-fe167d1f6547-istio-data\") pod \"87552227-fce6-474b-9b1b-fe167d1f6547\" (UID: \"87552227-fce6-474b-9b1b-fe167d1f6547\") " Nov 03 13:45:40.032220 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:40.032189 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87552227-fce6-474b-9b1b-fe167d1f6547-workload-certs" pod "87552227-fce6-474b-9b1b-fe167d1f6547" (UID: "87552227-fce6-474b-9b1b-fe167d1f6547"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:45:40.032316 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:40.032262 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87552227-fce6-474b-9b1b-fe167d1f6547-istiod-ca-cert" pod "87552227-fce6-474b-9b1b-fe167d1f6547" (UID: "87552227-fce6-474b-9b1b-fe167d1f6547"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:45:40.032415 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:40.032392 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87552227-fce6-474b-9b1b-fe167d1f6547-istio-data" pod "87552227-fce6-474b-9b1b-fe167d1f6547" (UID: "87552227-fce6-474b-9b1b-fe167d1f6547"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:45:40.032563 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:40.032539 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87552227-fce6-474b-9b1b-fe167d1f6547-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:40.032648 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:40.032567 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87552227-fce6-474b-9b1b-fe167d1f6547-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:40.032648 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:40.032581 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87552227-fce6-474b-9b1b-fe167d1f6547-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:40.032648 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:40.032594 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87552227-fce6-474b-9b1b-fe167d1f6547-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:40.032648 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:40.032608 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87552227-fce6-474b-9b1b-fe167d1f6547-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:40.034098 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:40.034077 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87552227-fce6-474b-9b1b-fe167d1f6547-istio-token" pod "87552227-fce6-474b-9b1b-fe167d1f6547" (UID: "87552227-fce6-474b-9b1b-fe167d1f6547"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:45:40.034519 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:40.034463 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87552227-fce6-474b-9b1b-fe167d1f6547-kube-api-access-8hdb2" pod "87552227-fce6-474b-9b1b-fe167d1f6547" (UID: "87552227-fce6-474b-9b1b-fe167d1f6547"). InnerVolumeSpecName "kube-api-access-8hdb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:45:40.034642 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:40.034613 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87552227-fce6-474b-9b1b-fe167d1f6547-istio-envoy" pod "87552227-fce6-474b-9b1b-fe167d1f6547" (UID: "87552227-fce6-474b-9b1b-fe167d1f6547"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:45:40.034642 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:40.034618 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87552227-fce6-474b-9b1b-fe167d1f6547-istio-podinfo" pod "87552227-fce6-474b-9b1b-fe167d1f6547" (UID: "87552227-fce6-474b-9b1b-fe167d1f6547"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:45:40.134203 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:40.134173 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hdb2\" (UniqueName: \"kubernetes.io/projected/87552227-fce6-474b-9b1b-fe167d1f6547-kube-api-access-8hdb2\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:40.134203 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:40.134202 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87552227-fce6-474b-9b1b-fe167d1f6547-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:40.134401 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:40.134212 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87552227-fce6-474b-9b1b-fe167d1f6547-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:40.134401 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:40.134222 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87552227-fce6-474b-9b1b-fe167d1f6547-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:40.326521 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:40.326484 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686"] Nov 03 13:45:40.332288 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:40.332258 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-8d8af2-openshift-default-65cc65598f-t2686"] Nov 03 13:45:41.073349 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:41.073318 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c5861a-c9qg5" Nov 03 13:45:41.248859 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:41.248772 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1cfb1a0-cef9-4bf9-8392-e671178ef46e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1cfb1a0-cef9-4bf9-8392-e671178ef46e-maas-api-xks-ca\") pod \"c1cfb1a0-cef9-4bf9-8392-e671178ef46e\" (UID: \"c1cfb1a0-cef9-4bf9-8392-e671178ef46e\") " Nov 03 13:45:41.248859 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:41.248847 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1cfb1a0-cef9-4bf9-8392-e671178ef46e-kube-api-access-gncwv\" (UniqueName: \"kubernetes.io/projected/c1cfb1a0-cef9-4bf9-8392-e671178ef46e-kube-api-access-gncwv\") pod \"c1cfb1a0-cef9-4bf9-8392-e671178ef46e\" (UID: \"c1cfb1a0-cef9-4bf9-8392-e671178ef46e\") " Nov 03 13:45:41.249128 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:41.248972 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1cfb1a0-cef9-4bf9-8392-e671178ef46e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1cfb1a0-cef9-4bf9-8392-e671178ef46e-maas-api-openshift-ca\") pod \"c1cfb1a0-cef9-4bf9-8392-e671178ef46e\" (UID: \"c1cfb1a0-cef9-4bf9-8392-e671178ef46e\") " Nov 03 13:45:41.249376 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:41.249348 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1cfb1a0-cef9-4bf9-8392-e671178ef46e-maas-api-openshift-ca" pod "c1cfb1a0-cef9-4bf9-8392-e671178ef46e" (UID: "c1cfb1a0-cef9-4bf9-8392-e671178ef46e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:45:41.249890 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:41.249856 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1cfb1a0-cef9-4bf9-8392-e671178ef46e-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:41.251436 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:41.251409 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1cfb1a0-cef9-4bf9-8392-e671178ef46e-maas-api-xks-ca" pod "c1cfb1a0-cef9-4bf9-8392-e671178ef46e" (UID: "c1cfb1a0-cef9-4bf9-8392-e671178ef46e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:45:41.251798 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:41.251773 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1cfb1a0-cef9-4bf9-8392-e671178ef46e-kube-api-access-gncwv" pod "c1cfb1a0-cef9-4bf9-8392-e671178ef46e" (UID: "c1cfb1a0-cef9-4bf9-8392-e671178ef46e"). InnerVolumeSpecName "kube-api-access-gncwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:45:41.351716 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:41.351639 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1cfb1a0-cef9-4bf9-8392-e671178ef46e-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:41.351716 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:41.351669 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gncwv\" (UniqueName: \"kubernetes.io/projected/c1cfb1a0-cef9-4bf9-8392-e671178ef46e-kube-api-access-gncwv\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:45:41.933364 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:41.933329 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c5861a-c9qg5" Nov 03 13:45:41.933364 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:41.933352 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80c5861a-c9qg5" event={"ID":"c1cfb1a0-cef9-4bf9-8392-e671178ef46e","Type":"ContainerDied","Data":"3142b0d1026c74a97a5a9312ebc86366c1a7a30582fbbf141e2bcdf44c1833a2"} Nov 03 13:45:41.933640 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:41.933402 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3142b0d1026c74a97a5a9312ebc86366c1a7a30582fbbf141e2bcdf44c1833a2" Nov 03 13:45:41.935389 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:41.935355 5979 generic.go:356] "Generic (PLEG): container finished" podID="0403729d-8803-40c2-9e5d-1cb2cc9914a1" containerID="2d13cb7796b4dd992f24aa0a404f34946439c117c567f652df7148d34bf9bbd0" exitCode=0 Nov 03 13:45:41.935532 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:41.935416 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6" event={"ID":"0403729d-8803-40c2-9e5d-1cb2cc9914a1","Type":"ContainerDied","Data":"2d13cb7796b4dd992f24aa0a404f34946439c117c567f652df7148d34bf9bbd0"} Nov 03 13:45:41.935847 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:41.935830 5979 scope.go:122] "RemoveContainer" containerID="2d13cb7796b4dd992f24aa0a404f34946439c117c567f652df7148d34bf9bbd0" Nov 03 13:45:41.937538 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:41.937506 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:45:41.937621 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:41.937594 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:45:41.937661 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:41.937642 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:45:42.116230 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:42.116194 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87552227-fce6-474b-9b1b-fe167d1f6547" path="/var/lib/kubelet/pods/87552227-fce6-474b-9b1b-fe167d1f6547/volumes" Nov 03 13:45:42.943270 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:42.943238 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6" event={"ID":"0403729d-8803-40c2-9e5d-1cb2cc9914a1","Type":"ContainerStarted","Data":"7c05b13cbe2c7dd9aa32bae96ebb5782e0e1d851f11f7e37ecca8cb492b71074"} Nov 03 13:45:43.814333 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:43.814297 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6" Nov 03 13:45:43.815117 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:43.815092 5979 patch_prober.go:29] interesting pod/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Nov 03 13:45:43.815228 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:43.815139 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6" podUID="0403729d-8803-40c2-9e5d-1cb2cc9914a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Nov 03 13:45:44.814946 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:44.814906 5979 patch_prober.go:29] interesting pod/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Nov 03 13:45:44.815374 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:44.814970 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6" podUID="0403729d-8803-40c2-9e5d-1cb2cc9914a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Nov 03 13:45:45.814758 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:45.814721 5979 patch_prober.go:29] interesting pod/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Nov 03 13:45:45.814942 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:45.814785 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6" podUID="0403729d-8803-40c2-9e5d-1cb2cc9914a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Nov 03 13:45:46.814612 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:46.814572 5979 patch_prober.go:29] interesting pod/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Nov 03 13:45:46.815066 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:46.814634 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6" podUID="0403729d-8803-40c2-9e5d-1cb2cc9914a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Nov 03 13:45:47.814439 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:47.814403 5979 patch_prober.go:29] interesting pod/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Nov 03 13:45:47.814645 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:47.814456 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6" podUID="0403729d-8803-40c2-9e5d-1cb2cc9914a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Nov 03 13:45:48.818901 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:48.818872 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6" Nov 03 13:45:48.819313 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:48.819147 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6" Nov 03 13:45:48.819818 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:48.819800 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6" Nov 03 13:45:50.656374 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:50.656334 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz"] Nov 03 13:45:50.656918 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:50.656904 5979 cpu_manager.go:414] "containerMap: removing container" podUID="87552227-fce6-474b-9b1b-fe167d1f6547" containerName="istio-proxy" Nov 03 13:45:50.656965 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:50.656919 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87552227-fce6-474b-9b1b-fe167d1f6547" containerName="istio-proxy" Nov 03 13:45:50.656965 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:50.656939 5979 cpu_manager.go:414] "containerMap: removing container" podUID="c1cfb1a0-cef9-4bf9-8392-e671178ef46e" containerName="revoke-keys" Nov 03 13:45:50.656965 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:50.656945 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1cfb1a0-cef9-4bf9-8392-e671178ef46e" containerName="revoke-keys" Nov 03 13:45:50.657067 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:50.657056 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="87552227-fce6-474b-9b1b-fe167d1f6547" containerName="istio-proxy" Nov 03 13:45:50.657067 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:50.657066 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="87552227-fce6-474b-9b1b-fe167d1f6547" containerName="istio-proxy" Nov 03 13:45:50.657133 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:50.657076 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1cfb1a0-cef9-4bf9-8392-e671178ef46e" containerName="revoke-keys" Nov 03 13:45:50.660742 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:50.660715 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz" Nov 03 13:45:50.726858 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:50.726814 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70ed2a89-8231-4e71-862a-02968573d4fe-istiod-ca-cert\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz\" (UID: \"70ed2a89-8231-4e71-862a-02968573d4fe\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz" Nov 03 13:45:50.726858 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:50.726864 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70ed2a89-8231-4e71-862a-02968573d4fe-istio-podinfo\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz\" (UID: \"70ed2a89-8231-4e71-862a-02968573d4fe\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz" Nov 03 13:45:50.727091 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:50.726990 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70ed2a89-8231-4e71-862a-02968573d4fe-istio-envoy\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz\" (UID: \"70ed2a89-8231-4e71-862a-02968573d4fe\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz" Nov 03 13:45:50.727091 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:50.727043 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70ed2a89-8231-4e71-862a-02968573d4fe-credential-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz\" (UID: \"70ed2a89-8231-4e71-862a-02968573d4fe\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz" Nov 03 13:45:50.727164 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:50.727103 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70ed2a89-8231-4e71-862a-02968573d4fe-istio-token\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz\" (UID: \"70ed2a89-8231-4e71-862a-02968573d4fe\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz" Nov 03 13:45:50.727206 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:50.727166 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70ed2a89-8231-4e71-862a-02968573d4fe-istio-data\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz\" (UID: \"70ed2a89-8231-4e71-862a-02968573d4fe\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz" Nov 03 13:45:50.727265 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:50.727252 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70ed2a89-8231-4e71-862a-02968573d4fe-workload-certs\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz\" (UID: \"70ed2a89-8231-4e71-862a-02968573d4fe\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz" Nov 03 13:45:50.727301 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:50.727273 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70ed2a89-8231-4e71-862a-02968573d4fe-workload-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz\" (UID: \"70ed2a89-8231-4e71-862a-02968573d4fe\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz" Nov 03 13:45:50.727379 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:50.727366 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjg2b\" (UniqueName: \"kubernetes.io/projected/70ed2a89-8231-4e71-862a-02968573d4fe-kube-api-access-sjg2b\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz\" (UID: \"70ed2a89-8231-4e71-862a-02968573d4fe\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz" Nov 03 13:45:50.763556 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:50.763519 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz"] Nov 03 13:45:50.829088 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:50.829049 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70ed2a89-8231-4e71-862a-02968573d4fe-workload-certs\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz\" (UID: \"70ed2a89-8231-4e71-862a-02968573d4fe\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz" Nov 03 13:45:50.829088 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:50.829087 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70ed2a89-8231-4e71-862a-02968573d4fe-workload-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz\" (UID: \"70ed2a89-8231-4e71-862a-02968573d4fe\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz" Nov 03 13:45:50.829361 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:50.829134 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjg2b\" (UniqueName: \"kubernetes.io/projected/70ed2a89-8231-4e71-862a-02968573d4fe-kube-api-access-sjg2b\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz\" (UID: \"70ed2a89-8231-4e71-862a-02968573d4fe\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz" Nov 03 13:45:50.829361 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:50.829174 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70ed2a89-8231-4e71-862a-02968573d4fe-istiod-ca-cert\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz\" (UID: \"70ed2a89-8231-4e71-862a-02968573d4fe\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz" Nov 03 13:45:50.829361 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:50.829200 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70ed2a89-8231-4e71-862a-02968573d4fe-istio-podinfo\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz\" (UID: \"70ed2a89-8231-4e71-862a-02968573d4fe\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz" Nov 03 13:45:50.829361 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:50.829280 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70ed2a89-8231-4e71-862a-02968573d4fe-istio-envoy\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz\" (UID: \"70ed2a89-8231-4e71-862a-02968573d4fe\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz" Nov 03 13:45:50.829361 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:50.829332 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70ed2a89-8231-4e71-862a-02968573d4fe-credential-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz\" (UID: \"70ed2a89-8231-4e71-862a-02968573d4fe\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz" Nov 03 13:45:50.829361 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:50.829355 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70ed2a89-8231-4e71-862a-02968573d4fe-istio-token\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz\" (UID: \"70ed2a89-8231-4e71-862a-02968573d4fe\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz" Nov 03 13:45:50.829728 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:50.829411 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70ed2a89-8231-4e71-862a-02968573d4fe-istio-data\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz\" (UID: \"70ed2a89-8231-4e71-862a-02968573d4fe\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz" Nov 03 13:45:50.829728 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:50.829613 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70ed2a89-8231-4e71-862a-02968573d4fe-workload-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz\" (UID: \"70ed2a89-8231-4e71-862a-02968573d4fe\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz" Nov 03 13:45:50.829728 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:50.829661 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70ed2a89-8231-4e71-862a-02968573d4fe-workload-certs\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz\" (UID: \"70ed2a89-8231-4e71-862a-02968573d4fe\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz" Nov 03 13:45:50.829885 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:50.829790 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70ed2a89-8231-4e71-862a-02968573d4fe-credential-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz\" (UID: \"70ed2a89-8231-4e71-862a-02968573d4fe\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz" Nov 03 13:45:50.829885 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:50.829857 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70ed2a89-8231-4e71-862a-02968573d4fe-istio-data\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz\" (UID: \"70ed2a89-8231-4e71-862a-02968573d4fe\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz" Nov 03 13:45:50.830045 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:50.830022 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70ed2a89-8231-4e71-862a-02968573d4fe-istiod-ca-cert\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz\" (UID: \"70ed2a89-8231-4e71-862a-02968573d4fe\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz" Nov 03 13:45:50.831737 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:50.831717 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70ed2a89-8231-4e71-862a-02968573d4fe-istio-envoy\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz\" (UID: \"70ed2a89-8231-4e71-862a-02968573d4fe\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz" Nov 03 13:45:50.831794 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:50.831779 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70ed2a89-8231-4e71-862a-02968573d4fe-istio-podinfo\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz\" (UID: \"70ed2a89-8231-4e71-862a-02968573d4fe\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz" Nov 03 13:45:50.888951 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:50.888917 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70ed2a89-8231-4e71-862a-02968573d4fe-istio-token\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz\" (UID: \"70ed2a89-8231-4e71-862a-02968573d4fe\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz" Nov 03 13:45:50.889139 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:50.889107 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjg2b\" (UniqueName: \"kubernetes.io/projected/70ed2a89-8231-4e71-862a-02968573d4fe-kube-api-access-sjg2b\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz\" (UID: \"70ed2a89-8231-4e71-862a-02968573d4fe\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz" Nov 03 13:45:50.977214 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:50.977120 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz" Nov 03 13:45:51.409441 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:51.409401 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz"] Nov 03 13:45:51.416411 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:45:51.416383 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70ed2a89_8231_4e71_862a_02968573d4fe.slice/crio-cf9c4ab20df8907ce397820809e836ee8cd31d574e198c5d79cd6cf37e472308 WatchSource:0}: Error finding container cf9c4ab20df8907ce397820809e836ee8cd31d574e198c5d79cd6cf37e472308: Status 404 returned error can't find the container with id cf9c4ab20df8907ce397820809e836ee8cd31d574e198c5d79cd6cf37e472308 Nov 03 13:45:51.419072 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:51.419034 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:45:51.419175 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:51.419104 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:45:51.419175 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:51.419134 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:45:52.030796 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:52.030760 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz" event={"ID":"70ed2a89-8231-4e71-862a-02968573d4fe","Type":"ContainerStarted","Data":"ff2997e72260c1176d52a45923ef02aabeb3a6cc98d253876911eab57073374b"} Nov 03 13:45:52.031258 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:52.030813 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz" event={"ID":"70ed2a89-8231-4e71-862a-02968573d4fe","Type":"ContainerStarted","Data":"cf9c4ab20df8907ce397820809e836ee8cd31d574e198c5d79cd6cf37e472308"} Nov 03 13:45:52.977580 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:52.977538 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz" Nov 03 13:45:52.979384 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:52.979355 5979 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" start-of-body= Nov 03 13:45:52.979540 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:52.979405 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz" podUID="70ed2a89-8231-4e71-862a-02968573d4fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" Nov 03 13:45:53.977868 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:53.977831 5979 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" start-of-body= Nov 03 13:45:53.978310 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:53.977885 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz" podUID="70ed2a89-8231-4e71-862a-02968573d4fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" Nov 03 13:45:54.978333 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:54.978294 5979 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" start-of-body= Nov 03 13:45:54.978772 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:54.978351 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz" podUID="70ed2a89-8231-4e71-862a-02968573d4fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" Nov 03 13:45:55.977623 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:55.977578 5979 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" start-of-body= Nov 03 13:45:55.977810 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:55.977636 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz" podUID="70ed2a89-8231-4e71-862a-02968573d4fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" Nov 03 13:45:56.978188 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:56.978150 5979 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" start-of-body= Nov 03 13:45:56.978650 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:56.978212 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz" podUID="70ed2a89-8231-4e71-862a-02968573d4fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" Nov 03 13:45:57.978159 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:57.978125 5979 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" start-of-body= Nov 03 13:45:57.978323 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:57.978179 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz" podUID="70ed2a89-8231-4e71-862a-02968573d4fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.240:15021/healthz/ready\": dial tcp 10.134.0.240:15021: connect: connection refused" Nov 03 13:45:58.115773 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:58.115722 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz" podStartSLOduration=8.115708312 podStartE2EDuration="8.115708312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:45:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:45:52.169978386 +0000 UTC m=+2434.180666646" watchObservedRunningTime="2026-11-03 13:45:58.115708312 +0000 UTC m=+2440.126396559" Nov 03 13:45:58.117217 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:58.117189 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a770b9-6vls9"] Nov 03 13:45:58.117851 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:58.117823 5979 cpu_manager.go:414] "containerMap: removing container" podUID="87552227-fce6-474b-9b1b-fe167d1f6547" containerName="istio-proxy" Nov 03 13:45:58.117851 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:58.117846 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87552227-fce6-474b-9b1b-fe167d1f6547" containerName="istio-proxy" Nov 03 13:45:58.121702 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:58.121681 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a770b9-6vls9" Nov 03 13:45:58.170771 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:58.170738 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:45:58.217370 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:58.217338 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:45:58.220241 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:58.220216 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a770b9-6vls9"] Nov 03 13:45:58.245769 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:58.245679 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzsqh\" (UniqueName: \"kubernetes.io/projected/0f7b92f8-adb4-4b0f-b59d-2949c7bad86e-kube-api-access-xzsqh\") pod \"maas-api-revoke-keys-e2e-ipp-a-a770b9-6vls9\" (UID: \"0f7b92f8-adb4-4b0f-b59d-2949c7bad86e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a770b9-6vls9" Nov 03 13:45:58.245769 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:58.245729 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f7b92f8-adb4-4b0f-b59d-2949c7bad86e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a770b9-6vls9\" (UID: \"0f7b92f8-adb4-4b0f-b59d-2949c7bad86e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a770b9-6vls9" Nov 03 13:45:58.246002 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:58.245954 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f7b92f8-adb4-4b0f-b59d-2949c7bad86e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a770b9-6vls9\" (UID: \"0f7b92f8-adb4-4b0f-b59d-2949c7bad86e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a770b9-6vls9" Nov 03 13:45:58.347008 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:58.346959 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f7b92f8-adb4-4b0f-b59d-2949c7bad86e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a770b9-6vls9\" (UID: \"0f7b92f8-adb4-4b0f-b59d-2949c7bad86e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a770b9-6vls9" Nov 03 13:45:58.347217 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:58.347194 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f7b92f8-adb4-4b0f-b59d-2949c7bad86e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a770b9-6vls9\" (UID: \"0f7b92f8-adb4-4b0f-b59d-2949c7bad86e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a770b9-6vls9" Nov 03 13:45:58.347315 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:58.347300 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzsqh\" (UniqueName: \"kubernetes.io/projected/0f7b92f8-adb4-4b0f-b59d-2949c7bad86e-kube-api-access-xzsqh\") pod \"maas-api-revoke-keys-e2e-ipp-a-a770b9-6vls9\" (UID: \"0f7b92f8-adb4-4b0f-b59d-2949c7bad86e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a770b9-6vls9" Nov 03 13:45:58.347879 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:58.347844 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f7b92f8-adb4-4b0f-b59d-2949c7bad86e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a770b9-6vls9\" (UID: \"0f7b92f8-adb4-4b0f-b59d-2949c7bad86e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a770b9-6vls9" Nov 03 13:45:58.349333 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:58.349310 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f7b92f8-adb4-4b0f-b59d-2949c7bad86e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a770b9-6vls9\" (UID: \"0f7b92f8-adb4-4b0f-b59d-2949c7bad86e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a770b9-6vls9" Nov 03 13:45:58.403566 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:58.403525 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzsqh\" (UniqueName: \"kubernetes.io/projected/0f7b92f8-adb4-4b0f-b59d-2949c7bad86e-kube-api-access-xzsqh\") pod \"maas-api-revoke-keys-e2e-ipp-a-a770b9-6vls9\" (UID: \"0f7b92f8-adb4-4b0f-b59d-2949c7bad86e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a770b9-6vls9" Nov 03 13:45:58.435881 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:58.435851 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a770b9-6vls9" Nov 03 13:45:58.835503 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:58.835365 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a770b9-6vls9"] Nov 03 13:45:58.841789 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:45:58.841759 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f7b92f8_adb4_4b0f_b59d_2949c7bad86e.slice/crio-7bc692acfe4e4222f7ca01bb31b560233422aa565bf8ea522bd224cdc3f95796 WatchSource:0}: Error finding container 7bc692acfe4e4222f7ca01bb31b560233422aa565bf8ea522bd224cdc3f95796: Status 404 returned error can't find the container with id 7bc692acfe4e4222f7ca01bb31b560233422aa565bf8ea522bd224cdc3f95796 Nov 03 13:45:58.982788 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:58.982756 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz" Nov 03 13:45:58.983197 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:58.983096 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz" Nov 03 13:45:58.983933 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:58.983913 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz" Nov 03 13:45:59.114597 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:45:59.114507 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a770b9-6vls9" event={"ID":"0f7b92f8-adb4-4b0f-b59d-2949c7bad86e","Type":"ContainerStarted","Data":"7bc692acfe4e4222f7ca01bb31b560233422aa565bf8ea522bd224cdc3f95796"} Nov 03 13:46:00.121999 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:00.121911 5979 generic.go:356] "Generic (PLEG): container finished" podID="0f7b92f8-adb4-4b0f-b59d-2949c7bad86e" containerID="a6915fe74a323f051267790257d49c2ac8b94cc70c3cf26cad9128c19341c4cd" exitCode=0 Nov 03 13:46:00.122403 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:00.121992 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a770b9-6vls9" event={"ID":"0f7b92f8-adb4-4b0f-b59d-2949c7bad86e","Type":"ContainerDied","Data":"a6915fe74a323f051267790257d49c2ac8b94cc70c3cf26cad9128c19341c4cd"} Nov 03 13:46:00.485599 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:00.485542 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74d9e037-96c4b"] Nov 03 13:46:00.490026 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:00.490000 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d9e037-96c4b" Nov 03 13:46:00.543908 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:00.543877 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74d9e037-96c4b"] Nov 03 13:46:00.583916 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:00.583868 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dgj9\" (UniqueName: \"kubernetes.io/projected/c03eda48-e87b-4847-a9be-da542cba2654-kube-api-access-4dgj9\") pod \"maas-api-revoke-sub-74d9e037-96c4b\" (UID: \"c03eda48-e87b-4847-a9be-da542cba2654\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d9e037-96c4b" Nov 03 13:46:00.584364 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:00.584331 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c03eda48-e87b-4847-a9be-da542cba2654-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74d9e037-96c4b\" (UID: \"c03eda48-e87b-4847-a9be-da542cba2654\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d9e037-96c4b" Nov 03 13:46:00.584534 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:00.584508 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c03eda48-e87b-4847-a9be-da542cba2654-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74d9e037-96c4b\" (UID: \"c03eda48-e87b-4847-a9be-da542cba2654\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d9e037-96c4b" Nov 03 13:46:00.685914 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:00.685855 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c03eda48-e87b-4847-a9be-da542cba2654-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74d9e037-96c4b\" (UID: \"c03eda48-e87b-4847-a9be-da542cba2654\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d9e037-96c4b" Nov 03 13:46:00.686122 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:00.685973 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c03eda48-e87b-4847-a9be-da542cba2654-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74d9e037-96c4b\" (UID: \"c03eda48-e87b-4847-a9be-da542cba2654\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d9e037-96c4b" Nov 03 13:46:00.686122 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:00.686038 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dgj9\" (UniqueName: \"kubernetes.io/projected/c03eda48-e87b-4847-a9be-da542cba2654-kube-api-access-4dgj9\") pod \"maas-api-revoke-sub-74d9e037-96c4b\" (UID: \"c03eda48-e87b-4847-a9be-da542cba2654\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d9e037-96c4b" Nov 03 13:46:00.686929 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:00.686897 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c03eda48-e87b-4847-a9be-da542cba2654-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74d9e037-96c4b\" (UID: \"c03eda48-e87b-4847-a9be-da542cba2654\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d9e037-96c4b" Nov 03 13:46:00.688574 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:00.688547 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c03eda48-e87b-4847-a9be-da542cba2654-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74d9e037-96c4b\" (UID: \"c03eda48-e87b-4847-a9be-da542cba2654\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d9e037-96c4b" Nov 03 13:46:00.739997 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:00.739914 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dgj9\" (UniqueName: \"kubernetes.io/projected/c03eda48-e87b-4847-a9be-da542cba2654-kube-api-access-4dgj9\") pod \"maas-api-revoke-sub-74d9e037-96c4b\" (UID: \"c03eda48-e87b-4847-a9be-da542cba2654\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d9e037-96c4b" Nov 03 13:46:00.808033 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:00.807982 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d9e037-96c4b" Nov 03 13:46:01.239157 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:01.239118 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74d9e037-96c4b"] Nov 03 13:46:01.313896 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:46:01.307318 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc03eda48_e87b_4847_a9be_da542cba2654.slice/crio-9d81952db54bc57fa853933f35e78ac23932a9eaa471c258bbbe402dac979684 WatchSource:0}: Error finding container 9d81952db54bc57fa853933f35e78ac23932a9eaa471c258bbbe402dac979684: Status 404 returned error can't find the container with id 9d81952db54bc57fa853933f35e78ac23932a9eaa471c258bbbe402dac979684 Nov 03 13:46:01.331586 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:01.331560 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a770b9-6vls9" Nov 03 13:46:01.509271 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:01.505902 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f7b92f8-adb4-4b0f-b59d-2949c7bad86e-kube-api-access-xzsqh\" (UniqueName: \"kubernetes.io/projected/0f7b92f8-adb4-4b0f-b59d-2949c7bad86e-kube-api-access-xzsqh\") pod \"0f7b92f8-adb4-4b0f-b59d-2949c7bad86e\" (UID: \"0f7b92f8-adb4-4b0f-b59d-2949c7bad86e\") " Nov 03 13:46:01.509271 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:01.506026 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f7b92f8-adb4-4b0f-b59d-2949c7bad86e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f7b92f8-adb4-4b0f-b59d-2949c7bad86e-maas-api-openshift-ca\") pod \"0f7b92f8-adb4-4b0f-b59d-2949c7bad86e\" (UID: \"0f7b92f8-adb4-4b0f-b59d-2949c7bad86e\") " Nov 03 13:46:01.509271 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:01.506126 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f7b92f8-adb4-4b0f-b59d-2949c7bad86e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f7b92f8-adb4-4b0f-b59d-2949c7bad86e-maas-api-xks-ca\") pod \"0f7b92f8-adb4-4b0f-b59d-2949c7bad86e\" (UID: \"0f7b92f8-adb4-4b0f-b59d-2949c7bad86e\") " Nov 03 13:46:01.509271 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:01.507539 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f7b92f8-adb4-4b0f-b59d-2949c7bad86e-maas-api-openshift-ca" pod "0f7b92f8-adb4-4b0f-b59d-2949c7bad86e" (UID: "0f7b92f8-adb4-4b0f-b59d-2949c7bad86e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:46:01.567304 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:01.565627 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f7b92f8-adb4-4b0f-b59d-2949c7bad86e-kube-api-access-xzsqh" pod "0f7b92f8-adb4-4b0f-b59d-2949c7bad86e" (UID: "0f7b92f8-adb4-4b0f-b59d-2949c7bad86e"). InnerVolumeSpecName "kube-api-access-xzsqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:46:01.567304 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:01.565882 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f7b92f8-adb4-4b0f-b59d-2949c7bad86e-maas-api-xks-ca" pod "0f7b92f8-adb4-4b0f-b59d-2949c7bad86e" (UID: "0f7b92f8-adb4-4b0f-b59d-2949c7bad86e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:46:01.608353 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:01.608316 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzsqh\" (UniqueName: \"kubernetes.io/projected/0f7b92f8-adb4-4b0f-b59d-2949c7bad86e-kube-api-access-xzsqh\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:01.608353 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:01.608351 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f7b92f8-adb4-4b0f-b59d-2949c7bad86e-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:01.608617 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:01.608363 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f7b92f8-adb4-4b0f-b59d-2949c7bad86e-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:02.138360 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:02.138315 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d9e037-96c4b" event={"ID":"c03eda48-e87b-4847-a9be-da542cba2654","Type":"ContainerStarted","Data":"fec1c6f4e0ecfa6773e04e38a58d359063d2f97ca0b942ebc15d8d66be00452d"} Nov 03 13:46:02.138571 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:02.138377 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d9e037-96c4b" event={"ID":"c03eda48-e87b-4847-a9be-da542cba2654","Type":"ContainerStarted","Data":"9d81952db54bc57fa853933f35e78ac23932a9eaa471c258bbbe402dac979684"} Nov 03 13:46:02.139723 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:02.139696 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a770b9-6vls9" Nov 03 13:46:02.139723 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:02.139710 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a770b9-6vls9" event={"ID":"0f7b92f8-adb4-4b0f-b59d-2949c7bad86e","Type":"ContainerDied","Data":"7bc692acfe4e4222f7ca01bb31b560233422aa565bf8ea522bd224cdc3f95796"} Nov 03 13:46:02.139899 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:02.139744 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7bc692acfe4e4222f7ca01bb31b560233422aa565bf8ea522bd224cdc3f95796" Nov 03 13:46:02.245582 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:02.245528 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d9e037-96c4b" podStartSLOduration=2.2455144049999998 podStartE2EDuration="2.245514405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:46:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:46:02.244521216 +0000 UTC m=+2444.255209473" watchObservedRunningTime="2026-11-03 13:46:02.245514405 +0000 UTC m=+2444.256202675" Nov 03 13:46:03.147458 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:03.147426 5979 generic.go:356] "Generic (PLEG): container finished" podID="c03eda48-e87b-4847-a9be-da542cba2654" containerID="fec1c6f4e0ecfa6773e04e38a58d359063d2f97ca0b942ebc15d8d66be00452d" exitCode=0 Nov 03 13:46:03.147458 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:03.147512 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d9e037-96c4b" event={"ID":"c03eda48-e87b-4847-a9be-da542cba2654","Type":"ContainerDied","Data":"fec1c6f4e0ecfa6773e04e38a58d359063d2f97ca0b942ebc15d8d66be00452d"} Nov 03 13:46:04.305746 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:04.305720 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d9e037-96c4b" Nov 03 13:46:04.441432 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:04.441348 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c03eda48-e87b-4847-a9be-da542cba2654-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c03eda48-e87b-4847-a9be-da542cba2654-maas-api-openshift-ca\") pod \"c03eda48-e87b-4847-a9be-da542cba2654\" (UID: \"c03eda48-e87b-4847-a9be-da542cba2654\") " Nov 03 13:46:04.441432 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:04.441392 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c03eda48-e87b-4847-a9be-da542cba2654-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c03eda48-e87b-4847-a9be-da542cba2654-maas-api-xks-ca\") pod \"c03eda48-e87b-4847-a9be-da542cba2654\" (UID: \"c03eda48-e87b-4847-a9be-da542cba2654\") " Nov 03 13:46:04.441716 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:04.441550 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c03eda48-e87b-4847-a9be-da542cba2654-kube-api-access-4dgj9\" (UniqueName: \"kubernetes.io/projected/c03eda48-e87b-4847-a9be-da542cba2654-kube-api-access-4dgj9\") pod \"c03eda48-e87b-4847-a9be-da542cba2654\" (UID: \"c03eda48-e87b-4847-a9be-da542cba2654\") " Nov 03 13:46:04.441806 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:04.441781 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c03eda48-e87b-4847-a9be-da542cba2654-maas-api-openshift-ca" pod "c03eda48-e87b-4847-a9be-da542cba2654" (UID: "c03eda48-e87b-4847-a9be-da542cba2654"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:46:04.442072 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:04.442054 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c03eda48-e87b-4847-a9be-da542cba2654-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:04.443756 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:04.443726 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c03eda48-e87b-4847-a9be-da542cba2654-kube-api-access-4dgj9" pod "c03eda48-e87b-4847-a9be-da542cba2654" (UID: "c03eda48-e87b-4847-a9be-da542cba2654"). InnerVolumeSpecName "kube-api-access-4dgj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:46:04.444038 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:04.444003 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c03eda48-e87b-4847-a9be-da542cba2654-maas-api-xks-ca" pod "c03eda48-e87b-4847-a9be-da542cba2654" (UID: "c03eda48-e87b-4847-a9be-da542cba2654"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:46:04.543597 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:04.543552 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c03eda48-e87b-4847-a9be-da542cba2654-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:04.543597 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:04.543594 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dgj9\" (UniqueName: \"kubernetes.io/projected/c03eda48-e87b-4847-a9be-da542cba2654-kube-api-access-4dgj9\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:05.162281 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:05.162251 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d9e037-96c4b" Nov 03 13:46:05.162462 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:05.162285 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d9e037-96c4b" event={"ID":"c03eda48-e87b-4847-a9be-da542cba2654","Type":"ContainerDied","Data":"9d81952db54bc57fa853933f35e78ac23932a9eaa471c258bbbe402dac979684"} Nov 03 13:46:05.162462 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:05.162332 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d81952db54bc57fa853933f35e78ac23932a9eaa471c258bbbe402dac979684" Nov 03 13:46:07.233827 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:46:07.233786 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:46:15.022744 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:15.022704 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d797192e-2jjgm"] Nov 03 13:46:15.023350 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:15.023334 5979 cpu_manager.go:414] "containerMap: removing container" podUID="0f7b92f8-adb4-4b0f-b59d-2949c7bad86e" containerName="revoke-keys" Nov 03 13:46:15.023393 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:15.023353 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f7b92f8-adb4-4b0f-b59d-2949c7bad86e" containerName="revoke-keys" Nov 03 13:46:15.023434 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:15.023395 5979 cpu_manager.go:414] "containerMap: removing container" podUID="c03eda48-e87b-4847-a9be-da542cba2654" containerName="revoke-keys" Nov 03 13:46:15.023434 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:15.023400 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c03eda48-e87b-4847-a9be-da542cba2654" containerName="revoke-keys" Nov 03 13:46:15.023558 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:15.023518 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f7b92f8-adb4-4b0f-b59d-2949c7bad86e" containerName="revoke-keys" Nov 03 13:46:15.023558 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:15.023534 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="c03eda48-e87b-4847-a9be-da542cba2654" containerName="revoke-keys" Nov 03 13:46:15.027034 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:15.027014 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d797192e-2jjgm" Nov 03 13:46:15.074095 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:15.074057 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmzgj\" (UniqueName: \"kubernetes.io/projected/8a6c7cba-c26f-41cc-af4f-e9e519eb9865-kube-api-access-fmzgj\") pod \"maas-api-revoke-sub-d797192e-2jjgm\" (UID: \"8a6c7cba-c26f-41cc-af4f-e9e519eb9865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d797192e-2jjgm" Nov 03 13:46:15.074254 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:15.074192 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a6c7cba-c26f-41cc-af4f-e9e519eb9865-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d797192e-2jjgm\" (UID: \"8a6c7cba-c26f-41cc-af4f-e9e519eb9865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d797192e-2jjgm" Nov 03 13:46:15.074254 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:15.074245 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a6c7cba-c26f-41cc-af4f-e9e519eb9865-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d797192e-2jjgm\" (UID: \"8a6c7cba-c26f-41cc-af4f-e9e519eb9865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d797192e-2jjgm" Nov 03 13:46:15.076256 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:15.076234 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:46:15.122756 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:15.122721 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:46:15.125699 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:15.125674 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d797192e-2jjgm"] Nov 03 13:46:15.175605 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:15.175575 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a6c7cba-c26f-41cc-af4f-e9e519eb9865-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d797192e-2jjgm\" (UID: \"8a6c7cba-c26f-41cc-af4f-e9e519eb9865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d797192e-2jjgm" Nov 03 13:46:15.175770 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:15.175643 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a6c7cba-c26f-41cc-af4f-e9e519eb9865-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d797192e-2jjgm\" (UID: \"8a6c7cba-c26f-41cc-af4f-e9e519eb9865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d797192e-2jjgm" Nov 03 13:46:15.175859 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:15.175830 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmzgj\" (UniqueName: \"kubernetes.io/projected/8a6c7cba-c26f-41cc-af4f-e9e519eb9865-kube-api-access-fmzgj\") pod \"maas-api-revoke-sub-d797192e-2jjgm\" (UID: \"8a6c7cba-c26f-41cc-af4f-e9e519eb9865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d797192e-2jjgm" Nov 03 13:46:15.176507 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:15.176457 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a6c7cba-c26f-41cc-af4f-e9e519eb9865-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d797192e-2jjgm\" (UID: \"8a6c7cba-c26f-41cc-af4f-e9e519eb9865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d797192e-2jjgm" Nov 03 13:46:15.178027 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:15.178005 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a6c7cba-c26f-41cc-af4f-e9e519eb9865-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d797192e-2jjgm\" (UID: \"8a6c7cba-c26f-41cc-af4f-e9e519eb9865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d797192e-2jjgm" Nov 03 13:46:15.230030 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:15.230001 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmzgj\" (UniqueName: \"kubernetes.io/projected/8a6c7cba-c26f-41cc-af4f-e9e519eb9865-kube-api-access-fmzgj\") pod \"maas-api-revoke-sub-d797192e-2jjgm\" (UID: \"8a6c7cba-c26f-41cc-af4f-e9e519eb9865\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d797192e-2jjgm" Nov 03 13:46:15.341245 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:15.341159 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d797192e-2jjgm" Nov 03 13:46:15.729371 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:15.729324 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d797192e-2jjgm"] Nov 03 13:46:15.736085 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:46:15.736059 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a6c7cba_c26f_41cc_af4f_e9e519eb9865.slice/crio-a139f8f8ad07420582f6b15860d6f0f0e3a95b377946e8264e675b5adfaf9ae1 WatchSource:0}: Error finding container a139f8f8ad07420582f6b15860d6f0f0e3a95b377946e8264e675b5adfaf9ae1: Status 404 returned error can't find the container with id a139f8f8ad07420582f6b15860d6f0f0e3a95b377946e8264e675b5adfaf9ae1 Nov 03 13:46:16.254509 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:16.254451 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d797192e-2jjgm" event={"ID":"8a6c7cba-c26f-41cc-af4f-e9e519eb9865","Type":"ContainerStarted","Data":"34fa15be788878a8d267de7acb38e62fc962ae0f975b81a8ba2c826e9b0008f3"} Nov 03 13:46:16.254898 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:16.254531 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d797192e-2jjgm" event={"ID":"8a6c7cba-c26f-41cc-af4f-e9e519eb9865","Type":"ContainerStarted","Data":"a139f8f8ad07420582f6b15860d6f0f0e3a95b377946e8264e675b5adfaf9ae1"} Nov 03 13:46:16.363223 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:16.363118 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d797192e-2jjgm" podStartSLOduration=2.363105782 podStartE2EDuration="2.363105782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:46:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:46:16.360648312 +0000 UTC m=+2458.371336554" watchObservedRunningTime="2026-11-03 13:46:16.363105782 +0000 UTC m=+2458.373794023" Nov 03 13:46:17.262258 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:17.262223 5979 generic.go:356] "Generic (PLEG): container finished" podID="8a6c7cba-c26f-41cc-af4f-e9e519eb9865" containerID="34fa15be788878a8d267de7acb38e62fc962ae0f975b81a8ba2c826e9b0008f3" exitCode=0 Nov 03 13:46:17.262749 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:17.262298 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d797192e-2jjgm" event={"ID":"8a6c7cba-c26f-41cc-af4f-e9e519eb9865","Type":"ContainerDied","Data":"34fa15be788878a8d267de7acb38e62fc962ae0f975b81a8ba2c826e9b0008f3"} Nov 03 13:46:18.054538 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:18.054491 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a770b9-774d768db-9zksn"] Nov 03 13:46:18.054785 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:18.054734 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-a770b9-774d768db-9zksn" podUID="5a9531dd-b724-4801-ac0d-71d42b7b3868" containerName="payload-processing" containerID="cri-o://037612024f52fde38513e0b0b3e6ac6721c15cd3043c91488e9411e2ad4e9cd9" gracePeriod=30 Nov 03 13:46:18.116778 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:18.116736 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a770b9-695d4dd84c-lzblq"] Nov 03 13:46:18.116958 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:18.116933 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a770b9-695d4dd84c-lzblq" podUID="9af873cf-0227-40cc-ba24-6df2b62f4676" containerName="payload-pre-processing" containerID="cri-o://a17fe2964ccdf910bc4838e42a185d6be9bdad884146e220954fe40a11f22b0d" gracePeriod=30 Nov 03 13:46:18.272680 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:18.272615 5979 generic.go:356] "Generic (PLEG): container finished" podID="9af873cf-0227-40cc-ba24-6df2b62f4676" containerID="a17fe2964ccdf910bc4838e42a185d6be9bdad884146e220954fe40a11f22b0d" exitCode=0 Nov 03 13:46:18.273249 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:18.272806 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a770b9-695d4dd84c-lzblq" event={"ID":"9af873cf-0227-40cc-ba24-6df2b62f4676","Type":"ContainerDied","Data":"a17fe2964ccdf910bc4838e42a185d6be9bdad884146e220954fe40a11f22b0d"} Nov 03 13:46:18.274616 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:18.274591 5979 generic.go:356] "Generic (PLEG): container finished" podID="5a9531dd-b724-4801-ac0d-71d42b7b3868" containerID="037612024f52fde38513e0b0b3e6ac6721c15cd3043c91488e9411e2ad4e9cd9" exitCode=0 Nov 03 13:46:18.274756 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:18.274659 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a770b9-774d768db-9zksn" event={"ID":"5a9531dd-b724-4801-ac0d-71d42b7b3868","Type":"ContainerDied","Data":"037612024f52fde38513e0b0b3e6ac6721c15cd3043c91488e9411e2ad4e9cd9"} Nov 03 13:46:18.403589 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:18.403561 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a770b9-774d768db-9zksn" Nov 03 13:46:18.437859 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:18.437834 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a770b9-695d4dd84c-lzblq" Nov 03 13:46:18.441404 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:18.441374 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d797192e-2jjgm" Nov 03 13:46:18.538139 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:18.538106 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a9531dd-b724-4801-ac0d-71d42b7b3868-kube-api-access-t2gr4\" (UniqueName: \"kubernetes.io/projected/5a9531dd-b724-4801-ac0d-71d42b7b3868-kube-api-access-t2gr4\") pod \"5a9531dd-b724-4801-ac0d-71d42b7b3868\" (UID: \"5a9531dd-b724-4801-ac0d-71d42b7b3868\") " Nov 03 13:46:18.538323 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:18.538170 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a9531dd-b724-4801-ac0d-71d42b7b3868-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a9531dd-b724-4801-ac0d-71d42b7b3868-plugins-config-volume\") pod \"5a9531dd-b724-4801-ac0d-71d42b7b3868\" (UID: \"5a9531dd-b724-4801-ac0d-71d42b7b3868\") " Nov 03 13:46:18.538323 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:18.538236 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9af873cf-0227-40cc-ba24-6df2b62f4676-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9af873cf-0227-40cc-ba24-6df2b62f4676-plugins-config-volume\") pod \"9af873cf-0227-40cc-ba24-6df2b62f4676\" (UID: \"9af873cf-0227-40cc-ba24-6df2b62f4676\") " Nov 03 13:46:18.538323 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:18.538270 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a6c7cba-c26f-41cc-af4f-e9e519eb9865-kube-api-access-fmzgj\" (UniqueName: \"kubernetes.io/projected/8a6c7cba-c26f-41cc-af4f-e9e519eb9865-kube-api-access-fmzgj\") pod \"8a6c7cba-c26f-41cc-af4f-e9e519eb9865\" (UID: \"8a6c7cba-c26f-41cc-af4f-e9e519eb9865\") " Nov 03 13:46:18.538535 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:18.538334 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a6c7cba-c26f-41cc-af4f-e9e519eb9865-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a6c7cba-c26f-41cc-af4f-e9e519eb9865-maas-api-xks-ca\") pod \"8a6c7cba-c26f-41cc-af4f-e9e519eb9865\" (UID: \"8a6c7cba-c26f-41cc-af4f-e9e519eb9865\") " Nov 03 13:46:18.538535 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:18.538399 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9af873cf-0227-40cc-ba24-6df2b62f4676-kube-api-access-b6vc6\" (UniqueName: \"kubernetes.io/projected/9af873cf-0227-40cc-ba24-6df2b62f4676-kube-api-access-b6vc6\") pod \"9af873cf-0227-40cc-ba24-6df2b62f4676\" (UID: \"9af873cf-0227-40cc-ba24-6df2b62f4676\") " Nov 03 13:46:18.538655 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:18.538626 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a6c7cba-c26f-41cc-af4f-e9e519eb9865-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a6c7cba-c26f-41cc-af4f-e9e519eb9865-maas-api-openshift-ca\") pod \"8a6c7cba-c26f-41cc-af4f-e9e519eb9865\" (UID: \"8a6c7cba-c26f-41cc-af4f-e9e519eb9865\") " Nov 03 13:46:18.539512 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:18.539210 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9af873cf-0227-40cc-ba24-6df2b62f4676-plugins-config-volume" pod "9af873cf-0227-40cc-ba24-6df2b62f4676" (UID: "9af873cf-0227-40cc-ba24-6df2b62f4676"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:46:18.539512 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:18.539432 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a6c7cba-c26f-41cc-af4f-e9e519eb9865-maas-api-openshift-ca" pod "8a6c7cba-c26f-41cc-af4f-e9e519eb9865" (UID: "8a6c7cba-c26f-41cc-af4f-e9e519eb9865"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:46:18.540038 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:18.540013 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a9531dd-b724-4801-ac0d-71d42b7b3868-plugins-config-volume" pod "5a9531dd-b724-4801-ac0d-71d42b7b3868" (UID: "5a9531dd-b724-4801-ac0d-71d42b7b3868"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:46:18.540956 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:18.540932 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a6c7cba-c26f-41cc-af4f-e9e519eb9865-kube-api-access-fmzgj" pod "8a6c7cba-c26f-41cc-af4f-e9e519eb9865" (UID: "8a6c7cba-c26f-41cc-af4f-e9e519eb9865"). InnerVolumeSpecName "kube-api-access-fmzgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:46:18.541100 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:18.540973 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a6c7cba-c26f-41cc-af4f-e9e519eb9865-maas-api-xks-ca" pod "8a6c7cba-c26f-41cc-af4f-e9e519eb9865" (UID: "8a6c7cba-c26f-41cc-af4f-e9e519eb9865"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:46:18.541180 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:18.541099 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a9531dd-b724-4801-ac0d-71d42b7b3868-kube-api-access-t2gr4" pod "5a9531dd-b724-4801-ac0d-71d42b7b3868" (UID: "5a9531dd-b724-4801-ac0d-71d42b7b3868"). InnerVolumeSpecName "kube-api-access-t2gr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:46:18.542431 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:18.542409 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9af873cf-0227-40cc-ba24-6df2b62f4676-kube-api-access-b6vc6" pod "9af873cf-0227-40cc-ba24-6df2b62f4676" (UID: "9af873cf-0227-40cc-ba24-6df2b62f4676"). InnerVolumeSpecName "kube-api-access-b6vc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:46:18.641417 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:18.641337 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a6c7cba-c26f-41cc-af4f-e9e519eb9865-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:18.641417 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:18.641371 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6vc6\" (UniqueName: \"kubernetes.io/projected/9af873cf-0227-40cc-ba24-6df2b62f4676-kube-api-access-b6vc6\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:18.641417 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:18.641381 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a6c7cba-c26f-41cc-af4f-e9e519eb9865-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:18.641417 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:18.641394 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2gr4\" (UniqueName: \"kubernetes.io/projected/5a9531dd-b724-4801-ac0d-71d42b7b3868-kube-api-access-t2gr4\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:18.641417 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:18.641407 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a9531dd-b724-4801-ac0d-71d42b7b3868-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:18.641417 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:18.641418 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9af873cf-0227-40cc-ba24-6df2b62f4676-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:18.641417 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:18.641427 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmzgj\" (UniqueName: \"kubernetes.io/projected/8a6c7cba-c26f-41cc-af4f-e9e519eb9865-kube-api-access-fmzgj\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:19.283511 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:19.283457 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a770b9-695d4dd84c-lzblq" Nov 03 13:46:19.283511 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:19.283496 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a770b9-695d4dd84c-lzblq" event={"ID":"9af873cf-0227-40cc-ba24-6df2b62f4676","Type":"ContainerDied","Data":"dfc44eeb2543024552f4ea9ecdf82184c5c19854fd137af9a550fc9722043b75"} Nov 03 13:46:19.284025 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:19.283560 5979 scope.go:122] "RemoveContainer" containerID="a17fe2964ccdf910bc4838e42a185d6be9bdad884146e220954fe40a11f22b0d" Nov 03 13:46:19.285088 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:19.285060 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a770b9-774d768db-9zksn" Nov 03 13:46:19.285194 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:19.285082 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a770b9-774d768db-9zksn" event={"ID":"5a9531dd-b724-4801-ac0d-71d42b7b3868","Type":"ContainerDied","Data":"0ea4d941755b689daeeb114604e744965eb6c30d7334cd43ecb5bf736804b339"} Nov 03 13:46:19.286660 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:19.286638 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d797192e-2jjgm" Nov 03 13:46:19.286787 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:19.286669 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d797192e-2jjgm" event={"ID":"8a6c7cba-c26f-41cc-af4f-e9e519eb9865","Type":"ContainerDied","Data":"a139f8f8ad07420582f6b15860d6f0f0e3a95b377946e8264e675b5adfaf9ae1"} Nov 03 13:46:19.286787 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:19.286716 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a139f8f8ad07420582f6b15860d6f0f0e3a95b377946e8264e675b5adfaf9ae1" Nov 03 13:46:19.301757 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:19.301731 5979 scope.go:122] "RemoveContainer" containerID="037612024f52fde38513e0b0b3e6ac6721c15cd3043c91488e9411e2ad4e9cd9" Nov 03 13:46:19.486680 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:19.486644 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a770b9-695d4dd84c-lzblq"] Nov 03 13:46:19.492291 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:19.492265 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a770b9-695d4dd84c-lzblq"] Nov 03 13:46:19.599619 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:19.599584 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a770b9-774d768db-9zksn"] Nov 03 13:46:19.604768 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:19.604740 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a770b9-774d768db-9zksn"] Nov 03 13:46:19.899203 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:19.899170 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a770b9-6vls9"] Nov 03 13:46:19.902613 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:19.902584 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a770b9-6vls9"] Nov 03 13:46:20.116084 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:20.116038 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f7b92f8-adb4-4b0f-b59d-2949c7bad86e" path="/var/lib/kubelet/pods/0f7b92f8-adb4-4b0f-b59d-2949c7bad86e/volumes" Nov 03 13:46:20.116644 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:20.116617 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a9531dd-b724-4801-ac0d-71d42b7b3868" path="/var/lib/kubelet/pods/5a9531dd-b724-4801-ac0d-71d42b7b3868/volumes" Nov 03 13:46:20.117070 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:20.117055 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9af873cf-0227-40cc-ba24-6df2b62f4676" path="/var/lib/kubelet/pods/9af873cf-0227-40cc-ba24-6df2b62f4676/volumes" Nov 03 13:46:25.815988 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:25.815945 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:46:32.753813 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:32.753776 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03057985-ccfbz"] Nov 03 13:46:32.754350 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:32.754335 5979 cpu_manager.go:414] "containerMap: removing container" podUID="8a6c7cba-c26f-41cc-af4f-e9e519eb9865" containerName="revoke-keys" Nov 03 13:46:32.754350 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:32.754351 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a6c7cba-c26f-41cc-af4f-e9e519eb9865" containerName="revoke-keys" Nov 03 13:46:32.754444 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:32.754386 5979 cpu_manager.go:414] "containerMap: removing container" podUID="9af873cf-0227-40cc-ba24-6df2b62f4676" containerName="payload-pre-processing" Nov 03 13:46:32.754444 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:32.754392 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9af873cf-0227-40cc-ba24-6df2b62f4676" containerName="payload-pre-processing" Nov 03 13:46:32.754444 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:32.754401 5979 cpu_manager.go:414] "containerMap: removing container" podUID="5a9531dd-b724-4801-ac0d-71d42b7b3868" containerName="payload-processing" Nov 03 13:46:32.754444 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:32.754406 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a9531dd-b724-4801-ac0d-71d42b7b3868" containerName="payload-processing" Nov 03 13:46:32.754612 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:32.754512 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="9af873cf-0227-40cc-ba24-6df2b62f4676" containerName="payload-pre-processing" Nov 03 13:46:32.754612 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:32.754522 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a9531dd-b724-4801-ac0d-71d42b7b3868" containerName="payload-processing" Nov 03 13:46:32.754612 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:32.754533 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a6c7cba-c26f-41cc-af4f-e9e519eb9865" containerName="revoke-keys" Nov 03 13:46:32.759511 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:32.759491 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03057985-ccfbz" Nov 03 13:46:32.808567 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:32.808530 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:46:32.812518 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:32.812486 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03057985-ccfbz"] Nov 03 13:46:32.850319 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:32.850280 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00e7d6b5-17dc-40e1-b6e0-ea2cdb359c79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03057985-ccfbz\" (UID: \"00e7d6b5-17dc-40e1-b6e0-ea2cdb359c79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03057985-ccfbz" Nov 03 13:46:32.850499 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:32.850404 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00e7d6b5-17dc-40e1-b6e0-ea2cdb359c79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03057985-ccfbz\" (UID: \"00e7d6b5-17dc-40e1-b6e0-ea2cdb359c79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03057985-ccfbz" Nov 03 13:46:32.850627 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:32.850605 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g7c5\" (UniqueName: \"kubernetes.io/projected/00e7d6b5-17dc-40e1-b6e0-ea2cdb359c79-kube-api-access-9g7c5\") pod \"maas-api-revoke-sub-03057985-ccfbz\" (UID: \"00e7d6b5-17dc-40e1-b6e0-ea2cdb359c79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03057985-ccfbz" Nov 03 13:46:32.855063 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:32.855040 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:46:32.952267 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:32.952227 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g7c5\" (UniqueName: \"kubernetes.io/projected/00e7d6b5-17dc-40e1-b6e0-ea2cdb359c79-kube-api-access-9g7c5\") pod \"maas-api-revoke-sub-03057985-ccfbz\" (UID: \"00e7d6b5-17dc-40e1-b6e0-ea2cdb359c79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03057985-ccfbz" Nov 03 13:46:32.952511 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:32.952281 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00e7d6b5-17dc-40e1-b6e0-ea2cdb359c79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03057985-ccfbz\" (UID: \"00e7d6b5-17dc-40e1-b6e0-ea2cdb359c79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03057985-ccfbz" Nov 03 13:46:32.952511 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:32.952361 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00e7d6b5-17dc-40e1-b6e0-ea2cdb359c79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03057985-ccfbz\" (UID: \"00e7d6b5-17dc-40e1-b6e0-ea2cdb359c79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03057985-ccfbz" Nov 03 13:46:32.952973 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:32.952940 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00e7d6b5-17dc-40e1-b6e0-ea2cdb359c79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03057985-ccfbz\" (UID: \"00e7d6b5-17dc-40e1-b6e0-ea2cdb359c79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03057985-ccfbz" Nov 03 13:46:32.954725 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:32.954703 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00e7d6b5-17dc-40e1-b6e0-ea2cdb359c79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03057985-ccfbz\" (UID: \"00e7d6b5-17dc-40e1-b6e0-ea2cdb359c79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03057985-ccfbz" Nov 03 13:46:33.009266 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:33.009193 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g7c5\" (UniqueName: \"kubernetes.io/projected/00e7d6b5-17dc-40e1-b6e0-ea2cdb359c79-kube-api-access-9g7c5\") pod \"maas-api-revoke-sub-03057985-ccfbz\" (UID: \"00e7d6b5-17dc-40e1-b6e0-ea2cdb359c79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03057985-ccfbz" Nov 03 13:46:33.072519 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:33.072456 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03057985-ccfbz" Nov 03 13:46:33.448336 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:33.448296 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03057985-ccfbz"] Nov 03 13:46:33.455456 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:46:33.455427 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00e7d6b5_17dc_40e1_b6e0_ea2cdb359c79.slice/crio-6ba0dd08df7ae8be86bcc0fae847d8ab129b4e545186af5efd221bb15c63b788 WatchSource:0}: Error finding container 6ba0dd08df7ae8be86bcc0fae847d8ab129b4e545186af5efd221bb15c63b788: Status 404 returned error can't find the container with id 6ba0dd08df7ae8be86bcc0fae847d8ab129b4e545186af5efd221bb15c63b788 Nov 03 13:46:34.383590 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:34.383551 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03057985-ccfbz" event={"ID":"00e7d6b5-17dc-40e1-b6e0-ea2cdb359c79","Type":"ContainerStarted","Data":"1f883628bc15b1215fad464d415fd5be2996c1dd7548e09417a424026d9aa331"} Nov 03 13:46:34.384105 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:34.383604 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03057985-ccfbz" event={"ID":"00e7d6b5-17dc-40e1-b6e0-ea2cdb359c79","Type":"ContainerStarted","Data":"6ba0dd08df7ae8be86bcc0fae847d8ab129b4e545186af5efd221bb15c63b788"} Nov 03 13:46:34.491376 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:34.491320 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03057985-ccfbz" podStartSLOduration=2.491305648 podStartE2EDuration="2.491305648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:46:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:46:34.489343407 +0000 UTC m=+2476.500031649" watchObservedRunningTime="2026-11-03 13:46:34.491305648 +0000 UTC m=+2476.501993887" Nov 03 13:46:35.391339 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:35.391300 5979 generic.go:356] "Generic (PLEG): container finished" podID="00e7d6b5-17dc-40e1-b6e0-ea2cdb359c79" containerID="1f883628bc15b1215fad464d415fd5be2996c1dd7548e09417a424026d9aa331" exitCode=0 Nov 03 13:46:35.391771 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:35.391389 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03057985-ccfbz" event={"ID":"00e7d6b5-17dc-40e1-b6e0-ea2cdb359c79","Type":"ContainerDied","Data":"1f883628bc15b1215fad464d415fd5be2996c1dd7548e09417a424026d9aa331"} Nov 03 13:46:35.568261 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:35.568232 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-32bec3-b5b998c44-7qx4z"] Nov 03 13:46:35.568501 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:35.568463 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-32bec3-b5b998c44-7qx4z" podUID="a47855e5-140b-478a-bd1e-33c43c14482e" containerName="maas-api" containerID="cri-o://193f2c4a7a239d17851d9b637744fb9040933a808938ff4ae37cb2be4f363f5e" gracePeriod=30 Nov 03 13:46:35.867377 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:35.867348 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-32bec3-b5b998c44-7qx4z" Nov 03 13:46:35.911078 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:35.911045 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a47855e5-140b-478a-bd1e-33c43c14482e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a47855e5-140b-478a-bd1e-33c43c14482e-maas-api-tls\") pod \"a47855e5-140b-478a-bd1e-33c43c14482e\" (UID: \"a47855e5-140b-478a-bd1e-33c43c14482e\") " Nov 03 13:46:35.911286 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:35.911209 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a47855e5-140b-478a-bd1e-33c43c14482e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a47855e5-140b-478a-bd1e-33c43c14482e-openshift-service-ca\") pod \"a47855e5-140b-478a-bd1e-33c43c14482e\" (UID: \"a47855e5-140b-478a-bd1e-33c43c14482e\") " Nov 03 13:46:35.911361 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:35.911284 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a47855e5-140b-478a-bd1e-33c43c14482e-kube-api-access-bddwx\" (UniqueName: \"kubernetes.io/projected/a47855e5-140b-478a-bd1e-33c43c14482e-kube-api-access-bddwx\") pod \"a47855e5-140b-478a-bd1e-33c43c14482e\" (UID: \"a47855e5-140b-478a-bd1e-33c43c14482e\") " Nov 03 13:46:35.911422 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:35.911386 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a47855e5-140b-478a-bd1e-33c43c14482e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a47855e5-140b-478a-bd1e-33c43c14482e-metrics-certs\") pod \"a47855e5-140b-478a-bd1e-33c43c14482e\" (UID: \"a47855e5-140b-478a-bd1e-33c43c14482e\") " Nov 03 13:46:35.911627 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:35.911599 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a47855e5-140b-478a-bd1e-33c43c14482e-openshift-service-ca" pod "a47855e5-140b-478a-bd1e-33c43c14482e" (UID: "a47855e5-140b-478a-bd1e-33c43c14482e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:46:35.911931 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:35.911909 5979 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a47855e5-140b-478a-bd1e-33c43c14482e-openshift-service-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:35.913288 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:35.913252 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a47855e5-140b-478a-bd1e-33c43c14482e-maas-api-tls" pod "a47855e5-140b-478a-bd1e-33c43c14482e" (UID: "a47855e5-140b-478a-bd1e-33c43c14482e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:46:35.913398 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:35.913379 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a47855e5-140b-478a-bd1e-33c43c14482e-kube-api-access-bddwx" pod "a47855e5-140b-478a-bd1e-33c43c14482e" (UID: "a47855e5-140b-478a-bd1e-33c43c14482e"). InnerVolumeSpecName "kube-api-access-bddwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:46:35.913444 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:35.913395 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a47855e5-140b-478a-bd1e-33c43c14482e-metrics-certs" pod "a47855e5-140b-478a-bd1e-33c43c14482e" (UID: "a47855e5-140b-478a-bd1e-33c43c14482e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:46:36.013200 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:36.013122 5979 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a47855e5-140b-478a-bd1e-33c43c14482e-metrics-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:36.013200 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:36.013149 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a47855e5-140b-478a-bd1e-33c43c14482e-maas-api-tls\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:36.013200 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:36.013159 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bddwx\" (UniqueName: \"kubernetes.io/projected/a47855e5-140b-478a-bd1e-33c43c14482e-kube-api-access-bddwx\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:36.042708 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:36.042673 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-32bec3-678877f9d4-qjwmv"] Nov 03 13:46:36.042921 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:36.042878 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-32bec3-678877f9d4-qjwmv" podUID="0f6351e2-84a8-414e-a0d8-dd394abec66a" containerName="payload-processing" containerID="cri-o://5c82f6ddfd174b221aa2b3176b14bbead8408213d0ec9f60ba10b33849cfa2fd" gracePeriod=30 Nov 03 13:46:36.100534 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:36.100502 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-32bec3-78c6df84dc-wr8qq"] Nov 03 13:46:36.100723 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:36.100686 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-32bec3-78c6df84dc-wr8qq" podUID="2b23dfaf-6b15-4c7e-8515-019f404475de" containerName="payload-pre-processing" containerID="cri-o://c9b7352f3a6f1a261ce0fed34cfcb3e619228a5c437fa76297aca47010704b5b" gracePeriod=30 Nov 03 13:46:36.367619 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:36.367591 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-32bec3-78c6df84dc-wr8qq" Nov 03 13:46:36.388077 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:36.388048 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-32bec3-678877f9d4-qjwmv" Nov 03 13:46:36.400345 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:36.400312 5979 generic.go:356] "Generic (PLEG): container finished" podID="2b23dfaf-6b15-4c7e-8515-019f404475de" containerID="c9b7352f3a6f1a261ce0fed34cfcb3e619228a5c437fa76297aca47010704b5b" exitCode=0 Nov 03 13:46:36.400789 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:36.400355 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-32bec3-78c6df84dc-wr8qq" Nov 03 13:46:36.400789 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:36.400398 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-32bec3-78c6df84dc-wr8qq" event={"ID":"2b23dfaf-6b15-4c7e-8515-019f404475de","Type":"ContainerDied","Data":"c9b7352f3a6f1a261ce0fed34cfcb3e619228a5c437fa76297aca47010704b5b"} Nov 03 13:46:36.400789 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:36.400447 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-32bec3-78c6df84dc-wr8qq" event={"ID":"2b23dfaf-6b15-4c7e-8515-019f404475de","Type":"ContainerDied","Data":"76ea25e7773097e46558e58dd4553bc263adc56b5bab54b259183cda55e5a4f4"} Nov 03 13:46:36.400789 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:36.400498 5979 scope.go:122] "RemoveContainer" containerID="c9b7352f3a6f1a261ce0fed34cfcb3e619228a5c437fa76297aca47010704b5b" Nov 03 13:46:36.401897 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:36.401874 5979 generic.go:356] "Generic (PLEG): container finished" podID="a47855e5-140b-478a-bd1e-33c43c14482e" containerID="193f2c4a7a239d17851d9b637744fb9040933a808938ff4ae37cb2be4f363f5e" exitCode=0 Nov 03 13:46:36.401999 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:36.401901 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-32bec3-b5b998c44-7qx4z" event={"ID":"a47855e5-140b-478a-bd1e-33c43c14482e","Type":"ContainerDied","Data":"193f2c4a7a239d17851d9b637744fb9040933a808938ff4ae37cb2be4f363f5e"} Nov 03 13:46:36.401999 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:36.401942 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-32bec3-b5b998c44-7qx4z" event={"ID":"a47855e5-140b-478a-bd1e-33c43c14482e","Type":"ContainerDied","Data":"b7ec1d39bfc4fbd04ae9060320a5014a22a04dfcb9b119a78b0d475a8e80c0c3"} Nov 03 13:46:36.402138 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:36.402117 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-32bec3-b5b998c44-7qx4z" Nov 03 13:46:36.403429 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:36.403407 5979 generic.go:356] "Generic (PLEG): container finished" podID="0f6351e2-84a8-414e-a0d8-dd394abec66a" containerID="5c82f6ddfd174b221aa2b3176b14bbead8408213d0ec9f60ba10b33849cfa2fd" exitCode=0 Nov 03 13:46:36.403560 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:36.403442 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-32bec3-678877f9d4-qjwmv" Nov 03 13:46:36.403560 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:36.403497 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-32bec3-678877f9d4-qjwmv" event={"ID":"0f6351e2-84a8-414e-a0d8-dd394abec66a","Type":"ContainerDied","Data":"5c82f6ddfd174b221aa2b3176b14bbead8408213d0ec9f60ba10b33849cfa2fd"} Nov 03 13:46:36.403560 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:36.403548 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-32bec3-678877f9d4-qjwmv" event={"ID":"0f6351e2-84a8-414e-a0d8-dd394abec66a","Type":"ContainerDied","Data":"d41fae08d372065827e5db7023954e8a7537b61a65b7659f36ff5cfa4b8afede"} Nov 03 13:46:36.410943 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:36.410916 5979 scope.go:122] "RemoveContainer" containerID="c9b7352f3a6f1a261ce0fed34cfcb3e619228a5c437fa76297aca47010704b5b" Nov 03 13:46:36.411244 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:46:36.411224 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9b7352f3a6f1a261ce0fed34cfcb3e619228a5c437fa76297aca47010704b5b\": container with ID starting with c9b7352f3a6f1a261ce0fed34cfcb3e619228a5c437fa76297aca47010704b5b not found: ID does not exist" containerID="c9b7352f3a6f1a261ce0fed34cfcb3e619228a5c437fa76297aca47010704b5b" Nov 03 13:46:36.411299 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:36.411253 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9b7352f3a6f1a261ce0fed34cfcb3e619228a5c437fa76297aca47010704b5b"} err="failed to get container status \"c9b7352f3a6f1a261ce0fed34cfcb3e619228a5c437fa76297aca47010704b5b\": rpc error: code = NotFound desc = could not find container \"c9b7352f3a6f1a261ce0fed34cfcb3e619228a5c437fa76297aca47010704b5b\": container with ID starting with c9b7352f3a6f1a261ce0fed34cfcb3e619228a5c437fa76297aca47010704b5b not found: ID does not exist" Nov 03 13:46:36.411299 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:36.411270 5979 scope.go:122] "RemoveContainer" containerID="193f2c4a7a239d17851d9b637744fb9040933a808938ff4ae37cb2be4f363f5e" Nov 03 13:46:36.420188 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:36.420158 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f6351e2-84a8-414e-a0d8-dd394abec66a-kube-api-access-4xjxx\" (UniqueName: \"kubernetes.io/projected/0f6351e2-84a8-414e-a0d8-dd394abec66a-kube-api-access-4xjxx\") pod \"0f6351e2-84a8-414e-a0d8-dd394abec66a\" (UID: \"0f6351e2-84a8-414e-a0d8-dd394abec66a\") " Nov 03 13:46:36.421342 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:36.420211 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b23dfaf-6b15-4c7e-8515-019f404475de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b23dfaf-6b15-4c7e-8515-019f404475de-plugins-config-volume\") pod \"2b23dfaf-6b15-4c7e-8515-019f404475de\" (UID: \"2b23dfaf-6b15-4c7e-8515-019f404475de\") " Nov 03 13:46:36.421342 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:36.420268 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b23dfaf-6b15-4c7e-8515-019f404475de-kube-api-access-tzj8l\" (UniqueName: \"kubernetes.io/projected/2b23dfaf-6b15-4c7e-8515-019f404475de-kube-api-access-tzj8l\") pod \"2b23dfaf-6b15-4c7e-8515-019f404475de\" (UID: \"2b23dfaf-6b15-4c7e-8515-019f404475de\") " Nov 03 13:46:36.421342 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:36.420324 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f6351e2-84a8-414e-a0d8-dd394abec66a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f6351e2-84a8-414e-a0d8-dd394abec66a-plugins-config-volume\") pod \"0f6351e2-84a8-414e-a0d8-dd394abec66a\" (UID: \"0f6351e2-84a8-414e-a0d8-dd394abec66a\") " Nov 03 13:46:36.421342 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:36.420894 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b23dfaf-6b15-4c7e-8515-019f404475de-plugins-config-volume" pod "2b23dfaf-6b15-4c7e-8515-019f404475de" (UID: "2b23dfaf-6b15-4c7e-8515-019f404475de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:46:36.421342 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:36.421191 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f6351e2-84a8-414e-a0d8-dd394abec66a-plugins-config-volume" pod "0f6351e2-84a8-414e-a0d8-dd394abec66a" (UID: "0f6351e2-84a8-414e-a0d8-dd394abec66a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:46:36.423129 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:36.423070 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f6351e2-84a8-414e-a0d8-dd394abec66a-kube-api-access-4xjxx" pod "0f6351e2-84a8-414e-a0d8-dd394abec66a" (UID: "0f6351e2-84a8-414e-a0d8-dd394abec66a"). InnerVolumeSpecName "kube-api-access-4xjxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:46:36.423223 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:36.423206 5979 scope.go:122] "RemoveContainer" containerID="193f2c4a7a239d17851d9b637744fb9040933a808938ff4ae37cb2be4f363f5e" Nov 03 13:46:36.423757 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:46:36.423727 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"193f2c4a7a239d17851d9b637744fb9040933a808938ff4ae37cb2be4f363f5e\": container with ID starting with 193f2c4a7a239d17851d9b637744fb9040933a808938ff4ae37cb2be4f363f5e not found: ID does not exist" containerID="193f2c4a7a239d17851d9b637744fb9040933a808938ff4ae37cb2be4f363f5e" Nov 03 13:46:36.423853 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:36.423770 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"193f2c4a7a239d17851d9b637744fb9040933a808938ff4ae37cb2be4f363f5e"} err="failed to get container status \"193f2c4a7a239d17851d9b637744fb9040933a808938ff4ae37cb2be4f363f5e\": rpc error: code = NotFound desc = could not find container \"193f2c4a7a239d17851d9b637744fb9040933a808938ff4ae37cb2be4f363f5e\": container with ID starting with 193f2c4a7a239d17851d9b637744fb9040933a808938ff4ae37cb2be4f363f5e not found: ID does not exist" Nov 03 13:46:36.423853 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:36.423807 5979 scope.go:122] "RemoveContainer" containerID="5c82f6ddfd174b221aa2b3176b14bbead8408213d0ec9f60ba10b33849cfa2fd" Nov 03 13:46:36.424602 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:36.424581 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b23dfaf-6b15-4c7e-8515-019f404475de-kube-api-access-tzj8l" pod "2b23dfaf-6b15-4c7e-8515-019f404475de" (UID: "2b23dfaf-6b15-4c7e-8515-019f404475de"). InnerVolumeSpecName "kube-api-access-tzj8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:46:36.435163 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:36.435048 5979 scope.go:122] "RemoveContainer" containerID="5c82f6ddfd174b221aa2b3176b14bbead8408213d0ec9f60ba10b33849cfa2fd" Nov 03 13:46:36.435785 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:46:36.435709 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c82f6ddfd174b221aa2b3176b14bbead8408213d0ec9f60ba10b33849cfa2fd\": container with ID starting with 5c82f6ddfd174b221aa2b3176b14bbead8408213d0ec9f60ba10b33849cfa2fd not found: ID does not exist" containerID="5c82f6ddfd174b221aa2b3176b14bbead8408213d0ec9f60ba10b33849cfa2fd" Nov 03 13:46:36.435785 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:36.435743 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c82f6ddfd174b221aa2b3176b14bbead8408213d0ec9f60ba10b33849cfa2fd"} err="failed to get container status \"5c82f6ddfd174b221aa2b3176b14bbead8408213d0ec9f60ba10b33849cfa2fd\": rpc error: code = NotFound desc = could not find container \"5c82f6ddfd174b221aa2b3176b14bbead8408213d0ec9f60ba10b33849cfa2fd\": container with ID starting with 5c82f6ddfd174b221aa2b3176b14bbead8408213d0ec9f60ba10b33849cfa2fd not found: ID does not exist" Nov 03 13:46:36.522258 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:36.522222 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xjxx\" (UniqueName: \"kubernetes.io/projected/0f6351e2-84a8-414e-a0d8-dd394abec66a-kube-api-access-4xjxx\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:36.522258 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:36.522253 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b23dfaf-6b15-4c7e-8515-019f404475de-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:36.522258 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:36.522263 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzj8l\" (UniqueName: \"kubernetes.io/projected/2b23dfaf-6b15-4c7e-8515-019f404475de-kube-api-access-tzj8l\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:36.522626 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:36.522272 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f6351e2-84a8-414e-a0d8-dd394abec66a-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:36.538646 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:36.538608 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03057985-ccfbz" Nov 03 13:46:36.629095 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:36.629058 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00e7d6b5-17dc-40e1-b6e0-ea2cdb359c79-kube-api-access-9g7c5\" (UniqueName: \"kubernetes.io/projected/00e7d6b5-17dc-40e1-b6e0-ea2cdb359c79-kube-api-access-9g7c5\") pod \"00e7d6b5-17dc-40e1-b6e0-ea2cdb359c79\" (UID: \"00e7d6b5-17dc-40e1-b6e0-ea2cdb359c79\") " Nov 03 13:46:36.629863 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:36.629839 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00e7d6b5-17dc-40e1-b6e0-ea2cdb359c79-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00e7d6b5-17dc-40e1-b6e0-ea2cdb359c79-maas-api-openshift-ca\") pod \"00e7d6b5-17dc-40e1-b6e0-ea2cdb359c79\" (UID: \"00e7d6b5-17dc-40e1-b6e0-ea2cdb359c79\") " Nov 03 13:46:36.630514 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:36.630388 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00e7d6b5-17dc-40e1-b6e0-ea2cdb359c79-maas-api-openshift-ca" pod "00e7d6b5-17dc-40e1-b6e0-ea2cdb359c79" (UID: "00e7d6b5-17dc-40e1-b6e0-ea2cdb359c79"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:46:36.630836 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:36.630811 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00e7d6b5-17dc-40e1-b6e0-ea2cdb359c79-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00e7d6b5-17dc-40e1-b6e0-ea2cdb359c79-maas-api-xks-ca\") pod \"00e7d6b5-17dc-40e1-b6e0-ea2cdb359c79\" (UID: \"00e7d6b5-17dc-40e1-b6e0-ea2cdb359c79\") " Nov 03 13:46:36.631725 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:36.631702 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00e7d6b5-17dc-40e1-b6e0-ea2cdb359c79-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:36.641361 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:36.636369 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00e7d6b5-17dc-40e1-b6e0-ea2cdb359c79-kube-api-access-9g7c5" pod "00e7d6b5-17dc-40e1-b6e0-ea2cdb359c79" (UID: "00e7d6b5-17dc-40e1-b6e0-ea2cdb359c79"). InnerVolumeSpecName "kube-api-access-9g7c5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:46:36.641361 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:36.638736 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00e7d6b5-17dc-40e1-b6e0-ea2cdb359c79-maas-api-xks-ca" pod "00e7d6b5-17dc-40e1-b6e0-ea2cdb359c79" (UID: "00e7d6b5-17dc-40e1-b6e0-ea2cdb359c79"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:46:36.732834 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:36.732801 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g7c5\" (UniqueName: \"kubernetes.io/projected/00e7d6b5-17dc-40e1-b6e0-ea2cdb359c79-kube-api-access-9g7c5\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:36.732834 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:36.732835 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00e7d6b5-17dc-40e1-b6e0-ea2cdb359c79-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:36.739869 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:36.739838 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-32bec3-b5b998c44-7qx4z"] Nov 03 13:46:36.743043 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:36.743020 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-32bec3-b5b998c44-7qx4z"] Nov 03 13:46:36.953815 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:36.953779 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-32bec3-678877f9d4-qjwmv"] Nov 03 13:46:36.959884 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:36.959854 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-32bec3-678877f9d4-qjwmv"] Nov 03 13:46:37.063161 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:37.063121 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-32bec3-78c6df84dc-wr8qq"] Nov 03 13:46:37.067629 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:37.067600 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-32bec3-78c6df84dc-wr8qq"] Nov 03 13:46:37.262950 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:46:37.262863 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:46:37.414561 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:37.414521 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03057985-ccfbz" event={"ID":"00e7d6b5-17dc-40e1-b6e0-ea2cdb359c79","Type":"ContainerDied","Data":"6ba0dd08df7ae8be86bcc0fae847d8ab129b4e545186af5efd221bb15c63b788"} Nov 03 13:46:37.414561 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:37.414547 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03057985-ccfbz" Nov 03 13:46:37.415084 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:37.414577 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ba0dd08df7ae8be86bcc0fae847d8ab129b4e545186af5efd221bb15c63b788" Nov 03 13:46:38.115863 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:38.115827 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f6351e2-84a8-414e-a0d8-dd394abec66a" path="/var/lib/kubelet/pods/0f6351e2-84a8-414e-a0d8-dd394abec66a/volumes" Nov 03 13:46:38.116440 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:38.116421 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b23dfaf-6b15-4c7e-8515-019f404475de" path="/var/lib/kubelet/pods/2b23dfaf-6b15-4c7e-8515-019f404475de/volumes" Nov 03 13:46:38.116993 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:38.116977 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a47855e5-140b-478a-bd1e-33c43c14482e" path="/var/lib/kubelet/pods/a47855e5-140b-478a-bd1e-33c43c14482e/volumes" Nov 03 13:46:41.330538 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:41.330502 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq"] Nov 03 13:46:41.330989 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:41.330720 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq" podUID="3a8276f7-206c-402e-8e82-087316e9fc3d" containerName="istio-proxy" containerID="cri-o://a363512eebfb6d0d498fe495407394c03d37a8119a9ae67361bfa781fdcac1fc" gracePeriod=30 Nov 03 13:46:44.260652 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:44.260615 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x"] Nov 03 13:46:44.261146 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:44.260881 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x" podUID="f1705a6f-0727-437b-af88-8fb5448050ac" containerName="istio-proxy" containerID="cri-o://bb8d311f146f5d6d55d0016dd801c0dd1de9f9dfb9ee79fb9a49906dea9f64ab" gracePeriod=30 Nov 03 13:46:46.436730 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.436694 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w"] Nov 03 13:46:46.437446 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.437426 5979 cpu_manager.go:414] "containerMap: removing container" podUID="00e7d6b5-17dc-40e1-b6e0-ea2cdb359c79" containerName="revoke-keys" Nov 03 13:46:46.437446 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.437447 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00e7d6b5-17dc-40e1-b6e0-ea2cdb359c79" containerName="revoke-keys" Nov 03 13:46:46.437589 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.437464 5979 cpu_manager.go:414] "containerMap: removing container" podUID="0f6351e2-84a8-414e-a0d8-dd394abec66a" containerName="payload-processing" Nov 03 13:46:46.437589 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.437484 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f6351e2-84a8-414e-a0d8-dd394abec66a" containerName="payload-processing" Nov 03 13:46:46.437589 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.437495 5979 cpu_manager.go:414] "containerMap: removing container" podUID="a47855e5-140b-478a-bd1e-33c43c14482e" containerName="maas-api" Nov 03 13:46:46.437589 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.437501 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a47855e5-140b-478a-bd1e-33c43c14482e" containerName="maas-api" Nov 03 13:46:46.437589 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.437525 5979 cpu_manager.go:414] "containerMap: removing container" podUID="2b23dfaf-6b15-4c7e-8515-019f404475de" containerName="payload-pre-processing" Nov 03 13:46:46.437589 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.437530 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b23dfaf-6b15-4c7e-8515-019f404475de" containerName="payload-pre-processing" Nov 03 13:46:46.437817 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.437619 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="00e7d6b5-17dc-40e1-b6e0-ea2cdb359c79" containerName="revoke-keys" Nov 03 13:46:46.437817 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.437627 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f6351e2-84a8-414e-a0d8-dd394abec66a" containerName="payload-processing" Nov 03 13:46:46.437817 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.437634 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="a47855e5-140b-478a-bd1e-33c43c14482e" containerName="maas-api" Nov 03 13:46:46.437817 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.437645 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b23dfaf-6b15-4c7e-8515-019f404475de" containerName="payload-pre-processing" Nov 03 13:46:46.440873 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.440854 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w" Nov 03 13:46:46.509935 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.506829 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-d83c93-openshift-default-dockercfg-pwl6m\"" Nov 03 13:46:46.557149 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.557113 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w"] Nov 03 13:46:46.561683 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.561464 5979 generic.go:356] "Generic (PLEG): container finished" podID="3a8276f7-206c-402e-8e82-087316e9fc3d" containerID="a363512eebfb6d0d498fe495407394c03d37a8119a9ae67361bfa781fdcac1fc" exitCode=0 Nov 03 13:46:46.561683 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.561565 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq" event={"ID":"3a8276f7-206c-402e-8e82-087316e9fc3d","Type":"ContainerDied","Data":"a363512eebfb6d0d498fe495407394c03d37a8119a9ae67361bfa781fdcac1fc"} Nov 03 13:46:46.561683 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.561619 5979 scope.go:122] "RemoveContainer" containerID="c4131785842b46468d82a45668859d90988a152807f235d3aafd576451608e2c" Nov 03 13:46:46.637406 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.637368 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-credential-socket\") pod \"e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w\" (UID: \"2fe2e1d4-9fac-456e-ba2b-e221b6c59e98\") " pod="openshift-ingress/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w" Nov 03 13:46:46.637829 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.637796 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-istio-envoy\") pod \"e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w\" (UID: \"2fe2e1d4-9fac-456e-ba2b-e221b6c59e98\") " pod="openshift-ingress/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w" Nov 03 13:46:46.638050 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.638033 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-istio-podinfo\") pod \"e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w\" (UID: \"2fe2e1d4-9fac-456e-ba2b-e221b6c59e98\") " pod="openshift-ingress/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w" Nov 03 13:46:46.638195 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.638181 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-istio-token\") pod \"e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w\" (UID: \"2fe2e1d4-9fac-456e-ba2b-e221b6c59e98\") " pod="openshift-ingress/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w" Nov 03 13:46:46.638334 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.638317 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-workload-socket\") pod \"e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w\" (UID: \"2fe2e1d4-9fac-456e-ba2b-e221b6c59e98\") " pod="openshift-ingress/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w" Nov 03 13:46:46.638492 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.638456 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ftqk\" (UniqueName: \"kubernetes.io/projected/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-kube-api-access-4ftqk\") pod \"e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w\" (UID: \"2fe2e1d4-9fac-456e-ba2b-e221b6c59e98\") " pod="openshift-ingress/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w" Nov 03 13:46:46.638753 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.638726 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-workload-certs\") pod \"e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w\" (UID: \"2fe2e1d4-9fac-456e-ba2b-e221b6c59e98\") " pod="openshift-ingress/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w" Nov 03 13:46:46.639145 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.639117 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-istiod-ca-cert\") pod \"e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w\" (UID: \"2fe2e1d4-9fac-456e-ba2b-e221b6c59e98\") " pod="openshift-ingress/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w" Nov 03 13:46:46.639453 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.639427 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-istio-data\") pod \"e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w\" (UID: \"2fe2e1d4-9fac-456e-ba2b-e221b6c59e98\") " pod="openshift-ingress/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w" Nov 03 13:46:46.717197 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.717172 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq" Nov 03 13:46:46.746058 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.746012 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-istio-data\") pod \"e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w\" (UID: \"2fe2e1d4-9fac-456e-ba2b-e221b6c59e98\") " pod="openshift-ingress/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w" Nov 03 13:46:46.746340 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.746318 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-credential-socket\") pod \"e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w\" (UID: \"2fe2e1d4-9fac-456e-ba2b-e221b6c59e98\") " pod="openshift-ingress/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w" Nov 03 13:46:46.746557 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.746537 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-istio-envoy\") pod \"e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w\" (UID: \"2fe2e1d4-9fac-456e-ba2b-e221b6c59e98\") " pod="openshift-ingress/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w" Nov 03 13:46:46.746692 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.746676 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-istio-podinfo\") pod \"e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w\" (UID: \"2fe2e1d4-9fac-456e-ba2b-e221b6c59e98\") " pod="openshift-ingress/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w" Nov 03 13:46:46.746833 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.746817 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-istio-token\") pod \"e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w\" (UID: \"2fe2e1d4-9fac-456e-ba2b-e221b6c59e98\") " pod="openshift-ingress/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w" Nov 03 13:46:46.746963 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.746945 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-workload-socket\") pod \"e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w\" (UID: \"2fe2e1d4-9fac-456e-ba2b-e221b6c59e98\") " pod="openshift-ingress/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w" Nov 03 13:46:46.747065 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.747051 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ftqk\" (UniqueName: \"kubernetes.io/projected/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-kube-api-access-4ftqk\") pod \"e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w\" (UID: \"2fe2e1d4-9fac-456e-ba2b-e221b6c59e98\") " pod="openshift-ingress/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w" Nov 03 13:46:46.747195 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.747183 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-workload-certs\") pod \"e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w\" (UID: \"2fe2e1d4-9fac-456e-ba2b-e221b6c59e98\") " pod="openshift-ingress/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w" Nov 03 13:46:46.747321 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.747308 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-istiod-ca-cert\") pod \"e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w\" (UID: \"2fe2e1d4-9fac-456e-ba2b-e221b6c59e98\") " pod="openshift-ingress/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w" Nov 03 13:46:46.751874 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.751847 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-istiod-ca-cert\") pod \"e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w\" (UID: \"2fe2e1d4-9fac-456e-ba2b-e221b6c59e98\") " pod="openshift-ingress/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w" Nov 03 13:46:46.752065 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.751853 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-workload-certs\") pod \"e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w\" (UID: \"2fe2e1d4-9fac-456e-ba2b-e221b6c59e98\") " pod="openshift-ingress/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w" Nov 03 13:46:46.752176 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.752056 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-credential-socket\") pod \"e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w\" (UID: \"2fe2e1d4-9fac-456e-ba2b-e221b6c59e98\") " pod="openshift-ingress/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w" Nov 03 13:46:46.756265 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.756225 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-istio-envoy\") pod \"e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w\" (UID: \"2fe2e1d4-9fac-456e-ba2b-e221b6c59e98\") " pod="openshift-ingress/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w" Nov 03 13:46:46.756456 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.756426 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-istio-data\") pod \"e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w\" (UID: \"2fe2e1d4-9fac-456e-ba2b-e221b6c59e98\") " pod="openshift-ingress/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w" Nov 03 13:46:46.756867 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.756831 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-workload-socket\") pod \"e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w\" (UID: \"2fe2e1d4-9fac-456e-ba2b-e221b6c59e98\") " pod="openshift-ingress/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w" Nov 03 13:46:46.758876 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.758839 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-istio-podinfo\") pod \"e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w\" (UID: \"2fe2e1d4-9fac-456e-ba2b-e221b6c59e98\") " pod="openshift-ingress/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w" Nov 03 13:46:46.806397 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.806370 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-istio-token\") pod \"e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w\" (UID: \"2fe2e1d4-9fac-456e-ba2b-e221b6c59e98\") " pod="openshift-ingress/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w" Nov 03 13:46:46.809089 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.809063 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ftqk\" (UniqueName: \"kubernetes.io/projected/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-kube-api-access-4ftqk\") pod \"e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w\" (UID: \"2fe2e1d4-9fac-456e-ba2b-e221b6c59e98\") " pod="openshift-ingress/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w" Nov 03 13:46:46.848451 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.848415 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a8276f7-206c-402e-8e82-087316e9fc3d-istio-token\" (UniqueName: \"kubernetes.io/projected/3a8276f7-206c-402e-8e82-087316e9fc3d-istio-token\") pod \"3a8276f7-206c-402e-8e82-087316e9fc3d\" (UID: \"3a8276f7-206c-402e-8e82-087316e9fc3d\") " Nov 03 13:46:46.848671 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.848482 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a8276f7-206c-402e-8e82-087316e9fc3d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a8276f7-206c-402e-8e82-087316e9fc3d-workload-socket\") pod \"3a8276f7-206c-402e-8e82-087316e9fc3d\" (UID: \"3a8276f7-206c-402e-8e82-087316e9fc3d\") " Nov 03 13:46:46.848671 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.848524 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a8276f7-206c-402e-8e82-087316e9fc3d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a8276f7-206c-402e-8e82-087316e9fc3d-credential-socket\") pod \"3a8276f7-206c-402e-8e82-087316e9fc3d\" (UID: \"3a8276f7-206c-402e-8e82-087316e9fc3d\") " Nov 03 13:46:46.848671 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.848589 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3a8276f7-206c-402e-8e82-087316e9fc3d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a8276f7-206c-402e-8e82-087316e9fc3d-istio-podinfo\") pod \"3a8276f7-206c-402e-8e82-087316e9fc3d\" (UID: \"3a8276f7-206c-402e-8e82-087316e9fc3d\") " Nov 03 13:46:46.848671 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.848631 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a8276f7-206c-402e-8e82-087316e9fc3d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a8276f7-206c-402e-8e82-087316e9fc3d-istio-data\") pod \"3a8276f7-206c-402e-8e82-087316e9fc3d\" (UID: \"3a8276f7-206c-402e-8e82-087316e9fc3d\") " Nov 03 13:46:46.848928 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.848680 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a8276f7-206c-402e-8e82-087316e9fc3d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a8276f7-206c-402e-8e82-087316e9fc3d-istiod-ca-cert\") pod \"3a8276f7-206c-402e-8e82-087316e9fc3d\" (UID: \"3a8276f7-206c-402e-8e82-087316e9fc3d\") " Nov 03 13:46:46.848928 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.848714 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a8276f7-206c-402e-8e82-087316e9fc3d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a8276f7-206c-402e-8e82-087316e9fc3d-istio-envoy\") pod \"3a8276f7-206c-402e-8e82-087316e9fc3d\" (UID: \"3a8276f7-206c-402e-8e82-087316e9fc3d\") " Nov 03 13:46:46.848928 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.848813 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a8276f7-206c-402e-8e82-087316e9fc3d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a8276f7-206c-402e-8e82-087316e9fc3d-workload-certs\") pod \"3a8276f7-206c-402e-8e82-087316e9fc3d\" (UID: \"3a8276f7-206c-402e-8e82-087316e9fc3d\") " Nov 03 13:46:46.848928 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.848883 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a8276f7-206c-402e-8e82-087316e9fc3d-kube-api-access-bc6l5\" (UniqueName: \"kubernetes.io/projected/3a8276f7-206c-402e-8e82-087316e9fc3d-kube-api-access-bc6l5\") pod \"3a8276f7-206c-402e-8e82-087316e9fc3d\" (UID: \"3a8276f7-206c-402e-8e82-087316e9fc3d\") " Nov 03 13:46:46.848928 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.848913 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a8276f7-206c-402e-8e82-087316e9fc3d-credential-socket" pod "3a8276f7-206c-402e-8e82-087316e9fc3d" (UID: "3a8276f7-206c-402e-8e82-087316e9fc3d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:46:46.849212 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.849019 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a8276f7-206c-402e-8e82-087316e9fc3d-istiod-ca-cert" pod "3a8276f7-206c-402e-8e82-087316e9fc3d" (UID: "3a8276f7-206c-402e-8e82-087316e9fc3d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:46:46.849301 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.849267 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a8276f7-206c-402e-8e82-087316e9fc3d-workload-socket" pod "3a8276f7-206c-402e-8e82-087316e9fc3d" (UID: "3a8276f7-206c-402e-8e82-087316e9fc3d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:46:46.849610 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.849532 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a8276f7-206c-402e-8e82-087316e9fc3d-workload-certs" pod "3a8276f7-206c-402e-8e82-087316e9fc3d" (UID: "3a8276f7-206c-402e-8e82-087316e9fc3d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:46:46.849718 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.849648 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a8276f7-206c-402e-8e82-087316e9fc3d-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:46.849718 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.849675 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a8276f7-206c-402e-8e82-087316e9fc3d-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:46.849718 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.849691 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a8276f7-206c-402e-8e82-087316e9fc3d-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:46.849960 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.849931 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a8276f7-206c-402e-8e82-087316e9fc3d-istio-data" pod "3a8276f7-206c-402e-8e82-087316e9fc3d" (UID: "3a8276f7-206c-402e-8e82-087316e9fc3d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:46:46.851065 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.851039 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a8276f7-206c-402e-8e82-087316e9fc3d-istio-podinfo" pod "3a8276f7-206c-402e-8e82-087316e9fc3d" (UID: "3a8276f7-206c-402e-8e82-087316e9fc3d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:46:46.851314 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.851293 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a8276f7-206c-402e-8e82-087316e9fc3d-kube-api-access-bc6l5" pod "3a8276f7-206c-402e-8e82-087316e9fc3d" (UID: "3a8276f7-206c-402e-8e82-087316e9fc3d"). InnerVolumeSpecName "kube-api-access-bc6l5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:46:46.851440 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.851318 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a8276f7-206c-402e-8e82-087316e9fc3d-istio-token" pod "3a8276f7-206c-402e-8e82-087316e9fc3d" (UID: "3a8276f7-206c-402e-8e82-087316e9fc3d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:46:46.851856 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.851828 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a8276f7-206c-402e-8e82-087316e9fc3d-istio-envoy" pod "3a8276f7-206c-402e-8e82-087316e9fc3d" (UID: "3a8276f7-206c-402e-8e82-087316e9fc3d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:46:46.950449 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.950359 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a8276f7-206c-402e-8e82-087316e9fc3d-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:46.950449 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.950391 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a8276f7-206c-402e-8e82-087316e9fc3d-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:46.950449 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.950401 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a8276f7-206c-402e-8e82-087316e9fc3d-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:46.950449 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.950410 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a8276f7-206c-402e-8e82-087316e9fc3d-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:46.950449 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.950422 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bc6l5\" (UniqueName: \"kubernetes.io/projected/3a8276f7-206c-402e-8e82-087316e9fc3d-kube-api-access-bc6l5\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:46.950449 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:46.950433 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a8276f7-206c-402e-8e82-087316e9fc3d-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:47.055242 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:47.055209 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w" Nov 03 13:46:47.484911 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:47.484875 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w"] Nov 03 13:46:47.491180 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:46:47.491145 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fe2e1d4_9fac_456e_ba2b_e221b6c59e98.slice/crio-bdad44da43f400a6ce5fb88b2e93068cc43ac3565ef917f56a5554cf7cf93eb1 WatchSource:0}: Error finding container bdad44da43f400a6ce5fb88b2e93068cc43ac3565ef917f56a5554cf7cf93eb1: Status 404 returned error can't find the container with id bdad44da43f400a6ce5fb88b2e93068cc43ac3565ef917f56a5554cf7cf93eb1 Nov 03 13:46:47.493983 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:47.493947 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:46:47.494092 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:47.494031 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:46:47.494092 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:47.494072 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:46:47.570130 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:47.570098 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq" Nov 03 13:46:47.570130 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:47.570115 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq" event={"ID":"3a8276f7-206c-402e-8e82-087316e9fc3d","Type":"ContainerDied","Data":"40a7520e1d800afe42be7a679b3a41979a7273ef053570f1c3e6dc11a003637f"} Nov 03 13:46:47.570357 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:47.570219 5979 scope.go:122] "RemoveContainer" containerID="a363512eebfb6d0d498fe495407394c03d37a8119a9ae67361bfa781fdcac1fc" Nov 03 13:46:47.571589 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:47.571561 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w" event={"ID":"2fe2e1d4-9fac-456e-ba2b-e221b6c59e98","Type":"ContainerStarted","Data":"bdad44da43f400a6ce5fb88b2e93068cc43ac3565ef917f56a5554cf7cf93eb1"} Nov 03 13:46:47.726859 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:47.726825 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq"] Nov 03 13:46:47.733672 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:47.733637 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-84e22c-openshift-default-75d4c59777-lnlxq"] Nov 03 13:46:48.116402 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:48.116312 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a8276f7-206c-402e-8e82-087316e9fc3d" path="/var/lib/kubelet/pods/3a8276f7-206c-402e-8e82-087316e9fc3d/volumes" Nov 03 13:46:48.580458 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:48.580416 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w" event={"ID":"2fe2e1d4-9fac-456e-ba2b-e221b6c59e98","Type":"ContainerStarted","Data":"e99df3ac5df64c94c69193b5bce556f78818c051cae6de0dc0d96f4a98bef169"} Nov 03 13:46:48.738451 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:48.738393 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w" podStartSLOduration=2.73837788 podStartE2EDuration="2.73837788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:46:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:46:48.737113164 +0000 UTC m=+2490.747801406" watchObservedRunningTime="2026-11-03 13:46:48.73837788 +0000 UTC m=+2490.749066122" Nov 03 13:46:49.055947 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:49.055905 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w" Nov 03 13:46:49.061326 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:49.061303 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w" Nov 03 13:46:49.532720 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:49.532693 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x" Nov 03 13:46:49.581002 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:49.580966 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1705a6f-0727-437b-af88-8fb5448050ac-kube-api-access-mdfr8\" (UniqueName: \"kubernetes.io/projected/f1705a6f-0727-437b-af88-8fb5448050ac-kube-api-access-mdfr8\") pod \"f1705a6f-0727-437b-af88-8fb5448050ac\" (UID: \"f1705a6f-0727-437b-af88-8fb5448050ac\") " Nov 03 13:46:49.581002 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:49.581030 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f1705a6f-0727-437b-af88-8fb5448050ac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1705a6f-0727-437b-af88-8fb5448050ac-istio-podinfo\") pod \"f1705a6f-0727-437b-af88-8fb5448050ac\" (UID: \"f1705a6f-0727-437b-af88-8fb5448050ac\") " Nov 03 13:46:49.582236 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:49.581123 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1705a6f-0727-437b-af88-8fb5448050ac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1705a6f-0727-437b-af88-8fb5448050ac-credential-socket\") pod \"f1705a6f-0727-437b-af88-8fb5448050ac\" (UID: \"f1705a6f-0727-437b-af88-8fb5448050ac\") " Nov 03 13:46:49.582236 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:49.581203 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1705a6f-0727-437b-af88-8fb5448050ac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1705a6f-0727-437b-af88-8fb5448050ac-istio-data\") pod \"f1705a6f-0727-437b-af88-8fb5448050ac\" (UID: \"f1705a6f-0727-437b-af88-8fb5448050ac\") " Nov 03 13:46:49.582236 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:49.581260 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1705a6f-0727-437b-af88-8fb5448050ac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1705a6f-0727-437b-af88-8fb5448050ac-istio-envoy\") pod \"f1705a6f-0727-437b-af88-8fb5448050ac\" (UID: \"f1705a6f-0727-437b-af88-8fb5448050ac\") " Nov 03 13:46:49.582236 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:49.581347 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1705a6f-0727-437b-af88-8fb5448050ac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1705a6f-0727-437b-af88-8fb5448050ac-workload-certs\") pod \"f1705a6f-0727-437b-af88-8fb5448050ac\" (UID: \"f1705a6f-0727-437b-af88-8fb5448050ac\") " Nov 03 13:46:49.582236 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:49.581380 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1705a6f-0727-437b-af88-8fb5448050ac-istio-token\" (UniqueName: \"kubernetes.io/projected/f1705a6f-0727-437b-af88-8fb5448050ac-istio-token\") pod \"f1705a6f-0727-437b-af88-8fb5448050ac\" (UID: \"f1705a6f-0727-437b-af88-8fb5448050ac\") " Nov 03 13:46:49.582236 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:49.581422 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1705a6f-0727-437b-af88-8fb5448050ac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1705a6f-0727-437b-af88-8fb5448050ac-istiod-ca-cert\") pod \"f1705a6f-0727-437b-af88-8fb5448050ac\" (UID: \"f1705a6f-0727-437b-af88-8fb5448050ac\") " Nov 03 13:46:49.582236 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:49.581466 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1705a6f-0727-437b-af88-8fb5448050ac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1705a6f-0727-437b-af88-8fb5448050ac-workload-socket\") pod \"f1705a6f-0727-437b-af88-8fb5448050ac\" (UID: \"f1705a6f-0727-437b-af88-8fb5448050ac\") " Nov 03 13:46:49.582236 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:49.581709 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1705a6f-0727-437b-af88-8fb5448050ac-credential-socket" pod "f1705a6f-0727-437b-af88-8fb5448050ac" (UID: "f1705a6f-0727-437b-af88-8fb5448050ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:46:49.582236 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:49.581813 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1705a6f-0727-437b-af88-8fb5448050ac-workload-certs" pod "f1705a6f-0727-437b-af88-8fb5448050ac" (UID: "f1705a6f-0727-437b-af88-8fb5448050ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:46:49.582236 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:49.581979 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1705a6f-0727-437b-af88-8fb5448050ac-istio-data" pod "f1705a6f-0727-437b-af88-8fb5448050ac" (UID: "f1705a6f-0727-437b-af88-8fb5448050ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:46:49.582236 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:49.582020 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1705a6f-0727-437b-af88-8fb5448050ac-istiod-ca-cert" pod "f1705a6f-0727-437b-af88-8fb5448050ac" (UID: "f1705a6f-0727-437b-af88-8fb5448050ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:46:49.583567 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:49.582397 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1705a6f-0727-437b-af88-8fb5448050ac-workload-socket" pod "f1705a6f-0727-437b-af88-8fb5448050ac" (UID: "f1705a6f-0727-437b-af88-8fb5448050ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:46:49.583567 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:49.582612 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1705a6f-0727-437b-af88-8fb5448050ac-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:49.583567 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:49.582636 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1705a6f-0727-437b-af88-8fb5448050ac-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:49.583567 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:49.582650 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1705a6f-0727-437b-af88-8fb5448050ac-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:49.583567 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:49.582703 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1705a6f-0727-437b-af88-8fb5448050ac-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:49.583567 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:49.582716 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1705a6f-0727-437b-af88-8fb5448050ac-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:49.584179 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:49.584155 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1705a6f-0727-437b-af88-8fb5448050ac-istio-podinfo" pod "f1705a6f-0727-437b-af88-8fb5448050ac" (UID: "f1705a6f-0727-437b-af88-8fb5448050ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:46:49.584270 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:49.584158 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1705a6f-0727-437b-af88-8fb5448050ac-kube-api-access-mdfr8" pod "f1705a6f-0727-437b-af88-8fb5448050ac" (UID: "f1705a6f-0727-437b-af88-8fb5448050ac"). InnerVolumeSpecName "kube-api-access-mdfr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:46:49.584528 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:49.584509 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1705a6f-0727-437b-af88-8fb5448050ac-istio-envoy" pod "f1705a6f-0727-437b-af88-8fb5448050ac" (UID: "f1705a6f-0727-437b-af88-8fb5448050ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:46:49.585276 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:49.585225 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1705a6f-0727-437b-af88-8fb5448050ac-istio-token" pod "f1705a6f-0727-437b-af88-8fb5448050ac" (UID: "f1705a6f-0727-437b-af88-8fb5448050ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:46:49.591037 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:49.591009 5979 generic.go:356] "Generic (PLEG): container finished" podID="f1705a6f-0727-437b-af88-8fb5448050ac" containerID="bb8d311f146f5d6d55d0016dd801c0dd1de9f9dfb9ee79fb9a49906dea9f64ab" exitCode=0 Nov 03 13:46:49.591149 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:49.591055 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x" Nov 03 13:46:49.591149 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:49.591103 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x" event={"ID":"f1705a6f-0727-437b-af88-8fb5448050ac","Type":"ContainerDied","Data":"bb8d311f146f5d6d55d0016dd801c0dd1de9f9dfb9ee79fb9a49906dea9f64ab"} Nov 03 13:46:49.591243 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:49.591177 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x" event={"ID":"f1705a6f-0727-437b-af88-8fb5448050ac","Type":"ContainerDied","Data":"5699936c1c50e15da059505660b97e57837daacfbd337810ab3a8b0395bf15b4"} Nov 03 13:46:49.591243 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:49.591208 5979 scope.go:122] "RemoveContainer" containerID="bb8d311f146f5d6d55d0016dd801c0dd1de9f9dfb9ee79fb9a49906dea9f64ab" Nov 03 13:46:49.591526 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:49.591505 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w" Nov 03 13:46:49.592767 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:49.592748 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w" Nov 03 13:46:49.612088 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:49.612067 5979 scope.go:122] "RemoveContainer" containerID="62d75487cbfbe1dd189530177ccc545a6ecbdeebe59d06f61632f1d1eee49d48" Nov 03 13:46:49.625982 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:49.625960 5979 scope.go:122] "RemoveContainer" containerID="bb8d311f146f5d6d55d0016dd801c0dd1de9f9dfb9ee79fb9a49906dea9f64ab" Nov 03 13:46:49.626274 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:46:49.626254 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb8d311f146f5d6d55d0016dd801c0dd1de9f9dfb9ee79fb9a49906dea9f64ab\": container with ID starting with bb8d311f146f5d6d55d0016dd801c0dd1de9f9dfb9ee79fb9a49906dea9f64ab not found: ID does not exist" containerID="bb8d311f146f5d6d55d0016dd801c0dd1de9f9dfb9ee79fb9a49906dea9f64ab" Nov 03 13:46:49.626329 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:49.626282 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb8d311f146f5d6d55d0016dd801c0dd1de9f9dfb9ee79fb9a49906dea9f64ab"} err="failed to get container status \"bb8d311f146f5d6d55d0016dd801c0dd1de9f9dfb9ee79fb9a49906dea9f64ab\": rpc error: code = NotFound desc = could not find container \"bb8d311f146f5d6d55d0016dd801c0dd1de9f9dfb9ee79fb9a49906dea9f64ab\": container with ID starting with bb8d311f146f5d6d55d0016dd801c0dd1de9f9dfb9ee79fb9a49906dea9f64ab not found: ID does not exist" Nov 03 13:46:49.626329 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:49.626302 5979 scope.go:122] "RemoveContainer" containerID="62d75487cbfbe1dd189530177ccc545a6ecbdeebe59d06f61632f1d1eee49d48" Nov 03 13:46:49.626582 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:46:49.626557 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62d75487cbfbe1dd189530177ccc545a6ecbdeebe59d06f61632f1d1eee49d48\": container with ID starting with 62d75487cbfbe1dd189530177ccc545a6ecbdeebe59d06f61632f1d1eee49d48 not found: ID does not exist" containerID="62d75487cbfbe1dd189530177ccc545a6ecbdeebe59d06f61632f1d1eee49d48" Nov 03 13:46:49.626671 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:49.626589 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62d75487cbfbe1dd189530177ccc545a6ecbdeebe59d06f61632f1d1eee49d48"} err="failed to get container status \"62d75487cbfbe1dd189530177ccc545a6ecbdeebe59d06f61632f1d1eee49d48\": rpc error: code = NotFound desc = could not find container \"62d75487cbfbe1dd189530177ccc545a6ecbdeebe59d06f61632f1d1eee49d48\": container with ID starting with 62d75487cbfbe1dd189530177ccc545a6ecbdeebe59d06f61632f1d1eee49d48 not found: ID does not exist" Nov 03 13:46:49.684796 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:49.684764 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1705a6f-0727-437b-af88-8fb5448050ac-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:49.685012 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:49.684996 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1705a6f-0727-437b-af88-8fb5448050ac-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:49.685128 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:49.685116 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1705a6f-0727-437b-af88-8fb5448050ac-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:49.685218 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:49.685203 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdfr8\" (UniqueName: \"kubernetes.io/projected/f1705a6f-0727-437b-af88-8fb5448050ac-kube-api-access-mdfr8\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:49.753850 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:49.753783 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x"] Nov 03 13:46:49.757711 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:49.757670 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-32bec3-openshift-default-ddf445ddf-j282x"] Nov 03 13:46:50.116534 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:50.116433 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1705a6f-0727-437b-af88-8fb5448050ac" path="/var/lib/kubelet/pods/f1705a6f-0727-437b-af88-8fb5448050ac/volumes" Nov 03 13:46:51.975401 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:51.975360 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e44b9eec-dwxj2"] Nov 03 13:46:51.976124 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:51.976102 5979 cpu_manager.go:414] "containerMap: removing container" podUID="3a8276f7-206c-402e-8e82-087316e9fc3d" containerName="istio-proxy" Nov 03 13:46:51.976242 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:51.976125 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a8276f7-206c-402e-8e82-087316e9fc3d" containerName="istio-proxy" Nov 03 13:46:51.976242 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:51.976158 5979 cpu_manager.go:414] "containerMap: removing container" podUID="f1705a6f-0727-437b-af88-8fb5448050ac" containerName="istio-proxy" Nov 03 13:46:51.976242 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:51.976166 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1705a6f-0727-437b-af88-8fb5448050ac" containerName="istio-proxy" Nov 03 13:46:51.976242 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:51.976178 5979 cpu_manager.go:414] "containerMap: removing container" podUID="3a8276f7-206c-402e-8e82-087316e9fc3d" containerName="istio-proxy" Nov 03 13:46:51.976242 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:51.976186 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a8276f7-206c-402e-8e82-087316e9fc3d" containerName="istio-proxy" Nov 03 13:46:51.976545 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:51.976353 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a8276f7-206c-402e-8e82-087316e9fc3d" containerName="istio-proxy" Nov 03 13:46:51.976545 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:51.976371 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1705a6f-0727-437b-af88-8fb5448050ac" containerName="istio-proxy" Nov 03 13:46:51.976545 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:51.976382 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a8276f7-206c-402e-8e82-087316e9fc3d" containerName="istio-proxy" Nov 03 13:46:51.981416 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:51.981393 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44b9eec-dwxj2" Nov 03 13:46:52.013969 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:52.013933 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39663db1-4bd2-43b0-9537-6a9cdd4223c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e44b9eec-dwxj2\" (UID: \"39663db1-4bd2-43b0-9537-6a9cdd4223c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44b9eec-dwxj2" Nov 03 13:46:52.014143 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:52.014096 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tknfr\" (UniqueName: \"kubernetes.io/projected/39663db1-4bd2-43b0-9537-6a9cdd4223c1-kube-api-access-tknfr\") pod \"maas-api-revoke-sub-e44b9eec-dwxj2\" (UID: \"39663db1-4bd2-43b0-9537-6a9cdd4223c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44b9eec-dwxj2" Nov 03 13:46:52.014192 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:52.014141 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39663db1-4bd2-43b0-9537-6a9cdd4223c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e44b9eec-dwxj2\" (UID: \"39663db1-4bd2-43b0-9537-6a9cdd4223c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44b9eec-dwxj2" Nov 03 13:46:52.034564 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:52.034531 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e44b9eec-dwxj2"] Nov 03 13:46:52.077266 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:52.077232 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:46:52.077421 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:52.077283 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:46:52.117495 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:52.116001 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39663db1-4bd2-43b0-9537-6a9cdd4223c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e44b9eec-dwxj2\" (UID: \"39663db1-4bd2-43b0-9537-6a9cdd4223c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44b9eec-dwxj2" Nov 03 13:46:52.117495 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:52.116176 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tknfr\" (UniqueName: \"kubernetes.io/projected/39663db1-4bd2-43b0-9537-6a9cdd4223c1-kube-api-access-tknfr\") pod \"maas-api-revoke-sub-e44b9eec-dwxj2\" (UID: \"39663db1-4bd2-43b0-9537-6a9cdd4223c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44b9eec-dwxj2" Nov 03 13:46:52.117495 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:52.116230 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39663db1-4bd2-43b0-9537-6a9cdd4223c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e44b9eec-dwxj2\" (UID: \"39663db1-4bd2-43b0-9537-6a9cdd4223c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44b9eec-dwxj2" Nov 03 13:46:52.119821 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:52.119580 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39663db1-4bd2-43b0-9537-6a9cdd4223c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e44b9eec-dwxj2\" (UID: \"39663db1-4bd2-43b0-9537-6a9cdd4223c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44b9eec-dwxj2" Nov 03 13:46:52.121491 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:52.120304 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39663db1-4bd2-43b0-9537-6a9cdd4223c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e44b9eec-dwxj2\" (UID: \"39663db1-4bd2-43b0-9537-6a9cdd4223c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44b9eec-dwxj2" Nov 03 13:46:52.174560 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:52.174523 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tknfr\" (UniqueName: \"kubernetes.io/projected/39663db1-4bd2-43b0-9537-6a9cdd4223c1-kube-api-access-tknfr\") pod \"maas-api-revoke-sub-e44b9eec-dwxj2\" (UID: \"39663db1-4bd2-43b0-9537-6a9cdd4223c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44b9eec-dwxj2" Nov 03 13:46:52.294510 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:52.294396 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44b9eec-dwxj2" Nov 03 13:46:52.674569 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:52.674524 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e44b9eec-dwxj2"] Nov 03 13:46:52.682118 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:46:52.682091 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39663db1_4bd2_43b0_9537_6a9cdd4223c1.slice/crio-283485aef05e638db9aa6f2f4cb3e2da8fc7d70d8d46f19b1e264b3bd2eb286b WatchSource:0}: Error finding container 283485aef05e638db9aa6f2f4cb3e2da8fc7d70d8d46f19b1e264b3bd2eb286b: Status 404 returned error can't find the container with id 283485aef05e638db9aa6f2f4cb3e2da8fc7d70d8d46f19b1e264b3bd2eb286b Nov 03 13:46:53.621251 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:53.621215 5979 generic.go:356] "Generic (PLEG): container finished" podID="39663db1-4bd2-43b0-9537-6a9cdd4223c1" containerID="a5058b5b5c261d8040f2bda6f9b8f8bf59483bde1ec1afb4bb753ffbbd3ce830" exitCode=0 Nov 03 13:46:53.621778 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:53.621285 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44b9eec-dwxj2" event={"ID":"39663db1-4bd2-43b0-9537-6a9cdd4223c1","Type":"ContainerDied","Data":"a5058b5b5c261d8040f2bda6f9b8f8bf59483bde1ec1afb4bb753ffbbd3ce830"} Nov 03 13:46:53.621778 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:53.621338 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44b9eec-dwxj2" event={"ID":"39663db1-4bd2-43b0-9537-6a9cdd4223c1","Type":"ContainerStarted","Data":"283485aef05e638db9aa6f2f4cb3e2da8fc7d70d8d46f19b1e264b3bd2eb286b"} Nov 03 13:46:54.599092 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:54.599049 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57f8a8e4-gnp6p"] Nov 03 13:46:54.599756 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:54.599700 5979 cpu_manager.go:414] "containerMap: removing container" podUID="f1705a6f-0727-437b-af88-8fb5448050ac" containerName="istio-proxy" Nov 03 13:46:54.599756 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:54.599756 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1705a6f-0727-437b-af88-8fb5448050ac" containerName="istio-proxy" Nov 03 13:46:54.599945 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:54.599925 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1705a6f-0727-437b-af88-8fb5448050ac" containerName="istio-proxy" Nov 03 13:46:54.603372 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:54.603352 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f8a8e4-gnp6p" Nov 03 13:46:54.645224 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:54.645179 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18ec454b-720d-499d-9884-4b47962d35b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57f8a8e4-gnp6p\" (UID: \"18ec454b-720d-499d-9884-4b47962d35b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f8a8e4-gnp6p" Nov 03 13:46:54.645690 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:54.645231 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18ec454b-720d-499d-9884-4b47962d35b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57f8a8e4-gnp6p\" (UID: \"18ec454b-720d-499d-9884-4b47962d35b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f8a8e4-gnp6p" Nov 03 13:46:54.645690 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:54.645411 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpbwc\" (UniqueName: \"kubernetes.io/projected/18ec454b-720d-499d-9884-4b47962d35b7-kube-api-access-lpbwc\") pod \"maas-api-revoke-sub-57f8a8e4-gnp6p\" (UID: \"18ec454b-720d-499d-9884-4b47962d35b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f8a8e4-gnp6p" Nov 03 13:46:54.659061 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:54.659017 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57f8a8e4-gnp6p"] Nov 03 13:46:54.747140 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:54.747096 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18ec454b-720d-499d-9884-4b47962d35b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57f8a8e4-gnp6p\" (UID: \"18ec454b-720d-499d-9884-4b47962d35b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f8a8e4-gnp6p" Nov 03 13:46:54.747350 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:54.747161 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18ec454b-720d-499d-9884-4b47962d35b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57f8a8e4-gnp6p\" (UID: \"18ec454b-720d-499d-9884-4b47962d35b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f8a8e4-gnp6p" Nov 03 13:46:54.747350 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:54.747269 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpbwc\" (UniqueName: \"kubernetes.io/projected/18ec454b-720d-499d-9884-4b47962d35b7-kube-api-access-lpbwc\") pod \"maas-api-revoke-sub-57f8a8e4-gnp6p\" (UID: \"18ec454b-720d-499d-9884-4b47962d35b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f8a8e4-gnp6p" Nov 03 13:46:54.748342 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:54.748313 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18ec454b-720d-499d-9884-4b47962d35b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57f8a8e4-gnp6p\" (UID: \"18ec454b-720d-499d-9884-4b47962d35b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f8a8e4-gnp6p" Nov 03 13:46:54.756218 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:54.756191 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18ec454b-720d-499d-9884-4b47962d35b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57f8a8e4-gnp6p\" (UID: \"18ec454b-720d-499d-9884-4b47962d35b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f8a8e4-gnp6p" Nov 03 13:46:54.801759 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:54.801727 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpbwc\" (UniqueName: \"kubernetes.io/projected/18ec454b-720d-499d-9884-4b47962d35b7-kube-api-access-lpbwc\") pod \"maas-api-revoke-sub-57f8a8e4-gnp6p\" (UID: \"18ec454b-720d-499d-9884-4b47962d35b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f8a8e4-gnp6p" Nov 03 13:46:54.819635 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:54.819608 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44b9eec-dwxj2" Nov 03 13:46:54.917361 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:54.917326 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f8a8e4-gnp6p" Nov 03 13:46:54.949662 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:54.949629 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39663db1-4bd2-43b0-9537-6a9cdd4223c1-kube-api-access-tknfr\" (UniqueName: \"kubernetes.io/projected/39663db1-4bd2-43b0-9537-6a9cdd4223c1-kube-api-access-tknfr\") pod \"39663db1-4bd2-43b0-9537-6a9cdd4223c1\" (UID: \"39663db1-4bd2-43b0-9537-6a9cdd4223c1\") " Nov 03 13:46:54.949856 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:54.949720 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39663db1-4bd2-43b0-9537-6a9cdd4223c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39663db1-4bd2-43b0-9537-6a9cdd4223c1-maas-api-xks-ca\") pod \"39663db1-4bd2-43b0-9537-6a9cdd4223c1\" (UID: \"39663db1-4bd2-43b0-9537-6a9cdd4223c1\") " Nov 03 13:46:54.950125 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:54.950103 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39663db1-4bd2-43b0-9537-6a9cdd4223c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39663db1-4bd2-43b0-9537-6a9cdd4223c1-maas-api-openshift-ca\") pod \"39663db1-4bd2-43b0-9537-6a9cdd4223c1\" (UID: \"39663db1-4bd2-43b0-9537-6a9cdd4223c1\") " Nov 03 13:46:54.950510 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:54.950488 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39663db1-4bd2-43b0-9537-6a9cdd4223c1-maas-api-openshift-ca" pod "39663db1-4bd2-43b0-9537-6a9cdd4223c1" (UID: "39663db1-4bd2-43b0-9537-6a9cdd4223c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:46:54.950624 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:54.950606 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39663db1-4bd2-43b0-9537-6a9cdd4223c1-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:54.951783 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:54.951757 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39663db1-4bd2-43b0-9537-6a9cdd4223c1-kube-api-access-tknfr" pod "39663db1-4bd2-43b0-9537-6a9cdd4223c1" (UID: "39663db1-4bd2-43b0-9537-6a9cdd4223c1"). InnerVolumeSpecName "kube-api-access-tknfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:46:54.956267 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:54.956240 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39663db1-4bd2-43b0-9537-6a9cdd4223c1-maas-api-xks-ca" pod "39663db1-4bd2-43b0-9537-6a9cdd4223c1" (UID: "39663db1-4bd2-43b0-9537-6a9cdd4223c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:46:55.052485 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:55.052432 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tknfr\" (UniqueName: \"kubernetes.io/projected/39663db1-4bd2-43b0-9537-6a9cdd4223c1-kube-api-access-tknfr\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:55.052485 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:55.052465 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39663db1-4bd2-43b0-9537-6a9cdd4223c1-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:55.294452 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:55.294361 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57f8a8e4-gnp6p"] Nov 03 13:46:55.302271 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:46:55.302243 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18ec454b_720d_499d_9884_4b47962d35b7.slice/crio-c03960c50a62b06cb0883c99765bf6ec1dfcf8d453576757f8ec0d97dea99730 WatchSource:0}: Error finding container c03960c50a62b06cb0883c99765bf6ec1dfcf8d453576757f8ec0d97dea99730: Status 404 returned error can't find the container with id c03960c50a62b06cb0883c99765bf6ec1dfcf8d453576757f8ec0d97dea99730 Nov 03 13:46:55.559235 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:55.559146 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:46:55.636262 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:55.636226 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f8a8e4-gnp6p" event={"ID":"18ec454b-720d-499d-9884-4b47962d35b7","Type":"ContainerStarted","Data":"c03960c50a62b06cb0883c99765bf6ec1dfcf8d453576757f8ec0d97dea99730"} Nov 03 13:46:55.637572 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:55.637550 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44b9eec-dwxj2" Nov 03 13:46:55.637719 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:55.637586 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e44b9eec-dwxj2" event={"ID":"39663db1-4bd2-43b0-9537-6a9cdd4223c1","Type":"ContainerDied","Data":"283485aef05e638db9aa6f2f4cb3e2da8fc7d70d8d46f19b1e264b3bd2eb286b"} Nov 03 13:46:55.637719 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:55.637625 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="283485aef05e638db9aa6f2f4cb3e2da8fc7d70d8d46f19b1e264b3bd2eb286b" Nov 03 13:46:55.982692 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:55.982658 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d83c93-6f4df847c5-gjhcw"] Nov 03 13:46:55.983355 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:55.983323 5979 cpu_manager.go:414] "containerMap: removing container" podUID="39663db1-4bd2-43b0-9537-6a9cdd4223c1" containerName="revoke-keys" Nov 03 13:46:55.983355 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:55.983344 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39663db1-4bd2-43b0-9537-6a9cdd4223c1" containerName="revoke-keys" Nov 03 13:46:55.983451 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:55.983429 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="39663db1-4bd2-43b0-9537-6a9cdd4223c1" containerName="revoke-keys" Nov 03 13:46:55.986844 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:55.986822 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d83c93-6f4df847c5-gjhcw" Nov 03 13:46:56.041566 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:56.041531 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-d83c93\"" Nov 03 13:46:56.041972 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:56.041531 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-d83c93-dockercfg-b2gg7\"" Nov 03 13:46:56.042180 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:56.042158 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d83c93-6f4df847c5-gjhcw"] Nov 03 13:46:56.065603 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:56.065564 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e79b25c7-5f37-4147-8349-c557fdf621e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-d83c93-6f4df847c5-gjhcw\" (UID: \"e79b25c7-5f37-4147-8349-c557fdf621e3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d83c93-6f4df847c5-gjhcw" Nov 03 13:46:56.065867 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:56.065837 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mpm9\" (UniqueName: \"kubernetes.io/projected/e79b25c7-5f37-4147-8349-c557fdf621e3-kube-api-access-5mpm9\") pod \"payload-pre-processing-e2e-praxis-a-d83c93-6f4df847c5-gjhcw\" (UID: \"e79b25c7-5f37-4147-8349-c557fdf621e3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d83c93-6f4df847c5-gjhcw" Nov 03 13:46:56.162967 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:56.162928 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d83c93-6f499df6bc-t88p4"] Nov 03 13:46:56.167282 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:56.167259 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d83c93-6f499df6bc-t88p4" Nov 03 13:46:56.167428 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:56.167400 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e79b25c7-5f37-4147-8349-c557fdf621e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-d83c93-6f4df847c5-gjhcw\" (UID: \"e79b25c7-5f37-4147-8349-c557fdf621e3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d83c93-6f4df847c5-gjhcw" Nov 03 13:46:56.167505 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:56.167451 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mpm9\" (UniqueName: \"kubernetes.io/projected/e79b25c7-5f37-4147-8349-c557fdf621e3-kube-api-access-5mpm9\") pod \"payload-pre-processing-e2e-praxis-a-d83c93-6f4df847c5-gjhcw\" (UID: \"e79b25c7-5f37-4147-8349-c557fdf621e3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d83c93-6f4df847c5-gjhcw" Nov 03 13:46:56.168143 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:56.168121 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e79b25c7-5f37-4147-8349-c557fdf621e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-d83c93-6f4df847c5-gjhcw\" (UID: \"e79b25c7-5f37-4147-8349-c557fdf621e3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d83c93-6f4df847c5-gjhcw" Nov 03 13:46:56.194542 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:56.194509 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d83c93-6f499df6bc-t88p4"] Nov 03 13:46:56.220877 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:56.220843 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mpm9\" (UniqueName: \"kubernetes.io/projected/e79b25c7-5f37-4147-8349-c557fdf621e3-kube-api-access-5mpm9\") pod \"payload-pre-processing-e2e-praxis-a-d83c93-6f4df847c5-gjhcw\" (UID: \"e79b25c7-5f37-4147-8349-c557fdf621e3\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d83c93-6f4df847c5-gjhcw" Nov 03 13:46:56.269395 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:56.269301 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfjdf\" (UniqueName: \"kubernetes.io/projected/52195f70-2bc0-42c8-b5b6-b215f849e526-kube-api-access-rfjdf\") pod \"payload-processing-e2e-praxis-a-d83c93-6f499df6bc-t88p4\" (UID: \"52195f70-2bc0-42c8-b5b6-b215f849e526\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d83c93-6f499df6bc-t88p4" Nov 03 13:46:56.269596 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:56.269542 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52195f70-2bc0-42c8-b5b6-b215f849e526-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-d83c93-6f499df6bc-t88p4\" (UID: \"52195f70-2bc0-42c8-b5b6-b215f849e526\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d83c93-6f499df6bc-t88p4" Nov 03 13:46:56.302264 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:56.302230 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d83c93-6f4df847c5-gjhcw" Nov 03 13:46:56.371174 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:56.371138 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52195f70-2bc0-42c8-b5b6-b215f849e526-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-d83c93-6f499df6bc-t88p4\" (UID: \"52195f70-2bc0-42c8-b5b6-b215f849e526\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d83c93-6f499df6bc-t88p4" Nov 03 13:46:56.371294 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:56.371259 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfjdf\" (UniqueName: \"kubernetes.io/projected/52195f70-2bc0-42c8-b5b6-b215f849e526-kube-api-access-rfjdf\") pod \"payload-processing-e2e-praxis-a-d83c93-6f499df6bc-t88p4\" (UID: \"52195f70-2bc0-42c8-b5b6-b215f849e526\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d83c93-6f499df6bc-t88p4" Nov 03 13:46:56.372449 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:56.372411 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52195f70-2bc0-42c8-b5b6-b215f849e526-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-d83c93-6f499df6bc-t88p4\" (UID: \"52195f70-2bc0-42c8-b5b6-b215f849e526\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d83c93-6f499df6bc-t88p4" Nov 03 13:46:56.425052 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:56.425013 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfjdf\" (UniqueName: \"kubernetes.io/projected/52195f70-2bc0-42c8-b5b6-b215f849e526-kube-api-access-rfjdf\") pod \"payload-processing-e2e-praxis-a-d83c93-6f499df6bc-t88p4\" (UID: \"52195f70-2bc0-42c8-b5b6-b215f849e526\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d83c93-6f499df6bc-t88p4" Nov 03 13:46:56.481353 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:56.481316 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d83c93-6f499df6bc-t88p4" Nov 03 13:46:56.646332 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:56.646298 5979 generic.go:356] "Generic (PLEG): container finished" podID="18ec454b-720d-499d-9884-4b47962d35b7" containerID="75e7ea3cb7180751c87665b504b7eff8d8d831b85e38dbf2e33bbc8bd1b350e9" exitCode=0 Nov 03 13:46:56.646554 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:56.646389 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f8a8e4-gnp6p" event={"ID":"18ec454b-720d-499d-9884-4b47962d35b7","Type":"ContainerDied","Data":"75e7ea3cb7180751c87665b504b7eff8d8d831b85e38dbf2e33bbc8bd1b350e9"} Nov 03 13:46:56.688203 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:56.688166 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d83c93-6f4df847c5-gjhcw"] Nov 03 13:46:56.694649 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:46:56.694621 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode79b25c7_5f37_4147_8349_c557fdf621e3.slice/crio-df6defe4e650c2b4955f54e914ce17f03e2960c61c7dd5ed2dd60333227f2beb WatchSource:0}: Error finding container df6defe4e650c2b4955f54e914ce17f03e2960c61c7dd5ed2dd60333227f2beb: Status 404 returned error can't find the container with id df6defe4e650c2b4955f54e914ce17f03e2960c61c7dd5ed2dd60333227f2beb Nov 03 13:46:56.863277 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:56.863189 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d83c93-6f499df6bc-t88p4"] Nov 03 13:46:56.870986 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:46:56.870955 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52195f70_2bc0_42c8_b5b6_b215f849e526.slice/crio-767ea39d883dc46142f26ef5622daba9e460314bcca16743c7d80460067fbcde WatchSource:0}: Error finding container 767ea39d883dc46142f26ef5622daba9e460314bcca16743c7d80460067fbcde: Status 404 returned error can't find the container with id 767ea39d883dc46142f26ef5622daba9e460314bcca16743c7d80460067fbcde Nov 03 13:46:57.655303 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:57.655266 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d83c93-6f4df847c5-gjhcw" event={"ID":"e79b25c7-5f37-4147-8349-c557fdf621e3","Type":"ContainerStarted","Data":"9688dfcf0cccc16acfc9eec53c4960addbb2dad965be12dfdd158a2839833da8"} Nov 03 13:46:57.655768 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:57.655335 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d83c93-6f4df847c5-gjhcw" Nov 03 13:46:57.655768 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:57.655359 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d83c93-6f4df847c5-gjhcw" event={"ID":"e79b25c7-5f37-4147-8349-c557fdf621e3","Type":"ContainerStarted","Data":"df6defe4e650c2b4955f54e914ce17f03e2960c61c7dd5ed2dd60333227f2beb"} Nov 03 13:46:57.657151 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:57.657122 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d83c93-6f499df6bc-t88p4" event={"ID":"52195f70-2bc0-42c8-b5b6-b215f849e526","Type":"ContainerStarted","Data":"3a93e09c722df39f3e1ec5564492a84c9d154df97801012466143ff77b86e4b1"} Nov 03 13:46:57.657308 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:57.657167 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d83c93-6f499df6bc-t88p4" event={"ID":"52195f70-2bc0-42c8-b5b6-b215f849e526","Type":"ContainerStarted","Data":"767ea39d883dc46142f26ef5622daba9e460314bcca16743c7d80460067fbcde"} Nov 03 13:46:57.657379 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:57.657325 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-d83c93-6f499df6bc-t88p4" Nov 03 13:46:57.763401 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:57.763342 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d83c93-6f4df847c5-gjhcw" podStartSLOduration=2.763328116 podStartE2EDuration="2.763328116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:46:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:46:57.761774928 +0000 UTC m=+2499.772463171" watchObservedRunningTime="2026-11-03 13:46:57.763328116 +0000 UTC m=+2499.774016358" Nov 03 13:46:57.808419 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:57.808392 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f8a8e4-gnp6p" Nov 03 13:46:57.869093 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:57.869034 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-d83c93-6f499df6bc-t88p4" podStartSLOduration=1.8689896419999998 podStartE2EDuration="1.868989642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:46:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:46:57.868265458 +0000 UTC m=+2499.878953700" watchObservedRunningTime="2026-11-03 13:46:57.868989642 +0000 UTC m=+2499.879677878" Nov 03 13:46:57.900171 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:57.897959 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18ec454b-720d-499d-9884-4b47962d35b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18ec454b-720d-499d-9884-4b47962d35b7-maas-api-openshift-ca\") pod \"18ec454b-720d-499d-9884-4b47962d35b7\" (UID: \"18ec454b-720d-499d-9884-4b47962d35b7\") " Nov 03 13:46:57.900171 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:57.898111 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18ec454b-720d-499d-9884-4b47962d35b7-kube-api-access-lpbwc\" (UniqueName: \"kubernetes.io/projected/18ec454b-720d-499d-9884-4b47962d35b7-kube-api-access-lpbwc\") pod \"18ec454b-720d-499d-9884-4b47962d35b7\" (UID: \"18ec454b-720d-499d-9884-4b47962d35b7\") " Nov 03 13:46:57.900171 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:57.898190 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18ec454b-720d-499d-9884-4b47962d35b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18ec454b-720d-499d-9884-4b47962d35b7-maas-api-xks-ca\") pod \"18ec454b-720d-499d-9884-4b47962d35b7\" (UID: \"18ec454b-720d-499d-9884-4b47962d35b7\") " Nov 03 13:46:57.905036 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:57.900864 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18ec454b-720d-499d-9884-4b47962d35b7-maas-api-openshift-ca" pod "18ec454b-720d-499d-9884-4b47962d35b7" (UID: "18ec454b-720d-499d-9884-4b47962d35b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:46:57.905036 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:57.903822 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18ec454b-720d-499d-9884-4b47962d35b7-maas-api-xks-ca" pod "18ec454b-720d-499d-9884-4b47962d35b7" (UID: "18ec454b-720d-499d-9884-4b47962d35b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:46:57.905441 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:57.905377 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18ec454b-720d-499d-9884-4b47962d35b7-kube-api-access-lpbwc" pod "18ec454b-720d-499d-9884-4b47962d35b7" (UID: "18ec454b-720d-499d-9884-4b47962d35b7"). InnerVolumeSpecName "kube-api-access-lpbwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:46:58.001500 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:58.000238 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpbwc\" (UniqueName: \"kubernetes.io/projected/18ec454b-720d-499d-9884-4b47962d35b7-kube-api-access-lpbwc\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:58.001500 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:58.000280 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18ec454b-720d-499d-9884-4b47962d35b7-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:58.001500 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:58.000294 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18ec454b-720d-499d-9884-4b47962d35b7-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:46:58.665061 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:58.665016 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f8a8e4-gnp6p" event={"ID":"18ec454b-720d-499d-9884-4b47962d35b7","Type":"ContainerDied","Data":"c03960c50a62b06cb0883c99765bf6ec1dfcf8d453576757f8ec0d97dea99730"} Nov 03 13:46:58.665585 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:58.665082 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57f8a8e4-gnp6p" Nov 03 13:46:58.665585 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:46:58.665108 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c03960c50a62b06cb0883c99765bf6ec1dfcf8d453576757f8ec0d97dea99730" Nov 03 13:47:05.398762 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:05.398715 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef5ed956-pc2gz"] Nov 03 13:47:05.399350 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:05.399334 5979 cpu_manager.go:414] "containerMap: removing container" podUID="18ec454b-720d-499d-9884-4b47962d35b7" containerName="revoke-keys" Nov 03 13:47:05.399350 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:05.399351 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18ec454b-720d-499d-9884-4b47962d35b7" containerName="revoke-keys" Nov 03 13:47:05.399452 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:05.399440 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="18ec454b-720d-499d-9884-4b47962d35b7" containerName="revoke-keys" Nov 03 13:47:05.403199 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:05.403179 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5ed956-pc2gz" Nov 03 13:47:05.497810 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:05.497766 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7nzz\" (UniqueName: \"kubernetes.io/projected/b26f627f-b967-45b9-80d3-847cc513bdfe-kube-api-access-w7nzz\") pod \"maas-api-revoke-sub-ef5ed956-pc2gz\" (UID: \"b26f627f-b967-45b9-80d3-847cc513bdfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5ed956-pc2gz" Nov 03 13:47:05.498011 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:05.497938 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b26f627f-b967-45b9-80d3-847cc513bdfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef5ed956-pc2gz\" (UID: \"b26f627f-b967-45b9-80d3-847cc513bdfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5ed956-pc2gz" Nov 03 13:47:05.498156 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:05.498122 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b26f627f-b967-45b9-80d3-847cc513bdfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef5ed956-pc2gz\" (UID: \"b26f627f-b967-45b9-80d3-847cc513bdfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5ed956-pc2gz" Nov 03 13:47:05.499632 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:05.499609 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:47:05.499744 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:05.499682 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:47:05.502568 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:05.502532 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef5ed956-pc2gz"] Nov 03 13:47:05.599143 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:05.599104 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7nzz\" (UniqueName: \"kubernetes.io/projected/b26f627f-b967-45b9-80d3-847cc513bdfe-kube-api-access-w7nzz\") pod \"maas-api-revoke-sub-ef5ed956-pc2gz\" (UID: \"b26f627f-b967-45b9-80d3-847cc513bdfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5ed956-pc2gz" Nov 03 13:47:05.599297 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:05.599190 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b26f627f-b967-45b9-80d3-847cc513bdfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef5ed956-pc2gz\" (UID: \"b26f627f-b967-45b9-80d3-847cc513bdfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5ed956-pc2gz" Nov 03 13:47:05.599361 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:05.599334 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b26f627f-b967-45b9-80d3-847cc513bdfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef5ed956-pc2gz\" (UID: \"b26f627f-b967-45b9-80d3-847cc513bdfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5ed956-pc2gz" Nov 03 13:47:05.599938 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:05.599910 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b26f627f-b967-45b9-80d3-847cc513bdfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef5ed956-pc2gz\" (UID: \"b26f627f-b967-45b9-80d3-847cc513bdfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5ed956-pc2gz" Nov 03 13:47:05.601593 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:05.601571 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b26f627f-b967-45b9-80d3-847cc513bdfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef5ed956-pc2gz\" (UID: \"b26f627f-b967-45b9-80d3-847cc513bdfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5ed956-pc2gz" Nov 03 13:47:05.653082 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:05.653010 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7nzz\" (UniqueName: \"kubernetes.io/projected/b26f627f-b967-45b9-80d3-847cc513bdfe-kube-api-access-w7nzz\") pod \"maas-api-revoke-sub-ef5ed956-pc2gz\" (UID: \"b26f627f-b967-45b9-80d3-847cc513bdfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5ed956-pc2gz" Nov 03 13:47:05.716659 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:05.716628 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5ed956-pc2gz" Nov 03 13:47:06.102068 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:06.101971 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef5ed956-pc2gz"] Nov 03 13:47:06.109140 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:47:06.109102 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb26f627f_b967_45b9_80d3_847cc513bdfe.slice/crio-a3e1e3571d73af529a22a58d9814932886fcfcdc339f6495e1d5d44de0867392 WatchSource:0}: Error finding container a3e1e3571d73af529a22a58d9814932886fcfcdc339f6495e1d5d44de0867392: Status 404 returned error can't find the container with id a3e1e3571d73af529a22a58d9814932886fcfcdc339f6495e1d5d44de0867392 Nov 03 13:47:06.720859 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:06.720821 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5ed956-pc2gz" event={"ID":"b26f627f-b967-45b9-80d3-847cc513bdfe","Type":"ContainerStarted","Data":"7af69ad90395a627d7c34ae8e167449836759f918f045317c6f117efba44edfb"} Nov 03 13:47:06.721269 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:06.720876 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5ed956-pc2gz" event={"ID":"b26f627f-b967-45b9-80d3-847cc513bdfe","Type":"ContainerStarted","Data":"a3e1e3571d73af529a22a58d9814932886fcfcdc339f6495e1d5d44de0867392"} Nov 03 13:47:06.827971 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:06.827913 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5ed956-pc2gz" podStartSLOduration=1.827899078 podStartE2EDuration="1.827899078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:47:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:47:06.826581401 +0000 UTC m=+2508.837269640" watchObservedRunningTime="2026-11-03 13:47:06.827899078 +0000 UTC m=+2508.838587354" Nov 03 13:47:07.234675 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:47:07.234633 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:47:07.261831 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:07.261797 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5606cf0b-fkwr8"] Nov 03 13:47:07.266024 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:07.266000 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5606cf0b-fkwr8" Nov 03 13:47:07.320140 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:07.320097 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5606cf0b-fkwr8"] Nov 03 13:47:07.320958 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:07.320932 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46qcg\" (UniqueName: \"kubernetes.io/projected/bb80b6d0-af59-468e-93a8-7ed4b40454ce-kube-api-access-46qcg\") pod \"maas-api-revoke-sub-5606cf0b-fkwr8\" (UID: \"bb80b6d0-af59-468e-93a8-7ed4b40454ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5606cf0b-fkwr8" Nov 03 13:47:07.321254 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:07.321232 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb80b6d0-af59-468e-93a8-7ed4b40454ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5606cf0b-fkwr8\" (UID: \"bb80b6d0-af59-468e-93a8-7ed4b40454ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5606cf0b-fkwr8" Nov 03 13:47:07.321410 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:07.321297 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb80b6d0-af59-468e-93a8-7ed4b40454ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5606cf0b-fkwr8\" (UID: \"bb80b6d0-af59-468e-93a8-7ed4b40454ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5606cf0b-fkwr8" Nov 03 13:47:07.422294 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:07.422214 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46qcg\" (UniqueName: \"kubernetes.io/projected/bb80b6d0-af59-468e-93a8-7ed4b40454ce-kube-api-access-46qcg\") pod \"maas-api-revoke-sub-5606cf0b-fkwr8\" (UID: \"bb80b6d0-af59-468e-93a8-7ed4b40454ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5606cf0b-fkwr8" Nov 03 13:47:07.422505 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:07.422315 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb80b6d0-af59-468e-93a8-7ed4b40454ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5606cf0b-fkwr8\" (UID: \"bb80b6d0-af59-468e-93a8-7ed4b40454ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5606cf0b-fkwr8" Nov 03 13:47:07.422505 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:07.422350 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb80b6d0-af59-468e-93a8-7ed4b40454ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5606cf0b-fkwr8\" (UID: \"bb80b6d0-af59-468e-93a8-7ed4b40454ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5606cf0b-fkwr8" Nov 03 13:47:07.422958 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:07.422938 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb80b6d0-af59-468e-93a8-7ed4b40454ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5606cf0b-fkwr8\" (UID: \"bb80b6d0-af59-468e-93a8-7ed4b40454ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5606cf0b-fkwr8" Nov 03 13:47:07.424680 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:07.424660 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb80b6d0-af59-468e-93a8-7ed4b40454ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5606cf0b-fkwr8\" (UID: \"bb80b6d0-af59-468e-93a8-7ed4b40454ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5606cf0b-fkwr8" Nov 03 13:47:07.476927 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:07.476893 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46qcg\" (UniqueName: \"kubernetes.io/projected/bb80b6d0-af59-468e-93a8-7ed4b40454ce-kube-api-access-46qcg\") pod \"maas-api-revoke-sub-5606cf0b-fkwr8\" (UID: \"bb80b6d0-af59-468e-93a8-7ed4b40454ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5606cf0b-fkwr8" Nov 03 13:47:07.580806 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:07.580771 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5606cf0b-fkwr8" Nov 03 13:47:07.728290 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:07.728202 5979 generic.go:356] "Generic (PLEG): container finished" podID="b26f627f-b967-45b9-80d3-847cc513bdfe" containerID="7af69ad90395a627d7c34ae8e167449836759f918f045317c6f117efba44edfb" exitCode=0 Nov 03 13:47:07.728738 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:07.728282 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5ed956-pc2gz" event={"ID":"b26f627f-b967-45b9-80d3-847cc513bdfe","Type":"ContainerDied","Data":"7af69ad90395a627d7c34ae8e167449836759f918f045317c6f117efba44edfb"} Nov 03 13:47:07.955617 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:07.955575 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5606cf0b-fkwr8"] Nov 03 13:47:07.961604 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:47:07.961575 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb80b6d0_af59_468e_93a8_7ed4b40454ce.slice/crio-0a6386bd80c2a919b7afe4a8ac35ae6811f18f6cf72dc5474872927759ffcdcc WatchSource:0}: Error finding container 0a6386bd80c2a919b7afe4a8ac35ae6811f18f6cf72dc5474872927759ffcdcc: Status 404 returned error can't find the container with id 0a6386bd80c2a919b7afe4a8ac35ae6811f18f6cf72dc5474872927759ffcdcc Nov 03 13:47:08.668224 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:08.668191 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d83c93-6f4df847c5-gjhcw" Nov 03 13:47:08.668407 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:08.668266 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-d83c93-6f499df6bc-t88p4" Nov 03 13:47:08.736723 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:08.736685 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5606cf0b-fkwr8" event={"ID":"bb80b6d0-af59-468e-93a8-7ed4b40454ce","Type":"ContainerStarted","Data":"96b70c6dbf0a8a6ef489e62e555fb0ab4454210809c0dd94be7540845cd9382b"} Nov 03 13:47:08.737132 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:08.736734 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5606cf0b-fkwr8" event={"ID":"bb80b6d0-af59-468e-93a8-7ed4b40454ce","Type":"ContainerStarted","Data":"0a6386bd80c2a919b7afe4a8ac35ae6811f18f6cf72dc5474872927759ffcdcc"} Nov 03 13:47:08.919880 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:08.919789 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5ed956-pc2gz" Nov 03 13:47:08.939098 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:08.939070 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b26f627f-b967-45b9-80d3-847cc513bdfe-kube-api-access-w7nzz\" (UniqueName: \"kubernetes.io/projected/b26f627f-b967-45b9-80d3-847cc513bdfe-kube-api-access-w7nzz\") pod \"b26f627f-b967-45b9-80d3-847cc513bdfe\" (UID: \"b26f627f-b967-45b9-80d3-847cc513bdfe\") " Nov 03 13:47:08.939260 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:08.939127 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b26f627f-b967-45b9-80d3-847cc513bdfe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b26f627f-b967-45b9-80d3-847cc513bdfe-maas-api-openshift-ca\") pod \"b26f627f-b967-45b9-80d3-847cc513bdfe\" (UID: \"b26f627f-b967-45b9-80d3-847cc513bdfe\") " Nov 03 13:47:08.939346 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:08.939248 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b26f627f-b967-45b9-80d3-847cc513bdfe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b26f627f-b967-45b9-80d3-847cc513bdfe-maas-api-xks-ca\") pod \"b26f627f-b967-45b9-80d3-847cc513bdfe\" (UID: \"b26f627f-b967-45b9-80d3-847cc513bdfe\") " Nov 03 13:47:08.939549 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:08.939525 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b26f627f-b967-45b9-80d3-847cc513bdfe-maas-api-openshift-ca" pod "b26f627f-b967-45b9-80d3-847cc513bdfe" (UID: "b26f627f-b967-45b9-80d3-847cc513bdfe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:47:08.939819 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:08.939796 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b26f627f-b967-45b9-80d3-847cc513bdfe-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:47:08.941271 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:08.941249 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b26f627f-b967-45b9-80d3-847cc513bdfe-maas-api-xks-ca" pod "b26f627f-b967-45b9-80d3-847cc513bdfe" (UID: "b26f627f-b967-45b9-80d3-847cc513bdfe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:47:08.941365 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:08.941354 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b26f627f-b967-45b9-80d3-847cc513bdfe-kube-api-access-w7nzz" pod "b26f627f-b967-45b9-80d3-847cc513bdfe" (UID: "b26f627f-b967-45b9-80d3-847cc513bdfe"). InnerVolumeSpecName "kube-api-access-w7nzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:47:08.991092 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:08.991035 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5606cf0b-fkwr8" podStartSLOduration=1.991019866 podStartE2EDuration="1.991019866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:47:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:47:08.988159676 +0000 UTC m=+2510.998847928" watchObservedRunningTime="2026-11-03 13:47:08.991019866 +0000 UTC m=+2511.001708108" Nov 03 13:47:09.041211 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:09.041176 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b26f627f-b967-45b9-80d3-847cc513bdfe-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:47:09.041211 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:09.041207 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7nzz\" (UniqueName: \"kubernetes.io/projected/b26f627f-b967-45b9-80d3-847cc513bdfe-kube-api-access-w7nzz\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:47:09.755326 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:09.755291 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5ed956-pc2gz" Nov 03 13:47:09.755851 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:09.755325 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef5ed956-pc2gz" event={"ID":"b26f627f-b967-45b9-80d3-847cc513bdfe","Type":"ContainerDied","Data":"a3e1e3571d73af529a22a58d9814932886fcfcdc339f6495e1d5d44de0867392"} Nov 03 13:47:09.755851 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:09.755401 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3e1e3571d73af529a22a58d9814932886fcfcdc339f6495e1d5d44de0867392" Nov 03 13:47:09.757119 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:09.757093 5979 generic.go:356] "Generic (PLEG): container finished" podID="bb80b6d0-af59-468e-93a8-7ed4b40454ce" containerID="96b70c6dbf0a8a6ef489e62e555fb0ab4454210809c0dd94be7540845cd9382b" exitCode=0 Nov 03 13:47:09.757233 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:09.757156 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5606cf0b-fkwr8" event={"ID":"bb80b6d0-af59-468e-93a8-7ed4b40454ce","Type":"ContainerDied","Data":"96b70c6dbf0a8a6ef489e62e555fb0ab4454210809c0dd94be7540845cd9382b"} Nov 03 13:47:10.903939 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:10.903914 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5606cf0b-fkwr8" Nov 03 13:47:10.962067 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:10.962030 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb80b6d0-af59-468e-93a8-7ed4b40454ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb80b6d0-af59-468e-93a8-7ed4b40454ce-maas-api-openshift-ca\") pod \"bb80b6d0-af59-468e-93a8-7ed4b40454ce\" (UID: \"bb80b6d0-af59-468e-93a8-7ed4b40454ce\") " Nov 03 13:47:10.962248 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:10.962092 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb80b6d0-af59-468e-93a8-7ed4b40454ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb80b6d0-af59-468e-93a8-7ed4b40454ce-maas-api-xks-ca\") pod \"bb80b6d0-af59-468e-93a8-7ed4b40454ce\" (UID: \"bb80b6d0-af59-468e-93a8-7ed4b40454ce\") " Nov 03 13:47:10.962248 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:10.962241 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb80b6d0-af59-468e-93a8-7ed4b40454ce-kube-api-access-46qcg\" (UniqueName: \"kubernetes.io/projected/bb80b6d0-af59-468e-93a8-7ed4b40454ce-kube-api-access-46qcg\") pod \"bb80b6d0-af59-468e-93a8-7ed4b40454ce\" (UID: \"bb80b6d0-af59-468e-93a8-7ed4b40454ce\") " Nov 03 13:47:10.962596 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:10.962568 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb80b6d0-af59-468e-93a8-7ed4b40454ce-maas-api-openshift-ca" pod "bb80b6d0-af59-468e-93a8-7ed4b40454ce" (UID: "bb80b6d0-af59-468e-93a8-7ed4b40454ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:47:10.964207 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:10.964184 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb80b6d0-af59-468e-93a8-7ed4b40454ce-maas-api-xks-ca" pod "bb80b6d0-af59-468e-93a8-7ed4b40454ce" (UID: "bb80b6d0-af59-468e-93a8-7ed4b40454ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:47:10.964356 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:10.964336 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb80b6d0-af59-468e-93a8-7ed4b40454ce-kube-api-access-46qcg" pod "bb80b6d0-af59-468e-93a8-7ed4b40454ce" (UID: "bb80b6d0-af59-468e-93a8-7ed4b40454ce"). InnerVolumeSpecName "kube-api-access-46qcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:47:11.064206 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:11.064124 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb80b6d0-af59-468e-93a8-7ed4b40454ce-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:47:11.064206 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:11.064155 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb80b6d0-af59-468e-93a8-7ed4b40454ce-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:47:11.064206 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:11.064165 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46qcg\" (UniqueName: \"kubernetes.io/projected/bb80b6d0-af59-468e-93a8-7ed4b40454ce-kube-api-access-46qcg\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:47:11.771715 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:11.771676 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5606cf0b-fkwr8" event={"ID":"bb80b6d0-af59-468e-93a8-7ed4b40454ce","Type":"ContainerDied","Data":"0a6386bd80c2a919b7afe4a8ac35ae6811f18f6cf72dc5474872927759ffcdcc"} Nov 03 13:47:11.771715 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:11.771710 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5606cf0b-fkwr8" Nov 03 13:47:11.771942 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:11.771734 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a6386bd80c2a919b7afe4a8ac35ae6811f18f6cf72dc5474872927759ffcdcc" Nov 03 13:47:12.381599 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:12.381514 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr"] Nov 03 13:47:12.382168 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:12.382073 5979 cpu_manager.go:414] "containerMap: removing container" podUID="bb80b6d0-af59-468e-93a8-7ed4b40454ce" containerName="revoke-keys" Nov 03 13:47:12.382168 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:12.382086 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb80b6d0-af59-468e-93a8-7ed4b40454ce" containerName="revoke-keys" Nov 03 13:47:12.382168 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:12.382122 5979 cpu_manager.go:414] "containerMap: removing container" podUID="b26f627f-b967-45b9-80d3-847cc513bdfe" containerName="revoke-keys" Nov 03 13:47:12.382168 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:12.382129 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b26f627f-b967-45b9-80d3-847cc513bdfe" containerName="revoke-keys" Nov 03 13:47:12.382424 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:12.382227 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="b26f627f-b967-45b9-80d3-847cc513bdfe" containerName="revoke-keys" Nov 03 13:47:12.382424 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:12.382250 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb80b6d0-af59-468e-93a8-7ed4b40454ce" containerName="revoke-keys" Nov 03 13:47:12.386363 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:12.386341 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr" Nov 03 13:47:12.461029 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:12.460942 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-10c688-openshift-default-dockercfg-svvh7\"" Nov 03 13:47:12.494239 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:12.493511 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9af6a95-df9b-4525-a080-edf5abab2c86-workload-certs\") pod \"e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr\" (UID: \"f9af6a95-df9b-4525-a080-edf5abab2c86\") " pod="openshift-ingress/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr" Nov 03 13:47:12.494239 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:12.493601 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9af6a95-df9b-4525-a080-edf5abab2c86-istiod-ca-cert\") pod \"e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr\" (UID: \"f9af6a95-df9b-4525-a080-edf5abab2c86\") " pod="openshift-ingress/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr" Nov 03 13:47:12.494239 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:12.493699 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9af6a95-df9b-4525-a080-edf5abab2c86-workload-socket\") pod \"e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr\" (UID: \"f9af6a95-df9b-4525-a080-edf5abab2c86\") " pod="openshift-ingress/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr" Nov 03 13:47:12.494239 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:12.493739 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9af6a95-df9b-4525-a080-edf5abab2c86-istio-envoy\") pod \"e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr\" (UID: \"f9af6a95-df9b-4525-a080-edf5abab2c86\") " pod="openshift-ingress/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr" Nov 03 13:47:12.494239 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:12.493798 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9af6a95-df9b-4525-a080-edf5abab2c86-istio-podinfo\") pod \"e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr\" (UID: \"f9af6a95-df9b-4525-a080-edf5abab2c86\") " pod="openshift-ingress/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr" Nov 03 13:47:12.494239 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:12.493861 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqks7\" (UniqueName: \"kubernetes.io/projected/f9af6a95-df9b-4525-a080-edf5abab2c86-kube-api-access-hqks7\") pod \"e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr\" (UID: \"f9af6a95-df9b-4525-a080-edf5abab2c86\") " pod="openshift-ingress/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr" Nov 03 13:47:12.494239 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:12.493947 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9af6a95-df9b-4525-a080-edf5abab2c86-istio-token\") pod \"e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr\" (UID: \"f9af6a95-df9b-4525-a080-edf5abab2c86\") " pod="openshift-ingress/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr" Nov 03 13:47:12.494239 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:12.494018 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9af6a95-df9b-4525-a080-edf5abab2c86-istio-data\") pod \"e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr\" (UID: \"f9af6a95-df9b-4525-a080-edf5abab2c86\") " pod="openshift-ingress/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr" Nov 03 13:47:12.494239 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:12.494051 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9af6a95-df9b-4525-a080-edf5abab2c86-credential-socket\") pod \"e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr\" (UID: \"f9af6a95-df9b-4525-a080-edf5abab2c86\") " pod="openshift-ingress/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr" Nov 03 13:47:12.512550 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:12.512462 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr"] Nov 03 13:47:12.595239 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:12.595197 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9af6a95-df9b-4525-a080-edf5abab2c86-workload-certs\") pod \"e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr\" (UID: \"f9af6a95-df9b-4525-a080-edf5abab2c86\") " pod="openshift-ingress/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr" Nov 03 13:47:12.595434 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:12.595276 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9af6a95-df9b-4525-a080-edf5abab2c86-istiod-ca-cert\") pod \"e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr\" (UID: \"f9af6a95-df9b-4525-a080-edf5abab2c86\") " pod="openshift-ingress/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr" Nov 03 13:47:12.595434 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:12.595384 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9af6a95-df9b-4525-a080-edf5abab2c86-workload-socket\") pod \"e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr\" (UID: \"f9af6a95-df9b-4525-a080-edf5abab2c86\") " pod="openshift-ingress/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr" Nov 03 13:47:12.595434 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:12.595410 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9af6a95-df9b-4525-a080-edf5abab2c86-istio-envoy\") pod \"e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr\" (UID: \"f9af6a95-df9b-4525-a080-edf5abab2c86\") " pod="openshift-ingress/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr" Nov 03 13:47:12.595644 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:12.595440 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9af6a95-df9b-4525-a080-edf5abab2c86-istio-podinfo\") pod \"e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr\" (UID: \"f9af6a95-df9b-4525-a080-edf5abab2c86\") " pod="openshift-ingress/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr" Nov 03 13:47:12.595644 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:12.595538 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqks7\" (UniqueName: \"kubernetes.io/projected/f9af6a95-df9b-4525-a080-edf5abab2c86-kube-api-access-hqks7\") pod \"e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr\" (UID: \"f9af6a95-df9b-4525-a080-edf5abab2c86\") " pod="openshift-ingress/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr" Nov 03 13:47:12.595644 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:12.595627 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9af6a95-df9b-4525-a080-edf5abab2c86-istio-token\") pod \"e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr\" (UID: \"f9af6a95-df9b-4525-a080-edf5abab2c86\") " pod="openshift-ingress/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr" Nov 03 13:47:12.595802 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:12.595700 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9af6a95-df9b-4525-a080-edf5abab2c86-workload-certs\") pod \"e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr\" (UID: \"f9af6a95-df9b-4525-a080-edf5abab2c86\") " pod="openshift-ingress/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr" Nov 03 13:47:12.595802 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:12.595709 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9af6a95-df9b-4525-a080-edf5abab2c86-istio-data\") pod \"e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr\" (UID: \"f9af6a95-df9b-4525-a080-edf5abab2c86\") " pod="openshift-ingress/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr" Nov 03 13:47:12.595802 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:12.595736 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9af6a95-df9b-4525-a080-edf5abab2c86-workload-socket\") pod \"e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr\" (UID: \"f9af6a95-df9b-4525-a080-edf5abab2c86\") " pod="openshift-ingress/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr" Nov 03 13:47:12.595802 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:12.595761 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9af6a95-df9b-4525-a080-edf5abab2c86-credential-socket\") pod \"e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr\" (UID: \"f9af6a95-df9b-4525-a080-edf5abab2c86\") " pod="openshift-ingress/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr" Nov 03 13:47:12.596028 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:12.595999 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9af6a95-df9b-4525-a080-edf5abab2c86-istio-data\") pod \"e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr\" (UID: \"f9af6a95-df9b-4525-a080-edf5abab2c86\") " pod="openshift-ingress/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr" Nov 03 13:47:12.596121 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:12.596105 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9af6a95-df9b-4525-a080-edf5abab2c86-credential-socket\") pod \"e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr\" (UID: \"f9af6a95-df9b-4525-a080-edf5abab2c86\") " pod="openshift-ingress/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr" Nov 03 13:47:12.596176 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:12.596141 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9af6a95-df9b-4525-a080-edf5abab2c86-istiod-ca-cert\") pod \"e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr\" (UID: \"f9af6a95-df9b-4525-a080-edf5abab2c86\") " pod="openshift-ingress/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr" Nov 03 13:47:12.597829 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:12.597808 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9af6a95-df9b-4525-a080-edf5abab2c86-istio-envoy\") pod \"e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr\" (UID: \"f9af6a95-df9b-4525-a080-edf5abab2c86\") " pod="openshift-ingress/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr" Nov 03 13:47:12.598054 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:12.598029 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9af6a95-df9b-4525-a080-edf5abab2c86-istio-podinfo\") pod \"e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr\" (UID: \"f9af6a95-df9b-4525-a080-edf5abab2c86\") " pod="openshift-ingress/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr" Nov 03 13:47:12.655029 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:12.654484 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqks7\" (UniqueName: \"kubernetes.io/projected/f9af6a95-df9b-4525-a080-edf5abab2c86-kube-api-access-hqks7\") pod \"e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr\" (UID: \"f9af6a95-df9b-4525-a080-edf5abab2c86\") " pod="openshift-ingress/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr" Nov 03 13:47:12.655972 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:12.655938 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9af6a95-df9b-4525-a080-edf5abab2c86-istio-token\") pod \"e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr\" (UID: \"f9af6a95-df9b-4525-a080-edf5abab2c86\") " pod="openshift-ingress/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr" Nov 03 13:47:12.703596 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:12.703560 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr" Nov 03 13:47:13.144880 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:13.144495 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr"] Nov 03 13:47:13.149016 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:47:13.148984 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9af6a95_df9b_4525_a080_edf5abab2c86.slice/crio-f48685f66853591fc268801f156628057fe5e40fa2167c0de75259ec4556a64b WatchSource:0}: Error finding container f48685f66853591fc268801f156628057fe5e40fa2167c0de75259ec4556a64b: Status 404 returned error can't find the container with id f48685f66853591fc268801f156628057fe5e40fa2167c0de75259ec4556a64b Nov 03 13:47:13.151560 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:13.151525 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:47:13.151665 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:13.151605 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:47:13.151665 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:13.151647 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:47:13.787341 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:13.787300 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr" event={"ID":"f9af6a95-df9b-4525-a080-edf5abab2c86","Type":"ContainerStarted","Data":"ff8c0f43d62843b89727cd62c24862fa2a5c58550afe7ed0db5b1f36d43e6d75"} Nov 03 13:47:13.787852 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:13.787363 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr" event={"ID":"f9af6a95-df9b-4525-a080-edf5abab2c86","Type":"ContainerStarted","Data":"f48685f66853591fc268801f156628057fe5e40fa2167c0de75259ec4556a64b"} Nov 03 13:47:13.897927 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:13.897871 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr" podStartSLOduration=1.897856405 podStartE2EDuration="1.897856405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:47:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:47:13.896817188 +0000 UTC m=+2515.907505430" watchObservedRunningTime="2026-11-03 13:47:13.897856405 +0000 UTC m=+2515.908544720" Nov 03 13:47:14.704748 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:14.704710 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr" Nov 03 13:47:14.710007 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:14.709982 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr" Nov 03 13:47:14.795537 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:14.795505 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr" Nov 03 13:47:14.796673 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:14.796652 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr" Nov 03 13:47:19.948370 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:19.948335 5979 scope.go:122] "RemoveContainer" containerID="dcc5f881440c295b18c0d52ff88aa73887e90de29ee1509c7cda4e4b1a9a91db" Nov 03 13:47:21.167334 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:21.167296 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-10c688-775c68b957-8lm7w"] Nov 03 13:47:21.173899 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:21.173874 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-10c688-775c68b957-8lm7w" Nov 03 13:47:21.223695 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:21.223658 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-10c688-775c68b957-8lm7w"] Nov 03 13:47:21.266824 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:21.266786 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-10c688\"" Nov 03 13:47:21.267001 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:21.266853 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-10c688-dockercfg-hg92w\"" Nov 03 13:47:21.323449 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:21.323403 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcsqv\" (UniqueName: \"kubernetes.io/projected/2ba80e36-d422-4942-b385-c0910c848a82-kube-api-access-tcsqv\") pod \"payload-processing-e2e-praxis-b-10c688-775c68b957-8lm7w\" (UID: \"2ba80e36-d422-4942-b385-c0910c848a82\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-10c688-775c68b957-8lm7w" Nov 03 13:47:21.323661 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:21.323592 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ba80e36-d422-4942-b385-c0910c848a82-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-10c688-775c68b957-8lm7w\" (UID: \"2ba80e36-d422-4942-b385-c0910c848a82\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-10c688-775c68b957-8lm7w" Nov 03 13:47:21.424359 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:21.424259 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ba80e36-d422-4942-b385-c0910c848a82-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-10c688-775c68b957-8lm7w\" (UID: \"2ba80e36-d422-4942-b385-c0910c848a82\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-10c688-775c68b957-8lm7w" Nov 03 13:47:21.424524 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:21.424402 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcsqv\" (UniqueName: \"kubernetes.io/projected/2ba80e36-d422-4942-b385-c0910c848a82-kube-api-access-tcsqv\") pod \"payload-processing-e2e-praxis-b-10c688-775c68b957-8lm7w\" (UID: \"2ba80e36-d422-4942-b385-c0910c848a82\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-10c688-775c68b957-8lm7w" Nov 03 13:47:21.425184 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:21.425162 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ba80e36-d422-4942-b385-c0910c848a82-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-10c688-775c68b957-8lm7w\" (UID: \"2ba80e36-d422-4942-b385-c0910c848a82\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-10c688-775c68b957-8lm7w" Nov 03 13:47:21.477901 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:21.477868 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcsqv\" (UniqueName: \"kubernetes.io/projected/2ba80e36-d422-4942-b385-c0910c848a82-kube-api-access-tcsqv\") pod \"payload-processing-e2e-praxis-b-10c688-775c68b957-8lm7w\" (UID: \"2ba80e36-d422-4942-b385-c0910c848a82\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-10c688-775c68b957-8lm7w" Nov 03 13:47:21.487754 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:21.487725 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-10c688-775c68b957-8lm7w" Nov 03 13:47:21.865602 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:21.865516 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-10c688-775c68b957-8lm7w"] Nov 03 13:47:21.870985 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:47:21.870958 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ba80e36_d422_4942_b385_c0910c848a82.slice/crio-74730b41eeec5fc0de98dd06cf29f9eb5a5f43c8eb261a23e5029ef04779b38c WatchSource:0}: Error finding container 74730b41eeec5fc0de98dd06cf29f9eb5a5f43c8eb261a23e5029ef04779b38c: Status 404 returned error can't find the container with id 74730b41eeec5fc0de98dd06cf29f9eb5a5f43c8eb261a23e5029ef04779b38c Nov 03 13:47:22.876375 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:22.876329 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-10c688-775c68b957-8lm7w" event={"ID":"2ba80e36-d422-4942-b385-c0910c848a82","Type":"ContainerStarted","Data":"4e612024cd9730859b4ac14282b9e3593ea2c55764e403c6704b02e3899f7e51"} Nov 03 13:47:22.876929 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:22.876408 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-10c688-775c68b957-8lm7w" event={"ID":"2ba80e36-d422-4942-b385-c0910c848a82","Type":"ContainerStarted","Data":"74730b41eeec5fc0de98dd06cf29f9eb5a5f43c8eb261a23e5029ef04779b38c"} Nov 03 13:47:22.876929 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:22.876603 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-10c688-775c68b957-8lm7w" Nov 03 13:47:22.988785 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:22.988724 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-10c688-775c68b957-8lm7w" podStartSLOduration=1.9887067950000001 podStartE2EDuration="1.988706795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:47:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:47:22.985254937 +0000 UTC m=+2524.995943180" watchObservedRunningTime="2026-11-03 13:47:22.988706795 +0000 UTC m=+2524.999395036" Nov 03 13:47:30.494831 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:30.494785 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a94fa233-wl58w"] Nov 03 13:47:30.500414 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:30.500389 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a94fa233-wl58w" Nov 03 13:47:30.553155 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:30.553120 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a94fa233-wl58w"] Nov 03 13:47:30.558859 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:30.558830 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbpmh\" (UniqueName: \"kubernetes.io/projected/ecf690af-2ffa-482c-883c-7f4df7da17e9-kube-api-access-sbpmh\") pod \"maas-api-revoke-sub-a94fa233-wl58w\" (UID: \"ecf690af-2ffa-482c-883c-7f4df7da17e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a94fa233-wl58w" Nov 03 13:47:30.559032 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:30.558992 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecf690af-2ffa-482c-883c-7f4df7da17e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a94fa233-wl58w\" (UID: \"ecf690af-2ffa-482c-883c-7f4df7da17e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a94fa233-wl58w" Nov 03 13:47:30.559153 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:30.559138 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecf690af-2ffa-482c-883c-7f4df7da17e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a94fa233-wl58w\" (UID: \"ecf690af-2ffa-482c-883c-7f4df7da17e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a94fa233-wl58w" Nov 03 13:47:30.595591 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:30.595550 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:47:30.595776 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:30.595675 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:47:30.660325 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:30.660283 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecf690af-2ffa-482c-883c-7f4df7da17e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a94fa233-wl58w\" (UID: \"ecf690af-2ffa-482c-883c-7f4df7da17e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a94fa233-wl58w" Nov 03 13:47:30.660547 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:30.660399 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecf690af-2ffa-482c-883c-7f4df7da17e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a94fa233-wl58w\" (UID: \"ecf690af-2ffa-482c-883c-7f4df7da17e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a94fa233-wl58w" Nov 03 13:47:30.660547 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:30.660442 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbpmh\" (UniqueName: \"kubernetes.io/projected/ecf690af-2ffa-482c-883c-7f4df7da17e9-kube-api-access-sbpmh\") pod \"maas-api-revoke-sub-a94fa233-wl58w\" (UID: \"ecf690af-2ffa-482c-883c-7f4df7da17e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a94fa233-wl58w" Nov 03 13:47:30.661073 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:30.661050 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecf690af-2ffa-482c-883c-7f4df7da17e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a94fa233-wl58w\" (UID: \"ecf690af-2ffa-482c-883c-7f4df7da17e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a94fa233-wl58w" Nov 03 13:47:30.662712 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:30.662692 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecf690af-2ffa-482c-883c-7f4df7da17e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a94fa233-wl58w\" (UID: \"ecf690af-2ffa-482c-883c-7f4df7da17e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a94fa233-wl58w" Nov 03 13:47:30.714883 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:30.714848 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbpmh\" (UniqueName: \"kubernetes.io/projected/ecf690af-2ffa-482c-883c-7f4df7da17e9-kube-api-access-sbpmh\") pod \"maas-api-revoke-sub-a94fa233-wl58w\" (UID: \"ecf690af-2ffa-482c-883c-7f4df7da17e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a94fa233-wl58w" Nov 03 13:47:30.814259 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:30.814172 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a94fa233-wl58w" Nov 03 13:47:31.186673 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:31.186635 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a94fa233-wl58w"] Nov 03 13:47:31.192961 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:47:31.192931 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecf690af_2ffa_482c_883c_7f4df7da17e9.slice/crio-1af57b3c10083c4d564de93a1bc778e55e979e736340c3531efe1ef3a2fb6fb4 WatchSource:0}: Error finding container 1af57b3c10083c4d564de93a1bc778e55e979e736340c3531efe1ef3a2fb6fb4: Status 404 returned error can't find the container with id 1af57b3c10083c4d564de93a1bc778e55e979e736340c3531efe1ef3a2fb6fb4 Nov 03 13:47:31.943396 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:31.943358 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a94fa233-wl58w" event={"ID":"ecf690af-2ffa-482c-883c-7f4df7da17e9","Type":"ContainerStarted","Data":"be7ff0f1196ea421b2441046129310b141803d33079d3759406bfed84aaf4fea"} Nov 03 13:47:31.943920 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:31.943415 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a94fa233-wl58w" event={"ID":"ecf690af-2ffa-482c-883c-7f4df7da17e9","Type":"ContainerStarted","Data":"1af57b3c10083c4d564de93a1bc778e55e979e736340c3531efe1ef3a2fb6fb4"} Nov 03 13:47:32.053873 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:32.053815 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a94fa233-wl58w" podStartSLOduration=2.053799481 podStartE2EDuration="2.053799481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:47:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:47:32.051718126 +0000 UTC m=+2534.062406369" watchObservedRunningTime="2026-11-03 13:47:32.053799481 +0000 UTC m=+2534.064487723" Nov 03 13:47:32.952067 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:32.952031 5979 generic.go:356] "Generic (PLEG): container finished" podID="ecf690af-2ffa-482c-883c-7f4df7da17e9" containerID="be7ff0f1196ea421b2441046129310b141803d33079d3759406bfed84aaf4fea" exitCode=0 Nov 03 13:47:32.952500 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:32.952117 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a94fa233-wl58w" event={"ID":"ecf690af-2ffa-482c-883c-7f4df7da17e9","Type":"ContainerDied","Data":"be7ff0f1196ea421b2441046129310b141803d33079d3759406bfed84aaf4fea"} Nov 03 13:47:33.891983 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:33.891946 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-10c688-775c68b957-8lm7w" Nov 03 13:47:34.116252 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:34.116222 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a94fa233-wl58w" Nov 03 13:47:34.201798 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:34.201709 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecf690af-2ffa-482c-883c-7f4df7da17e9-kube-api-access-sbpmh\" (UniqueName: \"kubernetes.io/projected/ecf690af-2ffa-482c-883c-7f4df7da17e9-kube-api-access-sbpmh\") pod \"ecf690af-2ffa-482c-883c-7f4df7da17e9\" (UID: \"ecf690af-2ffa-482c-883c-7f4df7da17e9\") " Nov 03 13:47:34.201972 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:34.201892 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecf690af-2ffa-482c-883c-7f4df7da17e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecf690af-2ffa-482c-883c-7f4df7da17e9-maas-api-xks-ca\") pod \"ecf690af-2ffa-482c-883c-7f4df7da17e9\" (UID: \"ecf690af-2ffa-482c-883c-7f4df7da17e9\") " Nov 03 13:47:34.201972 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:34.201943 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecf690af-2ffa-482c-883c-7f4df7da17e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecf690af-2ffa-482c-883c-7f4df7da17e9-maas-api-openshift-ca\") pod \"ecf690af-2ffa-482c-883c-7f4df7da17e9\" (UID: \"ecf690af-2ffa-482c-883c-7f4df7da17e9\") " Nov 03 13:47:34.202345 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:34.202316 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecf690af-2ffa-482c-883c-7f4df7da17e9-maas-api-openshift-ca" pod "ecf690af-2ffa-482c-883c-7f4df7da17e9" (UID: "ecf690af-2ffa-482c-883c-7f4df7da17e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:47:34.204020 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:34.203996 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecf690af-2ffa-482c-883c-7f4df7da17e9-kube-api-access-sbpmh" pod "ecf690af-2ffa-482c-883c-7f4df7da17e9" (UID: "ecf690af-2ffa-482c-883c-7f4df7da17e9"). InnerVolumeSpecName "kube-api-access-sbpmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:47:34.204110 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:34.204044 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecf690af-2ffa-482c-883c-7f4df7da17e9-maas-api-xks-ca" pod "ecf690af-2ffa-482c-883c-7f4df7da17e9" (UID: "ecf690af-2ffa-482c-883c-7f4df7da17e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:47:34.303824 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:34.303789 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecf690af-2ffa-482c-883c-7f4df7da17e9-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:47:34.303824 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:34.303818 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbpmh\" (UniqueName: \"kubernetes.io/projected/ecf690af-2ffa-482c-883c-7f4df7da17e9-kube-api-access-sbpmh\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:47:34.303824 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:34.303828 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecf690af-2ffa-482c-883c-7f4df7da17e9-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:47:34.966811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:34.966779 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a94fa233-wl58w" Nov 03 13:47:34.966999 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:34.966818 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a94fa233-wl58w" event={"ID":"ecf690af-2ffa-482c-883c-7f4df7da17e9","Type":"ContainerDied","Data":"1af57b3c10083c4d564de93a1bc778e55e979e736340c3531efe1ef3a2fb6fb4"} Nov 03 13:47:34.966999 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:34.966866 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1af57b3c10083c4d564de93a1bc778e55e979e736340c3531efe1ef3a2fb6fb4" Nov 03 13:47:37.234675 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:47:37.234637 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:47:43.872085 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:43.872048 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82e848cb-n96lb"] Nov 03 13:47:43.872903 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:43.872844 5979 cpu_manager.go:414] "containerMap: removing container" podUID="ecf690af-2ffa-482c-883c-7f4df7da17e9" containerName="revoke-keys" Nov 03 13:47:43.872903 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:43.872867 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecf690af-2ffa-482c-883c-7f4df7da17e9" containerName="revoke-keys" Nov 03 13:47:43.873109 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:43.873037 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecf690af-2ffa-482c-883c-7f4df7da17e9" containerName="revoke-keys" Nov 03 13:47:43.877779 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:43.877757 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e848cb-n96lb" Nov 03 13:47:43.922222 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:43.922191 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r9bd\" (UniqueName: \"kubernetes.io/projected/abcb30e6-16a3-48a2-ab22-2c524ba95084-kube-api-access-7r9bd\") pod \"maas-api-revoke-sub-82e848cb-n96lb\" (UID: \"abcb30e6-16a3-48a2-ab22-2c524ba95084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e848cb-n96lb" Nov 03 13:47:43.922385 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:43.922324 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abcb30e6-16a3-48a2-ab22-2c524ba95084-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82e848cb-n96lb\" (UID: \"abcb30e6-16a3-48a2-ab22-2c524ba95084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e848cb-n96lb" Nov 03 13:47:43.922438 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:43.922422 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abcb30e6-16a3-48a2-ab22-2c524ba95084-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82e848cb-n96lb\" (UID: \"abcb30e6-16a3-48a2-ab22-2c524ba95084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e848cb-n96lb" Nov 03 13:47:43.930188 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:43.930158 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82e848cb-n96lb"] Nov 03 13:47:43.972755 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:43.972717 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:47:43.972998 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:43.972978 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:47:44.024019 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:44.023991 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r9bd\" (UniqueName: \"kubernetes.io/projected/abcb30e6-16a3-48a2-ab22-2c524ba95084-kube-api-access-7r9bd\") pod \"maas-api-revoke-sub-82e848cb-n96lb\" (UID: \"abcb30e6-16a3-48a2-ab22-2c524ba95084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e848cb-n96lb" Nov 03 13:47:44.024213 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:44.024072 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abcb30e6-16a3-48a2-ab22-2c524ba95084-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82e848cb-n96lb\" (UID: \"abcb30e6-16a3-48a2-ab22-2c524ba95084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e848cb-n96lb" Nov 03 13:47:44.024213 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:44.024136 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abcb30e6-16a3-48a2-ab22-2c524ba95084-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82e848cb-n96lb\" (UID: \"abcb30e6-16a3-48a2-ab22-2c524ba95084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e848cb-n96lb" Nov 03 13:47:44.024829 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:44.024799 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abcb30e6-16a3-48a2-ab22-2c524ba95084-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82e848cb-n96lb\" (UID: \"abcb30e6-16a3-48a2-ab22-2c524ba95084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e848cb-n96lb" Nov 03 13:47:44.026548 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:44.026523 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abcb30e6-16a3-48a2-ab22-2c524ba95084-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82e848cb-n96lb\" (UID: \"abcb30e6-16a3-48a2-ab22-2c524ba95084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e848cb-n96lb" Nov 03 13:47:44.077568 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:44.077534 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r9bd\" (UniqueName: \"kubernetes.io/projected/abcb30e6-16a3-48a2-ab22-2c524ba95084-kube-api-access-7r9bd\") pod \"maas-api-revoke-sub-82e848cb-n96lb\" (UID: \"abcb30e6-16a3-48a2-ab22-2c524ba95084\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e848cb-n96lb" Nov 03 13:47:44.191027 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:44.190979 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e848cb-n96lb" Nov 03 13:47:44.564217 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:44.564126 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82e848cb-n96lb"] Nov 03 13:47:44.571736 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:47:44.571709 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabcb30e6_16a3_48a2_ab22_2c524ba95084.slice/crio-8c2c40d541bb5812acadbceb7e900ca16763dadd1a1ad0ecbdeb7c54a91e6431 WatchSource:0}: Error finding container 8c2c40d541bb5812acadbceb7e900ca16763dadd1a1ad0ecbdeb7c54a91e6431: Status 404 returned error can't find the container with id 8c2c40d541bb5812acadbceb7e900ca16763dadd1a1ad0ecbdeb7c54a91e6431 Nov 03 13:47:45.035105 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:45.035071 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e848cb-n96lb" event={"ID":"abcb30e6-16a3-48a2-ab22-2c524ba95084","Type":"ContainerStarted","Data":"a4fd78ebdfe8de9cfb24a8769e5439855d154c6e9379144ddaec022448864fa4"} Nov 03 13:47:45.035575 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:45.035132 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e848cb-n96lb" event={"ID":"abcb30e6-16a3-48a2-ab22-2c524ba95084","Type":"ContainerStarted","Data":"8c2c40d541bb5812acadbceb7e900ca16763dadd1a1ad0ecbdeb7c54a91e6431"} Nov 03 13:47:45.148194 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:45.148141 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e848cb-n96lb" podStartSLOduration=2.148126945 podStartE2EDuration="2.148126945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:47:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:47:45.144626481 +0000 UTC m=+2547.155314762" watchObservedRunningTime="2026-11-03 13:47:45.148126945 +0000 UTC m=+2547.158815186" Nov 03 13:47:45.274130 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:45.274086 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:47:46.046530 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:46.046466 5979 generic.go:356] "Generic (PLEG): container finished" podID="abcb30e6-16a3-48a2-ab22-2c524ba95084" containerID="a4fd78ebdfe8de9cfb24a8769e5439855d154c6e9379144ddaec022448864fa4" exitCode=0 Nov 03 13:47:46.047159 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:46.046559 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e848cb-n96lb" event={"ID":"abcb30e6-16a3-48a2-ab22-2c524ba95084","Type":"ContainerDied","Data":"a4fd78ebdfe8de9cfb24a8769e5439855d154c6e9379144ddaec022448864fa4"} Nov 03 13:47:47.325219 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:47.325149 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e848cb-n96lb" Nov 03 13:47:47.476690 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:47.476660 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abcb30e6-16a3-48a2-ab22-2c524ba95084-kube-api-access-7r9bd\" (UniqueName: \"kubernetes.io/projected/abcb30e6-16a3-48a2-ab22-2c524ba95084-kube-api-access-7r9bd\") pod \"abcb30e6-16a3-48a2-ab22-2c524ba95084\" (UID: \"abcb30e6-16a3-48a2-ab22-2c524ba95084\") " Nov 03 13:47:47.476912 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:47.476715 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abcb30e6-16a3-48a2-ab22-2c524ba95084-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abcb30e6-16a3-48a2-ab22-2c524ba95084-maas-api-xks-ca\") pod \"abcb30e6-16a3-48a2-ab22-2c524ba95084\" (UID: \"abcb30e6-16a3-48a2-ab22-2c524ba95084\") " Nov 03 13:47:47.476912 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:47.476826 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abcb30e6-16a3-48a2-ab22-2c524ba95084-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abcb30e6-16a3-48a2-ab22-2c524ba95084-maas-api-openshift-ca\") pod \"abcb30e6-16a3-48a2-ab22-2c524ba95084\" (UID: \"abcb30e6-16a3-48a2-ab22-2c524ba95084\") " Nov 03 13:47:47.477277 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:47.477245 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abcb30e6-16a3-48a2-ab22-2c524ba95084-maas-api-openshift-ca" pod "abcb30e6-16a3-48a2-ab22-2c524ba95084" (UID: "abcb30e6-16a3-48a2-ab22-2c524ba95084"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:47:47.477402 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:47.477359 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abcb30e6-16a3-48a2-ab22-2c524ba95084-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:47:47.478913 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:47.478885 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abcb30e6-16a3-48a2-ab22-2c524ba95084-maas-api-xks-ca" pod "abcb30e6-16a3-48a2-ab22-2c524ba95084" (UID: "abcb30e6-16a3-48a2-ab22-2c524ba95084"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:47:47.479381 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:47.479356 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abcb30e6-16a3-48a2-ab22-2c524ba95084-kube-api-access-7r9bd" pod "abcb30e6-16a3-48a2-ab22-2c524ba95084" (UID: "abcb30e6-16a3-48a2-ab22-2c524ba95084"). InnerVolumeSpecName "kube-api-access-7r9bd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:47:47.578943 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:47.578854 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r9bd\" (UniqueName: \"kubernetes.io/projected/abcb30e6-16a3-48a2-ab22-2c524ba95084-kube-api-access-7r9bd\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:47:47.578943 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:47.578892 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abcb30e6-16a3-48a2-ab22-2c524ba95084-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:47:48.077810 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:48.077766 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e848cb-n96lb" Nov 03 13:47:48.077810 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:48.077786 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82e848cb-n96lb" event={"ID":"abcb30e6-16a3-48a2-ab22-2c524ba95084","Type":"ContainerDied","Data":"8c2c40d541bb5812acadbceb7e900ca16763dadd1a1ad0ecbdeb7c54a91e6431"} Nov 03 13:47:48.078026 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:48.077839 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c2c40d541bb5812acadbceb7e900ca16763dadd1a1ad0ecbdeb7c54a91e6431" Nov 03 13:47:49.886798 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:49.886757 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae12be2d-sgcvn"] Nov 03 13:47:49.887544 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:49.887523 5979 cpu_manager.go:414] "containerMap: removing container" podUID="abcb30e6-16a3-48a2-ab22-2c524ba95084" containerName="revoke-keys" Nov 03 13:47:49.887544 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:49.887544 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abcb30e6-16a3-48a2-ab22-2c524ba95084" containerName="revoke-keys" Nov 03 13:47:49.887767 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:49.887753 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="abcb30e6-16a3-48a2-ab22-2c524ba95084" containerName="revoke-keys" Nov 03 13:47:49.891591 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:49.891570 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae12be2d-sgcvn" Nov 03 13:47:49.944074 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:49.944041 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae12be2d-sgcvn"] Nov 03 13:47:49.986657 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:49.986625 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:47:49.986870 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:49.986642 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:47:50.006872 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:50.006830 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csglq\" (UniqueName: \"kubernetes.io/projected/44b0445d-b9fe-460f-968b-9e4a950ff35a-kube-api-access-csglq\") pod \"maas-api-revoke-sub-ae12be2d-sgcvn\" (UID: \"44b0445d-b9fe-460f-968b-9e4a950ff35a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae12be2d-sgcvn" Nov 03 13:47:50.007040 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:50.006976 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44b0445d-b9fe-460f-968b-9e4a950ff35a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae12be2d-sgcvn\" (UID: \"44b0445d-b9fe-460f-968b-9e4a950ff35a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae12be2d-sgcvn" Nov 03 13:47:50.007143 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:50.007128 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44b0445d-b9fe-460f-968b-9e4a950ff35a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae12be2d-sgcvn\" (UID: \"44b0445d-b9fe-460f-968b-9e4a950ff35a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae12be2d-sgcvn" Nov 03 13:47:50.108959 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:50.108925 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44b0445d-b9fe-460f-968b-9e4a950ff35a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae12be2d-sgcvn\" (UID: \"44b0445d-b9fe-460f-968b-9e4a950ff35a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae12be2d-sgcvn" Nov 03 13:47:50.109141 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:50.108989 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csglq\" (UniqueName: \"kubernetes.io/projected/44b0445d-b9fe-460f-968b-9e4a950ff35a-kube-api-access-csglq\") pod \"maas-api-revoke-sub-ae12be2d-sgcvn\" (UID: \"44b0445d-b9fe-460f-968b-9e4a950ff35a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae12be2d-sgcvn" Nov 03 13:47:50.109141 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:50.109056 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44b0445d-b9fe-460f-968b-9e4a950ff35a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae12be2d-sgcvn\" (UID: \"44b0445d-b9fe-460f-968b-9e4a950ff35a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae12be2d-sgcvn" Nov 03 13:47:50.109773 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:50.109744 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44b0445d-b9fe-460f-968b-9e4a950ff35a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae12be2d-sgcvn\" (UID: \"44b0445d-b9fe-460f-968b-9e4a950ff35a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae12be2d-sgcvn" Nov 03 13:47:50.111379 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:50.111357 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44b0445d-b9fe-460f-968b-9e4a950ff35a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae12be2d-sgcvn\" (UID: \"44b0445d-b9fe-460f-968b-9e4a950ff35a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae12be2d-sgcvn" Nov 03 13:47:50.163674 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:50.163604 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csglq\" (UniqueName: \"kubernetes.io/projected/44b0445d-b9fe-460f-968b-9e4a950ff35a-kube-api-access-csglq\") pod \"maas-api-revoke-sub-ae12be2d-sgcvn\" (UID: \"44b0445d-b9fe-460f-968b-9e4a950ff35a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae12be2d-sgcvn" Nov 03 13:47:50.205067 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:50.205029 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae12be2d-sgcvn" Nov 03 13:47:50.584502 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:50.584388 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae12be2d-sgcvn"] Nov 03 13:47:50.591567 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:47:50.591529 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44b0445d_b9fe_460f_968b_9e4a950ff35a.slice/crio-bfac6b12fbbd4a731f05995bf7c5eecb62de1cc062b7bedafd855d0f6e6301d4 WatchSource:0}: Error finding container bfac6b12fbbd4a731f05995bf7c5eecb62de1cc062b7bedafd855d0f6e6301d4: Status 404 returned error can't find the container with id bfac6b12fbbd4a731f05995bf7c5eecb62de1cc062b7bedafd855d0f6e6301d4 Nov 03 13:47:51.098511 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.098445 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae12be2d-sgcvn" event={"ID":"44b0445d-b9fe-460f-968b-9e4a950ff35a","Type":"ContainerStarted","Data":"f8209fab595619d9b798f11adbb4023946a28f69c8c9cfc88cc0a67620aeda42"} Nov 03 13:47:51.098969 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.098530 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae12be2d-sgcvn" event={"ID":"44b0445d-b9fe-460f-968b-9e4a950ff35a","Type":"ContainerStarted","Data":"bfac6b12fbbd4a731f05995bf7c5eecb62de1cc062b7bedafd855d0f6e6301d4"} Nov 03 13:47:51.205970 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.205912 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae12be2d-sgcvn" podStartSLOduration=2.205896746 podStartE2EDuration="2.205896746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:47:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:47:51.205359525 +0000 UTC m=+2553.216047767" watchObservedRunningTime="2026-11-03 13:47:51.205896746 +0000 UTC m=+2553.216584989" Nov 03 13:47:51.457602 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.457572 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6"] Nov 03 13:47:51.462038 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.462017 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6" Nov 03 13:47:51.512808 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.512766 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-42c707-openshift-default-dockercfg-7t7mv\"" Nov 03 13:47:51.519835 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.519808 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6"] Nov 03 13:47:51.571799 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.571760 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m"] Nov 03 13:47:51.576277 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.576251 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m" Nov 03 13:47:51.600511 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.600452 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz"] Nov 03 13:47:51.605113 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.605089 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz" Nov 03 13:47:51.635280 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.635240 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c837757-208a-4cbe-a6b7-2b0014ae29bf-istio-podinfo\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6\" (UID: \"0c837757-208a-4cbe-a6b7-2b0014ae29bf\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6" Nov 03 13:47:51.635564 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.635400 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c837757-208a-4cbe-a6b7-2b0014ae29bf-istio-data\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6\" (UID: \"0c837757-208a-4cbe-a6b7-2b0014ae29bf\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6" Nov 03 13:47:51.635564 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.635435 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdcf9\" (UniqueName: \"kubernetes.io/projected/0c837757-208a-4cbe-a6b7-2b0014ae29bf-kube-api-access-bdcf9\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6\" (UID: \"0c837757-208a-4cbe-a6b7-2b0014ae29bf\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6" Nov 03 13:47:51.635564 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.635463 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c837757-208a-4cbe-a6b7-2b0014ae29bf-workload-certs\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6\" (UID: \"0c837757-208a-4cbe-a6b7-2b0014ae29bf\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6" Nov 03 13:47:51.635564 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.635519 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m"] Nov 03 13:47:51.635846 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.635586 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c837757-208a-4cbe-a6b7-2b0014ae29bf-istiod-ca-cert\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6\" (UID: \"0c837757-208a-4cbe-a6b7-2b0014ae29bf\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6" Nov 03 13:47:51.635846 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.635646 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c837757-208a-4cbe-a6b7-2b0014ae29bf-istio-envoy\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6\" (UID: \"0c837757-208a-4cbe-a6b7-2b0014ae29bf\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6" Nov 03 13:47:51.635846 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.635766 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c837757-208a-4cbe-a6b7-2b0014ae29bf-credential-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6\" (UID: \"0c837757-208a-4cbe-a6b7-2b0014ae29bf\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6" Nov 03 13:47:51.635846 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.635804 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c837757-208a-4cbe-a6b7-2b0014ae29bf-workload-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6\" (UID: \"0c837757-208a-4cbe-a6b7-2b0014ae29bf\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6" Nov 03 13:47:51.635846 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.635835 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c837757-208a-4cbe-a6b7-2b0014ae29bf-istio-token\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6\" (UID: \"0c837757-208a-4cbe-a6b7-2b0014ae29bf\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6" Nov 03 13:47:51.668072 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.667950 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz"] Nov 03 13:47:51.736788 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.736704 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79ff3314-693d-415c-a02d-b249b9a4748c-istio-podinfo\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m\" (UID: \"79ff3314-693d-415c-a02d-b249b9a4748c\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m" Nov 03 13:47:51.736788 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.736744 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b95b7f2-8b31-487f-ba23-b551e9406cc8-istiod-ca-cert\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz\" (UID: \"0b95b7f2-8b31-487f-ba23-b551e9406cc8\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz" Nov 03 13:47:51.736788 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.736773 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79ff3314-693d-415c-a02d-b249b9a4748c-istio-envoy\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m\" (UID: \"79ff3314-693d-415c-a02d-b249b9a4748c\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m" Nov 03 13:47:51.737019 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.736877 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c837757-208a-4cbe-a6b7-2b0014ae29bf-istio-data\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6\" (UID: \"0c837757-208a-4cbe-a6b7-2b0014ae29bf\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6" Nov 03 13:47:51.737019 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.736900 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdcf9\" (UniqueName: \"kubernetes.io/projected/0c837757-208a-4cbe-a6b7-2b0014ae29bf-kube-api-access-bdcf9\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6\" (UID: \"0c837757-208a-4cbe-a6b7-2b0014ae29bf\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6" Nov 03 13:47:51.737019 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.736920 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c837757-208a-4cbe-a6b7-2b0014ae29bf-workload-certs\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6\" (UID: \"0c837757-208a-4cbe-a6b7-2b0014ae29bf\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6" Nov 03 13:47:51.737019 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.736953 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ff3314-693d-415c-a02d-b249b9a4748c-credential-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m\" (UID: \"79ff3314-693d-415c-a02d-b249b9a4748c\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m" Nov 03 13:47:51.737019 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.736989 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b95b7f2-8b31-487f-ba23-b551e9406cc8-istio-data\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz\" (UID: \"0b95b7f2-8b31-487f-ba23-b551e9406cc8\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz" Nov 03 13:47:51.737019 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.737015 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c837757-208a-4cbe-a6b7-2b0014ae29bf-istiod-ca-cert\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6\" (UID: \"0c837757-208a-4cbe-a6b7-2b0014ae29bf\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6" Nov 03 13:47:51.737301 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.737048 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b95b7f2-8b31-487f-ba23-b551e9406cc8-workload-certs\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz\" (UID: \"0b95b7f2-8b31-487f-ba23-b551e9406cc8\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz" Nov 03 13:47:51.737301 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.737279 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c837757-208a-4cbe-a6b7-2b0014ae29bf-istio-envoy\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6\" (UID: \"0c837757-208a-4cbe-a6b7-2b0014ae29bf\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6" Nov 03 13:47:51.737430 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.737306 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c837757-208a-4cbe-a6b7-2b0014ae29bf-istio-data\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6\" (UID: \"0c837757-208a-4cbe-a6b7-2b0014ae29bf\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6" Nov 03 13:47:51.737430 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.737344 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79ff3314-693d-415c-a02d-b249b9a4748c-workload-certs\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m\" (UID: \"79ff3314-693d-415c-a02d-b249b9a4748c\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m" Nov 03 13:47:51.737430 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.737292 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c837757-208a-4cbe-a6b7-2b0014ae29bf-workload-certs\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6\" (UID: \"0c837757-208a-4cbe-a6b7-2b0014ae29bf\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6" Nov 03 13:47:51.737430 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.737383 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c837757-208a-4cbe-a6b7-2b0014ae29bf-credential-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6\" (UID: \"0c837757-208a-4cbe-a6b7-2b0014ae29bf\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6" Nov 03 13:47:51.737691 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.737491 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c837757-208a-4cbe-a6b7-2b0014ae29bf-workload-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6\" (UID: \"0c837757-208a-4cbe-a6b7-2b0014ae29bf\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6" Nov 03 13:47:51.737691 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.737547 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c837757-208a-4cbe-a6b7-2b0014ae29bf-istio-token\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6\" (UID: \"0c837757-208a-4cbe-a6b7-2b0014ae29bf\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6" Nov 03 13:47:51.737691 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.737583 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79ff3314-693d-415c-a02d-b249b9a4748c-istio-data\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m\" (UID: \"79ff3314-693d-415c-a02d-b249b9a4748c\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m" Nov 03 13:47:51.737691 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.737625 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b95b7f2-8b31-487f-ba23-b551e9406cc8-workload-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz\" (UID: \"0b95b7f2-8b31-487f-ba23-b551e9406cc8\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz" Nov 03 13:47:51.737691 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.737657 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79ff3314-693d-415c-a02d-b249b9a4748c-istio-token\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m\" (UID: \"79ff3314-693d-415c-a02d-b249b9a4748c\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m" Nov 03 13:47:51.737917 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.737704 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c837757-208a-4cbe-a6b7-2b0014ae29bf-istiod-ca-cert\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6\" (UID: \"0c837757-208a-4cbe-a6b7-2b0014ae29bf\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6" Nov 03 13:47:51.737917 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.737712 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c837757-208a-4cbe-a6b7-2b0014ae29bf-credential-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6\" (UID: \"0c837757-208a-4cbe-a6b7-2b0014ae29bf\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6" Nov 03 13:47:51.737917 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.737759 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c837757-208a-4cbe-a6b7-2b0014ae29bf-workload-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6\" (UID: \"0c837757-208a-4cbe-a6b7-2b0014ae29bf\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6" Nov 03 13:47:51.737917 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.737812 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6g4n\" (UniqueName: \"kubernetes.io/projected/79ff3314-693d-415c-a02d-b249b9a4748c-kube-api-access-g6g4n\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m\" (UID: \"79ff3314-693d-415c-a02d-b249b9a4748c\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m" Nov 03 13:47:51.737917 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.737845 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ff3314-693d-415c-a02d-b249b9a4748c-workload-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m\" (UID: \"79ff3314-693d-415c-a02d-b249b9a4748c\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m" Nov 03 13:47:51.737917 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.737881 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b95b7f2-8b31-487f-ba23-b551e9406cc8-istio-podinfo\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz\" (UID: \"0b95b7f2-8b31-487f-ba23-b551e9406cc8\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz" Nov 03 13:47:51.738196 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.738009 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b95b7f2-8b31-487f-ba23-b551e9406cc8-credential-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz\" (UID: \"0b95b7f2-8b31-487f-ba23-b551e9406cc8\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz" Nov 03 13:47:51.738196 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.738055 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dm58\" (UniqueName: \"kubernetes.io/projected/0b95b7f2-8b31-487f-ba23-b551e9406cc8-kube-api-access-7dm58\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz\" (UID: \"0b95b7f2-8b31-487f-ba23-b551e9406cc8\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz" Nov 03 13:47:51.738196 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.738120 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b95b7f2-8b31-487f-ba23-b551e9406cc8-istio-envoy\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz\" (UID: \"0b95b7f2-8b31-487f-ba23-b551e9406cc8\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz" Nov 03 13:47:51.738196 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.738166 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79ff3314-693d-415c-a02d-b249b9a4748c-istiod-ca-cert\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m\" (UID: \"79ff3314-693d-415c-a02d-b249b9a4748c\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m" Nov 03 13:47:51.738343 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.738223 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c837757-208a-4cbe-a6b7-2b0014ae29bf-istio-podinfo\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6\" (UID: \"0c837757-208a-4cbe-a6b7-2b0014ae29bf\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6" Nov 03 13:47:51.738343 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.738254 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b95b7f2-8b31-487f-ba23-b551e9406cc8-istio-token\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz\" (UID: \"0b95b7f2-8b31-487f-ba23-b551e9406cc8\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz" Nov 03 13:47:51.739538 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.739517 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c837757-208a-4cbe-a6b7-2b0014ae29bf-istio-envoy\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6\" (UID: \"0c837757-208a-4cbe-a6b7-2b0014ae29bf\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6" Nov 03 13:47:51.740328 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.740310 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c837757-208a-4cbe-a6b7-2b0014ae29bf-istio-podinfo\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6\" (UID: \"0c837757-208a-4cbe-a6b7-2b0014ae29bf\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6" Nov 03 13:47:51.792995 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.792952 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c837757-208a-4cbe-a6b7-2b0014ae29bf-istio-token\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6\" (UID: \"0c837757-208a-4cbe-a6b7-2b0014ae29bf\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6" Nov 03 13:47:51.793158 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.793006 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdcf9\" (UniqueName: \"kubernetes.io/projected/0c837757-208a-4cbe-a6b7-2b0014ae29bf-kube-api-access-bdcf9\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6\" (UID: \"0c837757-208a-4cbe-a6b7-2b0014ae29bf\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6" Nov 03 13:47:51.836747 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.836713 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6" Nov 03 13:47:51.839247 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.839217 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6g4n\" (UniqueName: \"kubernetes.io/projected/79ff3314-693d-415c-a02d-b249b9a4748c-kube-api-access-g6g4n\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m\" (UID: \"79ff3314-693d-415c-a02d-b249b9a4748c\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m" Nov 03 13:47:51.839366 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.839259 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ff3314-693d-415c-a02d-b249b9a4748c-workload-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m\" (UID: \"79ff3314-693d-415c-a02d-b249b9a4748c\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m" Nov 03 13:47:51.839366 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.839292 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b95b7f2-8b31-487f-ba23-b551e9406cc8-istio-podinfo\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz\" (UID: \"0b95b7f2-8b31-487f-ba23-b551e9406cc8\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz" Nov 03 13:47:51.839366 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.839330 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b95b7f2-8b31-487f-ba23-b551e9406cc8-credential-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz\" (UID: \"0b95b7f2-8b31-487f-ba23-b551e9406cc8\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz" Nov 03 13:47:51.839666 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.839366 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dm58\" (UniqueName: \"kubernetes.io/projected/0b95b7f2-8b31-487f-ba23-b551e9406cc8-kube-api-access-7dm58\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz\" (UID: \"0b95b7f2-8b31-487f-ba23-b551e9406cc8\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz" Nov 03 13:47:51.839666 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.839465 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b95b7f2-8b31-487f-ba23-b551e9406cc8-istio-envoy\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz\" (UID: \"0b95b7f2-8b31-487f-ba23-b551e9406cc8\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz" Nov 03 13:47:51.839666 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.839530 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79ff3314-693d-415c-a02d-b249b9a4748c-istiod-ca-cert\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m\" (UID: \"79ff3314-693d-415c-a02d-b249b9a4748c\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m" Nov 03 13:47:51.839666 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.839566 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b95b7f2-8b31-487f-ba23-b551e9406cc8-istio-token\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz\" (UID: \"0b95b7f2-8b31-487f-ba23-b551e9406cc8\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz" Nov 03 13:47:51.839666 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.839636 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79ff3314-693d-415c-a02d-b249b9a4748c-istio-podinfo\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m\" (UID: \"79ff3314-693d-415c-a02d-b249b9a4748c\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m" Nov 03 13:47:51.840641 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.839795 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ff3314-693d-415c-a02d-b249b9a4748c-workload-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m\" (UID: \"79ff3314-693d-415c-a02d-b249b9a4748c\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m" Nov 03 13:47:51.840641 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.839805 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b95b7f2-8b31-487f-ba23-b551e9406cc8-istiod-ca-cert\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz\" (UID: \"0b95b7f2-8b31-487f-ba23-b551e9406cc8\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz" Nov 03 13:47:51.840641 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.839827 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b95b7f2-8b31-487f-ba23-b551e9406cc8-credential-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz\" (UID: \"0b95b7f2-8b31-487f-ba23-b551e9406cc8\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz" Nov 03 13:47:51.840641 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.839884 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79ff3314-693d-415c-a02d-b249b9a4748c-istio-envoy\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m\" (UID: \"79ff3314-693d-415c-a02d-b249b9a4748c\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m" Nov 03 13:47:51.840641 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.840217 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ff3314-693d-415c-a02d-b249b9a4748c-credential-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m\" (UID: \"79ff3314-693d-415c-a02d-b249b9a4748c\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m" Nov 03 13:47:51.840641 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.840264 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b95b7f2-8b31-487f-ba23-b551e9406cc8-istio-data\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz\" (UID: \"0b95b7f2-8b31-487f-ba23-b551e9406cc8\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz" Nov 03 13:47:51.840641 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.840276 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79ff3314-693d-415c-a02d-b249b9a4748c-istiod-ca-cert\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m\" (UID: \"79ff3314-693d-415c-a02d-b249b9a4748c\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m" Nov 03 13:47:51.840641 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.840312 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b95b7f2-8b31-487f-ba23-b551e9406cc8-workload-certs\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz\" (UID: \"0b95b7f2-8b31-487f-ba23-b551e9406cc8\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz" Nov 03 13:47:51.840641 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.840425 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b95b7f2-8b31-487f-ba23-b551e9406cc8-istiod-ca-cert\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz\" (UID: \"0b95b7f2-8b31-487f-ba23-b551e9406cc8\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz" Nov 03 13:47:51.840641 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.840553 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ff3314-693d-415c-a02d-b249b9a4748c-credential-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m\" (UID: \"79ff3314-693d-415c-a02d-b249b9a4748c\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m" Nov 03 13:47:51.841205 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.840681 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79ff3314-693d-415c-a02d-b249b9a4748c-workload-certs\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m\" (UID: \"79ff3314-693d-415c-a02d-b249b9a4748c\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m" Nov 03 13:47:51.841205 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.840718 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b95b7f2-8b31-487f-ba23-b551e9406cc8-istio-data\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz\" (UID: \"0b95b7f2-8b31-487f-ba23-b551e9406cc8\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz" Nov 03 13:47:51.841205 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.840748 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79ff3314-693d-415c-a02d-b249b9a4748c-istio-data\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m\" (UID: \"79ff3314-693d-415c-a02d-b249b9a4748c\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m" Nov 03 13:47:51.841205 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.840946 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b95b7f2-8b31-487f-ba23-b551e9406cc8-workload-certs\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz\" (UID: \"0b95b7f2-8b31-487f-ba23-b551e9406cc8\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz" Nov 03 13:47:51.841205 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.841017 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b95b7f2-8b31-487f-ba23-b551e9406cc8-workload-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz\" (UID: \"0b95b7f2-8b31-487f-ba23-b551e9406cc8\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz" Nov 03 13:47:51.841205 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.841047 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79ff3314-693d-415c-a02d-b249b9a4748c-istio-token\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m\" (UID: \"79ff3314-693d-415c-a02d-b249b9a4748c\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m" Nov 03 13:47:51.841205 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.841049 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79ff3314-693d-415c-a02d-b249b9a4748c-workload-certs\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m\" (UID: \"79ff3314-693d-415c-a02d-b249b9a4748c\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m" Nov 03 13:47:51.842000 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.841387 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b95b7f2-8b31-487f-ba23-b551e9406cc8-workload-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz\" (UID: \"0b95b7f2-8b31-487f-ba23-b551e9406cc8\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz" Nov 03 13:47:51.842000 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.841621 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79ff3314-693d-415c-a02d-b249b9a4748c-istio-data\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m\" (UID: \"79ff3314-693d-415c-a02d-b249b9a4748c\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m" Nov 03 13:47:51.842270 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.842247 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b95b7f2-8b31-487f-ba23-b551e9406cc8-istio-envoy\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz\" (UID: \"0b95b7f2-8b31-487f-ba23-b551e9406cc8\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz" Nov 03 13:47:51.842363 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.842299 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79ff3314-693d-415c-a02d-b249b9a4748c-istio-envoy\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m\" (UID: \"79ff3314-693d-415c-a02d-b249b9a4748c\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m" Nov 03 13:47:51.842557 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.842538 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79ff3314-693d-415c-a02d-b249b9a4748c-istio-podinfo\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m\" (UID: \"79ff3314-693d-415c-a02d-b249b9a4748c\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m" Nov 03 13:47:51.842629 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.842568 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b95b7f2-8b31-487f-ba23-b551e9406cc8-istio-podinfo\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz\" (UID: \"0b95b7f2-8b31-487f-ba23-b551e9406cc8\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz" Nov 03 13:47:51.897650 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.897621 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b95b7f2-8b31-487f-ba23-b551e9406cc8-istio-token\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz\" (UID: \"0b95b7f2-8b31-487f-ba23-b551e9406cc8\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz" Nov 03 13:47:51.898056 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.898027 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dm58\" (UniqueName: \"kubernetes.io/projected/0b95b7f2-8b31-487f-ba23-b551e9406cc8-kube-api-access-7dm58\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz\" (UID: \"0b95b7f2-8b31-487f-ba23-b551e9406cc8\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz" Nov 03 13:47:51.899950 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.899922 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6g4n\" (UniqueName: \"kubernetes.io/projected/79ff3314-693d-415c-a02d-b249b9a4748c-kube-api-access-g6g4n\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m\" (UID: \"79ff3314-693d-415c-a02d-b249b9a4748c\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m" Nov 03 13:47:51.900299 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.900280 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79ff3314-693d-415c-a02d-b249b9a4748c-istio-token\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m\" (UID: \"79ff3314-693d-415c-a02d-b249b9a4748c\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m" Nov 03 13:47:51.919889 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:51.919855 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz" Nov 03 13:47:52.107635 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:52.107544 5979 generic.go:356] "Generic (PLEG): container finished" podID="44b0445d-b9fe-460f-968b-9e4a950ff35a" containerID="f8209fab595619d9b798f11adbb4023946a28f69c8c9cfc88cc0a67620aeda42" exitCode=0 Nov 03 13:47:52.116191 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:52.116156 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae12be2d-sgcvn" event={"ID":"44b0445d-b9fe-460f-968b-9e4a950ff35a","Type":"ContainerDied","Data":"f8209fab595619d9b798f11adbb4023946a28f69c8c9cfc88cc0a67620aeda42"} Nov 03 13:47:52.194657 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:52.194626 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m" Nov 03 13:47:52.271140 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:52.271099 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6"] Nov 03 13:47:52.289168 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:47:52.277977 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c837757_208a_4cbe_a6b7_2b0014ae29bf.slice/crio-328b7ec9408595fc407dc9a23a5d4395f3921514e49fc92f1a250383a4e1920d WatchSource:0}: Error finding container 328b7ec9408595fc407dc9a23a5d4395f3921514e49fc92f1a250383a4e1920d: Status 404 returned error can't find the container with id 328b7ec9408595fc407dc9a23a5d4395f3921514e49fc92f1a250383a4e1920d Nov 03 13:47:52.289168 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:52.281300 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:47:52.289168 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:52.281426 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:47:52.289168 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:52.281493 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:47:52.324409 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:52.324374 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz"] Nov 03 13:47:52.363252 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:52.363216 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:47:52.363397 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:52.363297 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:47:52.363397 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:52.363346 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:47:52.582839 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:52.582804 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m"] Nov 03 13:47:52.598748 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:47:52.598603 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79ff3314_693d_415c_a02d_b249b9a4748c.slice/crio-b0ea5c9c8037065247b73345063d0c99212009ab966bbf7c9d84f1b074471c81 WatchSource:0}: Error finding container b0ea5c9c8037065247b73345063d0c99212009ab966bbf7c9d84f1b074471c81: Status 404 returned error can't find the container with id b0ea5c9c8037065247b73345063d0c99212009ab966bbf7c9d84f1b074471c81 Nov 03 13:47:52.601790 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:52.601750 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:47:52.601881 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:52.601848 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:47:52.601933 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:52.601890 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:47:53.120416 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:53.120374 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m" event={"ID":"79ff3314-693d-415c-a02d-b249b9a4748c","Type":"ContainerStarted","Data":"10ed8dfb524cd3c7fc0f1152890c143cbfa4391bc0a4eea1b19c1441278fd45b"} Nov 03 13:47:53.120916 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:53.120868 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m" event={"ID":"79ff3314-693d-415c-a02d-b249b9a4748c","Type":"ContainerStarted","Data":"b0ea5c9c8037065247b73345063d0c99212009ab966bbf7c9d84f1b074471c81"} Nov 03 13:47:53.124524 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:53.124452 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz" event={"ID":"0b95b7f2-8b31-487f-ba23-b551e9406cc8","Type":"ContainerStarted","Data":"f22122c5534f610328f6f44d3bf0d6b4649d8e91dee4b81e038142277ffb2409"} Nov 03 13:47:53.124699 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:53.124565 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz" event={"ID":"0b95b7f2-8b31-487f-ba23-b551e9406cc8","Type":"ContainerStarted","Data":"fafe846d5d1cee428d221f58dc5929f8415cf9cbb5acbb75d917c2ef22b83939"} Nov 03 13:47:53.128307 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:53.128281 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6" event={"ID":"0c837757-208a-4cbe-a6b7-2b0014ae29bf","Type":"ContainerStarted","Data":"ff314bcb937698a6a1bd2547d8fa021aecb6b13d446d0683c6e3b4b3a76faa94"} Nov 03 13:47:53.128449 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:53.128333 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6" event={"ID":"0c837757-208a-4cbe-a6b7-2b0014ae29bf","Type":"ContainerStarted","Data":"328b7ec9408595fc407dc9a23a5d4395f3921514e49fc92f1a250383a4e1920d"} Nov 03 13:47:53.199496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:53.197339 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m" Nov 03 13:47:53.199496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:53.197497 5979 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body= Nov 03 13:47:53.199496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:53.197563 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m" podUID="79ff3314-693d-415c-a02d-b249b9a4748c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" Nov 03 13:47:53.285994 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:53.285913 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m" podStartSLOduration=2.285888656 podStartE2EDuration="2.285888656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:47:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:47:53.253360736 +0000 UTC m=+2555.264048980" watchObservedRunningTime="2026-11-03 13:47:53.285888656 +0000 UTC m=+2555.296576899" Nov 03 13:47:53.394823 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:53.385077 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz" podStartSLOduration=2.385057104 podStartE2EDuration="2.385057104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:47:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:47:53.379946042 +0000 UTC m=+2555.390634286" watchObservedRunningTime="2026-11-03 13:47:53.385057104 +0000 UTC m=+2555.395745347" Nov 03 13:47:53.595495 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:53.555866 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6" podStartSLOduration=2.555843664 podStartE2EDuration="2.555843664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:47:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:47:53.552966775 +0000 UTC m=+2555.563655052" watchObservedRunningTime="2026-11-03 13:47:53.555843664 +0000 UTC m=+2555.566531909" Nov 03 13:47:53.731698 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:53.731664 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae12be2d-sgcvn" Nov 03 13:47:53.733910 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:53.733870 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44b0445d-b9fe-460f-968b-9e4a950ff35a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44b0445d-b9fe-460f-968b-9e4a950ff35a-maas-api-openshift-ca\") pod \"44b0445d-b9fe-460f-968b-9e4a950ff35a\" (UID: \"44b0445d-b9fe-460f-968b-9e4a950ff35a\") " Nov 03 13:47:53.734104 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:53.733985 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44b0445d-b9fe-460f-968b-9e4a950ff35a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44b0445d-b9fe-460f-968b-9e4a950ff35a-maas-api-xks-ca\") pod \"44b0445d-b9fe-460f-968b-9e4a950ff35a\" (UID: \"44b0445d-b9fe-460f-968b-9e4a950ff35a\") " Nov 03 13:47:53.734104 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:53.734044 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44b0445d-b9fe-460f-968b-9e4a950ff35a-kube-api-access-csglq\" (UniqueName: \"kubernetes.io/projected/44b0445d-b9fe-460f-968b-9e4a950ff35a-kube-api-access-csglq\") pod \"44b0445d-b9fe-460f-968b-9e4a950ff35a\" (UID: \"44b0445d-b9fe-460f-968b-9e4a950ff35a\") " Nov 03 13:47:53.734295 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:53.734265 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44b0445d-b9fe-460f-968b-9e4a950ff35a-maas-api-openshift-ca" pod "44b0445d-b9fe-460f-968b-9e4a950ff35a" (UID: "44b0445d-b9fe-460f-968b-9e4a950ff35a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:47:53.734556 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:53.734531 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44b0445d-b9fe-460f-968b-9e4a950ff35a-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:47:53.736360 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:53.736332 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44b0445d-b9fe-460f-968b-9e4a950ff35a-maas-api-xks-ca" pod "44b0445d-b9fe-460f-968b-9e4a950ff35a" (UID: "44b0445d-b9fe-460f-968b-9e4a950ff35a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:47:53.736616 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:53.736591 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44b0445d-b9fe-460f-968b-9e4a950ff35a-kube-api-access-csglq" pod "44b0445d-b9fe-460f-968b-9e4a950ff35a" (UID: "44b0445d-b9fe-460f-968b-9e4a950ff35a"). InnerVolumeSpecName "kube-api-access-csglq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:47:53.836214 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:53.836157 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44b0445d-b9fe-460f-968b-9e4a950ff35a-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:47:53.836214 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:53.836198 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csglq\" (UniqueName: \"kubernetes.io/projected/44b0445d-b9fe-460f-968b-9e4a950ff35a-kube-api-access-csglq\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:47:53.837551 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:53.837529 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6" Nov 03 13:47:53.839229 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:53.839202 5979 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Nov 03 13:47:53.839320 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:53.839263 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6" podUID="0c837757-208a-4cbe-a6b7-2b0014ae29bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Nov 03 13:47:53.929497 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:53.928738 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz" Nov 03 13:47:53.931074 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:53.931048 5979 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" start-of-body= Nov 03 13:47:53.931496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:53.931221 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz" podUID="0b95b7f2-8b31-487f-ba23-b551e9406cc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" Nov 03 13:47:54.136074 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:54.136043 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae12be2d-sgcvn" Nov 03 13:47:54.137522 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:54.137495 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae12be2d-sgcvn" event={"ID":"44b0445d-b9fe-460f-968b-9e4a950ff35a","Type":"ContainerDied","Data":"bfac6b12fbbd4a731f05995bf7c5eecb62de1cc062b7bedafd855d0f6e6301d4"} Nov 03 13:47:54.137700 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:54.137685 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bfac6b12fbbd4a731f05995bf7c5eecb62de1cc062b7bedafd855d0f6e6301d4" Nov 03 13:47:54.198770 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:54.198735 5979 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body= Nov 03 13:47:54.204605 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:54.204561 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m" podUID="79ff3314-693d-415c-a02d-b249b9a4748c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" Nov 03 13:47:54.837810 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:54.837770 5979 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Nov 03 13:47:54.837978 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:54.837832 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6" podUID="0c837757-208a-4cbe-a6b7-2b0014ae29bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Nov 03 13:47:54.926979 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:54.926934 5979 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" start-of-body= Nov 03 13:47:54.927319 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:54.927280 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz" podUID="0b95b7f2-8b31-487f-ba23-b551e9406cc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" Nov 03 13:47:55.202940 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:55.202900 5979 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body= Nov 03 13:47:55.203648 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:55.203612 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m" podUID="79ff3314-693d-415c-a02d-b249b9a4748c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" Nov 03 13:47:55.838557 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:55.837857 5979 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Nov 03 13:47:55.838557 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:55.837912 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6" podUID="0c837757-208a-4cbe-a6b7-2b0014ae29bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Nov 03 13:47:55.921496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:55.920989 5979 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" start-of-body= Nov 03 13:47:55.921496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:55.921050 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz" podUID="0b95b7f2-8b31-487f-ba23-b551e9406cc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" Nov 03 13:47:56.204500 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:56.203874 5979 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body= Nov 03 13:47:56.204500 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:56.203939 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m" podUID="79ff3314-693d-415c-a02d-b249b9a4748c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" Nov 03 13:47:56.837779 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:56.837743 5979 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Nov 03 13:47:56.837950 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:56.837796 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6" podUID="0c837757-208a-4cbe-a6b7-2b0014ae29bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Nov 03 13:47:56.921116 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:56.920930 5979 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" start-of-body= Nov 03 13:47:56.921116 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:56.920998 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz" podUID="0b95b7f2-8b31-487f-ba23-b551e9406cc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" Nov 03 13:47:57.219497 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:57.219152 5979 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body= Nov 03 13:47:57.219497 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:57.219212 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m" podUID="79ff3314-693d-415c-a02d-b249b9a4748c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" Nov 03 13:47:57.837863 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:57.837827 5979 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Nov 03 13:47:57.838089 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:57.837885 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6" podUID="0c837757-208a-4cbe-a6b7-2b0014ae29bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Nov 03 13:47:57.920937 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:57.920897 5979 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" start-of-body= Nov 03 13:47:57.921222 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:57.921192 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz" podUID="0b95b7f2-8b31-487f-ba23-b551e9406cc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" Nov 03 13:47:58.201989 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:58.201949 5979 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body= Nov 03 13:47:58.202286 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:58.202254 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m" podUID="79ff3314-693d-415c-a02d-b249b9a4748c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" Nov 03 13:47:58.219403 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:58.219346 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd59e770-b2pvj"] Nov 03 13:47:58.220583 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:58.220555 5979 cpu_manager.go:414] "containerMap: removing container" podUID="44b0445d-b9fe-460f-968b-9e4a950ff35a" containerName="revoke-keys" Nov 03 13:47:58.221066 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:58.221048 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44b0445d-b9fe-460f-968b-9e4a950ff35a" containerName="revoke-keys" Nov 03 13:47:58.221387 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:58.221365 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="44b0445d-b9fe-460f-968b-9e4a950ff35a" containerName="revoke-keys" Nov 03 13:47:58.228043 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:58.228014 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd59e770-b2pvj" Nov 03 13:47:58.289531 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:58.281127 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:47:58.297645 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:58.297607 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd59e770-b2pvj"] Nov 03 13:47:58.323658 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:58.303783 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:47:58.324827 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:58.324793 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/effdd48b-ffed-472e-8e08-b6214d3de071-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd59e770-b2pvj\" (UID: \"effdd48b-ffed-472e-8e08-b6214d3de071\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd59e770-b2pvj" Nov 03 13:47:58.325014 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:58.324993 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/effdd48b-ffed-472e-8e08-b6214d3de071-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd59e770-b2pvj\" (UID: \"effdd48b-ffed-472e-8e08-b6214d3de071\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd59e770-b2pvj" Nov 03 13:47:58.325319 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:58.325300 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7qh6\" (UniqueName: \"kubernetes.io/projected/effdd48b-ffed-472e-8e08-b6214d3de071-kube-api-access-p7qh6\") pod \"maas-api-revoke-sub-bd59e770-b2pvj\" (UID: \"effdd48b-ffed-472e-8e08-b6214d3de071\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd59e770-b2pvj" Nov 03 13:47:58.426856 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:58.426799 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/effdd48b-ffed-472e-8e08-b6214d3de071-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd59e770-b2pvj\" (UID: \"effdd48b-ffed-472e-8e08-b6214d3de071\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd59e770-b2pvj" Nov 03 13:47:58.427028 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:58.426888 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/effdd48b-ffed-472e-8e08-b6214d3de071-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd59e770-b2pvj\" (UID: \"effdd48b-ffed-472e-8e08-b6214d3de071\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd59e770-b2pvj" Nov 03 13:47:58.427118 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:58.427085 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7qh6\" (UniqueName: \"kubernetes.io/projected/effdd48b-ffed-472e-8e08-b6214d3de071-kube-api-access-p7qh6\") pod \"maas-api-revoke-sub-bd59e770-b2pvj\" (UID: \"effdd48b-ffed-472e-8e08-b6214d3de071\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd59e770-b2pvj" Nov 03 13:47:58.428095 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:58.428058 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/effdd48b-ffed-472e-8e08-b6214d3de071-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd59e770-b2pvj\" (UID: \"effdd48b-ffed-472e-8e08-b6214d3de071\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd59e770-b2pvj" Nov 03 13:47:58.430296 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:58.430271 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/effdd48b-ffed-472e-8e08-b6214d3de071-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd59e770-b2pvj\" (UID: \"effdd48b-ffed-472e-8e08-b6214d3de071\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd59e770-b2pvj" Nov 03 13:47:58.519951 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:58.519695 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7qh6\" (UniqueName: \"kubernetes.io/projected/effdd48b-ffed-472e-8e08-b6214d3de071-kube-api-access-p7qh6\") pod \"maas-api-revoke-sub-bd59e770-b2pvj\" (UID: \"effdd48b-ffed-472e-8e08-b6214d3de071\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd59e770-b2pvj" Nov 03 13:47:58.542827 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:58.542794 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd59e770-b2pvj" Nov 03 13:47:58.890553 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:58.865825 5979 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" start-of-body= Nov 03 13:47:58.890553 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:58.865908 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6" podUID="0c837757-208a-4cbe-a6b7-2b0014ae29bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.1:15021/healthz/ready\": dial tcp 10.134.1.1:15021: connect: connection refused" Nov 03 13:47:58.925051 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:58.925010 5979 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" start-of-body= Nov 03 13:47:58.925239 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:58.925094 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz" podUID="0b95b7f2-8b31-487f-ba23-b551e9406cc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.3:15021/healthz/ready\": dial tcp 10.134.1.3:15021: connect: connection refused" Nov 03 13:47:59.041393 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:59.041358 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd59e770-b2pvj"] Nov 03 13:47:59.195226 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:59.195137 5979 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" start-of-body= Nov 03 13:47:59.195525 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:59.195488 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m" podUID="79ff3314-693d-415c-a02d-b249b9a4748c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.2:15021/healthz/ready\": dial tcp 10.134.1.2:15021: connect: connection refused" Nov 03 13:47:59.245855 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:47:59.245813 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd59e770-b2pvj" event={"ID":"effdd48b-ffed-472e-8e08-b6214d3de071","Type":"ContainerStarted","Data":"12abecbf50bb1bc3997d973660420271a796f305495560982672b870d8033d36"} Nov 03 13:48:00.148154 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:00.148118 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6" Nov 03 13:48:00.148997 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:00.148895 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6" Nov 03 13:48:00.151059 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:00.151032 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6" Nov 03 13:48:00.288097 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:00.288017 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd59e770-b2pvj" event={"ID":"effdd48b-ffed-472e-8e08-b6214d3de071","Type":"ContainerStarted","Data":"7d14c6b7f64f14d72f770a740d55185df80ecfa51d22161f3fd54f0d3de9b735"} Nov 03 13:48:00.488097 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:00.488005 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz" Nov 03 13:48:00.489350 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:00.489323 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz" Nov 03 13:48:00.493724 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:00.493703 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz" Nov 03 13:48:00.500802 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:00.500753 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd59e770-b2pvj" podStartSLOduration=2.500741573 podStartE2EDuration="2.500741573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:47:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:48:00.49764683 +0000 UTC m=+2562.508335073" watchObservedRunningTime="2026-11-03 13:48:00.500741573 +0000 UTC m=+2562.511429814" Nov 03 13:48:00.814113 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:00.814015 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m" Nov 03 13:48:00.814971 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:00.814943 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m" Nov 03 13:48:00.815734 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:00.815708 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m" Nov 03 13:48:01.295828 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:01.295795 5979 generic.go:356] "Generic (PLEG): container finished" podID="effdd48b-ffed-472e-8e08-b6214d3de071" containerID="7d14c6b7f64f14d72f770a740d55185df80ecfa51d22161f3fd54f0d3de9b735" exitCode=0 Nov 03 13:48:01.296318 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:01.295877 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd59e770-b2pvj" event={"ID":"effdd48b-ffed-472e-8e08-b6214d3de071","Type":"ContainerDied","Data":"7d14c6b7f64f14d72f770a740d55185df80ecfa51d22161f3fd54f0d3de9b735"} Nov 03 13:48:02.444081 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:02.444056 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd59e770-b2pvj" Nov 03 13:48:02.503081 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:02.503038 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/effdd48b-ffed-472e-8e08-b6214d3de071-kube-api-access-p7qh6\" (UniqueName: \"kubernetes.io/projected/effdd48b-ffed-472e-8e08-b6214d3de071-kube-api-access-p7qh6\") pod \"effdd48b-ffed-472e-8e08-b6214d3de071\" (UID: \"effdd48b-ffed-472e-8e08-b6214d3de071\") " Nov 03 13:48:02.503274 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:02.503167 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/effdd48b-ffed-472e-8e08-b6214d3de071-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/effdd48b-ffed-472e-8e08-b6214d3de071-maas-api-xks-ca\") pod \"effdd48b-ffed-472e-8e08-b6214d3de071\" (UID: \"effdd48b-ffed-472e-8e08-b6214d3de071\") " Nov 03 13:48:02.503274 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:02.503209 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/effdd48b-ffed-472e-8e08-b6214d3de071-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/effdd48b-ffed-472e-8e08-b6214d3de071-maas-api-openshift-ca\") pod \"effdd48b-ffed-472e-8e08-b6214d3de071\" (UID: \"effdd48b-ffed-472e-8e08-b6214d3de071\") " Nov 03 13:48:02.503668 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:02.503640 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/effdd48b-ffed-472e-8e08-b6214d3de071-maas-api-openshift-ca" pod "effdd48b-ffed-472e-8e08-b6214d3de071" (UID: "effdd48b-ffed-472e-8e08-b6214d3de071"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:48:02.505218 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:02.505196 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/effdd48b-ffed-472e-8e08-b6214d3de071-kube-api-access-p7qh6" pod "effdd48b-ffed-472e-8e08-b6214d3de071" (UID: "effdd48b-ffed-472e-8e08-b6214d3de071"). InnerVolumeSpecName "kube-api-access-p7qh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:48:02.505302 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:02.505246 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/effdd48b-ffed-472e-8e08-b6214d3de071-maas-api-xks-ca" pod "effdd48b-ffed-472e-8e08-b6214d3de071" (UID: "effdd48b-ffed-472e-8e08-b6214d3de071"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:48:02.605330 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:02.605232 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/effdd48b-ffed-472e-8e08-b6214d3de071-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:48:02.605330 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:02.605278 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/effdd48b-ffed-472e-8e08-b6214d3de071-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:48:02.605330 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:02.605289 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7qh6\" (UniqueName: \"kubernetes.io/projected/effdd48b-ffed-472e-8e08-b6214d3de071-kube-api-access-p7qh6\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:48:03.310701 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:03.310669 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd59e770-b2pvj" Nov 03 13:48:03.310902 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:03.310698 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd59e770-b2pvj" event={"ID":"effdd48b-ffed-472e-8e08-b6214d3de071","Type":"ContainerDied","Data":"12abecbf50bb1bc3997d973660420271a796f305495560982672b870d8033d36"} Nov 03 13:48:03.310902 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:03.310755 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12abecbf50bb1bc3997d973660420271a796f305495560982672b870d8033d36" Nov 03 13:48:07.247147 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:48:07.247070 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:48:08.974078 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:08.974042 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24a93d8e-q9rpd"] Nov 03 13:48:08.974674 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:08.974661 5979 cpu_manager.go:414] "containerMap: removing container" podUID="effdd48b-ffed-472e-8e08-b6214d3de071" containerName="revoke-keys" Nov 03 13:48:08.974733 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:08.974675 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="effdd48b-ffed-472e-8e08-b6214d3de071" containerName="revoke-keys" Nov 03 13:48:08.974819 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:08.974810 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="effdd48b-ffed-472e-8e08-b6214d3de071" containerName="revoke-keys" Nov 03 13:48:08.978433 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:08.978415 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a93d8e-q9rpd" Nov 03 13:48:09.027149 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:09.027117 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:48:09.027325 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:09.027170 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:48:09.039220 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:09.039190 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3becf35-a2d0-4144-b9d8-89f69321b056-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24a93d8e-q9rpd\" (UID: \"c3becf35-a2d0-4144-b9d8-89f69321b056\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a93d8e-q9rpd" Nov 03 13:48:09.039384 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:09.039349 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkz4b\" (UniqueName: \"kubernetes.io/projected/c3becf35-a2d0-4144-b9d8-89f69321b056-kube-api-access-lkz4b\") pod \"maas-api-revoke-sub-24a93d8e-q9rpd\" (UID: \"c3becf35-a2d0-4144-b9d8-89f69321b056\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a93d8e-q9rpd" Nov 03 13:48:09.039428 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:09.039409 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3becf35-a2d0-4144-b9d8-89f69321b056-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24a93d8e-q9rpd\" (UID: \"c3becf35-a2d0-4144-b9d8-89f69321b056\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a93d8e-q9rpd" Nov 03 13:48:09.079067 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:09.078593 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24a93d8e-q9rpd"] Nov 03 13:48:09.140905 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:09.140869 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkz4b\" (UniqueName: \"kubernetes.io/projected/c3becf35-a2d0-4144-b9d8-89f69321b056-kube-api-access-lkz4b\") pod \"maas-api-revoke-sub-24a93d8e-q9rpd\" (UID: \"c3becf35-a2d0-4144-b9d8-89f69321b056\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a93d8e-q9rpd" Nov 03 13:48:09.141104 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:09.140948 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3becf35-a2d0-4144-b9d8-89f69321b056-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24a93d8e-q9rpd\" (UID: \"c3becf35-a2d0-4144-b9d8-89f69321b056\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a93d8e-q9rpd" Nov 03 13:48:09.141104 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:09.141053 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3becf35-a2d0-4144-b9d8-89f69321b056-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24a93d8e-q9rpd\" (UID: \"c3becf35-a2d0-4144-b9d8-89f69321b056\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a93d8e-q9rpd" Nov 03 13:48:09.141792 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:09.141766 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3becf35-a2d0-4144-b9d8-89f69321b056-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24a93d8e-q9rpd\" (UID: \"c3becf35-a2d0-4144-b9d8-89f69321b056\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a93d8e-q9rpd" Nov 03 13:48:09.161520 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:09.143757 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3becf35-a2d0-4144-b9d8-89f69321b056-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24a93d8e-q9rpd\" (UID: \"c3becf35-a2d0-4144-b9d8-89f69321b056\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a93d8e-q9rpd" Nov 03 13:48:09.193982 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:09.193945 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkz4b\" (UniqueName: \"kubernetes.io/projected/c3becf35-a2d0-4144-b9d8-89f69321b056-kube-api-access-lkz4b\") pod \"maas-api-revoke-sub-24a93d8e-q9rpd\" (UID: \"c3becf35-a2d0-4144-b9d8-89f69321b056\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a93d8e-q9rpd" Nov 03 13:48:09.293020 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:09.292931 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a93d8e-q9rpd" Nov 03 13:48:09.690217 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:09.690180 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24a93d8e-q9rpd"] Nov 03 13:48:09.696354 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:48:09.696313 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3becf35_a2d0_4144_b9d8_89f69321b056.slice/crio-7b8699865f9e316a0252fa07a091f11c8ec810adad39abebab7948af790dc5d0 WatchSource:0}: Error finding container 7b8699865f9e316a0252fa07a091f11c8ec810adad39abebab7948af790dc5d0: Status 404 returned error can't find the container with id 7b8699865f9e316a0252fa07a091f11c8ec810adad39abebab7948af790dc5d0 Nov 03 13:48:10.389369 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:10.389330 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a93d8e-q9rpd" event={"ID":"c3becf35-a2d0-4144-b9d8-89f69321b056","Type":"ContainerStarted","Data":"90caf5f74e76d1591e7778c86c252e408a5da8cb5694cefc9d013050886782a8"} Nov 03 13:48:10.389808 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:10.389395 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a93d8e-q9rpd" event={"ID":"c3becf35-a2d0-4144-b9d8-89f69321b056","Type":"ContainerStarted","Data":"7b8699865f9e316a0252fa07a091f11c8ec810adad39abebab7948af790dc5d0"} Nov 03 13:48:10.497256 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:10.497202 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a93d8e-q9rpd" podStartSLOduration=2.497186964 podStartE2EDuration="2.497186964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:48:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:48:10.495642869 +0000 UTC m=+2572.506331110" watchObservedRunningTime="2026-11-03 13:48:10.497186964 +0000 UTC m=+2572.507875251" Nov 03 13:48:11.397910 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:11.397874 5979 generic.go:356] "Generic (PLEG): container finished" podID="c3becf35-a2d0-4144-b9d8-89f69321b056" containerID="90caf5f74e76d1591e7778c86c252e408a5da8cb5694cefc9d013050886782a8" exitCode=0 Nov 03 13:48:11.398327 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:11.397961 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a93d8e-q9rpd" event={"ID":"c3becf35-a2d0-4144-b9d8-89f69321b056","Type":"ContainerDied","Data":"90caf5f74e76d1591e7778c86c252e408a5da8cb5694cefc9d013050886782a8"} Nov 03 13:48:12.593237 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:12.593210 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a93d8e-q9rpd" Nov 03 13:48:12.610231 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:12.610198 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3becf35-a2d0-4144-b9d8-89f69321b056-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3becf35-a2d0-4144-b9d8-89f69321b056-maas-api-xks-ca\") pod \"c3becf35-a2d0-4144-b9d8-89f69321b056\" (UID: \"c3becf35-a2d0-4144-b9d8-89f69321b056\") " Nov 03 13:48:12.610422 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:12.610266 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3becf35-a2d0-4144-b9d8-89f69321b056-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3becf35-a2d0-4144-b9d8-89f69321b056-maas-api-openshift-ca\") pod \"c3becf35-a2d0-4144-b9d8-89f69321b056\" (UID: \"c3becf35-a2d0-4144-b9d8-89f69321b056\") " Nov 03 13:48:12.610422 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:12.610375 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3becf35-a2d0-4144-b9d8-89f69321b056-kube-api-access-lkz4b\" (UniqueName: \"kubernetes.io/projected/c3becf35-a2d0-4144-b9d8-89f69321b056-kube-api-access-lkz4b\") pod \"c3becf35-a2d0-4144-b9d8-89f69321b056\" (UID: \"c3becf35-a2d0-4144-b9d8-89f69321b056\") " Nov 03 13:48:12.610683 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:12.610654 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3becf35-a2d0-4144-b9d8-89f69321b056-maas-api-openshift-ca" pod "c3becf35-a2d0-4144-b9d8-89f69321b056" (UID: "c3becf35-a2d0-4144-b9d8-89f69321b056"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:48:12.610915 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:12.610895 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3becf35-a2d0-4144-b9d8-89f69321b056-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:48:12.612628 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:12.612605 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3becf35-a2d0-4144-b9d8-89f69321b056-kube-api-access-lkz4b" pod "c3becf35-a2d0-4144-b9d8-89f69321b056" (UID: "c3becf35-a2d0-4144-b9d8-89f69321b056"). InnerVolumeSpecName "kube-api-access-lkz4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:48:12.612771 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:12.612746 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3becf35-a2d0-4144-b9d8-89f69321b056-maas-api-xks-ca" pod "c3becf35-a2d0-4144-b9d8-89f69321b056" (UID: "c3becf35-a2d0-4144-b9d8-89f69321b056"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:48:12.712066 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:12.711977 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkz4b\" (UniqueName: \"kubernetes.io/projected/c3becf35-a2d0-4144-b9d8-89f69321b056-kube-api-access-lkz4b\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:48:12.712066 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:12.712018 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3becf35-a2d0-4144-b9d8-89f69321b056-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:48:12.971819 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:12.971736 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-261647f7-7286h"] Nov 03 13:48:12.972728 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:12.972695 5979 cpu_manager.go:414] "containerMap: removing container" podUID="c3becf35-a2d0-4144-b9d8-89f69321b056" containerName="revoke-keys" Nov 03 13:48:12.972728 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:12.972723 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3becf35-a2d0-4144-b9d8-89f69321b056" containerName="revoke-keys" Nov 03 13:48:12.972985 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:12.972966 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3becf35-a2d0-4144-b9d8-89f69321b056" containerName="revoke-keys" Nov 03 13:48:12.976806 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:12.976784 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-261647f7-7286h" Nov 03 13:48:13.019493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:13.019307 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54920681-626a-41cc-8a46-d0b3c13237be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-261647f7-7286h\" (UID: \"54920681-626a-41cc-8a46-d0b3c13237be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-261647f7-7286h" Nov 03 13:48:13.019713 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:13.019514 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz2gc\" (UniqueName: \"kubernetes.io/projected/54920681-626a-41cc-8a46-d0b3c13237be-kube-api-access-wz2gc\") pod \"maas-api-revoke-sub-261647f7-7286h\" (UID: \"54920681-626a-41cc-8a46-d0b3c13237be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-261647f7-7286h" Nov 03 13:48:13.019713 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:13.019576 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54920681-626a-41cc-8a46-d0b3c13237be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-261647f7-7286h\" (UID: \"54920681-626a-41cc-8a46-d0b3c13237be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-261647f7-7286h" Nov 03 13:48:13.047497 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:13.044872 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-261647f7-7286h"] Nov 03 13:48:13.121130 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:13.121095 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54920681-626a-41cc-8a46-d0b3c13237be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-261647f7-7286h\" (UID: \"54920681-626a-41cc-8a46-d0b3c13237be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-261647f7-7286h" Nov 03 13:48:13.121341 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:13.121246 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz2gc\" (UniqueName: \"kubernetes.io/projected/54920681-626a-41cc-8a46-d0b3c13237be-kube-api-access-wz2gc\") pod \"maas-api-revoke-sub-261647f7-7286h\" (UID: \"54920681-626a-41cc-8a46-d0b3c13237be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-261647f7-7286h" Nov 03 13:48:13.121341 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:13.121299 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54920681-626a-41cc-8a46-d0b3c13237be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-261647f7-7286h\" (UID: \"54920681-626a-41cc-8a46-d0b3c13237be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-261647f7-7286h" Nov 03 13:48:13.122130 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:13.122099 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54920681-626a-41cc-8a46-d0b3c13237be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-261647f7-7286h\" (UID: \"54920681-626a-41cc-8a46-d0b3c13237be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-261647f7-7286h" Nov 03 13:48:13.124506 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:13.123661 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54920681-626a-41cc-8a46-d0b3c13237be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-261647f7-7286h\" (UID: \"54920681-626a-41cc-8a46-d0b3c13237be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-261647f7-7286h" Nov 03 13:48:13.175999 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:13.175969 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz2gc\" (UniqueName: \"kubernetes.io/projected/54920681-626a-41cc-8a46-d0b3c13237be-kube-api-access-wz2gc\") pod \"maas-api-revoke-sub-261647f7-7286h\" (UID: \"54920681-626a-41cc-8a46-d0b3c13237be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-261647f7-7286h" Nov 03 13:48:13.303685 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:13.303592 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-261647f7-7286h" Nov 03 13:48:13.417510 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:13.416807 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a93d8e-q9rpd" Nov 03 13:48:13.417510 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:13.416831 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24a93d8e-q9rpd" event={"ID":"c3becf35-a2d0-4144-b9d8-89f69321b056","Type":"ContainerDied","Data":"7b8699865f9e316a0252fa07a091f11c8ec810adad39abebab7948af790dc5d0"} Nov 03 13:48:13.417510 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:13.416873 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b8699865f9e316a0252fa07a091f11c8ec810adad39abebab7948af790dc5d0" Nov 03 13:48:13.689707 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:13.689673 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-261647f7-7286h"] Nov 03 13:48:13.694899 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:48:13.694869 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54920681_626a_41cc_8a46_d0b3c13237be.slice/crio-605bb39ef858dada87dddff3a8727e228539834a6db4fe2610994b30ad1a4895 WatchSource:0}: Error finding container 605bb39ef858dada87dddff3a8727e228539834a6db4fe2610994b30ad1a4895: Status 404 returned error can't find the container with id 605bb39ef858dada87dddff3a8727e228539834a6db4fe2610994b30ad1a4895 Nov 03 13:48:14.426310 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:14.426267 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-261647f7-7286h" event={"ID":"54920681-626a-41cc-8a46-d0b3c13237be","Type":"ContainerStarted","Data":"ed97c789227603f89237b448cbdc85a8d1ee25668242f86236a7aa9d6dbf14c8"} Nov 03 13:48:14.426490 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:14.426336 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-261647f7-7286h" event={"ID":"54920681-626a-41cc-8a46-d0b3c13237be","Type":"ContainerStarted","Data":"605bb39ef858dada87dddff3a8727e228539834a6db4fe2610994b30ad1a4895"} Nov 03 13:48:14.533048 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:14.532992 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-261647f7-7286h" podStartSLOduration=2.532978799 podStartE2EDuration="2.532978799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:48:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:48:14.532454106 +0000 UTC m=+2576.543142347" watchObservedRunningTime="2026-11-03 13:48:14.532978799 +0000 UTC m=+2576.543667041" Nov 03 13:48:15.436716 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:15.436685 5979 generic.go:356] "Generic (PLEG): container finished" podID="2fe2e1d4-9fac-456e-ba2b-e221b6c59e98" containerID="e99df3ac5df64c94c69193b5bce556f78818c051cae6de0dc0d96f4a98bef169" exitCode=0 Nov 03 13:48:15.437218 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:15.436737 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w" event={"ID":"2fe2e1d4-9fac-456e-ba2b-e221b6c59e98","Type":"ContainerDied","Data":"e99df3ac5df64c94c69193b5bce556f78818c051cae6de0dc0d96f4a98bef169"} Nov 03 13:48:15.437306 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:15.437221 5979 scope.go:122] "RemoveContainer" containerID="e99df3ac5df64c94c69193b5bce556f78818c051cae6de0dc0d96f4a98bef169" Nov 03 13:48:15.438519 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:15.438496 5979 generic.go:356] "Generic (PLEG): container finished" podID="54920681-626a-41cc-8a46-d0b3c13237be" containerID="ed97c789227603f89237b448cbdc85a8d1ee25668242f86236a7aa9d6dbf14c8" exitCode=0 Nov 03 13:48:15.438636 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:15.438497 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-261647f7-7286h" event={"ID":"54920681-626a-41cc-8a46-d0b3c13237be","Type":"ContainerDied","Data":"ed97c789227603f89237b448cbdc85a8d1ee25668242f86236a7aa9d6dbf14c8"} Nov 03 13:48:15.438778 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:15.438754 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:48:15.438830 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:15.438817 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:48:15.438866 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:15.438852 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:48:16.473220 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:16.473177 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w" event={"ID":"2fe2e1d4-9fac-456e-ba2b-e221b6c59e98","Type":"ContainerStarted","Data":"1a57ea83cab3b1e78bf6133cd4eaf65965e89db715e28489cfdcb325932997c7"} Nov 03 13:48:16.667496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:16.665678 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-261647f7-7286h" Nov 03 13:48:16.680739 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:16.680703 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54920681-626a-41cc-8a46-d0b3c13237be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54920681-626a-41cc-8a46-d0b3c13237be-maas-api-openshift-ca\") pod \"54920681-626a-41cc-8a46-d0b3c13237be\" (UID: \"54920681-626a-41cc-8a46-d0b3c13237be\") " Nov 03 13:48:16.680739 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:16.680742 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54920681-626a-41cc-8a46-d0b3c13237be-kube-api-access-wz2gc\" (UniqueName: \"kubernetes.io/projected/54920681-626a-41cc-8a46-d0b3c13237be-kube-api-access-wz2gc\") pod \"54920681-626a-41cc-8a46-d0b3c13237be\" (UID: \"54920681-626a-41cc-8a46-d0b3c13237be\") " Nov 03 13:48:16.681011 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:16.680801 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54920681-626a-41cc-8a46-d0b3c13237be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54920681-626a-41cc-8a46-d0b3c13237be-maas-api-xks-ca\") pod \"54920681-626a-41cc-8a46-d0b3c13237be\" (UID: \"54920681-626a-41cc-8a46-d0b3c13237be\") " Nov 03 13:48:16.681337 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:16.681311 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54920681-626a-41cc-8a46-d0b3c13237be-maas-api-openshift-ca" pod "54920681-626a-41cc-8a46-d0b3c13237be" (UID: "54920681-626a-41cc-8a46-d0b3c13237be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:48:16.682879 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:16.682855 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54920681-626a-41cc-8a46-d0b3c13237be-maas-api-xks-ca" pod "54920681-626a-41cc-8a46-d0b3c13237be" (UID: "54920681-626a-41cc-8a46-d0b3c13237be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:48:16.682968 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:16.682942 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54920681-626a-41cc-8a46-d0b3c13237be-kube-api-access-wz2gc" pod "54920681-626a-41cc-8a46-d0b3c13237be" (UID: "54920681-626a-41cc-8a46-d0b3c13237be"). InnerVolumeSpecName "kube-api-access-wz2gc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:48:16.782645 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:16.782572 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54920681-626a-41cc-8a46-d0b3c13237be-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:48:16.782645 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:16.782598 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54920681-626a-41cc-8a46-d0b3c13237be-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:48:16.782645 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:16.782614 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wz2gc\" (UniqueName: \"kubernetes.io/projected/54920681-626a-41cc-8a46-d0b3c13237be-kube-api-access-wz2gc\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:48:17.068553 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:17.065618 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w" Nov 03 13:48:17.068706 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:17.068646 5979 patch_prober.go:29] interesting pod/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Nov 03 13:48:17.068751 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:17.068708 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w" podUID="2fe2e1d4-9fac-456e-ba2b-e221b6c59e98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Nov 03 13:48:17.486502 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:17.482608 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-261647f7-7286h" Nov 03 13:48:17.486502 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:17.482623 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-261647f7-7286h" event={"ID":"54920681-626a-41cc-8a46-d0b3c13237be","Type":"ContainerDied","Data":"605bb39ef858dada87dddff3a8727e228539834a6db4fe2610994b30ad1a4895"} Nov 03 13:48:17.486502 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:17.482668 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="605bb39ef858dada87dddff3a8727e228539834a6db4fe2610994b30ad1a4895" Nov 03 13:48:18.061203 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:18.061165 5979 patch_prober.go:29] interesting pod/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Nov 03 13:48:18.061533 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:18.061504 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w" podUID="2fe2e1d4-9fac-456e-ba2b-e221b6c59e98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Nov 03 13:48:19.056579 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:19.056540 5979 patch_prober.go:29] interesting pod/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Nov 03 13:48:19.057162 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:19.057129 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w" podUID="2fe2e1d4-9fac-456e-ba2b-e221b6c59e98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Nov 03 13:48:19.591242 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:19.591212 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w" Nov 03 13:48:20.065994 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:20.065954 5979 patch_prober.go:29] interesting pod/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Nov 03 13:48:20.066656 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:20.066623 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w" podUID="2fe2e1d4-9fac-456e-ba2b-e221b6c59e98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Nov 03 13:48:21.068716 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:21.068625 5979 patch_prober.go:29] interesting pod/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" start-of-body= Nov 03 13:48:21.068716 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:21.068686 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w" podUID="2fe2e1d4-9fac-456e-ba2b-e221b6c59e98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.245:15021/healthz/ready\": dial tcp 10.134.0.245:15021: connect: connection refused" Nov 03 13:48:22.060009 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:22.059979 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w" Nov 03 13:48:22.061138 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:22.061115 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w" Nov 03 13:48:28.699270 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:28.699235 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78202144-lppdj"] Nov 03 13:48:28.699988 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:28.699953 5979 cpu_manager.go:414] "containerMap: removing container" podUID="54920681-626a-41cc-8a46-d0b3c13237be" containerName="revoke-keys" Nov 03 13:48:28.699988 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:28.699984 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54920681-626a-41cc-8a46-d0b3c13237be" containerName="revoke-keys" Nov 03 13:48:28.700163 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:28.700115 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="54920681-626a-41cc-8a46-d0b3c13237be" containerName="revoke-keys" Nov 03 13:48:28.703646 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:28.703627 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78202144-lppdj" Nov 03 13:48:28.799229 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:28.799201 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:48:28.799416 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:28.799324 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:48:28.803123 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:28.803099 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78202144-lppdj"] Nov 03 13:48:28.857417 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:28.857380 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06ffc052-27e6-4de3-a4de-002f346c9f02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78202144-lppdj\" (UID: \"06ffc052-27e6-4de3-a4de-002f346c9f02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78202144-lppdj" Nov 03 13:48:28.857657 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:28.857636 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgk4d\" (UniqueName: \"kubernetes.io/projected/06ffc052-27e6-4de3-a4de-002f346c9f02-kube-api-access-tgk4d\") pod \"maas-api-revoke-sub-78202144-lppdj\" (UID: \"06ffc052-27e6-4de3-a4de-002f346c9f02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78202144-lppdj" Nov 03 13:48:28.857705 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:28.857668 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06ffc052-27e6-4de3-a4de-002f346c9f02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78202144-lppdj\" (UID: \"06ffc052-27e6-4de3-a4de-002f346c9f02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78202144-lppdj" Nov 03 13:48:28.959349 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:28.959263 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06ffc052-27e6-4de3-a4de-002f346c9f02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78202144-lppdj\" (UID: \"06ffc052-27e6-4de3-a4de-002f346c9f02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78202144-lppdj" Nov 03 13:48:28.959540 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:28.959419 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgk4d\" (UniqueName: \"kubernetes.io/projected/06ffc052-27e6-4de3-a4de-002f346c9f02-kube-api-access-tgk4d\") pod \"maas-api-revoke-sub-78202144-lppdj\" (UID: \"06ffc052-27e6-4de3-a4de-002f346c9f02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78202144-lppdj" Nov 03 13:48:28.959540 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:28.959438 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06ffc052-27e6-4de3-a4de-002f346c9f02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78202144-lppdj\" (UID: \"06ffc052-27e6-4de3-a4de-002f346c9f02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78202144-lppdj" Nov 03 13:48:28.959939 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:28.959916 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06ffc052-27e6-4de3-a4de-002f346c9f02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-78202144-lppdj\" (UID: \"06ffc052-27e6-4de3-a4de-002f346c9f02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78202144-lppdj" Nov 03 13:48:28.961918 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:28.961893 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06ffc052-27e6-4de3-a4de-002f346c9f02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-78202144-lppdj\" (UID: \"06ffc052-27e6-4de3-a4de-002f346c9f02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78202144-lppdj" Nov 03 13:48:29.013832 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:29.013794 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgk4d\" (UniqueName: \"kubernetes.io/projected/06ffc052-27e6-4de3-a4de-002f346c9f02-kube-api-access-tgk4d\") pod \"maas-api-revoke-sub-78202144-lppdj\" (UID: \"06ffc052-27e6-4de3-a4de-002f346c9f02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-78202144-lppdj" Nov 03 13:48:29.017626 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:29.017588 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78202144-lppdj" Nov 03 13:48:29.391648 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:29.391604 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78202144-lppdj"] Nov 03 13:48:29.397859 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:48:29.397823 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06ffc052_27e6_4de3_a4de_002f346c9f02.slice/crio-50aa0f164d1e8d350abb0d66004a73d208590693f48f01d38600cc4a2fad8dce WatchSource:0}: Error finding container 50aa0f164d1e8d350abb0d66004a73d208590693f48f01d38600cc4a2fad8dce: Status 404 returned error can't find the container with id 50aa0f164d1e8d350abb0d66004a73d208590693f48f01d38600cc4a2fad8dce Nov 03 13:48:29.634263 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:29.634228 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78202144-lppdj" event={"ID":"06ffc052-27e6-4de3-a4de-002f346c9f02","Type":"ContainerStarted","Data":"50aa0f164d1e8d350abb0d66004a73d208590693f48f01d38600cc4a2fad8dce"} Nov 03 13:48:29.975899 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:29.975815 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:48:30.642659 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:30.642628 5979 generic.go:356] "Generic (PLEG): container finished" podID="06ffc052-27e6-4de3-a4de-002f346c9f02" containerID="c84e56fbead790292df2cf6619c1195b77e24295d8e89f50a5058f5df5fc942c" exitCode=0 Nov 03 13:48:30.642828 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:30.642713 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78202144-lppdj" event={"ID":"06ffc052-27e6-4de3-a4de-002f346c9f02","Type":"ContainerDied","Data":"c84e56fbead790292df2cf6619c1195b77e24295d8e89f50a5058f5df5fc942c"} Nov 03 13:48:31.927756 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:31.927693 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78202144-lppdj" Nov 03 13:48:32.022102 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:32.022064 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06ffc052-27e6-4de3-a4de-002f346c9f02-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06ffc052-27e6-4de3-a4de-002f346c9f02-maas-api-openshift-ca\") pod \"06ffc052-27e6-4de3-a4de-002f346c9f02\" (UID: \"06ffc052-27e6-4de3-a4de-002f346c9f02\") " Nov 03 13:48:32.022277 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:32.022261 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06ffc052-27e6-4de3-a4de-002f346c9f02-kube-api-access-tgk4d\" (UniqueName: \"kubernetes.io/projected/06ffc052-27e6-4de3-a4de-002f346c9f02-kube-api-access-tgk4d\") pod \"06ffc052-27e6-4de3-a4de-002f346c9f02\" (UID: \"06ffc052-27e6-4de3-a4de-002f346c9f02\") " Nov 03 13:48:32.022322 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:32.022310 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06ffc052-27e6-4de3-a4de-002f346c9f02-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06ffc052-27e6-4de3-a4de-002f346c9f02-maas-api-xks-ca\") pod \"06ffc052-27e6-4de3-a4de-002f346c9f02\" (UID: \"06ffc052-27e6-4de3-a4de-002f346c9f02\") " Nov 03 13:48:32.022523 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:32.022458 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06ffc052-27e6-4de3-a4de-002f346c9f02-maas-api-openshift-ca" pod "06ffc052-27e6-4de3-a4de-002f346c9f02" (UID: "06ffc052-27e6-4de3-a4de-002f346c9f02"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:48:32.022885 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:32.022867 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06ffc052-27e6-4de3-a4de-002f346c9f02-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:48:32.024515 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:32.024482 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06ffc052-27e6-4de3-a4de-002f346c9f02-maas-api-xks-ca" pod "06ffc052-27e6-4de3-a4de-002f346c9f02" (UID: "06ffc052-27e6-4de3-a4de-002f346c9f02"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:48:32.024736 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:32.024716 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06ffc052-27e6-4de3-a4de-002f346c9f02-kube-api-access-tgk4d" pod "06ffc052-27e6-4de3-a4de-002f346c9f02" (UID: "06ffc052-27e6-4de3-a4de-002f346c9f02"). InnerVolumeSpecName "kube-api-access-tgk4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:48:32.126114 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:32.126063 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgk4d\" (UniqueName: \"kubernetes.io/projected/06ffc052-27e6-4de3-a4de-002f346c9f02-kube-api-access-tgk4d\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:48:32.126114 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:32.126106 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06ffc052-27e6-4de3-a4de-002f346c9f02-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:48:32.188859 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:48:32.188760 5979 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06ffc052_27e6_4de3_a4de_002f346c9f02.slice\": RecentStats: unable to find data in memory cache]" Nov 03 13:48:32.679351 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:32.679315 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78202144-lppdj" Nov 03 13:48:32.679351 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:32.679337 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-78202144-lppdj" event={"ID":"06ffc052-27e6-4de3-a4de-002f346c9f02","Type":"ContainerDied","Data":"50aa0f164d1e8d350abb0d66004a73d208590693f48f01d38600cc4a2fad8dce"} Nov 03 13:48:32.679681 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:32.679417 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50aa0f164d1e8d350abb0d66004a73d208590693f48f01d38600cc4a2fad8dce" Nov 03 13:48:33.243649 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:33.243611 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-2bceb1-6f4bbf98ff-k6vzq"] Nov 03 13:48:33.245185 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:33.245156 5979 cpu_manager.go:414] "containerMap: removing container" podUID="06ffc052-27e6-4de3-a4de-002f346c9f02" containerName="revoke-keys" Nov 03 13:48:33.245404 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:33.245388 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06ffc052-27e6-4de3-a4de-002f346c9f02" containerName="revoke-keys" Nov 03 13:48:33.245734 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:33.245718 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="06ffc052-27e6-4de3-a4de-002f346c9f02" containerName="revoke-keys" Nov 03 13:48:33.263909 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:33.263880 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2bceb1-6f4bbf98ff-k6vzq" Nov 03 13:48:33.303654 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:33.303609 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-2bceb1-6f4bbf98ff-k6vzq"] Nov 03 13:48:33.321927 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:33.321893 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-2bceb1\"" Nov 03 13:48:33.322122 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:33.321893 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-2bceb1-dockercfg-8g5j4\"" Nov 03 13:48:33.375499 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:33.370399 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58lbl\" (UniqueName: \"kubernetes.io/projected/92497d85-07e4-4286-aabc-6ad4036733e4-kube-api-access-58lbl\") pod \"payload-processing-e2e-shared-a-w2-2bceb1-6f4bbf98ff-k6vzq\" (UID: \"92497d85-07e4-4286-aabc-6ad4036733e4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2bceb1-6f4bbf98ff-k6vzq" Nov 03 13:48:33.375499 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:33.373761 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92497d85-07e4-4286-aabc-6ad4036733e4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-2bceb1-6f4bbf98ff-k6vzq\" (UID: \"92497d85-07e4-4286-aabc-6ad4036733e4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2bceb1-6f4bbf98ff-k6vzq" Nov 03 13:48:33.474877 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:33.474808 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58lbl\" (UniqueName: \"kubernetes.io/projected/92497d85-07e4-4286-aabc-6ad4036733e4-kube-api-access-58lbl\") pod \"payload-processing-e2e-shared-a-w2-2bceb1-6f4bbf98ff-k6vzq\" (UID: \"92497d85-07e4-4286-aabc-6ad4036733e4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2bceb1-6f4bbf98ff-k6vzq" Nov 03 13:48:33.475124 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:33.475037 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92497d85-07e4-4286-aabc-6ad4036733e4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-2bceb1-6f4bbf98ff-k6vzq\" (UID: \"92497d85-07e4-4286-aabc-6ad4036733e4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2bceb1-6f4bbf98ff-k6vzq" Nov 03 13:48:33.475765 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:33.475744 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92497d85-07e4-4286-aabc-6ad4036733e4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-2bceb1-6f4bbf98ff-k6vzq\" (UID: \"92497d85-07e4-4286-aabc-6ad4036733e4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2bceb1-6f4bbf98ff-k6vzq" Nov 03 13:48:33.528378 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:33.528299 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58lbl\" (UniqueName: \"kubernetes.io/projected/92497d85-07e4-4286-aabc-6ad4036733e4-kube-api-access-58lbl\") pod \"payload-processing-e2e-shared-a-w2-2bceb1-6f4bbf98ff-k6vzq\" (UID: \"92497d85-07e4-4286-aabc-6ad4036733e4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2bceb1-6f4bbf98ff-k6vzq" Nov 03 13:48:33.629488 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:33.629441 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2bceb1-6f4bbf98ff-k6vzq" Nov 03 13:48:34.034941 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:34.034905 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-2bceb1-6f4bbf98ff-k6vzq"] Nov 03 13:48:34.041029 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:48:34.040998 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92497d85_07e4_4286_aabc_6ad4036733e4.slice/crio-99061d2e934f5ce18826b935dd2114ebb47d5c1b96cd21cc5365733dec72a0c8 WatchSource:0}: Error finding container 99061d2e934f5ce18826b935dd2114ebb47d5c1b96cd21cc5365733dec72a0c8: Status 404 returned error can't find the container with id 99061d2e934f5ce18826b935dd2114ebb47d5c1b96cd21cc5365733dec72a0c8 Nov 03 13:48:34.698265 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:34.698228 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2bceb1-6f4bbf98ff-k6vzq" event={"ID":"92497d85-07e4-4286-aabc-6ad4036733e4","Type":"ContainerStarted","Data":"83e5c9dfc47b7ae4cf91360c3bc28373737b434015b5ad435f80a721b81804fe"} Nov 03 13:48:34.698716 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:34.698292 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2bceb1-6f4bbf98ff-k6vzq" event={"ID":"92497d85-07e4-4286-aabc-6ad4036733e4","Type":"ContainerStarted","Data":"99061d2e934f5ce18826b935dd2114ebb47d5c1b96cd21cc5365733dec72a0c8"} Nov 03 13:48:34.698716 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:34.698321 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2bceb1-6f4bbf98ff-k6vzq" Nov 03 13:48:34.813120 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:34.813067 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2bceb1-6f4bbf98ff-k6vzq" podStartSLOduration=1.8130514789999999 podStartE2EDuration="1.813051479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:48:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:48:34.811219139 +0000 UTC m=+2596.821907372" watchObservedRunningTime="2026-11-03 13:48:34.813051479 +0000 UTC m=+2596.823739721" Nov 03 13:48:37.235651 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:48:37.235615 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:48:40.747385 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:40.747353 5979 generic.go:356] "Generic (PLEG): container finished" podID="f9af6a95-df9b-4525-a080-edf5abab2c86" containerID="ff8c0f43d62843b89727cd62c24862fa2a5c58550afe7ed0db5b1f36d43e6d75" exitCode=0 Nov 03 13:48:40.747939 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:40.747407 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr" event={"ID":"f9af6a95-df9b-4525-a080-edf5abab2c86","Type":"ContainerDied","Data":"ff8c0f43d62843b89727cd62c24862fa2a5c58550afe7ed0db5b1f36d43e6d75"} Nov 03 13:48:40.747939 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:40.747815 5979 scope.go:122] "RemoveContainer" containerID="ff8c0f43d62843b89727cd62c24862fa2a5c58550afe7ed0db5b1f36d43e6d75" Nov 03 13:48:40.749554 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:40.749523 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:48:40.749650 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:40.749595 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:48:40.749650 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:40.749628 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:48:41.774055 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:41.774015 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr" event={"ID":"f9af6a95-df9b-4525-a080-edf5abab2c86","Type":"ContainerStarted","Data":"eb2ef15ce912894a5f5e93b8b0c28247b3c015d0f8d624eb9b8785f8b79d28db"} Nov 03 13:48:42.704556 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:42.704511 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr" Nov 03 13:48:42.705874 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:42.705844 5979 patch_prober.go:29] interesting pod/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" start-of-body= Nov 03 13:48:42.706001 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:42.705911 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr" podUID="f9af6a95-df9b-4525-a080-edf5abab2c86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" Nov 03 13:48:43.704753 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:43.704718 5979 patch_prober.go:29] interesting pod/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" start-of-body= Nov 03 13:48:43.705165 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:43.704784 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr" podUID="f9af6a95-df9b-4525-a080-edf5abab2c86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" Nov 03 13:48:44.704346 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:44.704311 5979 patch_prober.go:29] interesting pod/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" start-of-body= Nov 03 13:48:44.704564 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:44.704364 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr" podUID="f9af6a95-df9b-4525-a080-edf5abab2c86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" Nov 03 13:48:44.796140 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:44.796107 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr" Nov 03 13:48:45.704150 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:45.704113 5979 patch_prober.go:29] interesting pod/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" start-of-body= Nov 03 13:48:45.704459 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:45.704425 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr" podUID="f9af6a95-df9b-4525-a080-edf5abab2c86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" Nov 03 13:48:45.716624 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:45.715426 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2bceb1-6f4bbf98ff-k6vzq" Nov 03 13:48:46.170777 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:46.170738 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t"] Nov 03 13:48:46.175080 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:46.175056 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t" Nov 03 13:48:46.209491 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:46.209440 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7512d231-097f-4192-99f7-44bb0f98bdda-istio-podinfo\") pod \"e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t\" (UID: \"7512d231-097f-4192-99f7-44bb0f98bdda\") " pod="openshift-ingress/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t" Nov 03 13:48:46.209741 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:46.209719 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7512d231-097f-4192-99f7-44bb0f98bdda-workload-socket\") pod \"e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t\" (UID: \"7512d231-097f-4192-99f7-44bb0f98bdda\") " pod="openshift-ingress/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t" Nov 03 13:48:46.209932 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:46.209913 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7512d231-097f-4192-99f7-44bb0f98bdda-workload-certs\") pod \"e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t\" (UID: \"7512d231-097f-4192-99f7-44bb0f98bdda\") " pod="openshift-ingress/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t" Nov 03 13:48:46.210065 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:46.209952 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7512d231-097f-4192-99f7-44bb0f98bdda-istiod-ca-cert\") pod \"e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t\" (UID: \"7512d231-097f-4192-99f7-44bb0f98bdda\") " pod="openshift-ingress/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t" Nov 03 13:48:46.210065 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:46.209998 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7512d231-097f-4192-99f7-44bb0f98bdda-istio-envoy\") pod \"e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t\" (UID: \"7512d231-097f-4192-99f7-44bb0f98bdda\") " pod="openshift-ingress/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t" Nov 03 13:48:46.210351 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:46.210198 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7512d231-097f-4192-99f7-44bb0f98bdda-istio-data\") pod \"e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t\" (UID: \"7512d231-097f-4192-99f7-44bb0f98bdda\") " pod="openshift-ingress/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t" Nov 03 13:48:46.210351 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:46.210252 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7512d231-097f-4192-99f7-44bb0f98bdda-istio-token\") pod \"e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t\" (UID: \"7512d231-097f-4192-99f7-44bb0f98bdda\") " pod="openshift-ingress/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t" Nov 03 13:48:46.210351 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:46.210323 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7512d231-097f-4192-99f7-44bb0f98bdda-credential-socket\") pod \"e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t\" (UID: \"7512d231-097f-4192-99f7-44bb0f98bdda\") " pod="openshift-ingress/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t" Nov 03 13:48:46.210578 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:46.210357 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6wk5\" (UniqueName: \"kubernetes.io/projected/7512d231-097f-4192-99f7-44bb0f98bdda-kube-api-access-q6wk5\") pod \"e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t\" (UID: \"7512d231-097f-4192-99f7-44bb0f98bdda\") " pod="openshift-ingress/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t" Nov 03 13:48:46.225333 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:46.225307 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-fb4ca0-openshift-default-dockercfg-x4j96\"" Nov 03 13:48:46.279685 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:46.279651 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t"] Nov 03 13:48:46.345501 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:46.345026 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7512d231-097f-4192-99f7-44bb0f98bdda-istio-podinfo\") pod \"e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t\" (UID: \"7512d231-097f-4192-99f7-44bb0f98bdda\") " pod="openshift-ingress/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t" Nov 03 13:48:46.345501 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:46.345092 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7512d231-097f-4192-99f7-44bb0f98bdda-workload-socket\") pod \"e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t\" (UID: \"7512d231-097f-4192-99f7-44bb0f98bdda\") " pod="openshift-ingress/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t" Nov 03 13:48:46.345501 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:46.345181 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7512d231-097f-4192-99f7-44bb0f98bdda-workload-certs\") pod \"e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t\" (UID: \"7512d231-097f-4192-99f7-44bb0f98bdda\") " pod="openshift-ingress/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t" Nov 03 13:48:46.345501 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:46.345220 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7512d231-097f-4192-99f7-44bb0f98bdda-istiod-ca-cert\") pod \"e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t\" (UID: \"7512d231-097f-4192-99f7-44bb0f98bdda\") " pod="openshift-ingress/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t" Nov 03 13:48:46.345501 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:46.345268 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7512d231-097f-4192-99f7-44bb0f98bdda-istio-envoy\") pod \"e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t\" (UID: \"7512d231-097f-4192-99f7-44bb0f98bdda\") " pod="openshift-ingress/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t" Nov 03 13:48:46.345501 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:46.345339 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7512d231-097f-4192-99f7-44bb0f98bdda-istio-data\") pod \"e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t\" (UID: \"7512d231-097f-4192-99f7-44bb0f98bdda\") " pod="openshift-ingress/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t" Nov 03 13:48:46.345501 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:46.345382 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7512d231-097f-4192-99f7-44bb0f98bdda-istio-token\") pod \"e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t\" (UID: \"7512d231-097f-4192-99f7-44bb0f98bdda\") " pod="openshift-ingress/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t" Nov 03 13:48:46.345501 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:46.345456 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7512d231-097f-4192-99f7-44bb0f98bdda-credential-socket\") pod \"e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t\" (UID: \"7512d231-097f-4192-99f7-44bb0f98bdda\") " pod="openshift-ingress/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t" Nov 03 13:48:46.345501 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:46.345515 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6wk5\" (UniqueName: \"kubernetes.io/projected/7512d231-097f-4192-99f7-44bb0f98bdda-kube-api-access-q6wk5\") pod \"e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t\" (UID: \"7512d231-097f-4192-99f7-44bb0f98bdda\") " pod="openshift-ingress/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t" Nov 03 13:48:46.378496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:46.346868 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7512d231-097f-4192-99f7-44bb0f98bdda-istiod-ca-cert\") pod \"e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t\" (UID: \"7512d231-097f-4192-99f7-44bb0f98bdda\") " pod="openshift-ingress/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t" Nov 03 13:48:46.385554 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:46.381106 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7512d231-097f-4192-99f7-44bb0f98bdda-istio-envoy\") pod \"e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t\" (UID: \"7512d231-097f-4192-99f7-44bb0f98bdda\") " pod="openshift-ingress/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t" Nov 03 13:48:46.385554 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:46.381136 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7512d231-097f-4192-99f7-44bb0f98bdda-workload-socket\") pod \"e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t\" (UID: \"7512d231-097f-4192-99f7-44bb0f98bdda\") " pod="openshift-ingress/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t" Nov 03 13:48:46.385554 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:46.381380 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7512d231-097f-4192-99f7-44bb0f98bdda-istio-data\") pod \"e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t\" (UID: \"7512d231-097f-4192-99f7-44bb0f98bdda\") " pod="openshift-ingress/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t" Nov 03 13:48:46.385554 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:46.381423 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7512d231-097f-4192-99f7-44bb0f98bdda-workload-certs\") pod \"e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t\" (UID: \"7512d231-097f-4192-99f7-44bb0f98bdda\") " pod="openshift-ingress/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t" Nov 03 13:48:46.385554 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:46.381699 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7512d231-097f-4192-99f7-44bb0f98bdda-credential-socket\") pod \"e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t\" (UID: \"7512d231-097f-4192-99f7-44bb0f98bdda\") " pod="openshift-ingress/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t" Nov 03 13:48:46.385554 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:46.384745 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7512d231-097f-4192-99f7-44bb0f98bdda-istio-podinfo\") pod \"e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t\" (UID: \"7512d231-097f-4192-99f7-44bb0f98bdda\") " pod="openshift-ingress/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t" Nov 03 13:48:46.416789 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:46.416754 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6wk5\" (UniqueName: \"kubernetes.io/projected/7512d231-097f-4192-99f7-44bb0f98bdda-kube-api-access-q6wk5\") pod \"e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t\" (UID: \"7512d231-097f-4192-99f7-44bb0f98bdda\") " pod="openshift-ingress/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t" Nov 03 13:48:46.456546 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:46.455178 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7512d231-097f-4192-99f7-44bb0f98bdda-istio-token\") pod \"e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t\" (UID: \"7512d231-097f-4192-99f7-44bb0f98bdda\") " pod="openshift-ingress/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t" Nov 03 13:48:46.505798 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:46.505758 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t" Nov 03 13:48:46.704731 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:46.704322 5979 patch_prober.go:29] interesting pod/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" start-of-body= Nov 03 13:48:46.704731 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:46.704398 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr" podUID="f9af6a95-df9b-4525-a080-edf5abab2c86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.252:15021/healthz/ready\": dial tcp 10.134.0.252:15021: connect: connection refused" Nov 03 13:48:46.972029 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:46.971991 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t"] Nov 03 13:48:46.979333 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:48:46.979304 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7512d231_097f_4192_99f7_44bb0f98bdda.slice/crio-76148efece7439fa0aa6651c91fd8bfa2d5ca4a9e20f7f5126d5c2dc8089a35c WatchSource:0}: Error finding container 76148efece7439fa0aa6651c91fd8bfa2d5ca4a9e20f7f5126d5c2dc8089a35c: Status 404 returned error can't find the container with id 76148efece7439fa0aa6651c91fd8bfa2d5ca4a9e20f7f5126d5c2dc8089a35c Nov 03 13:48:46.981872 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:46.981842 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:48:46.982141 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:46.981907 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:48:46.982141 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:46.981938 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:48:47.708305 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:47.708274 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr" Nov 03 13:48:47.709352 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:47.709331 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr" Nov 03 13:48:47.835576 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:47.835534 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t" event={"ID":"7512d231-097f-4192-99f7-44bb0f98bdda","Type":"ContainerStarted","Data":"599f67b0f10ae1ba88ca3a8777bf623a8dfae54ca11b20e5c4a82fc28b79a7cc"} Nov 03 13:48:47.835779 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:47.835593 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t" event={"ID":"7512d231-097f-4192-99f7-44bb0f98bdda","Type":"ContainerStarted","Data":"76148efece7439fa0aa6651c91fd8bfa2d5ca4a9e20f7f5126d5c2dc8089a35c"} Nov 03 13:48:48.041026 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:48.040905 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t" podStartSLOduration=2.0408905920000002 podStartE2EDuration="2.040890592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:48:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:48:48.039226718 +0000 UTC m=+2610.049914971" watchObservedRunningTime="2026-11-03 13:48:48.040890592 +0000 UTC m=+2610.051578833" Nov 03 13:48:48.510388 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:48.510341 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t" Nov 03 13:48:48.516677 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:48.516642 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t" Nov 03 13:48:48.842996 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:48.842903 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t" Nov 03 13:48:48.843989 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:48.843967 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t" Nov 03 13:48:49.789492 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:49.789439 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5faf5d-764fdf77d7-tm6xj"] Nov 03 13:48:49.798427 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:49.798398 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5faf5d-764fdf77d7-tm6xj" Nov 03 13:48:49.846629 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:49.846555 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5faf5d-764fdf77d7-tm6xj"] Nov 03 13:48:49.888357 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:49.888316 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-5faf5d\"" Nov 03 13:48:49.888584 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:49.888377 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-5faf5d-dockercfg-z6jbs\"" Nov 03 13:48:49.925628 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:49.925594 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e7dbc4b-d67d-4f15-bddc-088c23fdee66-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-5faf5d-764fdf77d7-tm6xj\" (UID: \"6e7dbc4b-d67d-4f15-bddc-088c23fdee66\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5faf5d-764fdf77d7-tm6xj" Nov 03 13:48:49.925628 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:49.925633 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sxtk\" (UniqueName: \"kubernetes.io/projected/6e7dbc4b-d67d-4f15-bddc-088c23fdee66-kube-api-access-8sxtk\") pod \"payload-pre-processing-e2e-shared-a-w0-5faf5d-764fdf77d7-tm6xj\" (UID: \"6e7dbc4b-d67d-4f15-bddc-088c23fdee66\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5faf5d-764fdf77d7-tm6xj" Nov 03 13:48:50.027207 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:50.027166 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e7dbc4b-d67d-4f15-bddc-088c23fdee66-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-5faf5d-764fdf77d7-tm6xj\" (UID: \"6e7dbc4b-d67d-4f15-bddc-088c23fdee66\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5faf5d-764fdf77d7-tm6xj" Nov 03 13:48:50.027207 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:50.027209 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sxtk\" (UniqueName: \"kubernetes.io/projected/6e7dbc4b-d67d-4f15-bddc-088c23fdee66-kube-api-access-8sxtk\") pod \"payload-pre-processing-e2e-shared-a-w0-5faf5d-764fdf77d7-tm6xj\" (UID: \"6e7dbc4b-d67d-4f15-bddc-088c23fdee66\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5faf5d-764fdf77d7-tm6xj" Nov 03 13:48:50.028014 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:50.027989 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e7dbc4b-d67d-4f15-bddc-088c23fdee66-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-5faf5d-764fdf77d7-tm6xj\" (UID: \"6e7dbc4b-d67d-4f15-bddc-088c23fdee66\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5faf5d-764fdf77d7-tm6xj" Nov 03 13:48:50.080851 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:50.080782 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sxtk\" (UniqueName: \"kubernetes.io/projected/6e7dbc4b-d67d-4f15-bddc-088c23fdee66-kube-api-access-8sxtk\") pod \"payload-pre-processing-e2e-shared-a-w0-5faf5d-764fdf77d7-tm6xj\" (UID: \"6e7dbc4b-d67d-4f15-bddc-088c23fdee66\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5faf5d-764fdf77d7-tm6xj" Nov 03 13:48:50.112749 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:50.112722 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5faf5d-764fdf77d7-tm6xj" Nov 03 13:48:50.489765 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:50.489726 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5faf5d-764fdf77d7-tm6xj"] Nov 03 13:48:50.495530 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:48:50.495498 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e7dbc4b_d67d_4f15_bddc_088c23fdee66.slice/crio-44b1f888a95cc4615fef8525ad1601ee7519c2014473b5c4881bcb9fb81c6dc5 WatchSource:0}: Error finding container 44b1f888a95cc4615fef8525ad1601ee7519c2014473b5c4881bcb9fb81c6dc5: Status 404 returned error can't find the container with id 44b1f888a95cc4615fef8525ad1601ee7519c2014473b5c4881bcb9fb81c6dc5 Nov 03 13:48:50.860402 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:50.860308 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5faf5d-764fdf77d7-tm6xj" event={"ID":"6e7dbc4b-d67d-4f15-bddc-088c23fdee66","Type":"ContainerStarted","Data":"bd872d66863ba45799ba82fda949726b450ba561baa4ad85cb810c6d59b208f5"} Nov 03 13:48:50.860402 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:50.860377 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5faf5d-764fdf77d7-tm6xj" event={"ID":"6e7dbc4b-d67d-4f15-bddc-088c23fdee66","Type":"ContainerStarted","Data":"44b1f888a95cc4615fef8525ad1601ee7519c2014473b5c4881bcb9fb81c6dc5"} Nov 03 13:48:50.860891 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:50.860407 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5faf5d-764fdf77d7-tm6xj" Nov 03 13:48:50.974764 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:50.974702 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5faf5d-764fdf77d7-tm6xj" podStartSLOduration=1.9746842500000001 podStartE2EDuration="1.97468425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:48:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:48:50.970053714 +0000 UTC m=+2612.980741956" watchObservedRunningTime="2026-11-03 13:48:50.97468425 +0000 UTC m=+2612.985372492" Nov 03 13:48:51.943031 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:51.942993 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj"] Nov 03 13:48:51.947784 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:51.947755 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj" Nov 03 13:48:52.080493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:52.071877 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm725\" (UniqueName: \"kubernetes.io/projected/210b9bed-6df7-4f71-982e-ce2f72287b17-kube-api-access-fm725\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj\" (UID: \"210b9bed-6df7-4f71-982e-ce2f72287b17\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj" Nov 03 13:48:52.080493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:52.072030 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/210b9bed-6df7-4f71-982e-ce2f72287b17-workload-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj\" (UID: \"210b9bed-6df7-4f71-982e-ce2f72287b17\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj" Nov 03 13:48:52.080493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:52.072097 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/210b9bed-6df7-4f71-982e-ce2f72287b17-credential-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj\" (UID: \"210b9bed-6df7-4f71-982e-ce2f72287b17\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj" Nov 03 13:48:52.080493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:52.072121 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/210b9bed-6df7-4f71-982e-ce2f72287b17-workload-certs\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj\" (UID: \"210b9bed-6df7-4f71-982e-ce2f72287b17\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj" Nov 03 13:48:52.080493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:52.072159 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/210b9bed-6df7-4f71-982e-ce2f72287b17-istio-data\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj\" (UID: \"210b9bed-6df7-4f71-982e-ce2f72287b17\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj" Nov 03 13:48:52.080493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:52.072201 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/210b9bed-6df7-4f71-982e-ce2f72287b17-istio-token\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj\" (UID: \"210b9bed-6df7-4f71-982e-ce2f72287b17\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj" Nov 03 13:48:52.080493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:52.072225 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/210b9bed-6df7-4f71-982e-ce2f72287b17-istio-envoy\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj\" (UID: \"210b9bed-6df7-4f71-982e-ce2f72287b17\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj" Nov 03 13:48:52.080493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:52.072291 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/210b9bed-6df7-4f71-982e-ce2f72287b17-istio-podinfo\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj\" (UID: \"210b9bed-6df7-4f71-982e-ce2f72287b17\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj" Nov 03 13:48:52.080493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:52.072345 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/210b9bed-6df7-4f71-982e-ce2f72287b17-istiod-ca-cert\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj\" (UID: \"210b9bed-6df7-4f71-982e-ce2f72287b17\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj" Nov 03 13:48:52.163659 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:52.163625 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj"] Nov 03 13:48:52.175674 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:52.175622 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/210b9bed-6df7-4f71-982e-ce2f72287b17-workload-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj\" (UID: \"210b9bed-6df7-4f71-982e-ce2f72287b17\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj" Nov 03 13:48:52.176029 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:52.176006 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/210b9bed-6df7-4f71-982e-ce2f72287b17-credential-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj\" (UID: \"210b9bed-6df7-4f71-982e-ce2f72287b17\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj" Nov 03 13:48:52.176196 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:52.176177 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/210b9bed-6df7-4f71-982e-ce2f72287b17-workload-certs\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj\" (UID: \"210b9bed-6df7-4f71-982e-ce2f72287b17\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj" Nov 03 13:48:52.176346 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:52.176332 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/210b9bed-6df7-4f71-982e-ce2f72287b17-istio-data\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj\" (UID: \"210b9bed-6df7-4f71-982e-ce2f72287b17\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj" Nov 03 13:48:52.176500 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:52.176463 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/210b9bed-6df7-4f71-982e-ce2f72287b17-istio-token\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj\" (UID: \"210b9bed-6df7-4f71-982e-ce2f72287b17\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj" Nov 03 13:48:52.176621 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:52.176608 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/210b9bed-6df7-4f71-982e-ce2f72287b17-istio-envoy\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj\" (UID: \"210b9bed-6df7-4f71-982e-ce2f72287b17\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj" Nov 03 13:48:52.176773 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:52.176760 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/210b9bed-6df7-4f71-982e-ce2f72287b17-istio-podinfo\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj\" (UID: \"210b9bed-6df7-4f71-982e-ce2f72287b17\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj" Nov 03 13:48:52.176938 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:52.176923 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/210b9bed-6df7-4f71-982e-ce2f72287b17-istiod-ca-cert\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj\" (UID: \"210b9bed-6df7-4f71-982e-ce2f72287b17\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj" Nov 03 13:48:52.177109 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:52.177089 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm725\" (UniqueName: \"kubernetes.io/projected/210b9bed-6df7-4f71-982e-ce2f72287b17-kube-api-access-fm725\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj\" (UID: \"210b9bed-6df7-4f71-982e-ce2f72287b17\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj" Nov 03 13:48:52.178083 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:52.178058 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/210b9bed-6df7-4f71-982e-ce2f72287b17-workload-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj\" (UID: \"210b9bed-6df7-4f71-982e-ce2f72287b17\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj" Nov 03 13:48:52.178486 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:52.178450 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/210b9bed-6df7-4f71-982e-ce2f72287b17-credential-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj\" (UID: \"210b9bed-6df7-4f71-982e-ce2f72287b17\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj" Nov 03 13:48:52.178847 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:52.178828 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/210b9bed-6df7-4f71-982e-ce2f72287b17-workload-certs\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj\" (UID: \"210b9bed-6df7-4f71-982e-ce2f72287b17\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj" Nov 03 13:48:52.179244 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:52.179225 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/210b9bed-6df7-4f71-982e-ce2f72287b17-istio-data\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj\" (UID: \"210b9bed-6df7-4f71-982e-ce2f72287b17\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj" Nov 03 13:48:52.186542 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:52.186512 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/210b9bed-6df7-4f71-982e-ce2f72287b17-istiod-ca-cert\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj\" (UID: \"210b9bed-6df7-4f71-982e-ce2f72287b17\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj" Nov 03 13:48:52.190142 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:52.190059 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/210b9bed-6df7-4f71-982e-ce2f72287b17-istio-envoy\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj\" (UID: \"210b9bed-6df7-4f71-982e-ce2f72287b17\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj" Nov 03 13:48:52.192769 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:52.192728 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/210b9bed-6df7-4f71-982e-ce2f72287b17-istio-podinfo\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj\" (UID: \"210b9bed-6df7-4f71-982e-ce2f72287b17\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj" Nov 03 13:48:52.231549 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:52.231025 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm725\" (UniqueName: \"kubernetes.io/projected/210b9bed-6df7-4f71-982e-ce2f72287b17-kube-api-access-fm725\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj\" (UID: \"210b9bed-6df7-4f71-982e-ce2f72287b17\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj" Nov 03 13:48:52.232437 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:52.232406 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/210b9bed-6df7-4f71-982e-ce2f72287b17-istio-token\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj\" (UID: \"210b9bed-6df7-4f71-982e-ce2f72287b17\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj" Nov 03 13:48:52.260961 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:52.260925 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj" Nov 03 13:48:52.697023 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:52.696980 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj"] Nov 03 13:48:52.707039 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:52.707003 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:48:52.707146 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:52.707074 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:48:52.707146 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:52.707111 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:48:52.877460 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:52.877427 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj" event={"ID":"210b9bed-6df7-4f71-982e-ce2f72287b17","Type":"ContainerStarted","Data":"a2db1f88c60565bd7643e74b1123e391ad271b0cd09eed4cd8718876b6c26f87"} Nov 03 13:48:52.877659 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:52.877500 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj" event={"ID":"210b9bed-6df7-4f71-982e-ce2f72287b17","Type":"ContainerStarted","Data":"9419e77ab6fdb1bb18a6cf93fd5748c40916bd44d01795dce8b7bd95f45a5b3d"} Nov 03 13:48:52.993967 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:52.993868 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj" podStartSLOduration=1.993851994 podStartE2EDuration="1.993851994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:48:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:48:52.992629406 +0000 UTC m=+2615.003317648" watchObservedRunningTime="2026-11-03 13:48:52.993851994 +0000 UTC m=+2615.004540236" Nov 03 13:48:53.262753 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:53.262626 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj" Nov 03 13:48:53.267634 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:53.267596 5979 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" start-of-body= Nov 03 13:48:53.267937 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:53.267907 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj" podUID="210b9bed-6df7-4f71-982e-ce2f72287b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" Nov 03 13:48:54.267908 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:54.267868 5979 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" start-of-body= Nov 03 13:48:54.268591 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:54.268556 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj" podUID="210b9bed-6df7-4f71-982e-ce2f72287b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" Nov 03 13:48:55.269364 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:55.268905 5979 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" start-of-body= Nov 03 13:48:55.269364 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:55.268981 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj" podUID="210b9bed-6df7-4f71-982e-ce2f72287b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" Nov 03 13:48:56.263493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:56.262858 5979 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" start-of-body= Nov 03 13:48:56.263493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:56.262920 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj" podUID="210b9bed-6df7-4f71-982e-ce2f72287b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" Nov 03 13:48:57.268822 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:57.268775 5979 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" start-of-body= Nov 03 13:48:57.269225 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:57.268841 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj" podUID="210b9bed-6df7-4f71-982e-ce2f72287b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" Nov 03 13:48:58.265508 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:58.264948 5979 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" start-of-body= Nov 03 13:48:58.265508 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:58.265030 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj" podUID="210b9bed-6df7-4f71-982e-ce2f72287b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" Nov 03 13:48:59.262052 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:59.261950 5979 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" start-of-body= Nov 03 13:48:59.262052 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:48:59.262010 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj" podUID="210b9bed-6df7-4f71-982e-ce2f72287b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.11:15021/healthz/ready\": dial tcp 10.134.1.11:15021: connect: connection refused" Nov 03 13:49:00.266658 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:00.266626 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj" Nov 03 13:49:00.267152 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:00.267133 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj" Nov 03 13:49:00.267992 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:00.267971 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj" Nov 03 13:49:01.871678 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:01.871649 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5faf5d-764fdf77d7-tm6xj" Nov 03 13:49:07.235199 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:49:07.235156 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:49:07.328755 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:07.328720 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:49:17.621502 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:17.621447 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd6096f9-dbjmz"] Nov 03 13:49:17.626232 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:17.626203 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd6096f9-dbjmz" Nov 03 13:49:17.675795 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:17.675760 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:49:17.747511 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:17.747448 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd6096f9-dbjmz"] Nov 03 13:49:17.751086 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:17.750355 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb1baabd-140e-4516-8044-ea044b02afab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd6096f9-dbjmz\" (UID: \"cb1baabd-140e-4516-8044-ea044b02afab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd6096f9-dbjmz" Nov 03 13:49:17.751455 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:17.751410 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb1baabd-140e-4516-8044-ea044b02afab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd6096f9-dbjmz\" (UID: \"cb1baabd-140e-4516-8044-ea044b02afab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd6096f9-dbjmz" Nov 03 13:49:17.751712 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:17.751681 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svvbc\" (UniqueName: \"kubernetes.io/projected/cb1baabd-140e-4516-8044-ea044b02afab-kube-api-access-svvbc\") pod \"maas-api-revoke-sub-bd6096f9-dbjmz\" (UID: \"cb1baabd-140e-4516-8044-ea044b02afab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd6096f9-dbjmz" Nov 03 13:49:17.751947 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:17.751929 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:49:17.757686 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:17.757657 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4b9658-999455b7f-l6wwv"] Nov 03 13:49:17.852915 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:17.852879 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb1baabd-140e-4516-8044-ea044b02afab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd6096f9-dbjmz\" (UID: \"cb1baabd-140e-4516-8044-ea044b02afab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd6096f9-dbjmz" Nov 03 13:49:17.853107 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:17.852943 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb1baabd-140e-4516-8044-ea044b02afab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd6096f9-dbjmz\" (UID: \"cb1baabd-140e-4516-8044-ea044b02afab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd6096f9-dbjmz" Nov 03 13:49:17.853107 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:17.852995 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svvbc\" (UniqueName: \"kubernetes.io/projected/cb1baabd-140e-4516-8044-ea044b02afab-kube-api-access-svvbc\") pod \"maas-api-revoke-sub-bd6096f9-dbjmz\" (UID: \"cb1baabd-140e-4516-8044-ea044b02afab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd6096f9-dbjmz" Nov 03 13:49:17.853690 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:17.853660 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb1baabd-140e-4516-8044-ea044b02afab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd6096f9-dbjmz\" (UID: \"cb1baabd-140e-4516-8044-ea044b02afab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd6096f9-dbjmz" Nov 03 13:49:17.855316 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:17.855292 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb1baabd-140e-4516-8044-ea044b02afab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd6096f9-dbjmz\" (UID: \"cb1baabd-140e-4516-8044-ea044b02afab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd6096f9-dbjmz" Nov 03 13:49:17.861196 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:17.861172 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4b9658-7c5887bdf9-8nldv"] Nov 03 13:49:17.906709 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:17.906680 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svvbc\" (UniqueName: \"kubernetes.io/projected/cb1baabd-140e-4516-8044-ea044b02afab-kube-api-access-svvbc\") pod \"maas-api-revoke-sub-bd6096f9-dbjmz\" (UID: \"cb1baabd-140e-4516-8044-ea044b02afab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd6096f9-dbjmz" Nov 03 13:49:17.944599 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:17.944566 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd6096f9-dbjmz" Nov 03 13:49:18.350906 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:18.347415 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd6096f9-dbjmz"] Nov 03 13:49:19.045121 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:19.045020 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65aae740-8thwr"] Nov 03 13:49:19.048741 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:19.048712 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65aae740-8thwr"] Nov 03 13:49:19.098300 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:19.098261 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd6096f9-dbjmz" event={"ID":"cb1baabd-140e-4516-8044-ea044b02afab","Type":"ContainerStarted","Data":"4186b6866a3ddff825ba1618ba27a4245046474dc8c67a7e89b6775fd4efa34b"} Nov 03 13:49:19.098503 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:19.098357 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd6096f9-dbjmz" event={"ID":"cb1baabd-140e-4516-8044-ea044b02afab","Type":"ContainerStarted","Data":"91b909f9d7b125196797c92ddb5e564e7f6aa619253346289e39578bbafe0c40"} Nov 03 13:49:19.206589 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:19.206538 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd6096f9-dbjmz" podStartSLOduration=2.206523801 podStartE2EDuration="2.206523801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:49:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:49:19.205834995 +0000 UTC m=+2641.216523237" watchObservedRunningTime="2026-11-03 13:49:19.206523801 +0000 UTC m=+2641.217212043" Nov 03 13:49:20.107363 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:20.107330 5979 generic.go:356] "Generic (PLEG): container finished" podID="cb1baabd-140e-4516-8044-ea044b02afab" containerID="4186b6866a3ddff825ba1618ba27a4245046474dc8c67a7e89b6775fd4efa34b" exitCode=0 Nov 03 13:49:20.107859 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:20.107383 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd6096f9-dbjmz" event={"ID":"cb1baabd-140e-4516-8044-ea044b02afab","Type":"ContainerDied","Data":"4186b6866a3ddff825ba1618ba27a4245046474dc8c67a7e89b6775fd4efa34b"} Nov 03 13:49:20.115892 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:20.115864 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46aeee7c-881c-4bb5-8f0c-99cda4c0f2d9" path="/var/lib/kubelet/pods/46aeee7c-881c-4bb5-8f0c-99cda4c0f2d9/volumes" Nov 03 13:49:21.258911 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:21.258881 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd6096f9-dbjmz" Nov 03 13:49:21.412349 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:21.412312 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb1baabd-140e-4516-8044-ea044b02afab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb1baabd-140e-4516-8044-ea044b02afab-maas-api-openshift-ca\") pod \"cb1baabd-140e-4516-8044-ea044b02afab\" (UID: \"cb1baabd-140e-4516-8044-ea044b02afab\") " Nov 03 13:49:21.412562 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:21.412374 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb1baabd-140e-4516-8044-ea044b02afab-kube-api-access-svvbc\" (UniqueName: \"kubernetes.io/projected/cb1baabd-140e-4516-8044-ea044b02afab-kube-api-access-svvbc\") pod \"cb1baabd-140e-4516-8044-ea044b02afab\" (UID: \"cb1baabd-140e-4516-8044-ea044b02afab\") " Nov 03 13:49:21.412634 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:21.412588 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb1baabd-140e-4516-8044-ea044b02afab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb1baabd-140e-4516-8044-ea044b02afab-maas-api-xks-ca\") pod \"cb1baabd-140e-4516-8044-ea044b02afab\" (UID: \"cb1baabd-140e-4516-8044-ea044b02afab\") " Nov 03 13:49:21.412779 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:21.412751 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb1baabd-140e-4516-8044-ea044b02afab-maas-api-openshift-ca" pod "cb1baabd-140e-4516-8044-ea044b02afab" (UID: "cb1baabd-140e-4516-8044-ea044b02afab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:49:21.413145 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:21.413121 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb1baabd-140e-4516-8044-ea044b02afab-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:49:21.414617 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:21.414586 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb1baabd-140e-4516-8044-ea044b02afab-maas-api-xks-ca" pod "cb1baabd-140e-4516-8044-ea044b02afab" (UID: "cb1baabd-140e-4516-8044-ea044b02afab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:49:21.414754 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:21.414736 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb1baabd-140e-4516-8044-ea044b02afab-kube-api-access-svvbc" pod "cb1baabd-140e-4516-8044-ea044b02afab" (UID: "cb1baabd-140e-4516-8044-ea044b02afab"). InnerVolumeSpecName "kube-api-access-svvbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:49:21.514934 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:21.514899 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svvbc\" (UniqueName: \"kubernetes.io/projected/cb1baabd-140e-4516-8044-ea044b02afab-kube-api-access-svvbc\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:49:21.514934 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:21.514927 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb1baabd-140e-4516-8044-ea044b02afab-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:49:22.122358 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:22.122326 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd6096f9-dbjmz" Nov 03 13:49:22.122569 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:22.122370 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd6096f9-dbjmz" event={"ID":"cb1baabd-140e-4516-8044-ea044b02afab","Type":"ContainerDied","Data":"91b909f9d7b125196797c92ddb5e564e7f6aa619253346289e39578bbafe0c40"} Nov 03 13:49:22.122569 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:22.122425 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91b909f9d7b125196797c92ddb5e564e7f6aa619253346289e39578bbafe0c40" Nov 03 13:49:22.757995 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:22.757951 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-4b9658-999455b7f-l6wwv" podUID="5f2c1f4c-f81b-46fc-9f29-22ad372565c9" containerName="payload-processing" containerID="cri-o://53c5ae4fc25865cec1716c0e7d3504fc7e1482a2b300f53e637bb50ee804d21c" gracePeriod=25 Nov 03 13:49:22.861449 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:22.861390 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4b9658-7c5887bdf9-8nldv" podUID="76ffa3a3-1a28-4468-aff0-b1e96f33d3e8" containerName="payload-pre-processing" containerID="cri-o://6252df7f84d7b9d27a8b06e80c4cff2f5eeec24816df2c3ea9fd4c1df7a8c1db" gracePeriod=25 Nov 03 13:49:27.483407 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:27.483364 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4b9658-7c5887bdf9-8nldv" podUID="76ffa3a3-1a28-4468-aff0-b1e96f33d3e8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:49:28.099908 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:28.099873 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-977f32-7bcd65dd7f-cjkdb"] Nov 03 13:49:28.210903 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:28.207802 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-977f32-576d49df6c-8khqq"] Nov 03 13:49:28.491151 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:28.491116 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-4b9658-999455b7f-l6wwv" podUID="5f2c1f4c-f81b-46fc-9f29-22ad372565c9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:49:33.100589 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:33.100528 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-977f32-7bcd65dd7f-cjkdb" podUID="dd8dc463-bd1d-4451-afba-337d5a6a74ba" containerName="payload-processing" containerID="cri-o://0495da0f0dc82e18f272f20a05249c9fdbbbf7a55e35fd6e8b4f87b509532521" gracePeriod=25 Nov 03 13:49:33.224041 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:33.223980 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-977f32-576d49df6c-8khqq" podUID="5dc7b3ba-e302-41a5-8ef5-555da33e6222" containerName="payload-pre-processing" containerID="cri-o://f697b95b7979a2d2a29fd81240a7e51820f24930950fd03163b2404089dd582b" gracePeriod=25 Nov 03 13:49:33.421628 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:33.421578 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-977f32-7bcd65dd7f-cjkdb" Nov 03 13:49:33.497046 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:33.497017 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd8dc463-bd1d-4451-afba-337d5a6a74ba-kube-api-access-zsd9z\" (UniqueName: \"kubernetes.io/projected/dd8dc463-bd1d-4451-afba-337d5a6a74ba-kube-api-access-zsd9z\") pod \"dd8dc463-bd1d-4451-afba-337d5a6a74ba\" (UID: \"dd8dc463-bd1d-4451-afba-337d5a6a74ba\") " Nov 03 13:49:33.497276 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:33.497260 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd8dc463-bd1d-4451-afba-337d5a6a74ba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd8dc463-bd1d-4451-afba-337d5a6a74ba-plugins-config-volume\") pod \"dd8dc463-bd1d-4451-afba-337d5a6a74ba\" (UID: \"dd8dc463-bd1d-4451-afba-337d5a6a74ba\") " Nov 03 13:49:33.497710 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:33.497684 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd8dc463-bd1d-4451-afba-337d5a6a74ba-plugins-config-volume" pod "dd8dc463-bd1d-4451-afba-337d5a6a74ba" (UID: "dd8dc463-bd1d-4451-afba-337d5a6a74ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:49:33.498989 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:33.498970 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd8dc463-bd1d-4451-afba-337d5a6a74ba-kube-api-access-zsd9z" pod "dd8dc463-bd1d-4451-afba-337d5a6a74ba" (UID: "dd8dc463-bd1d-4451-afba-337d5a6a74ba"). InnerVolumeSpecName "kube-api-access-zsd9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:49:33.510448 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:33.510425 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-977f32-576d49df6c-8khqq" Nov 03 13:49:33.599152 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:33.599115 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dc7b3ba-e302-41a5-8ef5-555da33e6222-kube-api-access-4xpwq\" (UniqueName: \"kubernetes.io/projected/5dc7b3ba-e302-41a5-8ef5-555da33e6222-kube-api-access-4xpwq\") pod \"5dc7b3ba-e302-41a5-8ef5-555da33e6222\" (UID: \"5dc7b3ba-e302-41a5-8ef5-555da33e6222\") " Nov 03 13:49:33.599333 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:33.599223 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dc7b3ba-e302-41a5-8ef5-555da33e6222-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dc7b3ba-e302-41a5-8ef5-555da33e6222-plugins-config-volume\") pod \"5dc7b3ba-e302-41a5-8ef5-555da33e6222\" (UID: \"5dc7b3ba-e302-41a5-8ef5-555da33e6222\") " Nov 03 13:49:33.599671 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:33.599647 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dc7b3ba-e302-41a5-8ef5-555da33e6222-plugins-config-volume" pod "5dc7b3ba-e302-41a5-8ef5-555da33e6222" (UID: "5dc7b3ba-e302-41a5-8ef5-555da33e6222"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:49:33.599775 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:33.599760 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd8dc463-bd1d-4451-afba-337d5a6a74ba-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:49:33.599819 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:33.599780 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsd9z\" (UniqueName: \"kubernetes.io/projected/dd8dc463-bd1d-4451-afba-337d5a6a74ba-kube-api-access-zsd9z\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:49:33.599819 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:33.599791 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dc7b3ba-e302-41a5-8ef5-555da33e6222-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:49:33.601224 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:33.601196 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dc7b3ba-e302-41a5-8ef5-555da33e6222-kube-api-access-4xpwq" pod "5dc7b3ba-e302-41a5-8ef5-555da33e6222" (UID: "5dc7b3ba-e302-41a5-8ef5-555da33e6222"). InnerVolumeSpecName "kube-api-access-4xpwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:49:33.700858 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:33.700822 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xpwq\" (UniqueName: \"kubernetes.io/projected/5dc7b3ba-e302-41a5-8ef5-555da33e6222-kube-api-access-4xpwq\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:49:34.255722 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:34.255678 5979 generic.go:356] "Generic (PLEG): container finished" podID="5dc7b3ba-e302-41a5-8ef5-555da33e6222" containerID="f697b95b7979a2d2a29fd81240a7e51820f24930950fd03163b2404089dd582b" exitCode=0 Nov 03 13:49:34.256369 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:34.256242 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-977f32-576d49df6c-8khqq" Nov 03 13:49:34.256601 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:34.256260 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-977f32-576d49df6c-8khqq" event={"ID":"5dc7b3ba-e302-41a5-8ef5-555da33e6222","Type":"ContainerDied","Data":"f697b95b7979a2d2a29fd81240a7e51820f24930950fd03163b2404089dd582b"} Nov 03 13:49:34.257037 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:34.257014 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-977f32-576d49df6c-8khqq" event={"ID":"5dc7b3ba-e302-41a5-8ef5-555da33e6222","Type":"ContainerDied","Data":"61ca111b33aeadc4d410371fc37c4ae8229e24a01fd3aa37ff850720f5e1375e"} Nov 03 13:49:34.257196 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:34.257145 5979 scope.go:122] "RemoveContainer" containerID="f697b95b7979a2d2a29fd81240a7e51820f24930950fd03163b2404089dd582b" Nov 03 13:49:34.260494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:34.260300 5979 generic.go:356] "Generic (PLEG): container finished" podID="dd8dc463-bd1d-4451-afba-337d5a6a74ba" containerID="0495da0f0dc82e18f272f20a05249c9fdbbbf7a55e35fd6e8b4f87b509532521" exitCode=0 Nov 03 13:49:34.260612 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:34.260597 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-977f32-7bcd65dd7f-cjkdb" event={"ID":"dd8dc463-bd1d-4451-afba-337d5a6a74ba","Type":"ContainerDied","Data":"0495da0f0dc82e18f272f20a05249c9fdbbbf7a55e35fd6e8b4f87b509532521"} Nov 03 13:49:34.260686 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:34.260645 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-977f32-7bcd65dd7f-cjkdb" event={"ID":"dd8dc463-bd1d-4451-afba-337d5a6a74ba","Type":"ContainerDied","Data":"9759a07aa9c88f8b1aba18331ea20c6969e2761b178ca90137dcea2418de55c7"} Nov 03 13:49:34.260751 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:34.260722 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-977f32-7bcd65dd7f-cjkdb" Nov 03 13:49:34.273746 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:34.272779 5979 scope.go:122] "RemoveContainer" containerID="f697b95b7979a2d2a29fd81240a7e51820f24930950fd03163b2404089dd582b" Nov 03 13:49:34.273746 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:49:34.273183 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f697b95b7979a2d2a29fd81240a7e51820f24930950fd03163b2404089dd582b\": container with ID starting with f697b95b7979a2d2a29fd81240a7e51820f24930950fd03163b2404089dd582b not found: ID does not exist" containerID="f697b95b7979a2d2a29fd81240a7e51820f24930950fd03163b2404089dd582b" Nov 03 13:49:34.273746 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:34.273214 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f697b95b7979a2d2a29fd81240a7e51820f24930950fd03163b2404089dd582b"} err="failed to get container status \"f697b95b7979a2d2a29fd81240a7e51820f24930950fd03163b2404089dd582b\": rpc error: code = NotFound desc = could not find container \"f697b95b7979a2d2a29fd81240a7e51820f24930950fd03163b2404089dd582b\": container with ID starting with f697b95b7979a2d2a29fd81240a7e51820f24930950fd03163b2404089dd582b not found: ID does not exist" Nov 03 13:49:34.273746 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:34.273233 5979 scope.go:122] "RemoveContainer" containerID="0495da0f0dc82e18f272f20a05249c9fdbbbf7a55e35fd6e8b4f87b509532521" Nov 03 13:49:34.283406 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:34.283388 5979 scope.go:122] "RemoveContainer" containerID="0495da0f0dc82e18f272f20a05249c9fdbbbf7a55e35fd6e8b4f87b509532521" Nov 03 13:49:34.283685 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:49:34.283662 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0495da0f0dc82e18f272f20a05249c9fdbbbf7a55e35fd6e8b4f87b509532521\": container with ID starting with 0495da0f0dc82e18f272f20a05249c9fdbbbf7a55e35fd6e8b4f87b509532521 not found: ID does not exist" containerID="0495da0f0dc82e18f272f20a05249c9fdbbbf7a55e35fd6e8b4f87b509532521" Nov 03 13:49:34.283754 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:34.283691 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0495da0f0dc82e18f272f20a05249c9fdbbbf7a55e35fd6e8b4f87b509532521"} err="failed to get container status \"0495da0f0dc82e18f272f20a05249c9fdbbbf7a55e35fd6e8b4f87b509532521\": rpc error: code = NotFound desc = could not find container \"0495da0f0dc82e18f272f20a05249c9fdbbbf7a55e35fd6e8b4f87b509532521\": container with ID starting with 0495da0f0dc82e18f272f20a05249c9fdbbbf7a55e35fd6e8b4f87b509532521 not found: ID does not exist" Nov 03 13:49:34.411178 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:34.411114 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-977f32-7bcd65dd7f-cjkdb"] Nov 03 13:49:34.457419 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:34.457389 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-977f32-7bcd65dd7f-cjkdb"] Nov 03 13:49:34.518329 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:34.518280 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-977f32-576d49df6c-8khqq"] Nov 03 13:49:34.521867 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:34.521834 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-977f32-576d49df6c-8khqq"] Nov 03 13:49:35.080424 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:35.080386 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4b9658-86c4d8b589-5fdvq"] Nov 03 13:49:35.080655 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:35.080628 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4b9658-86c4d8b589-5fdvq" podUID="f79a66f2-0c10-4260-ba7e-6cc21b2a52c4" containerName="maas-api" containerID="cri-o://e3727ad043e5cc405c1320025081fb513d2a43d0b312e4197751d5b07a965f60" gracePeriod=30 Nov 03 13:49:36.028531 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:36.028494 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4b9658-86c4d8b589-5fdvq" podUID="f79a66f2-0c10-4260-ba7e-6cc21b2a52c4" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.217:8443/health\": dial tcp 10.134.0.217:8443: connect: connection refused" Nov 03 13:49:36.174605 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:36.174564 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dc7b3ba-e302-41a5-8ef5-555da33e6222" path="/var/lib/kubelet/pods/5dc7b3ba-e302-41a5-8ef5-555da33e6222/volumes" Nov 03 13:49:36.175754 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:36.175606 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd8dc463-bd1d-4451-afba-337d5a6a74ba" path="/var/lib/kubelet/pods/dd8dc463-bd1d-4451-afba-337d5a6a74ba/volumes" Nov 03 13:49:36.280254 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:36.280179 5979 generic.go:356] "Generic (PLEG): container finished" podID="f79a66f2-0c10-4260-ba7e-6cc21b2a52c4" containerID="e3727ad043e5cc405c1320025081fb513d2a43d0b312e4197751d5b07a965f60" exitCode=0 Nov 03 13:49:36.280400 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:36.280247 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4b9658-86c4d8b589-5fdvq" event={"ID":"f79a66f2-0c10-4260-ba7e-6cc21b2a52c4","Type":"ContainerDied","Data":"e3727ad043e5cc405c1320025081fb513d2a43d0b312e4197751d5b07a965f60"} Nov 03 13:49:36.370817 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:36.370790 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4b9658-86c4d8b589-5fdvq" Nov 03 13:49:36.450850 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:36.450807 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f79a66f2-0c10-4260-ba7e-6cc21b2a52c4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f79a66f2-0c10-4260-ba7e-6cc21b2a52c4-metrics-certs\") pod \"f79a66f2-0c10-4260-ba7e-6cc21b2a52c4\" (UID: \"f79a66f2-0c10-4260-ba7e-6cc21b2a52c4\") " Nov 03 13:49:36.451064 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:36.450919 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f79a66f2-0c10-4260-ba7e-6cc21b2a52c4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f79a66f2-0c10-4260-ba7e-6cc21b2a52c4-maas-api-tls\") pod \"f79a66f2-0c10-4260-ba7e-6cc21b2a52c4\" (UID: \"f79a66f2-0c10-4260-ba7e-6cc21b2a52c4\") " Nov 03 13:49:36.451064 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:36.450952 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f79a66f2-0c10-4260-ba7e-6cc21b2a52c4-kube-api-access-hkwv9\" (UniqueName: \"kubernetes.io/projected/f79a66f2-0c10-4260-ba7e-6cc21b2a52c4-kube-api-access-hkwv9\") pod \"f79a66f2-0c10-4260-ba7e-6cc21b2a52c4\" (UID: \"f79a66f2-0c10-4260-ba7e-6cc21b2a52c4\") " Nov 03 13:49:36.451064 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:36.451009 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f79a66f2-0c10-4260-ba7e-6cc21b2a52c4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f79a66f2-0c10-4260-ba7e-6cc21b2a52c4-openshift-service-ca\") pod \"f79a66f2-0c10-4260-ba7e-6cc21b2a52c4\" (UID: \"f79a66f2-0c10-4260-ba7e-6cc21b2a52c4\") " Nov 03 13:49:36.451448 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:36.451415 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f79a66f2-0c10-4260-ba7e-6cc21b2a52c4-openshift-service-ca" pod "f79a66f2-0c10-4260-ba7e-6cc21b2a52c4" (UID: "f79a66f2-0c10-4260-ba7e-6cc21b2a52c4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:49:36.451672 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:36.451651 5979 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f79a66f2-0c10-4260-ba7e-6cc21b2a52c4-openshift-service-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:49:36.453253 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:36.453209 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f79a66f2-0c10-4260-ba7e-6cc21b2a52c4-maas-api-tls" pod "f79a66f2-0c10-4260-ba7e-6cc21b2a52c4" (UID: "f79a66f2-0c10-4260-ba7e-6cc21b2a52c4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:49:36.453606 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:36.453577 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f79a66f2-0c10-4260-ba7e-6cc21b2a52c4-metrics-certs" pod "f79a66f2-0c10-4260-ba7e-6cc21b2a52c4" (UID: "f79a66f2-0c10-4260-ba7e-6cc21b2a52c4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:49:36.453686 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:36.453642 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f79a66f2-0c10-4260-ba7e-6cc21b2a52c4-kube-api-access-hkwv9" pod "f79a66f2-0c10-4260-ba7e-6cc21b2a52c4" (UID: "f79a66f2-0c10-4260-ba7e-6cc21b2a52c4"). InnerVolumeSpecName "kube-api-access-hkwv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:49:36.553535 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:36.553428 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f79a66f2-0c10-4260-ba7e-6cc21b2a52c4-maas-api-tls\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:49:36.553535 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:36.553461 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkwv9\" (UniqueName: \"kubernetes.io/projected/f79a66f2-0c10-4260-ba7e-6cc21b2a52c4-kube-api-access-hkwv9\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:49:36.553535 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:36.553504 5979 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f79a66f2-0c10-4260-ba7e-6cc21b2a52c4-metrics-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:49:37.050231 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:37.050192 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1e4e03c-dpjrz"] Nov 03 13:49:37.079026 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:37.078991 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1e4e03c-dpjrz"] Nov 03 13:49:37.234420 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:49:37.234384 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:49:37.289739 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:37.289707 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4b9658-86c4d8b589-5fdvq" Nov 03 13:49:37.289739 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:37.289726 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4b9658-86c4d8b589-5fdvq" event={"ID":"f79a66f2-0c10-4260-ba7e-6cc21b2a52c4","Type":"ContainerDied","Data":"eedfd1696856db2722071c4cf6d046dbc43ef34b02b0799c2c3422d5fb125469"} Nov 03 13:49:37.290004 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:37.289795 5979 scope.go:122] "RemoveContainer" containerID="e3727ad043e5cc405c1320025081fb513d2a43d0b312e4197751d5b07a965f60" Nov 03 13:49:37.444418 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:37.444382 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4b9658-86c4d8b589-5fdvq"] Nov 03 13:49:37.448232 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:37.448204 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4b9658-86c4d8b589-5fdvq"] Nov 03 13:49:37.483771 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:37.483736 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4b9658-7c5887bdf9-8nldv" podUID="76ffa3a3-1a28-4468-aff0-b1e96f33d3e8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:49:38.118801 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:38.118766 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fa25a94-fb23-4745-b999-e77d4c54a035" path="/var/lib/kubelet/pods/2fa25a94-fb23-4745-b999-e77d4c54a035/volumes" Nov 03 13:49:38.119285 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:38.119267 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f79a66f2-0c10-4260-ba7e-6cc21b2a52c4" path="/var/lib/kubelet/pods/f79a66f2-0c10-4260-ba7e-6cc21b2a52c4/volumes" Nov 03 13:49:38.491452 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:38.491419 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-4b9658-999455b7f-l6wwv" podUID="5f2c1f4c-f81b-46fc-9f29-22ad372565c9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 13:49:43.032388 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:43.032310 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-4b9658-999455b7f-l6wwv" Nov 03 13:49:43.113646 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:43.113618 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4b9658-7c5887bdf9-8nldv" Nov 03 13:49:43.146975 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:43.146938 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f2c1f4c-f81b-46fc-9f29-22ad372565c9-kube-api-access-wvh5m\" (UniqueName: \"kubernetes.io/projected/5f2c1f4c-f81b-46fc-9f29-22ad372565c9-kube-api-access-wvh5m\") pod \"5f2c1f4c-f81b-46fc-9f29-22ad372565c9\" (UID: \"5f2c1f4c-f81b-46fc-9f29-22ad372565c9\") " Nov 03 13:49:43.147191 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:43.147172 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f2c1f4c-f81b-46fc-9f29-22ad372565c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f2c1f4c-f81b-46fc-9f29-22ad372565c9-plugins-config-volume\") pod \"5f2c1f4c-f81b-46fc-9f29-22ad372565c9\" (UID: \"5f2c1f4c-f81b-46fc-9f29-22ad372565c9\") " Nov 03 13:49:43.147668 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:43.147639 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f2c1f4c-f81b-46fc-9f29-22ad372565c9-plugins-config-volume" pod "5f2c1f4c-f81b-46fc-9f29-22ad372565c9" (UID: "5f2c1f4c-f81b-46fc-9f29-22ad372565c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:49:43.147803 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:43.147784 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f2c1f4c-f81b-46fc-9f29-22ad372565c9-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:49:43.149521 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:43.149484 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f2c1f4c-f81b-46fc-9f29-22ad372565c9-kube-api-access-wvh5m" pod "5f2c1f4c-f81b-46fc-9f29-22ad372565c9" (UID: "5f2c1f4c-f81b-46fc-9f29-22ad372565c9"). InnerVolumeSpecName "kube-api-access-wvh5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:49:43.248807 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:43.248776 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76ffa3a3-1a28-4468-aff0-b1e96f33d3e8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76ffa3a3-1a28-4468-aff0-b1e96f33d3e8-plugins-config-volume\") pod \"76ffa3a3-1a28-4468-aff0-b1e96f33d3e8\" (UID: \"76ffa3a3-1a28-4468-aff0-b1e96f33d3e8\") " Nov 03 13:49:43.248988 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:43.248829 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76ffa3a3-1a28-4468-aff0-b1e96f33d3e8-kube-api-access-h5prg\" (UniqueName: \"kubernetes.io/projected/76ffa3a3-1a28-4468-aff0-b1e96f33d3e8-kube-api-access-h5prg\") pod \"76ffa3a3-1a28-4468-aff0-b1e96f33d3e8\" (UID: \"76ffa3a3-1a28-4468-aff0-b1e96f33d3e8\") " Nov 03 13:49:43.249295 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:43.249267 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76ffa3a3-1a28-4468-aff0-b1e96f33d3e8-plugins-config-volume" pod "76ffa3a3-1a28-4468-aff0-b1e96f33d3e8" (UID: "76ffa3a3-1a28-4468-aff0-b1e96f33d3e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:49:43.249591 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:43.249567 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvh5m\" (UniqueName: \"kubernetes.io/projected/5f2c1f4c-f81b-46fc-9f29-22ad372565c9-kube-api-access-wvh5m\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:49:43.249706 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:43.249593 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76ffa3a3-1a28-4468-aff0-b1e96f33d3e8-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:49:43.251230 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:43.251200 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76ffa3a3-1a28-4468-aff0-b1e96f33d3e8-kube-api-access-h5prg" pod "76ffa3a3-1a28-4468-aff0-b1e96f33d3e8" (UID: "76ffa3a3-1a28-4468-aff0-b1e96f33d3e8"). InnerVolumeSpecName "kube-api-access-h5prg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:49:43.335570 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:43.335482 5979 generic.go:356] "Generic (PLEG): container finished" podID="5f2c1f4c-f81b-46fc-9f29-22ad372565c9" containerID="53c5ae4fc25865cec1716c0e7d3504fc7e1482a2b300f53e637bb50ee804d21c" exitCode=0 Nov 03 13:49:43.335570 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:43.335546 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-4b9658-999455b7f-l6wwv" Nov 03 13:49:43.335816 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:43.335539 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-4b9658-999455b7f-l6wwv" event={"ID":"5f2c1f4c-f81b-46fc-9f29-22ad372565c9","Type":"ContainerDied","Data":"53c5ae4fc25865cec1716c0e7d3504fc7e1482a2b300f53e637bb50ee804d21c"} Nov 03 13:49:43.335816 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:43.335687 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-4b9658-999455b7f-l6wwv" event={"ID":"5f2c1f4c-f81b-46fc-9f29-22ad372565c9","Type":"ContainerDied","Data":"70f2d5c170665fa7027137ec6b9c719a5544dce5c9095a0f10e969101ba25d87"} Nov 03 13:49:43.335816 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:43.335714 5979 scope.go:122] "RemoveContainer" containerID="53c5ae4fc25865cec1716c0e7d3504fc7e1482a2b300f53e637bb50ee804d21c" Nov 03 13:49:43.337181 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:43.337160 5979 generic.go:356] "Generic (PLEG): container finished" podID="76ffa3a3-1a28-4468-aff0-b1e96f33d3e8" containerID="6252df7f84d7b9d27a8b06e80c4cff2f5eeec24816df2c3ea9fd4c1df7a8c1db" exitCode=0 Nov 03 13:49:43.337321 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:43.337215 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4b9658-7c5887bdf9-8nldv" Nov 03 13:49:43.337321 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:43.337244 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4b9658-7c5887bdf9-8nldv" event={"ID":"76ffa3a3-1a28-4468-aff0-b1e96f33d3e8","Type":"ContainerDied","Data":"6252df7f84d7b9d27a8b06e80c4cff2f5eeec24816df2c3ea9fd4c1df7a8c1db"} Nov 03 13:49:43.337321 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:43.337281 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4b9658-7c5887bdf9-8nldv" event={"ID":"76ffa3a3-1a28-4468-aff0-b1e96f33d3e8","Type":"ContainerDied","Data":"91f454194e3124f00d2c2cfccd6b39065c0ba9b512071a51a391bdd3c07180e5"} Nov 03 13:49:43.349018 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:43.348992 5979 scope.go:122] "RemoveContainer" containerID="53c5ae4fc25865cec1716c0e7d3504fc7e1482a2b300f53e637bb50ee804d21c" Nov 03 13:49:43.349564 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:49:43.349537 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53c5ae4fc25865cec1716c0e7d3504fc7e1482a2b300f53e637bb50ee804d21c\": container with ID starting with 53c5ae4fc25865cec1716c0e7d3504fc7e1482a2b300f53e637bb50ee804d21c not found: ID does not exist" containerID="53c5ae4fc25865cec1716c0e7d3504fc7e1482a2b300f53e637bb50ee804d21c" Nov 03 13:49:43.349712 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:43.349571 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53c5ae4fc25865cec1716c0e7d3504fc7e1482a2b300f53e637bb50ee804d21c"} err="failed to get container status \"53c5ae4fc25865cec1716c0e7d3504fc7e1482a2b300f53e637bb50ee804d21c\": rpc error: code = NotFound desc = could not find container \"53c5ae4fc25865cec1716c0e7d3504fc7e1482a2b300f53e637bb50ee804d21c\": container with ID starting with 53c5ae4fc25865cec1716c0e7d3504fc7e1482a2b300f53e637bb50ee804d21c not found: ID does not exist" Nov 03 13:49:43.349712 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:43.349596 5979 scope.go:122] "RemoveContainer" containerID="6252df7f84d7b9d27a8b06e80c4cff2f5eeec24816df2c3ea9fd4c1df7a8c1db" Nov 03 13:49:43.350661 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:43.350627 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5prg\" (UniqueName: \"kubernetes.io/projected/76ffa3a3-1a28-4468-aff0-b1e96f33d3e8-kube-api-access-h5prg\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:49:43.359946 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:43.359924 5979 scope.go:122] "RemoveContainer" containerID="6252df7f84d7b9d27a8b06e80c4cff2f5eeec24816df2c3ea9fd4c1df7a8c1db" Nov 03 13:49:43.360245 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:49:43.360225 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6252df7f84d7b9d27a8b06e80c4cff2f5eeec24816df2c3ea9fd4c1df7a8c1db\": container with ID starting with 6252df7f84d7b9d27a8b06e80c4cff2f5eeec24816df2c3ea9fd4c1df7a8c1db not found: ID does not exist" containerID="6252df7f84d7b9d27a8b06e80c4cff2f5eeec24816df2c3ea9fd4c1df7a8c1db" Nov 03 13:49:43.360301 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:43.360251 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6252df7f84d7b9d27a8b06e80c4cff2f5eeec24816df2c3ea9fd4c1df7a8c1db"} err="failed to get container status \"6252df7f84d7b9d27a8b06e80c4cff2f5eeec24816df2c3ea9fd4c1df7a8c1db\": rpc error: code = NotFound desc = could not find container \"6252df7f84d7b9d27a8b06e80c4cff2f5eeec24816df2c3ea9fd4c1df7a8c1db\": container with ID starting with 6252df7f84d7b9d27a8b06e80c4cff2f5eeec24816df2c3ea9fd4c1df7a8c1db not found: ID does not exist" Nov 03 13:49:43.394938 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:49:43.394899 5979 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f2c1f4c_f81b_46fc_9f29_22ad372565c9.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f2c1f4c_f81b_46fc_9f29_22ad372565c9.slice/crio-70f2d5c170665fa7027137ec6b9c719a5544dce5c9095a0f10e969101ba25d87\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76ffa3a3_1a28_4468_aff0_b1e96f33d3e8.slice/crio-91f454194e3124f00d2c2cfccd6b39065c0ba9b512071a51a391bdd3c07180e5\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76ffa3a3_1a28_4468_aff0_b1e96f33d3e8.slice\": RecentStats: unable to find data in memory cache]" Nov 03 13:49:43.491399 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:43.491366 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4b9658-7c5887bdf9-8nldv"] Nov 03 13:49:43.495548 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:43.495521 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4b9658-7c5887bdf9-8nldv"] Nov 03 13:49:43.596972 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:43.596879 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4b9658-999455b7f-l6wwv"] Nov 03 13:49:43.600697 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:43.600668 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-4b9658-999455b7f-l6wwv"] Nov 03 13:49:44.115601 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:44.115566 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f2c1f4c-f81b-46fc-9f29-22ad372565c9" path="/var/lib/kubelet/pods/5f2c1f4c-f81b-46fc-9f29-22ad372565c9/volumes" Nov 03 13:49:44.116046 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:44.116032 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76ffa3a3-1a28-4468-aff0-b1e96f33d3e8" path="/var/lib/kubelet/pods/76ffa3a3-1a28-4468-aff0-b1e96f33d3e8/volumes" Nov 03 13:49:46.218326 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:46.218293 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq"] Nov 03 13:49:46.218846 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:46.218537 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq" podUID="763f2c9e-811d-49dc-b971-4782cb8a0964" containerName="istio-proxy" containerID="cri-o://15f9c496b01302ac6b60bb5c5af4e3fbfc26d47e7e01323191ac4bd585fcea48" gracePeriod=30 Nov 03 13:49:47.838494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:47.833289 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5faf5d-669fbdddd8-ms8tf"] Nov 03 13:49:47.838494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:47.834307 5979 cpu_manager.go:414] "containerMap: removing container" podUID="cb1baabd-140e-4516-8044-ea044b02afab" containerName="revoke-keys" Nov 03 13:49:47.838494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:47.834330 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb1baabd-140e-4516-8044-ea044b02afab" containerName="revoke-keys" Nov 03 13:49:47.838494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:47.834365 5979 cpu_manager.go:414] "containerMap: removing container" podUID="76ffa3a3-1a28-4468-aff0-b1e96f33d3e8" containerName="payload-pre-processing" Nov 03 13:49:47.838494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:47.834374 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76ffa3a3-1a28-4468-aff0-b1e96f33d3e8" containerName="payload-pre-processing" Nov 03 13:49:47.838494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:47.834391 5979 cpu_manager.go:414] "containerMap: removing container" podUID="dd8dc463-bd1d-4451-afba-337d5a6a74ba" containerName="payload-processing" Nov 03 13:49:47.838494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:47.834399 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd8dc463-bd1d-4451-afba-337d5a6a74ba" containerName="payload-processing" Nov 03 13:49:47.838494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:47.834432 5979 cpu_manager.go:414] "containerMap: removing container" podUID="5dc7b3ba-e302-41a5-8ef5-555da33e6222" containerName="payload-pre-processing" Nov 03 13:49:47.838494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:47.834440 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dc7b3ba-e302-41a5-8ef5-555da33e6222" containerName="payload-pre-processing" Nov 03 13:49:47.849786 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:47.849758 5979 cpu_manager.go:414] "containerMap: removing container" podUID="f79a66f2-0c10-4260-ba7e-6cc21b2a52c4" containerName="maas-api" Nov 03 13:49:47.850019 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:47.850003 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f79a66f2-0c10-4260-ba7e-6cc21b2a52c4" containerName="maas-api" Nov 03 13:49:47.850127 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:47.850115 5979 cpu_manager.go:414] "containerMap: removing container" podUID="5f2c1f4c-f81b-46fc-9f29-22ad372565c9" containerName="payload-processing" Nov 03 13:49:47.850209 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:47.850199 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f2c1f4c-f81b-46fc-9f29-22ad372565c9" containerName="payload-processing" Nov 03 13:49:47.850556 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:47.850533 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dc7b3ba-e302-41a5-8ef5-555da33e6222" containerName="payload-pre-processing" Nov 03 13:49:47.854570 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:47.854545 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="f79a66f2-0c10-4260-ba7e-6cc21b2a52c4" containerName="maas-api" Nov 03 13:49:47.854734 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:47.854714 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f2c1f4c-f81b-46fc-9f29-22ad372565c9" containerName="payload-processing" Nov 03 13:49:47.854971 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:47.854956 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb1baabd-140e-4516-8044-ea044b02afab" containerName="revoke-keys" Nov 03 13:49:47.855087 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:47.855074 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="76ffa3a3-1a28-4468-aff0-b1e96f33d3e8" containerName="payload-pre-processing" Nov 03 13:49:47.855187 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:47.855173 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd8dc463-bd1d-4451-afba-337d5a6a74ba" containerName="payload-processing" Nov 03 13:49:47.884495 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:47.880914 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5faf5d-669fbdddd8-ms8tf" Nov 03 13:49:47.895493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:47.887685 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5faf5d-669fbdddd8-ms8tf"] Nov 03 13:49:47.942928 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:47.942891 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 13:49:47.943082 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:47.942995 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 13:49:47.949707 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:47.949672 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49140ce2-70a3-43bc-8ad4-5c0ac5dff963-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-5faf5d-669fbdddd8-ms8tf\" (UID: \"49140ce2-70a3-43bc-8ad4-5c0ac5dff963\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5faf5d-669fbdddd8-ms8tf" Nov 03 13:49:47.949831 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:47.949732 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rq8z\" (UniqueName: \"kubernetes.io/projected/49140ce2-70a3-43bc-8ad4-5c0ac5dff963-kube-api-access-4rq8z\") pod \"maas-api-e2e-shared-a-w0-5faf5d-669fbdddd8-ms8tf\" (UID: \"49140ce2-70a3-43bc-8ad4-5c0ac5dff963\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5faf5d-669fbdddd8-ms8tf" Nov 03 13:49:47.949927 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:47.949907 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49140ce2-70a3-43bc-8ad4-5c0ac5dff963-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-5faf5d-669fbdddd8-ms8tf\" (UID: \"49140ce2-70a3-43bc-8ad4-5c0ac5dff963\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5faf5d-669fbdddd8-ms8tf" Nov 03 13:49:47.949967 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:47.949954 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49140ce2-70a3-43bc-8ad4-5c0ac5dff963-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-5faf5d-669fbdddd8-ms8tf\" (UID: \"49140ce2-70a3-43bc-8ad4-5c0ac5dff963\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5faf5d-669fbdddd8-ms8tf" Nov 03 13:49:47.966926 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:47.966897 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-5faf5d-serving-cert\"" Nov 03 13:49:47.966926 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:47.966909 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 13:49:47.967178 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:47.967161 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-dgx9m\"" Nov 03 13:49:48.015005 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:48.014975 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:49:48.051547 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:48.051515 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49140ce2-70a3-43bc-8ad4-5c0ac5dff963-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-5faf5d-669fbdddd8-ms8tf\" (UID: \"49140ce2-70a3-43bc-8ad4-5c0ac5dff963\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5faf5d-669fbdddd8-ms8tf" Nov 03 13:49:48.051732 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:48.051562 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49140ce2-70a3-43bc-8ad4-5c0ac5dff963-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-5faf5d-669fbdddd8-ms8tf\" (UID: \"49140ce2-70a3-43bc-8ad4-5c0ac5dff963\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5faf5d-669fbdddd8-ms8tf" Nov 03 13:49:48.051732 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:48.051636 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49140ce2-70a3-43bc-8ad4-5c0ac5dff963-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-5faf5d-669fbdddd8-ms8tf\" (UID: \"49140ce2-70a3-43bc-8ad4-5c0ac5dff963\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5faf5d-669fbdddd8-ms8tf" Nov 03 13:49:48.051732 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:48.051678 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rq8z\" (UniqueName: \"kubernetes.io/projected/49140ce2-70a3-43bc-8ad4-5c0ac5dff963-kube-api-access-4rq8z\") pod \"maas-api-e2e-shared-a-w0-5faf5d-669fbdddd8-ms8tf\" (UID: \"49140ce2-70a3-43bc-8ad4-5c0ac5dff963\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5faf5d-669fbdddd8-ms8tf" Nov 03 13:49:48.051732 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:49:48.051717 5979 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5faf5d-serving-cert: secret "maas-api-e2e-shared-a-w0-5faf5d-serving-cert" not found Nov 03 13:49:48.051970 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:49:48.051788 5979 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49140ce2-70a3-43bc-8ad4-5c0ac5dff963-maas-api-tls podName:49140ce2-70a3-43bc-8ad4-5c0ac5dff963 nodeName:}" failed. No retries permitted until 2026-11-03 13:49:48.551771795 +0000 UTC m=+2670.562460014 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/49140ce2-70a3-43bc-8ad4-5c0ac5dff963-maas-api-tls") pod "maas-api-e2e-shared-a-w0-5faf5d-669fbdddd8-ms8tf" (UID: "49140ce2-70a3-43bc-8ad4-5c0ac5dff963") : secret "maas-api-e2e-shared-a-w0-5faf5d-serving-cert" not found Nov 03 13:49:48.052354 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:48.052329 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49140ce2-70a3-43bc-8ad4-5c0ac5dff963-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-5faf5d-669fbdddd8-ms8tf\" (UID: \"49140ce2-70a3-43bc-8ad4-5c0ac5dff963\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5faf5d-669fbdddd8-ms8tf" Nov 03 13:49:48.053965 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:48.053944 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49140ce2-70a3-43bc-8ad4-5c0ac5dff963-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-5faf5d-669fbdddd8-ms8tf\" (UID: \"49140ce2-70a3-43bc-8ad4-5c0ac5dff963\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5faf5d-669fbdddd8-ms8tf" Nov 03 13:49:48.106297 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:48.106218 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rq8z\" (UniqueName: \"kubernetes.io/projected/49140ce2-70a3-43bc-8ad4-5c0ac5dff963-kube-api-access-4rq8z\") pod \"maas-api-e2e-shared-a-w0-5faf5d-669fbdddd8-ms8tf\" (UID: \"49140ce2-70a3-43bc-8ad4-5c0ac5dff963\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5faf5d-669fbdddd8-ms8tf" Nov 03 13:49:48.424492 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:48.424339 5979 patch_prober.go:29] interesting pod/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 13:49:48.424492 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:48.424403 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq" podUID="763f2c9e-811d-49dc-b971-4782cb8a0964" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 13:49:48.563173 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:48.563139 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49140ce2-70a3-43bc-8ad4-5c0ac5dff963-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-5faf5d-669fbdddd8-ms8tf\" (UID: \"49140ce2-70a3-43bc-8ad4-5c0ac5dff963\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5faf5d-669fbdddd8-ms8tf" Nov 03 13:49:48.565839 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:48.565808 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49140ce2-70a3-43bc-8ad4-5c0ac5dff963-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-5faf5d-669fbdddd8-ms8tf\" (UID: \"49140ce2-70a3-43bc-8ad4-5c0ac5dff963\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5faf5d-669fbdddd8-ms8tf" Nov 03 13:49:48.811115 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:48.811026 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5faf5d-669fbdddd8-ms8tf" Nov 03 13:49:49.187167 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:49.187132 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5faf5d-669fbdddd8-ms8tf"] Nov 03 13:49:49.192940 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:49:49.192910 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49140ce2_70a3_43bc_8ad4_5c0ac5dff963.slice/crio-2963f3405bbadca32cf5a0b12cfb3513cc6fbc8f2eebc4e987841580b7d9306b WatchSource:0}: Error finding container 2963f3405bbadca32cf5a0b12cfb3513cc6fbc8f2eebc4e987841580b7d9306b: Status 404 returned error can't find the container with id 2963f3405bbadca32cf5a0b12cfb3513cc6fbc8f2eebc4e987841580b7d9306b Nov 03 13:49:49.194521 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:49.194500 5979 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 13:49:49.474859 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:49.474764 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5faf5d-669fbdddd8-ms8tf" event={"ID":"49140ce2-70a3-43bc-8ad4-5c0ac5dff963","Type":"ContainerStarted","Data":"2963f3405bbadca32cf5a0b12cfb3513cc6fbc8f2eebc4e987841580b7d9306b"} Nov 03 13:49:50.483188 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:50.483150 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5faf5d-669fbdddd8-ms8tf" event={"ID":"49140ce2-70a3-43bc-8ad4-5c0ac5dff963","Type":"ContainerStarted","Data":"ab99e76474aaf8e48b3e986b05f74495aa8ce275bb53e96bdb1d48cb1dd176c9"} Nov 03 13:49:50.483686 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:50.483219 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5faf5d-669fbdddd8-ms8tf" Nov 03 13:49:50.592245 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:50.592190 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5faf5d-669fbdddd8-ms8tf" podStartSLOduration=2.740866066 podStartE2EDuration="3.592173426s" totalImagesPullingTime="851.30736ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:49:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:49:50.590502313 +0000 UTC m=+2672.601190555" watchObservedRunningTime="2026-11-03 13:49:50.592173426 +0000 UTC m=+2672.602861668" Nov 03 13:49:51.495275 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:51.495180 5979 generic.go:356] "Generic (PLEG): container finished" podID="763f2c9e-811d-49dc-b971-4782cb8a0964" containerID="15f9c496b01302ac6b60bb5c5af4e3fbfc26d47e7e01323191ac4bd585fcea48" exitCode=0 Nov 03 13:49:51.495275 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:51.495249 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq" event={"ID":"763f2c9e-811d-49dc-b971-4782cb8a0964","Type":"ContainerDied","Data":"15f9c496b01302ac6b60bb5c5af4e3fbfc26d47e7e01323191ac4bd585fcea48"} Nov 03 13:49:51.495788 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:51.495318 5979 scope.go:122] "RemoveContainer" containerID="f23261b12105120dd094972ca2c57abe63e04748f496e89d9653d7141b4088f1" Nov 03 13:49:51.539230 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:51.539203 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq" Nov 03 13:49:51.605441 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:51.605403 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/763f2c9e-811d-49dc-b971-4782cb8a0964-kube-api-access-nxjm9\" (UniqueName: \"kubernetes.io/projected/763f2c9e-811d-49dc-b971-4782cb8a0964-kube-api-access-nxjm9\") pod \"763f2c9e-811d-49dc-b971-4782cb8a0964\" (UID: \"763f2c9e-811d-49dc-b971-4782cb8a0964\") " Nov 03 13:49:51.605646 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:51.605536 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/763f2c9e-811d-49dc-b971-4782cb8a0964-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/763f2c9e-811d-49dc-b971-4782cb8a0964-istio-podinfo\") pod \"763f2c9e-811d-49dc-b971-4782cb8a0964\" (UID: \"763f2c9e-811d-49dc-b971-4782cb8a0964\") " Nov 03 13:49:51.605646 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:51.605599 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/763f2c9e-811d-49dc-b971-4782cb8a0964-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/763f2c9e-811d-49dc-b971-4782cb8a0964-credential-socket\") pod \"763f2c9e-811d-49dc-b971-4782cb8a0964\" (UID: \"763f2c9e-811d-49dc-b971-4782cb8a0964\") " Nov 03 13:49:51.605753 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:51.605678 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/763f2c9e-811d-49dc-b971-4782cb8a0964-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/763f2c9e-811d-49dc-b971-4782cb8a0964-istio-data\") pod \"763f2c9e-811d-49dc-b971-4782cb8a0964\" (UID: \"763f2c9e-811d-49dc-b971-4782cb8a0964\") " Nov 03 13:49:51.605753 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:51.605734 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/763f2c9e-811d-49dc-b971-4782cb8a0964-istio-token\" (UniqueName: \"kubernetes.io/projected/763f2c9e-811d-49dc-b971-4782cb8a0964-istio-token\") pod \"763f2c9e-811d-49dc-b971-4782cb8a0964\" (UID: \"763f2c9e-811d-49dc-b971-4782cb8a0964\") " Nov 03 13:49:51.605879 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:51.605800 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/763f2c9e-811d-49dc-b971-4782cb8a0964-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/763f2c9e-811d-49dc-b971-4782cb8a0964-workload-socket\") pod \"763f2c9e-811d-49dc-b971-4782cb8a0964\" (UID: \"763f2c9e-811d-49dc-b971-4782cb8a0964\") " Nov 03 13:49:51.605879 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:51.605842 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/763f2c9e-811d-49dc-b971-4782cb8a0964-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/763f2c9e-811d-49dc-b971-4782cb8a0964-istiod-ca-cert\") pod \"763f2c9e-811d-49dc-b971-4782cb8a0964\" (UID: \"763f2c9e-811d-49dc-b971-4782cb8a0964\") " Nov 03 13:49:51.605996 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:51.605886 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/763f2c9e-811d-49dc-b971-4782cb8a0964-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/763f2c9e-811d-49dc-b971-4782cb8a0964-istio-envoy\") pod \"763f2c9e-811d-49dc-b971-4782cb8a0964\" (UID: \"763f2c9e-811d-49dc-b971-4782cb8a0964\") " Nov 03 13:49:51.605996 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:51.605933 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/763f2c9e-811d-49dc-b971-4782cb8a0964-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/763f2c9e-811d-49dc-b971-4782cb8a0964-workload-certs\") pod \"763f2c9e-811d-49dc-b971-4782cb8a0964\" (UID: \"763f2c9e-811d-49dc-b971-4782cb8a0964\") " Nov 03 13:49:51.605996 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:51.605931 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/763f2c9e-811d-49dc-b971-4782cb8a0964-istio-data" pod "763f2c9e-811d-49dc-b971-4782cb8a0964" (UID: "763f2c9e-811d-49dc-b971-4782cb8a0964"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:49:51.605996 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:51.605950 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/763f2c9e-811d-49dc-b971-4782cb8a0964-credential-socket" pod "763f2c9e-811d-49dc-b971-4782cb8a0964" (UID: "763f2c9e-811d-49dc-b971-4782cb8a0964"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:49:51.606238 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:51.606073 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/763f2c9e-811d-49dc-b971-4782cb8a0964-workload-socket" pod "763f2c9e-811d-49dc-b971-4782cb8a0964" (UID: "763f2c9e-811d-49dc-b971-4782cb8a0964"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:49:51.606238 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:51.606219 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/763f2c9e-811d-49dc-b971-4782cb8a0964-workload-certs" pod "763f2c9e-811d-49dc-b971-4782cb8a0964" (UID: "763f2c9e-811d-49dc-b971-4782cb8a0964"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:49:51.606374 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:51.606239 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/763f2c9e-811d-49dc-b971-4782cb8a0964-istiod-ca-cert" pod "763f2c9e-811d-49dc-b971-4782cb8a0964" (UID: "763f2c9e-811d-49dc-b971-4782cb8a0964"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:49:51.606772 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:51.606744 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/763f2c9e-811d-49dc-b971-4782cb8a0964-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:49:51.606772 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:51.606773 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/763f2c9e-811d-49dc-b971-4782cb8a0964-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:49:51.606978 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:51.606787 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/763f2c9e-811d-49dc-b971-4782cb8a0964-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:49:51.606978 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:51.606801 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/763f2c9e-811d-49dc-b971-4782cb8a0964-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:49:51.606978 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:51.606817 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/763f2c9e-811d-49dc-b971-4782cb8a0964-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:49:51.608096 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:51.608071 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/763f2c9e-811d-49dc-b971-4782cb8a0964-kube-api-access-nxjm9" pod "763f2c9e-811d-49dc-b971-4782cb8a0964" (UID: "763f2c9e-811d-49dc-b971-4782cb8a0964"). InnerVolumeSpecName "kube-api-access-nxjm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:49:51.608218 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:51.608119 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/763f2c9e-811d-49dc-b971-4782cb8a0964-istio-envoy" pod "763f2c9e-811d-49dc-b971-4782cb8a0964" (UID: "763f2c9e-811d-49dc-b971-4782cb8a0964"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:49:51.608289 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:51.608268 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/763f2c9e-811d-49dc-b971-4782cb8a0964-istio-podinfo" pod "763f2c9e-811d-49dc-b971-4782cb8a0964" (UID: "763f2c9e-811d-49dc-b971-4782cb8a0964"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:49:51.608351 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:51.608289 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/763f2c9e-811d-49dc-b971-4782cb8a0964-istio-token" pod "763f2c9e-811d-49dc-b971-4782cb8a0964" (UID: "763f2c9e-811d-49dc-b971-4782cb8a0964"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:49:51.708106 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:51.708062 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/763f2c9e-811d-49dc-b971-4782cb8a0964-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:49:51.708106 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:51.708095 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxjm9\" (UniqueName: \"kubernetes.io/projected/763f2c9e-811d-49dc-b971-4782cb8a0964-kube-api-access-nxjm9\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:49:51.708106 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:51.708105 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/763f2c9e-811d-49dc-b971-4782cb8a0964-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:49:51.708106 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:51.708114 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/763f2c9e-811d-49dc-b971-4782cb8a0964-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:49:52.503704 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:52.503670 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq" Nov 03 13:49:52.503704 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:52.503688 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq" event={"ID":"763f2c9e-811d-49dc-b971-4782cb8a0964","Type":"ContainerDied","Data":"50a9d7ca7bdab0665b6a3ff0dd12689659520a0aeba5bc1cad02c9c43750e67d"} Nov 03 13:49:52.504249 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:52.503745 5979 scope.go:122] "RemoveContainer" containerID="15f9c496b01302ac6b60bb5c5af4e3fbfc26d47e7e01323191ac4bd585fcea48" Nov 03 13:49:52.606466 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:52.606393 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ba6fbd-7856699cbc-gpnbs"] Nov 03 13:49:52.698642 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:52.698596 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq"] Nov 03 13:49:52.702172 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:52.702145 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-4b9658-openshift-default-855d6bbdd5-cj8xq"] Nov 03 13:49:52.718981 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:52.718947 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ba6fbd-5f664c6899-7p4bb"] Nov 03 13:49:54.046924 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:54.046883 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28b9e6c0-s9dc6"] Nov 03 13:49:54.050326 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:54.050292 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28b9e6c0-s9dc6"] Nov 03 13:49:54.117051 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:54.117016 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="763f2c9e-811d-49dc-b971-4782cb8a0964" path="/var/lib/kubelet/pods/763f2c9e-811d-49dc-b971-4782cb8a0964/volumes" Nov 03 13:49:54.117586 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:54.117568 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bab0d665-2857-40ea-a07f-7c4b535d84e4" path="/var/lib/kubelet/pods/bab0d665-2857-40ea-a07f-7c4b535d84e4/volumes" Nov 03 13:49:56.502083 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:56.502055 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5faf5d-669fbdddd8-ms8tf" Nov 03 13:49:57.606838 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:57.606793 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-ba6fbd-7856699cbc-gpnbs" podUID="34740dc2-773c-4e06-b540-16752c227ba1" containerName="payload-processing" containerID="cri-o://c33486f9ff5339fc81c3d2593782c8b670010ca46858abc0b4a4badd63e0accb" gracePeriod=25 Nov 03 13:49:57.719634 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:57.719596 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ba6fbd-5f664c6899-7p4bb" podUID="9e256eb2-38e4-4f44-b796-4c47201cfe6d" containerName="payload-pre-processing" containerID="cri-o://2e4b81d6c17e69597f9c7b6165c6ac61163e6a25b370e4415fc8727ddbbcaa6c" gracePeriod=25 Nov 03 13:49:57.879830 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:57.879759 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ba6fbd-7856699cbc-gpnbs" Nov 03 13:49:57.976532 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:57.976501 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34740dc2-773c-4e06-b540-16752c227ba1-kube-api-access-9cc4z\" (UniqueName: \"kubernetes.io/projected/34740dc2-773c-4e06-b540-16752c227ba1-kube-api-access-9cc4z\") pod \"34740dc2-773c-4e06-b540-16752c227ba1\" (UID: \"34740dc2-773c-4e06-b540-16752c227ba1\") " Nov 03 13:49:57.976693 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:57.976556 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34740dc2-773c-4e06-b540-16752c227ba1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34740dc2-773c-4e06-b540-16752c227ba1-plugins-config-volume\") pod \"34740dc2-773c-4e06-b540-16752c227ba1\" (UID: \"34740dc2-773c-4e06-b540-16752c227ba1\") " Nov 03 13:49:57.977808 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:57.977775 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34740dc2-773c-4e06-b540-16752c227ba1-plugins-config-volume" pod "34740dc2-773c-4e06-b540-16752c227ba1" (UID: "34740dc2-773c-4e06-b540-16752c227ba1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:49:57.978235 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:57.978217 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34740dc2-773c-4e06-b540-16752c227ba1-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:49:57.979303 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:57.979277 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34740dc2-773c-4e06-b540-16752c227ba1-kube-api-access-9cc4z" pod "34740dc2-773c-4e06-b540-16752c227ba1" (UID: "34740dc2-773c-4e06-b540-16752c227ba1"). InnerVolumeSpecName "kube-api-access-9cc4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:49:57.987598 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:57.987573 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ba6fbd-5f664c6899-7p4bb" Nov 03 13:49:58.079253 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:58.079219 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e256eb2-38e4-4f44-b796-4c47201cfe6d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e256eb2-38e4-4f44-b796-4c47201cfe6d-plugins-config-volume\") pod \"9e256eb2-38e4-4f44-b796-4c47201cfe6d\" (UID: \"9e256eb2-38e4-4f44-b796-4c47201cfe6d\") " Nov 03 13:49:58.079462 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:58.079311 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e256eb2-38e4-4f44-b796-4c47201cfe6d-kube-api-access-99mns\" (UniqueName: \"kubernetes.io/projected/9e256eb2-38e4-4f44-b796-4c47201cfe6d-kube-api-access-99mns\") pod \"9e256eb2-38e4-4f44-b796-4c47201cfe6d\" (UID: \"9e256eb2-38e4-4f44-b796-4c47201cfe6d\") " Nov 03 13:49:58.079618 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:58.079596 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e256eb2-38e4-4f44-b796-4c47201cfe6d-plugins-config-volume" pod "9e256eb2-38e4-4f44-b796-4c47201cfe6d" (UID: "9e256eb2-38e4-4f44-b796-4c47201cfe6d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:49:58.079892 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:58.079873 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e256eb2-38e4-4f44-b796-4c47201cfe6d-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:49:58.079949 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:58.079898 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cc4z\" (UniqueName: \"kubernetes.io/projected/34740dc2-773c-4e06-b540-16752c227ba1-kube-api-access-9cc4z\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:49:58.081329 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:58.081308 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e256eb2-38e4-4f44-b796-4c47201cfe6d-kube-api-access-99mns" pod "9e256eb2-38e4-4f44-b796-4c47201cfe6d" (UID: "9e256eb2-38e4-4f44-b796-4c47201cfe6d"). InnerVolumeSpecName "kube-api-access-99mns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:49:58.181421 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:58.181390 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99mns\" (UniqueName: \"kubernetes.io/projected/9e256eb2-38e4-4f44-b796-4c47201cfe6d-kube-api-access-99mns\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:49:58.567874 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:58.567833 5979 generic.go:356] "Generic (PLEG): container finished" podID="34740dc2-773c-4e06-b540-16752c227ba1" containerID="c33486f9ff5339fc81c3d2593782c8b670010ca46858abc0b4a4badd63e0accb" exitCode=0 Nov 03 13:49:58.568093 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:58.567903 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ba6fbd-7856699cbc-gpnbs" Nov 03 13:49:58.568093 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:58.567913 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ba6fbd-7856699cbc-gpnbs" event={"ID":"34740dc2-773c-4e06-b540-16752c227ba1","Type":"ContainerDied","Data":"c33486f9ff5339fc81c3d2593782c8b670010ca46858abc0b4a4badd63e0accb"} Nov 03 13:49:58.568093 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:58.568002 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ba6fbd-7856699cbc-gpnbs" event={"ID":"34740dc2-773c-4e06-b540-16752c227ba1","Type":"ContainerDied","Data":"afb1ef657d0f4624a59123f4c2c004e29adcc28d02d00bfe8c43d86548a3ded5"} Nov 03 13:49:58.568093 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:58.568033 5979 scope.go:122] "RemoveContainer" containerID="c33486f9ff5339fc81c3d2593782c8b670010ca46858abc0b4a4badd63e0accb" Nov 03 13:49:58.571640 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:58.571607 5979 generic.go:356] "Generic (PLEG): container finished" podID="9e256eb2-38e4-4f44-b796-4c47201cfe6d" containerID="2e4b81d6c17e69597f9c7b6165c6ac61163e6a25b370e4415fc8727ddbbcaa6c" exitCode=0 Nov 03 13:49:58.571784 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:58.571658 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ba6fbd-5f664c6899-7p4bb" Nov 03 13:49:58.571784 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:58.571749 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ba6fbd-5f664c6899-7p4bb" event={"ID":"9e256eb2-38e4-4f44-b796-4c47201cfe6d","Type":"ContainerDied","Data":"2e4b81d6c17e69597f9c7b6165c6ac61163e6a25b370e4415fc8727ddbbcaa6c"} Nov 03 13:49:58.571905 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:58.571787 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ba6fbd-5f664c6899-7p4bb" event={"ID":"9e256eb2-38e4-4f44-b796-4c47201cfe6d","Type":"ContainerDied","Data":"ea0dd2a634e5fc62197dedb0a092039f11b7395111919fba6b7e81710978cd0b"} Nov 03 13:49:58.580062 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:58.580044 5979 scope.go:122] "RemoveContainer" containerID="c33486f9ff5339fc81c3d2593782c8b670010ca46858abc0b4a4badd63e0accb" Nov 03 13:49:58.580304 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:49:58.580283 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c33486f9ff5339fc81c3d2593782c8b670010ca46858abc0b4a4badd63e0accb\": container with ID starting with c33486f9ff5339fc81c3d2593782c8b670010ca46858abc0b4a4badd63e0accb not found: ID does not exist" containerID="c33486f9ff5339fc81c3d2593782c8b670010ca46858abc0b4a4badd63e0accb" Nov 03 13:49:58.580375 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:58.580317 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c33486f9ff5339fc81c3d2593782c8b670010ca46858abc0b4a4badd63e0accb"} err="failed to get container status \"c33486f9ff5339fc81c3d2593782c8b670010ca46858abc0b4a4badd63e0accb\": rpc error: code = NotFound desc = could not find container \"c33486f9ff5339fc81c3d2593782c8b670010ca46858abc0b4a4badd63e0accb\": container with ID starting with c33486f9ff5339fc81c3d2593782c8b670010ca46858abc0b4a4badd63e0accb not found: ID does not exist" Nov 03 13:49:58.580375 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:58.580338 5979 scope.go:122] "RemoveContainer" containerID="2e4b81d6c17e69597f9c7b6165c6ac61163e6a25b370e4415fc8727ddbbcaa6c" Nov 03 13:49:58.597629 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:58.592521 5979 scope.go:122] "RemoveContainer" containerID="2e4b81d6c17e69597f9c7b6165c6ac61163e6a25b370e4415fc8727ddbbcaa6c" Nov 03 13:49:58.597629 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:49:58.592997 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e4b81d6c17e69597f9c7b6165c6ac61163e6a25b370e4415fc8727ddbbcaa6c\": container with ID starting with 2e4b81d6c17e69597f9c7b6165c6ac61163e6a25b370e4415fc8727ddbbcaa6c not found: ID does not exist" containerID="2e4b81d6c17e69597f9c7b6165c6ac61163e6a25b370e4415fc8727ddbbcaa6c" Nov 03 13:49:58.597629 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:58.593033 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e4b81d6c17e69597f9c7b6165c6ac61163e6a25b370e4415fc8727ddbbcaa6c"} err="failed to get container status \"2e4b81d6c17e69597f9c7b6165c6ac61163e6a25b370e4415fc8727ddbbcaa6c\": rpc error: code = NotFound desc = could not find container \"2e4b81d6c17e69597f9c7b6165c6ac61163e6a25b370e4415fc8727ddbbcaa6c\": container with ID starting with 2e4b81d6c17e69597f9c7b6165c6ac61163e6a25b370e4415fc8727ddbbcaa6c not found: ID does not exist" Nov 03 13:49:58.723280 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:58.723243 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ba6fbd-5f664c6899-7p4bb"] Nov 03 13:49:58.727203 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:58.727174 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ba6fbd-5f664c6899-7p4bb"] Nov 03 13:49:58.872584 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:58.872492 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ba6fbd-7856699cbc-gpnbs"] Nov 03 13:49:58.876072 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:58.876044 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ba6fbd-7856699cbc-gpnbs"] Nov 03 13:49:59.203441 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:59.203408 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb4ca0-7cb4bfb5f4-j84xg"] Nov 03 13:49:59.204077 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:59.204058 5979 cpu_manager.go:414] "containerMap: removing container" podUID="34740dc2-773c-4e06-b540-16752c227ba1" containerName="payload-processing" Nov 03 13:49:59.204077 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:59.204075 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34740dc2-773c-4e06-b540-16752c227ba1" containerName="payload-processing" Nov 03 13:49:59.204211 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:59.204117 5979 cpu_manager.go:414] "containerMap: removing container" podUID="9e256eb2-38e4-4f44-b796-4c47201cfe6d" containerName="payload-pre-processing" Nov 03 13:49:59.204211 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:59.204123 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e256eb2-38e4-4f44-b796-4c47201cfe6d" containerName="payload-pre-processing" Nov 03 13:49:59.204211 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:59.204149 5979 cpu_manager.go:414] "containerMap: removing container" podUID="763f2c9e-811d-49dc-b971-4782cb8a0964" containerName="istio-proxy" Nov 03 13:49:59.204211 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:59.204154 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="763f2c9e-811d-49dc-b971-4782cb8a0964" containerName="istio-proxy" Nov 03 13:49:59.204211 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:59.204178 5979 cpu_manager.go:414] "containerMap: removing container" podUID="763f2c9e-811d-49dc-b971-4782cb8a0964" containerName="istio-proxy" Nov 03 13:49:59.204211 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:59.204182 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="763f2c9e-811d-49dc-b971-4782cb8a0964" containerName="istio-proxy" Nov 03 13:49:59.204405 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:59.204267 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="34740dc2-773c-4e06-b540-16752c227ba1" containerName="payload-processing" Nov 03 13:49:59.204405 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:59.204280 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e256eb2-38e4-4f44-b796-4c47201cfe6d" containerName="payload-pre-processing" Nov 03 13:49:59.204405 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:59.204288 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="763f2c9e-811d-49dc-b971-4782cb8a0964" containerName="istio-proxy" Nov 03 13:49:59.209319 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:59.209301 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb4ca0-7cb4bfb5f4-j84xg" Nov 03 13:49:59.258974 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:59.258911 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-fb4ca0-serving-cert\"" Nov 03 13:49:59.262845 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:59.262813 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb4ca0-7cb4bfb5f4-j84xg"] Nov 03 13:49:59.294180 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:59.294142 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40b93c39-a07f-4e43-a1cc-66cfb1469e51-maas-api-tls\") pod \"maas-api-e2e-praxis-b-fb4ca0-7cb4bfb5f4-j84xg\" (UID: \"40b93c39-a07f-4e43-a1cc-66cfb1469e51\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb4ca0-7cb4bfb5f4-j84xg" Nov 03 13:49:59.294427 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:59.294399 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40b93c39-a07f-4e43-a1cc-66cfb1469e51-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-fb4ca0-7cb4bfb5f4-j84xg\" (UID: \"40b93c39-a07f-4e43-a1cc-66cfb1469e51\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb4ca0-7cb4bfb5f4-j84xg" Nov 03 13:49:59.294659 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:59.294637 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40b93c39-a07f-4e43-a1cc-66cfb1469e51-metrics-certs\") pod \"maas-api-e2e-praxis-b-fb4ca0-7cb4bfb5f4-j84xg\" (UID: \"40b93c39-a07f-4e43-a1cc-66cfb1469e51\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb4ca0-7cb4bfb5f4-j84xg" Nov 03 13:49:59.294710 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:59.294679 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hhh8\" (UniqueName: \"kubernetes.io/projected/40b93c39-a07f-4e43-a1cc-66cfb1469e51-kube-api-access-9hhh8\") pod \"maas-api-e2e-praxis-b-fb4ca0-7cb4bfb5f4-j84xg\" (UID: \"40b93c39-a07f-4e43-a1cc-66cfb1469e51\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb4ca0-7cb4bfb5f4-j84xg" Nov 03 13:49:59.395679 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:59.395647 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40b93c39-a07f-4e43-a1cc-66cfb1469e51-maas-api-tls\") pod \"maas-api-e2e-praxis-b-fb4ca0-7cb4bfb5f4-j84xg\" (UID: \"40b93c39-a07f-4e43-a1cc-66cfb1469e51\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb4ca0-7cb4bfb5f4-j84xg" Nov 03 13:49:59.395878 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:59.395748 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40b93c39-a07f-4e43-a1cc-66cfb1469e51-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-fb4ca0-7cb4bfb5f4-j84xg\" (UID: \"40b93c39-a07f-4e43-a1cc-66cfb1469e51\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb4ca0-7cb4bfb5f4-j84xg" Nov 03 13:49:59.395878 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:49:59.395796 5979 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb4ca0-serving-cert: secret "maas-api-e2e-praxis-b-fb4ca0-serving-cert" not found Nov 03 13:49:59.395878 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:59.395842 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40b93c39-a07f-4e43-a1cc-66cfb1469e51-metrics-certs\") pod \"maas-api-e2e-praxis-b-fb4ca0-7cb4bfb5f4-j84xg\" (UID: \"40b93c39-a07f-4e43-a1cc-66cfb1469e51\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb4ca0-7cb4bfb5f4-j84xg" Nov 03 13:49:59.395878 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:49:59.395863 5979 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40b93c39-a07f-4e43-a1cc-66cfb1469e51-maas-api-tls podName:40b93c39-a07f-4e43-a1cc-66cfb1469e51 nodeName:}" failed. No retries permitted until 2026-11-03 13:49:59.89584713 +0000 UTC m=+2681.906535363 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/40b93c39-a07f-4e43-a1cc-66cfb1469e51-maas-api-tls") pod "maas-api-e2e-praxis-b-fb4ca0-7cb4bfb5f4-j84xg" (UID: "40b93c39-a07f-4e43-a1cc-66cfb1469e51") : secret "maas-api-e2e-praxis-b-fb4ca0-serving-cert" not found Nov 03 13:49:59.396136 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:59.395892 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hhh8\" (UniqueName: \"kubernetes.io/projected/40b93c39-a07f-4e43-a1cc-66cfb1469e51-kube-api-access-9hhh8\") pod \"maas-api-e2e-praxis-b-fb4ca0-7cb4bfb5f4-j84xg\" (UID: \"40b93c39-a07f-4e43-a1cc-66cfb1469e51\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb4ca0-7cb4bfb5f4-j84xg" Nov 03 13:49:59.396515 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:59.396492 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40b93c39-a07f-4e43-a1cc-66cfb1469e51-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-fb4ca0-7cb4bfb5f4-j84xg\" (UID: \"40b93c39-a07f-4e43-a1cc-66cfb1469e51\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb4ca0-7cb4bfb5f4-j84xg" Nov 03 13:49:59.398319 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:59.398301 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40b93c39-a07f-4e43-a1cc-66cfb1469e51-metrics-certs\") pod \"maas-api-e2e-praxis-b-fb4ca0-7cb4bfb5f4-j84xg\" (UID: \"40b93c39-a07f-4e43-a1cc-66cfb1469e51\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb4ca0-7cb4bfb5f4-j84xg" Nov 03 13:49:59.450271 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:59.450241 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hhh8\" (UniqueName: \"kubernetes.io/projected/40b93c39-a07f-4e43-a1cc-66cfb1469e51-kube-api-access-9hhh8\") pod \"maas-api-e2e-praxis-b-fb4ca0-7cb4bfb5f4-j84xg\" (UID: \"40b93c39-a07f-4e43-a1cc-66cfb1469e51\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb4ca0-7cb4bfb5f4-j84xg" Nov 03 13:49:59.902297 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:59.902259 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40b93c39-a07f-4e43-a1cc-66cfb1469e51-maas-api-tls\") pod \"maas-api-e2e-praxis-b-fb4ca0-7cb4bfb5f4-j84xg\" (UID: \"40b93c39-a07f-4e43-a1cc-66cfb1469e51\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb4ca0-7cb4bfb5f4-j84xg" Nov 03 13:49:59.904796 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:49:59.904775 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40b93c39-a07f-4e43-a1cc-66cfb1469e51-maas-api-tls\") pod \"maas-api-e2e-praxis-b-fb4ca0-7cb4bfb5f4-j84xg\" (UID: \"40b93c39-a07f-4e43-a1cc-66cfb1469e51\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb4ca0-7cb4bfb5f4-j84xg" Nov 03 13:50:00.116956 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:00.116920 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34740dc2-773c-4e06-b540-16752c227ba1" path="/var/lib/kubelet/pods/34740dc2-773c-4e06-b540-16752c227ba1/volumes" Nov 03 13:50:00.117435 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:00.117420 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e256eb2-38e4-4f44-b796-4c47201cfe6d" path="/var/lib/kubelet/pods/9e256eb2-38e4-4f44-b796-4c47201cfe6d/volumes" Nov 03 13:50:00.123364 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:00.123336 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb4ca0-7cb4bfb5f4-j84xg" Nov 03 13:50:00.503558 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:00.503456 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb4ca0-7cb4bfb5f4-j84xg"] Nov 03 13:50:00.509621 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:50:00.509590 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40b93c39_a07f_4e43_a1cc_66cfb1469e51.slice/crio-ff7bfb41dce1de4cac875262de99426af4f2df2dce660f9143e72570a3411fa5 WatchSource:0}: Error finding container ff7bfb41dce1de4cac875262de99426af4f2df2dce660f9143e72570a3411fa5: Status 404 returned error can't find the container with id ff7bfb41dce1de4cac875262de99426af4f2df2dce660f9143e72570a3411fa5 Nov 03 13:50:00.594153 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:00.594110 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb4ca0-7cb4bfb5f4-j84xg" event={"ID":"40b93c39-a07f-4e43-a1cc-66cfb1469e51","Type":"ContainerStarted","Data":"ff7bfb41dce1de4cac875262de99426af4f2df2dce660f9143e72570a3411fa5"} Nov 03 13:50:01.603116 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:01.603079 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb4ca0-7cb4bfb5f4-j84xg" event={"ID":"40b93c39-a07f-4e43-a1cc-66cfb1469e51","Type":"ContainerStarted","Data":"cfaa1145263e37611741492bccb4f61df847a434caa4d3ac8766db3bf01fe1c7"} Nov 03 13:50:01.603550 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:01.603166 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb4ca0-7cb4bfb5f4-j84xg" Nov 03 13:50:01.713221 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:01.713167 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb4ca0-7cb4bfb5f4-j84xg" podStartSLOduration=1.866251611 podStartE2EDuration="2.713147444s" totalImagesPullingTime="846.895833ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:49:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:50:01.710685967 +0000 UTC m=+2683.721374233" watchObservedRunningTime="2026-11-03 13:50:01.713147444 +0000 UTC m=+2683.723835686" Nov 03 13:50:07.235108 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:50:07.235070 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:50:07.616724 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:07.616654 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb4ca0-7cb4bfb5f4-j84xg" Nov 03 13:50:14.428884 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:14.428844 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-10c688-775c68b957-8lm7w"] Nov 03 13:50:15.632623 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:15.632585 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6"] Nov 03 13:50:15.633115 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:15.633052 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6" podUID="0403729d-8803-40c2-9e5d-1cb2cc9914a1" containerName="istio-proxy" containerID="cri-o://7c05b13cbe2c7dd9aa32bae96ebb5782e0e1d851f11f7e37ecca8cb492b71074" gracePeriod=30 Nov 03 13:50:17.418243 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:17.418199 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:50:18.821332 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:18.821299 5979 patch_prober.go:29] interesting pod/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 13:50:18.821725 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:18.821352 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6" podUID="0403729d-8803-40c2-9e5d-1cb2cc9914a1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 13:50:19.430076 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:19.430032 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-10c688-775c68b957-8lm7w" podUID="2ba80e36-d422-4942-b385-c0910c848a82" containerName="payload-processing" containerID="cri-o://4e612024cd9730859b4ac14282b9e3593ea2c55764e403c6704b02e3899f7e51" gracePeriod=25 Nov 03 13:50:19.775874 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:19.775844 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-10c688-775c68b957-8lm7w" Nov 03 13:50:19.874177 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:19.874138 5979 generic.go:356] "Generic (PLEG): container finished" podID="2ba80e36-d422-4942-b385-c0910c848a82" containerID="4e612024cd9730859b4ac14282b9e3593ea2c55764e403c6704b02e3899f7e51" exitCode=0 Nov 03 13:50:19.874611 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:19.874189 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-10c688-775c68b957-8lm7w" Nov 03 13:50:19.874611 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:19.874229 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-10c688-775c68b957-8lm7w" event={"ID":"2ba80e36-d422-4942-b385-c0910c848a82","Type":"ContainerDied","Data":"4e612024cd9730859b4ac14282b9e3593ea2c55764e403c6704b02e3899f7e51"} Nov 03 13:50:19.874611 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:19.874314 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-10c688-775c68b957-8lm7w" event={"ID":"2ba80e36-d422-4942-b385-c0910c848a82","Type":"ContainerDied","Data":"74730b41eeec5fc0de98dd06cf29f9eb5a5f43c8eb261a23e5029ef04779b38c"} Nov 03 13:50:19.874611 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:19.874334 5979 scope.go:122] "RemoveContainer" containerID="4e612024cd9730859b4ac14282b9e3593ea2c55764e403c6704b02e3899f7e51" Nov 03 13:50:19.880714 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:19.880692 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ba80e36-d422-4942-b385-c0910c848a82-kube-api-access-tcsqv\" (UniqueName: \"kubernetes.io/projected/2ba80e36-d422-4942-b385-c0910c848a82-kube-api-access-tcsqv\") pod \"2ba80e36-d422-4942-b385-c0910c848a82\" (UID: \"2ba80e36-d422-4942-b385-c0910c848a82\") " Nov 03 13:50:19.880815 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:19.880743 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ba80e36-d422-4942-b385-c0910c848a82-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ba80e36-d422-4942-b385-c0910c848a82-plugins-config-volume\") pod \"2ba80e36-d422-4942-b385-c0910c848a82\" (UID: \"2ba80e36-d422-4942-b385-c0910c848a82\") " Nov 03 13:50:19.881190 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:19.881163 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ba80e36-d422-4942-b385-c0910c848a82-plugins-config-volume" pod "2ba80e36-d422-4942-b385-c0910c848a82" (UID: "2ba80e36-d422-4942-b385-c0910c848a82"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:50:19.882788 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:19.882764 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ba80e36-d422-4942-b385-c0910c848a82-kube-api-access-tcsqv" pod "2ba80e36-d422-4942-b385-c0910c848a82" (UID: "2ba80e36-d422-4942-b385-c0910c848a82"). InnerVolumeSpecName "kube-api-access-tcsqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:50:19.885546 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:19.885524 5979 scope.go:122] "RemoveContainer" containerID="4e612024cd9730859b4ac14282b9e3593ea2c55764e403c6704b02e3899f7e51" Nov 03 13:50:19.885909 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:50:19.885885 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e612024cd9730859b4ac14282b9e3593ea2c55764e403c6704b02e3899f7e51\": container with ID starting with 4e612024cd9730859b4ac14282b9e3593ea2c55764e403c6704b02e3899f7e51 not found: ID does not exist" containerID="4e612024cd9730859b4ac14282b9e3593ea2c55764e403c6704b02e3899f7e51" Nov 03 13:50:19.885997 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:19.885921 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e612024cd9730859b4ac14282b9e3593ea2c55764e403c6704b02e3899f7e51"} err="failed to get container status \"4e612024cd9730859b4ac14282b9e3593ea2c55764e403c6704b02e3899f7e51\": rpc error: code = NotFound desc = could not find container \"4e612024cd9730859b4ac14282b9e3593ea2c55764e403c6704b02e3899f7e51\": container with ID starting with 4e612024cd9730859b4ac14282b9e3593ea2c55764e403c6704b02e3899f7e51 not found: ID does not exist" Nov 03 13:50:19.982548 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:19.982416 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcsqv\" (UniqueName: \"kubernetes.io/projected/2ba80e36-d422-4942-b385-c0910c848a82-kube-api-access-tcsqv\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:50:19.982548 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:19.982461 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ba80e36-d422-4942-b385-c0910c848a82-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:50:20.015993 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:20.015962 5979 scope.go:122] "RemoveContainer" containerID="93d89ac1509414e580d842e91d4271f2e51452aa1e7e3a54ed5911241e1529f2" Nov 03 13:50:20.282067 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:20.281970 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-10c688-775c68b957-8lm7w"] Nov 03 13:50:20.285315 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:20.285288 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-10c688-775c68b957-8lm7w"] Nov 03 13:50:20.890928 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:20.890751 5979 generic.go:356] "Generic (PLEG): container finished" podID="0403729d-8803-40c2-9e5d-1cb2cc9914a1" containerID="7c05b13cbe2c7dd9aa32bae96ebb5782e0e1d851f11f7e37ecca8cb492b71074" exitCode=0 Nov 03 13:50:20.890928 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:20.890812 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6" event={"ID":"0403729d-8803-40c2-9e5d-1cb2cc9914a1","Type":"ContainerDied","Data":"7c05b13cbe2c7dd9aa32bae96ebb5782e0e1d851f11f7e37ecca8cb492b71074"} Nov 03 13:50:20.890928 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:20.890857 5979 scope.go:122] "RemoveContainer" containerID="2d13cb7796b4dd992f24aa0a404f34946439c117c567f652df7148d34bf9bbd0" Nov 03 13:50:20.977108 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:20.977083 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6" Nov 03 13:50:20.995223 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:20.995196 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0403729d-8803-40c2-9e5d-1cb2cc9914a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0403729d-8803-40c2-9e5d-1cb2cc9914a1-credential-socket\") pod \"0403729d-8803-40c2-9e5d-1cb2cc9914a1\" (UID: \"0403729d-8803-40c2-9e5d-1cb2cc9914a1\") " Nov 03 13:50:20.995332 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:20.995236 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0403729d-8803-40c2-9e5d-1cb2cc9914a1-istio-token\" (UniqueName: \"kubernetes.io/projected/0403729d-8803-40c2-9e5d-1cb2cc9914a1-istio-token\") pod \"0403729d-8803-40c2-9e5d-1cb2cc9914a1\" (UID: \"0403729d-8803-40c2-9e5d-1cb2cc9914a1\") " Nov 03 13:50:20.995332 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:20.995292 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0403729d-8803-40c2-9e5d-1cb2cc9914a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0403729d-8803-40c2-9e5d-1cb2cc9914a1-istio-data\") pod \"0403729d-8803-40c2-9e5d-1cb2cc9914a1\" (UID: \"0403729d-8803-40c2-9e5d-1cb2cc9914a1\") " Nov 03 13:50:20.995453 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:20.995335 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0403729d-8803-40c2-9e5d-1cb2cc9914a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0403729d-8803-40c2-9e5d-1cb2cc9914a1-workload-certs\") pod \"0403729d-8803-40c2-9e5d-1cb2cc9914a1\" (UID: \"0403729d-8803-40c2-9e5d-1cb2cc9914a1\") " Nov 03 13:50:20.995453 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:20.995368 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0403729d-8803-40c2-9e5d-1cb2cc9914a1-kube-api-access-wv96m\" (UniqueName: \"kubernetes.io/projected/0403729d-8803-40c2-9e5d-1cb2cc9914a1-kube-api-access-wv96m\") pod \"0403729d-8803-40c2-9e5d-1cb2cc9914a1\" (UID: \"0403729d-8803-40c2-9e5d-1cb2cc9914a1\") " Nov 03 13:50:20.995453 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:20.995406 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0403729d-8803-40c2-9e5d-1cb2cc9914a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0403729d-8803-40c2-9e5d-1cb2cc9914a1-istiod-ca-cert\") pod \"0403729d-8803-40c2-9e5d-1cb2cc9914a1\" (UID: \"0403729d-8803-40c2-9e5d-1cb2cc9914a1\") " Nov 03 13:50:20.995656 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:20.995446 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0403729d-8803-40c2-9e5d-1cb2cc9914a1-credential-socket" pod "0403729d-8803-40c2-9e5d-1cb2cc9914a1" (UID: "0403729d-8803-40c2-9e5d-1cb2cc9914a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:50:20.995656 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:20.995490 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0403729d-8803-40c2-9e5d-1cb2cc9914a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0403729d-8803-40c2-9e5d-1cb2cc9914a1-istio-podinfo\") pod \"0403729d-8803-40c2-9e5d-1cb2cc9914a1\" (UID: \"0403729d-8803-40c2-9e5d-1cb2cc9914a1\") " Nov 03 13:50:20.995656 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:20.995635 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0403729d-8803-40c2-9e5d-1cb2cc9914a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0403729d-8803-40c2-9e5d-1cb2cc9914a1-workload-socket\") pod \"0403729d-8803-40c2-9e5d-1cb2cc9914a1\" (UID: \"0403729d-8803-40c2-9e5d-1cb2cc9914a1\") " Nov 03 13:50:20.995873 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:20.995678 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0403729d-8803-40c2-9e5d-1cb2cc9914a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0403729d-8803-40c2-9e5d-1cb2cc9914a1-istio-envoy\") pod \"0403729d-8803-40c2-9e5d-1cb2cc9914a1\" (UID: \"0403729d-8803-40c2-9e5d-1cb2cc9914a1\") " Nov 03 13:50:20.995873 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:20.995680 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0403729d-8803-40c2-9e5d-1cb2cc9914a1-workload-certs" pod "0403729d-8803-40c2-9e5d-1cb2cc9914a1" (UID: "0403729d-8803-40c2-9e5d-1cb2cc9914a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:50:20.995873 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:20.995714 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0403729d-8803-40c2-9e5d-1cb2cc9914a1-istio-data" pod "0403729d-8803-40c2-9e5d-1cb2cc9914a1" (UID: "0403729d-8803-40c2-9e5d-1cb2cc9914a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:50:20.996025 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:20.995917 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0403729d-8803-40c2-9e5d-1cb2cc9914a1-workload-socket" pod "0403729d-8803-40c2-9e5d-1cb2cc9914a1" (UID: "0403729d-8803-40c2-9e5d-1cb2cc9914a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:50:20.996025 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:20.995937 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0403729d-8803-40c2-9e5d-1cb2cc9914a1-istiod-ca-cert" pod "0403729d-8803-40c2-9e5d-1cb2cc9914a1" (UID: "0403729d-8803-40c2-9e5d-1cb2cc9914a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:50:20.996282 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:20.996260 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0403729d-8803-40c2-9e5d-1cb2cc9914a1-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:50:20.996374 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:20.996285 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0403729d-8803-40c2-9e5d-1cb2cc9914a1-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:50:20.996374 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:20.996301 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0403729d-8803-40c2-9e5d-1cb2cc9914a1-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:50:20.996374 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:20.996310 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0403729d-8803-40c2-9e5d-1cb2cc9914a1-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:50:20.996374 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:20.996319 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0403729d-8803-40c2-9e5d-1cb2cc9914a1-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:50:20.997620 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:20.997597 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0403729d-8803-40c2-9e5d-1cb2cc9914a1-istio-token" pod "0403729d-8803-40c2-9e5d-1cb2cc9914a1" (UID: "0403729d-8803-40c2-9e5d-1cb2cc9914a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:50:20.997798 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:20.997781 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0403729d-8803-40c2-9e5d-1cb2cc9914a1-kube-api-access-wv96m" pod "0403729d-8803-40c2-9e5d-1cb2cc9914a1" (UID: "0403729d-8803-40c2-9e5d-1cb2cc9914a1"). InnerVolumeSpecName "kube-api-access-wv96m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:50:20.998046 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:20.998021 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0403729d-8803-40c2-9e5d-1cb2cc9914a1-istio-envoy" pod "0403729d-8803-40c2-9e5d-1cb2cc9914a1" (UID: "0403729d-8803-40c2-9e5d-1cb2cc9914a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:50:20.998148 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:20.998024 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0403729d-8803-40c2-9e5d-1cb2cc9914a1-istio-podinfo" pod "0403729d-8803-40c2-9e5d-1cb2cc9914a1" (UID: "0403729d-8803-40c2-9e5d-1cb2cc9914a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:50:21.097211 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:21.097177 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0403729d-8803-40c2-9e5d-1cb2cc9914a1-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:50:21.097211 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:21.097210 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0403729d-8803-40c2-9e5d-1cb2cc9914a1-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:50:21.097430 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:21.097224 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv96m\" (UniqueName: \"kubernetes.io/projected/0403729d-8803-40c2-9e5d-1cb2cc9914a1-kube-api-access-wv96m\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:50:21.097430 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:21.097239 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0403729d-8803-40c2-9e5d-1cb2cc9914a1-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:50:21.899899 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:21.899869 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6" Nov 03 13:50:21.900385 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:21.899904 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6" event={"ID":"0403729d-8803-40c2-9e5d-1cb2cc9914a1","Type":"ContainerDied","Data":"adc778446fb7672951f3a07ad9a691642bb1bb3033c192f934e73715d0005f1c"} Nov 03 13:50:21.900385 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:21.899969 5979 scope.go:122] "RemoveContainer" containerID="7c05b13cbe2c7dd9aa32bae96ebb5782e0e1d851f11f7e37ecca8cb492b71074" Nov 03 13:50:22.115646 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:22.115614 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ba80e36-d422-4942-b385-c0910c848a82" path="/var/lib/kubelet/pods/2ba80e36-d422-4942-b385-c0910c848a82/volumes" Nov 03 13:50:22.123093 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:22.123061 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6"] Nov 03 13:50:22.128935 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:22.128910 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-ba6fbd-openshift-default-b596f957c-gpkd6"] Nov 03 13:50:24.116139 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:24.116102 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0403729d-8803-40c2-9e5d-1cb2cc9914a1" path="/var/lib/kubelet/pods/0403729d-8803-40c2-9e5d-1cb2cc9914a1/volumes" Nov 03 13:50:25.051412 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:25.051371 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2862039b-jpttz"] Nov 03 13:50:25.097835 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:25.097801 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2862039b-jpttz"] Nov 03 13:50:26.045912 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:26.045871 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e083cf9-lrpg7"] Nov 03 13:50:26.092412 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:26.092371 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e083cf9-lrpg7"] Nov 03 13:50:26.118704 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:26.118667 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9911db5-2c92-4007-ad7e-68e14df3b6dc" path="/var/lib/kubelet/pods/d9911db5-2c92-4007-ad7e-68e14df3b6dc/volumes" Nov 03 13:50:26.119347 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:26.119320 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3ff6305-7ae8-4569-acec-f5360482bcd4" path="/var/lib/kubelet/pods/e3ff6305-7ae8-4569-acec-f5360482bcd4/volumes" Nov 03 13:50:27.945910 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:27.945880 5979 generic.go:356] "Generic (PLEG): container finished" podID="7512d231-097f-4192-99f7-44bb0f98bdda" containerID="599f67b0f10ae1ba88ca3a8777bf623a8dfae54ca11b20e5c4a82fc28b79a7cc" exitCode=0 Nov 03 13:50:27.946309 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:27.945951 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t" event={"ID":"7512d231-097f-4192-99f7-44bb0f98bdda","Type":"ContainerDied","Data":"599f67b0f10ae1ba88ca3a8777bf623a8dfae54ca11b20e5c4a82fc28b79a7cc"} Nov 03 13:50:27.946385 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:27.946368 5979 scope.go:122] "RemoveContainer" containerID="599f67b0f10ae1ba88ca3a8777bf623a8dfae54ca11b20e5c4a82fc28b79a7cc" Nov 03 13:50:27.947936 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:27.947909 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:50:27.948022 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:27.947981 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:50:27.948022 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:27.948012 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:50:28.978016 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:28.977931 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t" event={"ID":"7512d231-097f-4192-99f7-44bb0f98bdda","Type":"ContainerStarted","Data":"154c304c370c17c8c81c1e288e693e21a4591eb3d5015f2087e536d9b015952d"} Nov 03 13:50:29.510997 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:29.510966 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t" Nov 03 13:50:29.512061 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:29.512032 5979 patch_prober.go:29] interesting pod/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" start-of-body= Nov 03 13:50:29.512233 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:29.512091 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t" podUID="7512d231-097f-4192-99f7-44bb0f98bdda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" Nov 03 13:50:30.510603 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:30.510566 5979 patch_prober.go:29] interesting pod/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" start-of-body= Nov 03 13:50:30.510998 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:30.510617 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t" podUID="7512d231-097f-4192-99f7-44bb0f98bdda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" Nov 03 13:50:31.511120 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:31.511083 5979 patch_prober.go:29] interesting pod/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" start-of-body= Nov 03 13:50:31.511576 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:31.511146 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t" podUID="7512d231-097f-4192-99f7-44bb0f98bdda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" Nov 03 13:50:32.510664 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:32.510628 5979 patch_prober.go:29] interesting pod/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" start-of-body= Nov 03 13:50:32.510876 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:32.510694 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t" podUID="7512d231-097f-4192-99f7-44bb0f98bdda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" Nov 03 13:50:33.510820 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:33.510782 5979 patch_prober.go:29] interesting pod/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" start-of-body= Nov 03 13:50:33.511268 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:33.510847 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t" podUID="7512d231-097f-4192-99f7-44bb0f98bdda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" Nov 03 13:50:33.844560 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:33.843647 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t" Nov 03 13:50:33.952905 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:33.952865 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj"] Nov 03 13:50:33.953523 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:33.953459 5979 cpu_manager.go:414] "containerMap: removing container" podUID="0403729d-8803-40c2-9e5d-1cb2cc9914a1" containerName="istio-proxy" Nov 03 13:50:33.953523 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:33.953490 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0403729d-8803-40c2-9e5d-1cb2cc9914a1" containerName="istio-proxy" Nov 03 13:50:33.953523 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:33.953511 5979 cpu_manager.go:414] "containerMap: removing container" podUID="2ba80e36-d422-4942-b385-c0910c848a82" containerName="payload-processing" Nov 03 13:50:33.953523 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:33.953519 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ba80e36-d422-4942-b385-c0910c848a82" containerName="payload-processing" Nov 03 13:50:33.953803 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:33.953543 5979 cpu_manager.go:414] "containerMap: removing container" podUID="0403729d-8803-40c2-9e5d-1cb2cc9914a1" containerName="istio-proxy" Nov 03 13:50:33.953803 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:33.953548 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0403729d-8803-40c2-9e5d-1cb2cc9914a1" containerName="istio-proxy" Nov 03 13:50:33.953803 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:33.953656 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="0403729d-8803-40c2-9e5d-1cb2cc9914a1" containerName="istio-proxy" Nov 03 13:50:33.953803 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:33.953673 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ba80e36-d422-4942-b385-c0910c848a82" containerName="payload-processing" Nov 03 13:50:33.953803 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:33.953680 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="763f2c9e-811d-49dc-b971-4782cb8a0964" containerName="istio-proxy" Nov 03 13:50:33.957334 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:33.957314 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj" Nov 03 13:50:34.088114 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:34.088078 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-60261a-openshift-default-dockercfg-czkth\"" Nov 03 13:50:34.106553 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:34.106433 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4723366a-b53b-40f7-a27a-d5cde461525d-istio-podinfo\") pod \"e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj\" (UID: \"4723366a-b53b-40f7-a27a-d5cde461525d\") " pod="openshift-ingress/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj" Nov 03 13:50:34.106872 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:34.106838 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4723366a-b53b-40f7-a27a-d5cde461525d-workload-socket\") pod \"e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj\" (UID: \"4723366a-b53b-40f7-a27a-d5cde461525d\") " pod="openshift-ingress/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj" Nov 03 13:50:34.107014 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:34.106997 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4723366a-b53b-40f7-a27a-d5cde461525d-istio-envoy\") pod \"e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj\" (UID: \"4723366a-b53b-40f7-a27a-d5cde461525d\") " pod="openshift-ingress/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj" Nov 03 13:50:34.107133 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:34.107121 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4723366a-b53b-40f7-a27a-d5cde461525d-istio-token\") pod \"e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj\" (UID: \"4723366a-b53b-40f7-a27a-d5cde461525d\") " pod="openshift-ingress/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj" Nov 03 13:50:34.107265 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:34.107250 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4723366a-b53b-40f7-a27a-d5cde461525d-istio-data\") pod \"e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj\" (UID: \"4723366a-b53b-40f7-a27a-d5cde461525d\") " pod="openshift-ingress/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj" Nov 03 13:50:34.107489 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:34.107450 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-266k5\" (UniqueName: \"kubernetes.io/projected/4723366a-b53b-40f7-a27a-d5cde461525d-kube-api-access-266k5\") pod \"e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj\" (UID: \"4723366a-b53b-40f7-a27a-d5cde461525d\") " pod="openshift-ingress/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj" Nov 03 13:50:34.107697 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:34.107676 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4723366a-b53b-40f7-a27a-d5cde461525d-credential-socket\") pod \"e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj\" (UID: \"4723366a-b53b-40f7-a27a-d5cde461525d\") " pod="openshift-ingress/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj" Nov 03 13:50:34.107875 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:34.107859 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4723366a-b53b-40f7-a27a-d5cde461525d-workload-certs\") pod \"e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj\" (UID: \"4723366a-b53b-40f7-a27a-d5cde461525d\") " pod="openshift-ingress/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj" Nov 03 13:50:34.108003 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:34.107986 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4723366a-b53b-40f7-a27a-d5cde461525d-istiod-ca-cert\") pod \"e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj\" (UID: \"4723366a-b53b-40f7-a27a-d5cde461525d\") " pod="openshift-ingress/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj" Nov 03 13:50:34.149128 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:34.149092 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj"] Nov 03 13:50:34.208847 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:34.208814 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4723366a-b53b-40f7-a27a-d5cde461525d-istio-podinfo\") pod \"e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj\" (UID: \"4723366a-b53b-40f7-a27a-d5cde461525d\") " pod="openshift-ingress/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj" Nov 03 13:50:34.209059 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:34.208857 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4723366a-b53b-40f7-a27a-d5cde461525d-workload-socket\") pod \"e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj\" (UID: \"4723366a-b53b-40f7-a27a-d5cde461525d\") " pod="openshift-ingress/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj" Nov 03 13:50:34.209059 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:34.208876 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4723366a-b53b-40f7-a27a-d5cde461525d-istio-envoy\") pod \"e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj\" (UID: \"4723366a-b53b-40f7-a27a-d5cde461525d\") " pod="openshift-ingress/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj" Nov 03 13:50:34.209059 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:34.208900 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4723366a-b53b-40f7-a27a-d5cde461525d-istio-token\") pod \"e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj\" (UID: \"4723366a-b53b-40f7-a27a-d5cde461525d\") " pod="openshift-ingress/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj" Nov 03 13:50:34.209059 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:34.208930 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4723366a-b53b-40f7-a27a-d5cde461525d-istio-data\") pod \"e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj\" (UID: \"4723366a-b53b-40f7-a27a-d5cde461525d\") " pod="openshift-ingress/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj" Nov 03 13:50:34.209059 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:34.208998 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-266k5\" (UniqueName: \"kubernetes.io/projected/4723366a-b53b-40f7-a27a-d5cde461525d-kube-api-access-266k5\") pod \"e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj\" (UID: \"4723366a-b53b-40f7-a27a-d5cde461525d\") " pod="openshift-ingress/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj" Nov 03 13:50:34.209059 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:34.209045 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4723366a-b53b-40f7-a27a-d5cde461525d-credential-socket\") pod \"e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj\" (UID: \"4723366a-b53b-40f7-a27a-d5cde461525d\") " pod="openshift-ingress/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj" Nov 03 13:50:34.209424 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:34.209109 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4723366a-b53b-40f7-a27a-d5cde461525d-workload-certs\") pod \"e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj\" (UID: \"4723366a-b53b-40f7-a27a-d5cde461525d\") " pod="openshift-ingress/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj" Nov 03 13:50:34.209424 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:34.209140 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4723366a-b53b-40f7-a27a-d5cde461525d-istiod-ca-cert\") pod \"e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj\" (UID: \"4723366a-b53b-40f7-a27a-d5cde461525d\") " pod="openshift-ingress/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj" Nov 03 13:50:34.209424 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:34.209310 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4723366a-b53b-40f7-a27a-d5cde461525d-workload-socket\") pod \"e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj\" (UID: \"4723366a-b53b-40f7-a27a-d5cde461525d\") " pod="openshift-ingress/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj" Nov 03 13:50:34.209635 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:34.209430 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4723366a-b53b-40f7-a27a-d5cde461525d-istio-data\") pod \"e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj\" (UID: \"4723366a-b53b-40f7-a27a-d5cde461525d\") " pod="openshift-ingress/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj" Nov 03 13:50:34.209635 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:34.209498 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4723366a-b53b-40f7-a27a-d5cde461525d-credential-socket\") pod \"e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj\" (UID: \"4723366a-b53b-40f7-a27a-d5cde461525d\") " pod="openshift-ingress/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj" Nov 03 13:50:34.209635 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:34.209511 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4723366a-b53b-40f7-a27a-d5cde461525d-workload-certs\") pod \"e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj\" (UID: \"4723366a-b53b-40f7-a27a-d5cde461525d\") " pod="openshift-ingress/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj" Nov 03 13:50:34.209847 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:34.209828 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4723366a-b53b-40f7-a27a-d5cde461525d-istiod-ca-cert\") pod \"e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj\" (UID: \"4723366a-b53b-40f7-a27a-d5cde461525d\") " pod="openshift-ingress/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj" Nov 03 13:50:34.211616 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:34.211592 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4723366a-b53b-40f7-a27a-d5cde461525d-istio-envoy\") pod \"e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj\" (UID: \"4723366a-b53b-40f7-a27a-d5cde461525d\") " pod="openshift-ingress/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj" Nov 03 13:50:34.211871 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:34.211855 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4723366a-b53b-40f7-a27a-d5cde461525d-istio-podinfo\") pod \"e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj\" (UID: \"4723366a-b53b-40f7-a27a-d5cde461525d\") " pod="openshift-ingress/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj" Nov 03 13:50:34.278239 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:34.278071 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-266k5\" (UniqueName: \"kubernetes.io/projected/4723366a-b53b-40f7-a27a-d5cde461525d-kube-api-access-266k5\") pod \"e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj\" (UID: \"4723366a-b53b-40f7-a27a-d5cde461525d\") " pod="openshift-ingress/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj" Nov 03 13:50:34.281600 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:34.280029 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4723366a-b53b-40f7-a27a-d5cde461525d-istio-token\") pod \"e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj\" (UID: \"4723366a-b53b-40f7-a27a-d5cde461525d\") " pod="openshift-ingress/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj" Nov 03 13:50:34.515237 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:34.515203 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t" Nov 03 13:50:34.516436 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:34.516415 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t" Nov 03 13:50:34.572558 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:34.572513 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj" Nov 03 13:50:35.004808 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:35.004775 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj"] Nov 03 13:50:35.010424 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:50:35.010395 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4723366a_b53b_40f7_a27a_d5cde461525d.slice/crio-58188e0fdcedc4b434a10c721fa559c81542c0c09a7cfb25b5eff520ecb0ed9d WatchSource:0}: Error finding container 58188e0fdcedc4b434a10c721fa559c81542c0c09a7cfb25b5eff520ecb0ed9d: Status 404 returned error can't find the container with id 58188e0fdcedc4b434a10c721fa559c81542c0c09a7cfb25b5eff520ecb0ed9d Nov 03 13:50:35.013105 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:35.013068 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:50:35.013186 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:35.013137 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:50:35.013186 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:35.013165 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:50:35.177661 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:35.174052 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj" event={"ID":"4723366a-b53b-40f7-a27a-d5cde461525d","Type":"ContainerStarted","Data":"bbccee5e13015c2853354a6cbc64fccec9b6655e2cd856a2b03a7bfb19300918"} Nov 03 13:50:35.177661 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:35.174117 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj" event={"ID":"4723366a-b53b-40f7-a27a-d5cde461525d","Type":"ContainerStarted","Data":"58188e0fdcedc4b434a10c721fa559c81542c0c09a7cfb25b5eff520ecb0ed9d"} Nov 03 13:50:35.289937 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:35.289831 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj" podStartSLOduration=2.289813068 podStartE2EDuration="2.289813068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:50:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:50:35.286524746 +0000 UTC m=+2717.297212985" watchObservedRunningTime="2026-11-03 13:50:35.289813068 +0000 UTC m=+2717.300501311" Nov 03 13:50:35.573407 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:35.573332 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj" Nov 03 13:50:36.578611 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:36.578576 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj" Nov 03 13:50:37.181868 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:37.181840 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj" Nov 03 13:50:37.182963 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:37.182940 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj" Nov 03 13:50:37.234211 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:50:37.234173 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:50:41.044629 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:41.044596 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80c5861a-c9qg5"] Nov 03 13:50:41.090978 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:41.090946 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80c5861a-c9qg5"] Nov 03 13:50:42.116757 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:42.116726 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1cfb1a0-cef9-4bf9-8392-e671178ef46e" path="/var/lib/kubelet/pods/c1cfb1a0-cef9-4bf9-8392-e671178ef46e/volumes" Nov 03 13:50:43.079936 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:43.079906 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc"] Nov 03 13:50:43.080740 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:43.080720 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="0403729d-8803-40c2-9e5d-1cb2cc9914a1" containerName="istio-proxy" Nov 03 13:50:43.084749 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:43.084730 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc" Nov 03 13:50:43.133638 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:43.133606 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-d341af-openshift-default-dockercfg-csn4v\"" Nov 03 13:50:43.187930 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:43.187879 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc"] Nov 03 13:50:43.292802 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:43.292686 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-workload-socket\") pod \"e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc\" (UID: \"8df65c37-2cc2-4bd1-9d05-4576dfaec9a6\") " pod="openshift-ingress/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc" Nov 03 13:50:43.293175 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:43.293134 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsxk4\" (UniqueName: \"kubernetes.io/projected/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-kube-api-access-gsxk4\") pod \"e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc\" (UID: \"8df65c37-2cc2-4bd1-9d05-4576dfaec9a6\") " pod="openshift-ingress/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc" Nov 03 13:50:43.293353 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:43.293324 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-istio-token\") pod \"e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc\" (UID: \"8df65c37-2cc2-4bd1-9d05-4576dfaec9a6\") " pod="openshift-ingress/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc" Nov 03 13:50:43.293559 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:43.293526 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-workload-certs\") pod \"e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc\" (UID: \"8df65c37-2cc2-4bd1-9d05-4576dfaec9a6\") " pod="openshift-ingress/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc" Nov 03 13:50:43.293705 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:43.293678 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-istiod-ca-cert\") pod \"e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc\" (UID: \"8df65c37-2cc2-4bd1-9d05-4576dfaec9a6\") " pod="openshift-ingress/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc" Nov 03 13:50:43.293872 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:43.293843 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-istio-envoy\") pod \"e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc\" (UID: \"8df65c37-2cc2-4bd1-9d05-4576dfaec9a6\") " pod="openshift-ingress/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc" Nov 03 13:50:43.294041 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:43.294013 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-istio-podinfo\") pod \"e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc\" (UID: \"8df65c37-2cc2-4bd1-9d05-4576dfaec9a6\") " pod="openshift-ingress/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc" Nov 03 13:50:43.294205 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:43.294178 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-credential-socket\") pod \"e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc\" (UID: \"8df65c37-2cc2-4bd1-9d05-4576dfaec9a6\") " pod="openshift-ingress/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc" Nov 03 13:50:43.297550 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:43.294357 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-istio-data\") pod \"e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc\" (UID: \"8df65c37-2cc2-4bd1-9d05-4576dfaec9a6\") " pod="openshift-ingress/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc" Nov 03 13:50:43.399218 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:43.399186 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-workload-socket\") pod \"e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc\" (UID: \"8df65c37-2cc2-4bd1-9d05-4576dfaec9a6\") " pod="openshift-ingress/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc" Nov 03 13:50:43.399379 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:43.399240 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsxk4\" (UniqueName: \"kubernetes.io/projected/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-kube-api-access-gsxk4\") pod \"e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc\" (UID: \"8df65c37-2cc2-4bd1-9d05-4576dfaec9a6\") " pod="openshift-ingress/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc" Nov 03 13:50:43.399379 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:43.399354 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-istio-token\") pod \"e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc\" (UID: \"8df65c37-2cc2-4bd1-9d05-4576dfaec9a6\") " pod="openshift-ingress/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc" Nov 03 13:50:43.399520 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:43.399409 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-workload-certs\") pod \"e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc\" (UID: \"8df65c37-2cc2-4bd1-9d05-4576dfaec9a6\") " pod="openshift-ingress/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc" Nov 03 13:50:43.399585 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:43.399516 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-istiod-ca-cert\") pod \"e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc\" (UID: \"8df65c37-2cc2-4bd1-9d05-4576dfaec9a6\") " pod="openshift-ingress/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc" Nov 03 13:50:43.399585 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:43.399569 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-istio-envoy\") pod \"e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc\" (UID: \"8df65c37-2cc2-4bd1-9d05-4576dfaec9a6\") " pod="openshift-ingress/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc" Nov 03 13:50:43.399699 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:43.399614 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-istio-podinfo\") pod \"e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc\" (UID: \"8df65c37-2cc2-4bd1-9d05-4576dfaec9a6\") " pod="openshift-ingress/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc" Nov 03 13:50:43.399699 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:43.399642 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-credential-socket\") pod \"e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc\" (UID: \"8df65c37-2cc2-4bd1-9d05-4576dfaec9a6\") " pod="openshift-ingress/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc" Nov 03 13:50:43.399699 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:43.399657 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-workload-socket\") pod \"e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc\" (UID: \"8df65c37-2cc2-4bd1-9d05-4576dfaec9a6\") " pod="openshift-ingress/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc" Nov 03 13:50:43.399855 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:43.399695 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-istio-data\") pod \"e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc\" (UID: \"8df65c37-2cc2-4bd1-9d05-4576dfaec9a6\") " pod="openshift-ingress/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc" Nov 03 13:50:43.399855 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:43.399754 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-workload-certs\") pod \"e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc\" (UID: \"8df65c37-2cc2-4bd1-9d05-4576dfaec9a6\") " pod="openshift-ingress/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc" Nov 03 13:50:43.400053 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:43.400020 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-credential-socket\") pod \"e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc\" (UID: \"8df65c37-2cc2-4bd1-9d05-4576dfaec9a6\") " pod="openshift-ingress/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc" Nov 03 13:50:43.400191 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:43.400088 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-istio-data\") pod \"e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc\" (UID: \"8df65c37-2cc2-4bd1-9d05-4576dfaec9a6\") " pod="openshift-ingress/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc" Nov 03 13:50:43.400191 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:43.400163 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-istiod-ca-cert\") pod \"e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc\" (UID: \"8df65c37-2cc2-4bd1-9d05-4576dfaec9a6\") " pod="openshift-ingress/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc" Nov 03 13:50:43.402003 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:43.401983 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-istio-envoy\") pod \"e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc\" (UID: \"8df65c37-2cc2-4bd1-9d05-4576dfaec9a6\") " pod="openshift-ingress/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc" Nov 03 13:50:43.402247 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:43.402224 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-istio-podinfo\") pod \"e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc\" (UID: \"8df65c37-2cc2-4bd1-9d05-4576dfaec9a6\") " pod="openshift-ingress/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc" Nov 03 13:50:43.464558 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:43.464521 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsxk4\" (UniqueName: \"kubernetes.io/projected/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-kube-api-access-gsxk4\") pod \"e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc\" (UID: \"8df65c37-2cc2-4bd1-9d05-4576dfaec9a6\") " pod="openshift-ingress/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc" Nov 03 13:50:43.464765 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:43.464586 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-istio-token\") pod \"e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc\" (UID: \"8df65c37-2cc2-4bd1-9d05-4576dfaec9a6\") " pod="openshift-ingress/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc" Nov 03 13:50:43.699390 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:43.699306 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc" Nov 03 13:50:44.020668 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:44.020587 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-60261a-c86bf8c5f-pxlmt"] Nov 03 13:50:44.026083 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:44.026058 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-60261a-c86bf8c5f-pxlmt" Nov 03 13:50:44.078250 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:44.078220 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-60261a\"" Nov 03 13:50:44.078422 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:44.078303 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-60261a-dockercfg-27j4m\"" Nov 03 13:50:44.082123 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:44.082097 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-60261a-c86bf8c5f-pxlmt"] Nov 03 13:50:44.184494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:44.184166 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e2ca1ff-46e0-4b67-a561-bebf0d231464-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-60261a-c86bf8c5f-pxlmt\" (UID: \"0e2ca1ff-46e0-4b67-a561-bebf0d231464\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-60261a-c86bf8c5f-pxlmt" Nov 03 13:50:44.184494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:44.184396 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv42p\" (UniqueName: \"kubernetes.io/projected/0e2ca1ff-46e0-4b67-a561-bebf0d231464-kube-api-access-dv42p\") pod \"payload-pre-processing-e2e-shared-a-w1-60261a-c86bf8c5f-pxlmt\" (UID: \"0e2ca1ff-46e0-4b67-a561-bebf0d231464\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-60261a-c86bf8c5f-pxlmt" Nov 03 13:50:44.242494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:44.242078 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-60261a-77454f94c-ccbgm"] Nov 03 13:50:44.254495 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:44.254037 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-60261a-77454f94c-ccbgm" Nov 03 13:50:44.258406 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:44.255448 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-60261a-77454f94c-ccbgm"] Nov 03 13:50:44.266520 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:44.266099 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc"] Nov 03 13:50:44.272349 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:44.272310 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:50:44.272419 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:44.272401 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:50:44.272457 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:44.272446 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:50:44.287039 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:44.286960 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv42p\" (UniqueName: \"kubernetes.io/projected/0e2ca1ff-46e0-4b67-a561-bebf0d231464-kube-api-access-dv42p\") pod \"payload-pre-processing-e2e-shared-a-w1-60261a-c86bf8c5f-pxlmt\" (UID: \"0e2ca1ff-46e0-4b67-a561-bebf0d231464\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-60261a-c86bf8c5f-pxlmt" Nov 03 13:50:44.287704 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:44.287238 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxktq\" (UniqueName: \"kubernetes.io/projected/cdc023c5-a79b-49d0-9bed-ea1b749d4416-kube-api-access-jxktq\") pod \"payload-processing-e2e-shared-a-w1-60261a-77454f94c-ccbgm\" (UID: \"cdc023c5-a79b-49d0-9bed-ea1b749d4416\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-60261a-77454f94c-ccbgm" Nov 03 13:50:44.287704 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:44.287359 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e2ca1ff-46e0-4b67-a561-bebf0d231464-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-60261a-c86bf8c5f-pxlmt\" (UID: \"0e2ca1ff-46e0-4b67-a561-bebf0d231464\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-60261a-c86bf8c5f-pxlmt" Nov 03 13:50:44.287704 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:44.287443 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdc023c5-a79b-49d0-9bed-ea1b749d4416-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-60261a-77454f94c-ccbgm\" (UID: \"cdc023c5-a79b-49d0-9bed-ea1b749d4416\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-60261a-77454f94c-ccbgm" Nov 03 13:50:44.288027 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:44.288002 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e2ca1ff-46e0-4b67-a561-bebf0d231464-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-60261a-c86bf8c5f-pxlmt\" (UID: \"0e2ca1ff-46e0-4b67-a561-bebf0d231464\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-60261a-c86bf8c5f-pxlmt" Nov 03 13:50:44.319037 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:44.319008 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc" event={"ID":"8df65c37-2cc2-4bd1-9d05-4576dfaec9a6","Type":"ContainerStarted","Data":"8bfbc78f08819bebec23464da6af413ce213c678322a5b08a3033beb15fe98f5"} Nov 03 13:50:44.340123 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:44.340098 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv42p\" (UniqueName: \"kubernetes.io/projected/0e2ca1ff-46e0-4b67-a561-bebf0d231464-kube-api-access-dv42p\") pod \"payload-pre-processing-e2e-shared-a-w1-60261a-c86bf8c5f-pxlmt\" (UID: \"0e2ca1ff-46e0-4b67-a561-bebf0d231464\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-60261a-c86bf8c5f-pxlmt" Nov 03 13:50:44.340324 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:44.340309 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-60261a-c86bf8c5f-pxlmt" Nov 03 13:50:44.388772 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:44.388731 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdc023c5-a79b-49d0-9bed-ea1b749d4416-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-60261a-77454f94c-ccbgm\" (UID: \"cdc023c5-a79b-49d0-9bed-ea1b749d4416\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-60261a-77454f94c-ccbgm" Nov 03 13:50:44.388985 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:44.388962 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxktq\" (UniqueName: \"kubernetes.io/projected/cdc023c5-a79b-49d0-9bed-ea1b749d4416-kube-api-access-jxktq\") pod \"payload-processing-e2e-shared-a-w1-60261a-77454f94c-ccbgm\" (UID: \"cdc023c5-a79b-49d0-9bed-ea1b749d4416\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-60261a-77454f94c-ccbgm" Nov 03 13:50:44.389619 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:44.389595 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdc023c5-a79b-49d0-9bed-ea1b749d4416-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-60261a-77454f94c-ccbgm\" (UID: \"cdc023c5-a79b-49d0-9bed-ea1b749d4416\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-60261a-77454f94c-ccbgm" Nov 03 13:50:44.452743 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:44.448595 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxktq\" (UniqueName: \"kubernetes.io/projected/cdc023c5-a79b-49d0-9bed-ea1b749d4416-kube-api-access-jxktq\") pod \"payload-processing-e2e-shared-a-w1-60261a-77454f94c-ccbgm\" (UID: \"cdc023c5-a79b-49d0-9bed-ea1b749d4416\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-60261a-77454f94c-ccbgm" Nov 03 13:50:44.576604 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:44.576529 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-60261a-77454f94c-ccbgm" Nov 03 13:50:44.720070 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:44.720025 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-60261a-c86bf8c5f-pxlmt"] Nov 03 13:50:44.726287 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:50:44.726255 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e2ca1ff_46e0_4b67_a561_bebf0d231464.slice/crio-c696a463be0e556ce53c685f65cc9bf7f451e92c3b22d2ce4c272149131bdcce WatchSource:0}: Error finding container c696a463be0e556ce53c685f65cc9bf7f451e92c3b22d2ce4c272149131bdcce: Status 404 returned error can't find the container with id c696a463be0e556ce53c685f65cc9bf7f451e92c3b22d2ce4c272149131bdcce Nov 03 13:50:44.963151 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:44.963114 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-60261a-77454f94c-ccbgm"] Nov 03 13:50:45.328308 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:45.328207 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-60261a-c86bf8c5f-pxlmt" event={"ID":"0e2ca1ff-46e0-4b67-a561-bebf0d231464","Type":"ContainerStarted","Data":"4a63747785a13b7b05bc1c66e6a2091613d8dc6142384b9a72dab5f003124ba9"} Nov 03 13:50:45.328308 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:45.328272 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-60261a-c86bf8c5f-pxlmt" Nov 03 13:50:45.328308 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:45.328294 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-60261a-c86bf8c5f-pxlmt" event={"ID":"0e2ca1ff-46e0-4b67-a561-bebf0d231464","Type":"ContainerStarted","Data":"c696a463be0e556ce53c685f65cc9bf7f451e92c3b22d2ce4c272149131bdcce"} Nov 03 13:50:45.329863 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:45.329831 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-60261a-77454f94c-ccbgm" event={"ID":"cdc023c5-a79b-49d0-9bed-ea1b749d4416","Type":"ContainerStarted","Data":"e97c86d573fc01a8fd6e0cb8193420852302aa4fa64486f60a4eb35f39d17156"} Nov 03 13:50:45.329988 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:45.329884 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-60261a-77454f94c-ccbgm" event={"ID":"cdc023c5-a79b-49d0-9bed-ea1b749d4416","Type":"ContainerStarted","Data":"94058f424c06681052900201b008ae5844149e9bfc60972aa6d72ee23502db3b"} Nov 03 13:50:45.329988 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:45.329912 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-60261a-77454f94c-ccbgm" Nov 03 13:50:45.331439 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:45.331412 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc" event={"ID":"8df65c37-2cc2-4bd1-9d05-4576dfaec9a6","Type":"ContainerStarted","Data":"285bb6e9355b6d8fdcaf8c55057073e92fb5e1074e1af38f22f3036fa5464d1a"} Nov 03 13:50:45.434213 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:45.434113 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-60261a-c86bf8c5f-pxlmt" podStartSLOduration=2.434097861 podStartE2EDuration="2.434097861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:50:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:50:45.434056273 +0000 UTC m=+2727.444744517" watchObservedRunningTime="2026-11-03 13:50:45.434097861 +0000 UTC m=+2727.444786103" Nov 03 13:50:45.544462 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:45.544398 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-60261a-77454f94c-ccbgm" podStartSLOduration=1.5443806119999999 podStartE2EDuration="1.544380612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:50:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:50:45.541028245 +0000 UTC m=+2727.551716490" watchObservedRunningTime="2026-11-03 13:50:45.544380612 +0000 UTC m=+2727.555068856" Nov 03 13:50:45.653691 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:45.653642 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc" podStartSLOduration=2.65362804 podStartE2EDuration="2.65362804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:50:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:50:45.650785177 +0000 UTC m=+2727.661473419" watchObservedRunningTime="2026-11-03 13:50:45.65362804 +0000 UTC m=+2727.664316279" Nov 03 13:50:45.702991 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:45.702950 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc" Nov 03 13:50:45.718502 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:45.718200 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc" Nov 03 13:50:46.338508 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:46.338462 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc" Nov 03 13:50:46.339530 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:46.339508 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc" Nov 03 13:50:53.596075 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:53.596043 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d341af-5c58b5c667-bnws4"] Nov 03 13:50:53.600406 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:53.600384 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d341af-5c58b5c667-bnws4" Nov 03 13:50:53.638925 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:53.638887 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f03e902b-a262-4d46-b393-dea566c264d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-d341af-5c58b5c667-bnws4\" (UID: \"f03e902b-a262-4d46-b393-dea566c264d2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d341af-5c58b5c667-bnws4" Nov 03 13:50:53.639137 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:53.638953 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c2t2\" (UniqueName: \"kubernetes.io/projected/f03e902b-a262-4d46-b393-dea566c264d2-kube-api-access-9c2t2\") pod \"payload-pre-processing-e2e-shared-b-w2-d341af-5c58b5c667-bnws4\" (UID: \"f03e902b-a262-4d46-b393-dea566c264d2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d341af-5c58b5c667-bnws4" Nov 03 13:50:53.654872 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:53.654834 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-d341af\"" Nov 03 13:50:53.729875 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:53.729824 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-d341af-dockercfg-wmk99\"" Nov 03 13:50:53.737451 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:53.737416 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d341af-5c58b5c667-bnws4"] Nov 03 13:50:53.744102 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:53.742678 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f03e902b-a262-4d46-b393-dea566c264d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-d341af-5c58b5c667-bnws4\" (UID: \"f03e902b-a262-4d46-b393-dea566c264d2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d341af-5c58b5c667-bnws4" Nov 03 13:50:53.744102 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:53.742756 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c2t2\" (UniqueName: \"kubernetes.io/projected/f03e902b-a262-4d46-b393-dea566c264d2-kube-api-access-9c2t2\") pod \"payload-pre-processing-e2e-shared-b-w2-d341af-5c58b5c667-bnws4\" (UID: \"f03e902b-a262-4d46-b393-dea566c264d2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d341af-5c58b5c667-bnws4" Nov 03 13:50:53.744102 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:53.744046 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f03e902b-a262-4d46-b393-dea566c264d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-d341af-5c58b5c667-bnws4\" (UID: \"f03e902b-a262-4d46-b393-dea566c264d2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d341af-5c58b5c667-bnws4" Nov 03 13:50:53.770549 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:53.770513 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d341af-7c4f477dcc-nvkxn"] Nov 03 13:50:53.776350 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:53.776323 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d341af-7c4f477dcc-nvkxn" Nov 03 13:50:53.800699 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:53.800665 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c2t2\" (UniqueName: \"kubernetes.io/projected/f03e902b-a262-4d46-b393-dea566c264d2-kube-api-access-9c2t2\") pod \"payload-pre-processing-e2e-shared-b-w2-d341af-5c58b5c667-bnws4\" (UID: \"f03e902b-a262-4d46-b393-dea566c264d2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d341af-5c58b5c667-bnws4" Nov 03 13:50:53.808980 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:53.808927 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d341af-7c4f477dcc-nvkxn"] Nov 03 13:50:53.844114 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:53.844084 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78dfe2a2-8938-4f14-a784-f07dbfcdd7f7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-d341af-7c4f477dcc-nvkxn\" (UID: \"78dfe2a2-8938-4f14-a784-f07dbfcdd7f7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d341af-7c4f477dcc-nvkxn" Nov 03 13:50:53.844288 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:53.844129 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stffb\" (UniqueName: \"kubernetes.io/projected/78dfe2a2-8938-4f14-a784-f07dbfcdd7f7-kube-api-access-stffb\") pod \"payload-processing-e2e-shared-b-w2-d341af-7c4f477dcc-nvkxn\" (UID: \"78dfe2a2-8938-4f14-a784-f07dbfcdd7f7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d341af-7c4f477dcc-nvkxn" Nov 03 13:50:53.915198 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:53.915163 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d341af-5c58b5c667-bnws4" Nov 03 13:50:53.945415 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:53.945371 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78dfe2a2-8938-4f14-a784-f07dbfcdd7f7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-d341af-7c4f477dcc-nvkxn\" (UID: \"78dfe2a2-8938-4f14-a784-f07dbfcdd7f7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d341af-7c4f477dcc-nvkxn" Nov 03 13:50:53.945596 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:53.945438 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stffb\" (UniqueName: \"kubernetes.io/projected/78dfe2a2-8938-4f14-a784-f07dbfcdd7f7-kube-api-access-stffb\") pod \"payload-processing-e2e-shared-b-w2-d341af-7c4f477dcc-nvkxn\" (UID: \"78dfe2a2-8938-4f14-a784-f07dbfcdd7f7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d341af-7c4f477dcc-nvkxn" Nov 03 13:50:53.946276 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:53.946253 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78dfe2a2-8938-4f14-a784-f07dbfcdd7f7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-d341af-7c4f477dcc-nvkxn\" (UID: \"78dfe2a2-8938-4f14-a784-f07dbfcdd7f7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d341af-7c4f477dcc-nvkxn" Nov 03 13:50:53.999646 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:53.999611 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stffb\" (UniqueName: \"kubernetes.io/projected/78dfe2a2-8938-4f14-a784-f07dbfcdd7f7-kube-api-access-stffb\") pod \"payload-processing-e2e-shared-b-w2-d341af-7c4f477dcc-nvkxn\" (UID: \"78dfe2a2-8938-4f14-a784-f07dbfcdd7f7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d341af-7c4f477dcc-nvkxn" Nov 03 13:50:54.117109 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:54.117072 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d341af-7c4f477dcc-nvkxn" Nov 03 13:50:54.292027 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:54.291995 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d341af-5c58b5c667-bnws4"] Nov 03 13:50:54.298193 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:50:54.298166 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf03e902b_a262_4d46_b393_dea566c264d2.slice/crio-32aee4de3dc45ecba210f688bda5e7689f20b62c3132c8ba50f27a203dec57c3 WatchSource:0}: Error finding container 32aee4de3dc45ecba210f688bda5e7689f20b62c3132c8ba50f27a203dec57c3: Status 404 returned error can't find the container with id 32aee4de3dc45ecba210f688bda5e7689f20b62c3132c8ba50f27a203dec57c3 Nov 03 13:50:54.394078 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:54.394048 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d341af-5c58b5c667-bnws4" event={"ID":"f03e902b-a262-4d46-b393-dea566c264d2","Type":"ContainerStarted","Data":"80392b315f206fec3a2f219e29a41a44ec383771b0281661a7276d8bd95b4fc4"} Nov 03 13:50:54.394219 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:54.394110 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d341af-5c58b5c667-bnws4" event={"ID":"f03e902b-a262-4d46-b393-dea566c264d2","Type":"ContainerStarted","Data":"32aee4de3dc45ecba210f688bda5e7689f20b62c3132c8ba50f27a203dec57c3"} Nov 03 13:50:54.394290 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:54.394250 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d341af-5c58b5c667-bnws4" Nov 03 13:50:54.493849 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:54.493765 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d341af-7c4f477dcc-nvkxn"] Nov 03 13:50:54.499810 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:50:54.499780 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78dfe2a2_8938_4f14_a784_f07dbfcdd7f7.slice/crio-b099f22e9a2ae5aecee38e251047a8cd09dbf5e4d92d625ea94e8fb083593b04 WatchSource:0}: Error finding container b099f22e9a2ae5aecee38e251047a8cd09dbf5e4d92d625ea94e8fb083593b04: Status 404 returned error can't find the container with id b099f22e9a2ae5aecee38e251047a8cd09dbf5e4d92d625ea94e8fb083593b04 Nov 03 13:50:54.503008 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:54.502959 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d341af-5c58b5c667-bnws4" podStartSLOduration=1.502943398 podStartE2EDuration="1.502943398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:50:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:50:54.501231553 +0000 UTC m=+2736.511919796" watchObservedRunningTime="2026-11-03 13:50:54.502943398 +0000 UTC m=+2736.513631641" Nov 03 13:50:55.403513 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:55.403449 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d341af-7c4f477dcc-nvkxn" event={"ID":"78dfe2a2-8938-4f14-a784-f07dbfcdd7f7","Type":"ContainerStarted","Data":"34a4e8ddb6cccb8cbe3cfcc341953f72d4417c64f1c860b77829001987fb5db5"} Nov 03 13:50:55.403942 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:55.403557 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d341af-7c4f477dcc-nvkxn" event={"ID":"78dfe2a2-8938-4f14-a784-f07dbfcdd7f7","Type":"ContainerStarted","Data":"b099f22e9a2ae5aecee38e251047a8cd09dbf5e4d92d625ea94e8fb083593b04"} Nov 03 13:50:55.403942 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:55.403596 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d341af-7c4f477dcc-nvkxn" Nov 03 13:50:55.512349 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:55.512291 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d341af-7c4f477dcc-nvkxn" podStartSLOduration=2.512277021 podStartE2EDuration="2.512277021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:50:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:50:55.511081519 +0000 UTC m=+2737.521769771" watchObservedRunningTime="2026-11-03 13:50:55.512277021 +0000 UTC m=+2737.522965263" Nov 03 13:50:56.368707 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:56.368676 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-60261a-77454f94c-ccbgm" Nov 03 13:50:56.381508 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:50:56.378642 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-60261a-c86bf8c5f-pxlmt" Nov 03 13:51:05.057022 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:05.056986 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74d9e037-96c4b"] Nov 03 13:51:05.103444 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:05.103409 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74d9e037-96c4b"] Nov 03 13:51:05.406336 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:05.406309 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d341af-5c58b5c667-bnws4" Nov 03 13:51:06.116298 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:06.116263 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c03eda48-e87b-4847-a9be-da542cba2654" path="/var/lib/kubelet/pods/c03eda48-e87b-4847-a9be-da542cba2654/volumes" Nov 03 13:51:06.431949 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:06.431917 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d341af-7c4f477dcc-nvkxn" Nov 03 13:51:07.235615 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:51:07.235580 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:51:19.044727 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:19.044688 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d797192e-2jjgm"] Nov 03 13:51:19.091440 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:19.091405 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d797192e-2jjgm"] Nov 03 13:51:20.116151 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:20.116115 5979 scope.go:122] "RemoveContainer" containerID="97cfb540abb72c1eb43176ff0db9e33f741735c57c56e0cd5931731cc7abfc7a" Nov 03 13:51:20.116599 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:20.116251 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a6c7cba-c26f-41cc-af4f-e9e519eb9865" path="/var/lib/kubelet/pods/8a6c7cba-c26f-41cc-af4f-e9e519eb9865/volumes" Nov 03 13:51:20.126188 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:20.126170 5979 scope.go:122] "RemoveContainer" containerID="1dd465655c278c52f3518ff064040817b4a706ec664c3200dac644078b84044f" Nov 03 13:51:22.419491 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:22.419450 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr"] Nov 03 13:51:22.419982 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:22.419665 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr" podUID="f9af6a95-df9b-4525-a080-edf5abab2c86" containerName="istio-proxy" containerID="cri-o://eb2ef15ce912894a5f5e93b8b0c28247b3c015d0f8d624eb9b8785f8b79d28db" gracePeriod=30 Nov 03 13:51:23.331204 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:23.331172 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d83c93-6f499df6bc-t88p4"] Nov 03 13:51:23.440411 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:23.440354 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d83c93-6f4df847c5-gjhcw"] Nov 03 13:51:27.701016 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:27.700964 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr" Nov 03 13:51:27.751014 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:27.750980 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9af6a95-df9b-4525-a080-edf5abab2c86-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9af6a95-df9b-4525-a080-edf5abab2c86-istio-podinfo\") pod \"f9af6a95-df9b-4525-a080-edf5abab2c86\" (UID: \"f9af6a95-df9b-4525-a080-edf5abab2c86\") " Nov 03 13:51:27.751185 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:27.751031 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9af6a95-df9b-4525-a080-edf5abab2c86-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9af6a95-df9b-4525-a080-edf5abab2c86-credential-socket\") pod \"f9af6a95-df9b-4525-a080-edf5abab2c86\" (UID: \"f9af6a95-df9b-4525-a080-edf5abab2c86\") " Nov 03 13:51:27.751185 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:27.751068 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9af6a95-df9b-4525-a080-edf5abab2c86-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9af6a95-df9b-4525-a080-edf5abab2c86-istiod-ca-cert\") pod \"f9af6a95-df9b-4525-a080-edf5abab2c86\" (UID: \"f9af6a95-df9b-4525-a080-edf5abab2c86\") " Nov 03 13:51:27.751324 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:27.751195 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9af6a95-df9b-4525-a080-edf5abab2c86-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9af6a95-df9b-4525-a080-edf5abab2c86-istio-data\") pod \"f9af6a95-df9b-4525-a080-edf5abab2c86\" (UID: \"f9af6a95-df9b-4525-a080-edf5abab2c86\") " Nov 03 13:51:27.751324 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:27.751249 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9af6a95-df9b-4525-a080-edf5abab2c86-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9af6a95-df9b-4525-a080-edf5abab2c86-workload-certs\") pod \"f9af6a95-df9b-4525-a080-edf5abab2c86\" (UID: \"f9af6a95-df9b-4525-a080-edf5abab2c86\") " Nov 03 13:51:27.751324 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:27.751308 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9af6a95-df9b-4525-a080-edf5abab2c86-kube-api-access-hqks7\" (UniqueName: \"kubernetes.io/projected/f9af6a95-df9b-4525-a080-edf5abab2c86-kube-api-access-hqks7\") pod \"f9af6a95-df9b-4525-a080-edf5abab2c86\" (UID: \"f9af6a95-df9b-4525-a080-edf5abab2c86\") " Nov 03 13:51:27.751532 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:27.751371 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9af6a95-df9b-4525-a080-edf5abab2c86-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9af6a95-df9b-4525-a080-edf5abab2c86-istio-envoy\") pod \"f9af6a95-df9b-4525-a080-edf5abab2c86\" (UID: \"f9af6a95-df9b-4525-a080-edf5abab2c86\") " Nov 03 13:51:27.751532 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:27.751403 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9af6a95-df9b-4525-a080-edf5abab2c86-credential-socket" pod "f9af6a95-df9b-4525-a080-edf5abab2c86" (UID: "f9af6a95-df9b-4525-a080-edf5abab2c86"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:51:27.751532 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:27.751453 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9af6a95-df9b-4525-a080-edf5abab2c86-istio-data" pod "f9af6a95-df9b-4525-a080-edf5abab2c86" (UID: "f9af6a95-df9b-4525-a080-edf5abab2c86"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:51:27.751532 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:27.751502 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9af6a95-df9b-4525-a080-edf5abab2c86-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9af6a95-df9b-4525-a080-edf5abab2c86-workload-socket\") pod \"f9af6a95-df9b-4525-a080-edf5abab2c86\" (UID: \"f9af6a95-df9b-4525-a080-edf5abab2c86\") " Nov 03 13:51:27.751532 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:27.751528 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9af6a95-df9b-4525-a080-edf5abab2c86-istio-token\" (UniqueName: \"kubernetes.io/projected/f9af6a95-df9b-4525-a080-edf5abab2c86-istio-token\") pod \"f9af6a95-df9b-4525-a080-edf5abab2c86\" (UID: \"f9af6a95-df9b-4525-a080-edf5abab2c86\") " Nov 03 13:51:27.751854 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:27.751568 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9af6a95-df9b-4525-a080-edf5abab2c86-istiod-ca-cert" pod "f9af6a95-df9b-4525-a080-edf5abab2c86" (UID: "f9af6a95-df9b-4525-a080-edf5abab2c86"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:51:27.751854 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:27.751761 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9af6a95-df9b-4525-a080-edf5abab2c86-workload-socket" pod "f9af6a95-df9b-4525-a080-edf5abab2c86" (UID: "f9af6a95-df9b-4525-a080-edf5abab2c86"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:51:27.752005 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:27.751927 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9af6a95-df9b-4525-a080-edf5abab2c86-workload-certs" pod "f9af6a95-df9b-4525-a080-edf5abab2c86" (UID: "f9af6a95-df9b-4525-a080-edf5abab2c86"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:51:27.752116 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:27.752091 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9af6a95-df9b-4525-a080-edf5abab2c86-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:51:27.752249 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:27.752118 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9af6a95-df9b-4525-a080-edf5abab2c86-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:51:27.752249 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:27.752134 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9af6a95-df9b-4525-a080-edf5abab2c86-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:51:27.752249 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:27.752151 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9af6a95-df9b-4525-a080-edf5abab2c86-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:51:27.752249 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:27.752166 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9af6a95-df9b-4525-a080-edf5abab2c86-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:51:27.753531 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:27.753499 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9af6a95-df9b-4525-a080-edf5abab2c86-istio-podinfo" pod "f9af6a95-df9b-4525-a080-edf5abab2c86" (UID: "f9af6a95-df9b-4525-a080-edf5abab2c86"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:51:27.753886 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:27.753870 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9af6a95-df9b-4525-a080-edf5abab2c86-istio-token" pod "f9af6a95-df9b-4525-a080-edf5abab2c86" (UID: "f9af6a95-df9b-4525-a080-edf5abab2c86"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:51:27.753996 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:27.753981 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9af6a95-df9b-4525-a080-edf5abab2c86-kube-api-access-hqks7" pod "f9af6a95-df9b-4525-a080-edf5abab2c86" (UID: "f9af6a95-df9b-4525-a080-edf5abab2c86"). InnerVolumeSpecName "kube-api-access-hqks7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:51:27.753996 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:27.753984 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9af6a95-df9b-4525-a080-edf5abab2c86-istio-envoy" pod "f9af6a95-df9b-4525-a080-edf5abab2c86" (UID: "f9af6a95-df9b-4525-a080-edf5abab2c86"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:51:27.853795 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:27.853767 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9af6a95-df9b-4525-a080-edf5abab2c86-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:51:27.853795 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:27.853792 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqks7\" (UniqueName: \"kubernetes.io/projected/f9af6a95-df9b-4525-a080-edf5abab2c86-kube-api-access-hqks7\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:51:27.853991 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:27.853802 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9af6a95-df9b-4525-a080-edf5abab2c86-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:51:27.853991 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:27.853812 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9af6a95-df9b-4525-a080-edf5abab2c86-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:51:27.888355 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:27.888308 5979 generic.go:356] "Generic (PLEG): container finished" podID="f9af6a95-df9b-4525-a080-edf5abab2c86" containerID="eb2ef15ce912894a5f5e93b8b0c28247b3c015d0f8d624eb9b8785f8b79d28db" exitCode=0 Nov 03 13:51:27.888573 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:27.888366 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr" Nov 03 13:51:27.888573 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:27.888400 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr" event={"ID":"f9af6a95-df9b-4525-a080-edf5abab2c86","Type":"ContainerDied","Data":"eb2ef15ce912894a5f5e93b8b0c28247b3c015d0f8d624eb9b8785f8b79d28db"} Nov 03 13:51:27.888573 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:27.888504 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr" event={"ID":"f9af6a95-df9b-4525-a080-edf5abab2c86","Type":"ContainerDied","Data":"f48685f66853591fc268801f156628057fe5e40fa2167c0de75259ec4556a64b"} Nov 03 13:51:27.888573 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:27.888511 5979 scope.go:122] "RemoveContainer" containerID="eb2ef15ce912894a5f5e93b8b0c28247b3c015d0f8d624eb9b8785f8b79d28db" Nov 03 13:51:27.907921 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:27.907903 5979 scope.go:122] "RemoveContainer" containerID="ff8c0f43d62843b89727cd62c24862fa2a5c58550afe7ed0db5b1f36d43e6d75" Nov 03 13:51:27.917790 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:27.917769 5979 scope.go:122] "RemoveContainer" containerID="eb2ef15ce912894a5f5e93b8b0c28247b3c015d0f8d624eb9b8785f8b79d28db" Nov 03 13:51:27.918067 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:51:27.918047 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb2ef15ce912894a5f5e93b8b0c28247b3c015d0f8d624eb9b8785f8b79d28db\": container with ID starting with eb2ef15ce912894a5f5e93b8b0c28247b3c015d0f8d624eb9b8785f8b79d28db not found: ID does not exist" containerID="eb2ef15ce912894a5f5e93b8b0c28247b3c015d0f8d624eb9b8785f8b79d28db" Nov 03 13:51:27.918111 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:27.918075 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb2ef15ce912894a5f5e93b8b0c28247b3c015d0f8d624eb9b8785f8b79d28db"} err="failed to get container status \"eb2ef15ce912894a5f5e93b8b0c28247b3c015d0f8d624eb9b8785f8b79d28db\": rpc error: code = NotFound desc = could not find container \"eb2ef15ce912894a5f5e93b8b0c28247b3c015d0f8d624eb9b8785f8b79d28db\": container with ID starting with eb2ef15ce912894a5f5e93b8b0c28247b3c015d0f8d624eb9b8785f8b79d28db not found: ID does not exist" Nov 03 13:51:27.918111 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:27.918091 5979 scope.go:122] "RemoveContainer" containerID="ff8c0f43d62843b89727cd62c24862fa2a5c58550afe7ed0db5b1f36d43e6d75" Nov 03 13:51:27.918324 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:51:27.918309 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff8c0f43d62843b89727cd62c24862fa2a5c58550afe7ed0db5b1f36d43e6d75\": container with ID starting with ff8c0f43d62843b89727cd62c24862fa2a5c58550afe7ed0db5b1f36d43e6d75 not found: ID does not exist" containerID="ff8c0f43d62843b89727cd62c24862fa2a5c58550afe7ed0db5b1f36d43e6d75" Nov 03 13:51:27.918369 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:27.918330 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff8c0f43d62843b89727cd62c24862fa2a5c58550afe7ed0db5b1f36d43e6d75"} err="failed to get container status \"ff8c0f43d62843b89727cd62c24862fa2a5c58550afe7ed0db5b1f36d43e6d75\": rpc error: code = NotFound desc = could not find container \"ff8c0f43d62843b89727cd62c24862fa2a5c58550afe7ed0db5b1f36d43e6d75\": container with ID starting with ff8c0f43d62843b89727cd62c24862fa2a5c58550afe7ed0db5b1f36d43e6d75 not found: ID does not exist" Nov 03 13:51:28.043755 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:28.043679 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr"] Nov 03 13:51:28.051985 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:28.051959 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-10c688-openshift-default-6c5df4bfcc-dtpwr"] Nov 03 13:51:28.116338 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:28.116308 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9af6a95-df9b-4525-a080-edf5abab2c86" path="/var/lib/kubelet/pods/f9af6a95-df9b-4525-a080-edf5abab2c86/volumes" Nov 03 13:51:28.331590 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:28.331434 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-d83c93-6f499df6bc-t88p4" podUID="52195f70-2bc0-42c8-b5b6-b215f849e526" containerName="payload-processing" containerID="cri-o://3a93e09c722df39f3e1ec5564492a84c9d154df97801012466143ff77b86e4b1" gracePeriod=25 Nov 03 13:51:28.444016 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:28.443924 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d83c93-6f4df847c5-gjhcw" podUID="e79b25c7-5f37-4147-8349-c557fdf621e3" containerName="payload-pre-processing" containerID="cri-o://9688dfcf0cccc16acfc9eec53c4960addbb2dad965be12dfdd158a2839833da8" gracePeriod=25 Nov 03 13:51:28.611293 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:28.611261 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d83c93-6f499df6bc-t88p4" Nov 03 13:51:28.663786 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:28.663747 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52195f70-2bc0-42c8-b5b6-b215f849e526-kube-api-access-rfjdf\" (UniqueName: \"kubernetes.io/projected/52195f70-2bc0-42c8-b5b6-b215f849e526-kube-api-access-rfjdf\") pod \"52195f70-2bc0-42c8-b5b6-b215f849e526\" (UID: \"52195f70-2bc0-42c8-b5b6-b215f849e526\") " Nov 03 13:51:28.664092 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:28.664071 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52195f70-2bc0-42c8-b5b6-b215f849e526-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52195f70-2bc0-42c8-b5b6-b215f849e526-plugins-config-volume\") pod \"52195f70-2bc0-42c8-b5b6-b215f849e526\" (UID: \"52195f70-2bc0-42c8-b5b6-b215f849e526\") " Nov 03 13:51:28.664526 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:28.664498 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52195f70-2bc0-42c8-b5b6-b215f849e526-plugins-config-volume" pod "52195f70-2bc0-42c8-b5b6-b215f849e526" (UID: "52195f70-2bc0-42c8-b5b6-b215f849e526"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:51:28.664650 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:28.664594 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52195f70-2bc0-42c8-b5b6-b215f849e526-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:51:28.667080 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:28.667053 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52195f70-2bc0-42c8-b5b6-b215f849e526-kube-api-access-rfjdf" pod "52195f70-2bc0-42c8-b5b6-b215f849e526" (UID: "52195f70-2bc0-42c8-b5b6-b215f849e526"). InnerVolumeSpecName "kube-api-access-rfjdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:51:28.701150 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:28.701124 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d83c93-6f4df847c5-gjhcw" Nov 03 13:51:28.765815 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:28.765784 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e79b25c7-5f37-4147-8349-c557fdf621e3-kube-api-access-5mpm9\" (UniqueName: \"kubernetes.io/projected/e79b25c7-5f37-4147-8349-c557fdf621e3-kube-api-access-5mpm9\") pod \"e79b25c7-5f37-4147-8349-c557fdf621e3\" (UID: \"e79b25c7-5f37-4147-8349-c557fdf621e3\") " Nov 03 13:51:28.766023 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:28.765914 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e79b25c7-5f37-4147-8349-c557fdf621e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e79b25c7-5f37-4147-8349-c557fdf621e3-plugins-config-volume\") pod \"e79b25c7-5f37-4147-8349-c557fdf621e3\" (UID: \"e79b25c7-5f37-4147-8349-c557fdf621e3\") " Nov 03 13:51:28.766296 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:28.766273 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e79b25c7-5f37-4147-8349-c557fdf621e3-plugins-config-volume" pod "e79b25c7-5f37-4147-8349-c557fdf621e3" (UID: "e79b25c7-5f37-4147-8349-c557fdf621e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:51:28.766419 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:28.766401 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfjdf\" (UniqueName: \"kubernetes.io/projected/52195f70-2bc0-42c8-b5b6-b215f849e526-kube-api-access-rfjdf\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:51:28.766492 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:28.766424 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e79b25c7-5f37-4147-8349-c557fdf621e3-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:51:28.767811 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:28.767791 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e79b25c7-5f37-4147-8349-c557fdf621e3-kube-api-access-5mpm9" pod "e79b25c7-5f37-4147-8349-c557fdf621e3" (UID: "e79b25c7-5f37-4147-8349-c557fdf621e3"). InnerVolumeSpecName "kube-api-access-5mpm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:51:28.868000 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:28.867901 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mpm9\" (UniqueName: \"kubernetes.io/projected/e79b25c7-5f37-4147-8349-c557fdf621e3-kube-api-access-5mpm9\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:51:28.897726 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:28.897688 5979 generic.go:356] "Generic (PLEG): container finished" podID="e79b25c7-5f37-4147-8349-c557fdf621e3" containerID="9688dfcf0cccc16acfc9eec53c4960addbb2dad965be12dfdd158a2839833da8" exitCode=0 Nov 03 13:51:28.897904 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:28.897755 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d83c93-6f4df847c5-gjhcw" Nov 03 13:51:28.897904 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:28.897775 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d83c93-6f4df847c5-gjhcw" event={"ID":"e79b25c7-5f37-4147-8349-c557fdf621e3","Type":"ContainerDied","Data":"9688dfcf0cccc16acfc9eec53c4960addbb2dad965be12dfdd158a2839833da8"} Nov 03 13:51:28.897904 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:28.897866 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d83c93-6f4df847c5-gjhcw" event={"ID":"e79b25c7-5f37-4147-8349-c557fdf621e3","Type":"ContainerDied","Data":"df6defe4e650c2b4955f54e914ce17f03e2960c61c7dd5ed2dd60333227f2beb"} Nov 03 13:51:28.897904 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:28.897896 5979 scope.go:122] "RemoveContainer" containerID="9688dfcf0cccc16acfc9eec53c4960addbb2dad965be12dfdd158a2839833da8" Nov 03 13:51:28.899138 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:28.899118 5979 generic.go:356] "Generic (PLEG): container finished" podID="52195f70-2bc0-42c8-b5b6-b215f849e526" containerID="3a93e09c722df39f3e1ec5564492a84c9d154df97801012466143ff77b86e4b1" exitCode=0 Nov 03 13:51:28.899252 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:28.899155 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d83c93-6f499df6bc-t88p4" event={"ID":"52195f70-2bc0-42c8-b5b6-b215f849e526","Type":"ContainerDied","Data":"3a93e09c722df39f3e1ec5564492a84c9d154df97801012466143ff77b86e4b1"} Nov 03 13:51:28.899252 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:28.899184 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d83c93-6f499df6bc-t88p4" Nov 03 13:51:28.899252 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:28.899210 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d83c93-6f499df6bc-t88p4" event={"ID":"52195f70-2bc0-42c8-b5b6-b215f849e526","Type":"ContainerDied","Data":"767ea39d883dc46142f26ef5622daba9e460314bcca16743c7d80460067fbcde"} Nov 03 13:51:28.909605 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:28.909586 5979 scope.go:122] "RemoveContainer" containerID="9688dfcf0cccc16acfc9eec53c4960addbb2dad965be12dfdd158a2839833da8" Nov 03 13:51:28.909906 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:51:28.909875 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9688dfcf0cccc16acfc9eec53c4960addbb2dad965be12dfdd158a2839833da8\": container with ID starting with 9688dfcf0cccc16acfc9eec53c4960addbb2dad965be12dfdd158a2839833da8 not found: ID does not exist" containerID="9688dfcf0cccc16acfc9eec53c4960addbb2dad965be12dfdd158a2839833da8" Nov 03 13:51:28.910027 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:28.909916 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9688dfcf0cccc16acfc9eec53c4960addbb2dad965be12dfdd158a2839833da8"} err="failed to get container status \"9688dfcf0cccc16acfc9eec53c4960addbb2dad965be12dfdd158a2839833da8\": rpc error: code = NotFound desc = could not find container \"9688dfcf0cccc16acfc9eec53c4960addbb2dad965be12dfdd158a2839833da8\": container with ID starting with 9688dfcf0cccc16acfc9eec53c4960addbb2dad965be12dfdd158a2839833da8 not found: ID does not exist" Nov 03 13:51:28.910027 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:28.909940 5979 scope.go:122] "RemoveContainer" containerID="3a93e09c722df39f3e1ec5564492a84c9d154df97801012466143ff77b86e4b1" Nov 03 13:51:28.919229 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:28.919204 5979 scope.go:122] "RemoveContainer" containerID="3a93e09c722df39f3e1ec5564492a84c9d154df97801012466143ff77b86e4b1" Nov 03 13:51:28.919500 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:51:28.919480 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a93e09c722df39f3e1ec5564492a84c9d154df97801012466143ff77b86e4b1\": container with ID starting with 3a93e09c722df39f3e1ec5564492a84c9d154df97801012466143ff77b86e4b1 not found: ID does not exist" containerID="3a93e09c722df39f3e1ec5564492a84c9d154df97801012466143ff77b86e4b1" Nov 03 13:51:28.919569 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:28.919506 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a93e09c722df39f3e1ec5564492a84c9d154df97801012466143ff77b86e4b1"} err="failed to get container status \"3a93e09c722df39f3e1ec5564492a84c9d154df97801012466143ff77b86e4b1\": rpc error: code = NotFound desc = could not find container \"3a93e09c722df39f3e1ec5564492a84c9d154df97801012466143ff77b86e4b1\": container with ID starting with 3a93e09c722df39f3e1ec5564492a84c9d154df97801012466143ff77b86e4b1 not found: ID does not exist" Nov 03 13:51:29.053003 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:29.052965 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d83c93-6f499df6bc-t88p4"] Nov 03 13:51:29.057583 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:29.057554 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d83c93-6f499df6bc-t88p4"] Nov 03 13:51:29.160763 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:29.160728 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d83c93-6f4df847c5-gjhcw"] Nov 03 13:51:29.165107 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:29.165079 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d83c93-6f4df847c5-gjhcw"] Nov 03 13:51:29.665421 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:29.665375 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d83c93-6f4df847c5-gjhcw" podUID="e79b25c7-5f37-4147-8349-c557fdf621e3" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.248:50052\" within 1s: context deadline exceeded" Nov 03 13:51:29.665626 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:51:29.665449 5979 logging.go:55] [core] [Channel #5445 SubChannel #5446]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.248:50052", ServerName: "10.134.0.248:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.248:50052: operation was canceled" Nov 03 13:51:30.115775 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:30.115743 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52195f70-2bc0-42c8-b5b6-b215f849e526" path="/var/lib/kubelet/pods/52195f70-2bc0-42c8-b5b6-b215f849e526/volumes" Nov 03 13:51:30.116164 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:30.116150 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e79b25c7-5f37-4147-8349-c557fdf621e3" path="/var/lib/kubelet/pods/e79b25c7-5f37-4147-8349-c557fdf621e3/volumes" Nov 03 13:51:31.501288 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:31.501256 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:51:34.154648 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:34.154614 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfc2cc3a-nbl96"] Nov 03 13:51:34.155222 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:34.155205 5979 cpu_manager.go:414] "containerMap: removing container" podUID="e79b25c7-5f37-4147-8349-c557fdf621e3" containerName="payload-pre-processing" Nov 03 13:51:34.155222 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:34.155222 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e79b25c7-5f37-4147-8349-c557fdf621e3" containerName="payload-pre-processing" Nov 03 13:51:34.155379 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:34.155265 5979 cpu_manager.go:414] "containerMap: removing container" podUID="f9af6a95-df9b-4525-a080-edf5abab2c86" containerName="istio-proxy" Nov 03 13:51:34.155379 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:34.155271 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9af6a95-df9b-4525-a080-edf5abab2c86" containerName="istio-proxy" Nov 03 13:51:34.155379 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:34.155286 5979 cpu_manager.go:414] "containerMap: removing container" podUID="f9af6a95-df9b-4525-a080-edf5abab2c86" containerName="istio-proxy" Nov 03 13:51:34.155379 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:34.155294 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9af6a95-df9b-4525-a080-edf5abab2c86" containerName="istio-proxy" Nov 03 13:51:34.155379 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:34.155312 5979 cpu_manager.go:414] "containerMap: removing container" podUID="52195f70-2bc0-42c8-b5b6-b215f849e526" containerName="payload-processing" Nov 03 13:51:34.155379 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:34.155317 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52195f70-2bc0-42c8-b5b6-b215f849e526" containerName="payload-processing" Nov 03 13:51:34.155687 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:34.155414 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="52195f70-2bc0-42c8-b5b6-b215f849e526" containerName="payload-processing" Nov 03 13:51:34.155687 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:34.155425 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="e79b25c7-5f37-4147-8349-c557fdf621e3" containerName="payload-pre-processing" Nov 03 13:51:34.155687 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:34.155442 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9af6a95-df9b-4525-a080-edf5abab2c86" containerName="istio-proxy" Nov 03 13:51:34.155687 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:34.155452 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9af6a95-df9b-4525-a080-edf5abab2c86" containerName="istio-proxy" Nov 03 13:51:34.160427 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:34.160405 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc2cc3a-nbl96" Nov 03 13:51:34.212894 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:34.212845 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfc2cc3a-nbl96"] Nov 03 13:51:34.237033 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:34.236997 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f285fcb6-7753-4725-9381-a484d084937a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfc2cc3a-nbl96\" (UID: \"f285fcb6-7753-4725-9381-a484d084937a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc2cc3a-nbl96" Nov 03 13:51:34.237494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:34.237440 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f285fcb6-7753-4725-9381-a484d084937a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfc2cc3a-nbl96\" (UID: \"f285fcb6-7753-4725-9381-a484d084937a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc2cc3a-nbl96" Nov 03 13:51:34.237612 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:34.237552 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9n9x\" (UniqueName: \"kubernetes.io/projected/f285fcb6-7753-4725-9381-a484d084937a-kube-api-access-m9n9x\") pod \"maas-api-revoke-sub-bfc2cc3a-nbl96\" (UID: \"f285fcb6-7753-4725-9381-a484d084937a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc2cc3a-nbl96" Nov 03 13:51:34.255161 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:34.255134 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:51:34.255285 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:34.255133 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:51:34.339288 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:34.339256 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9n9x\" (UniqueName: \"kubernetes.io/projected/f285fcb6-7753-4725-9381-a484d084937a-kube-api-access-m9n9x\") pod \"maas-api-revoke-sub-bfc2cc3a-nbl96\" (UID: \"f285fcb6-7753-4725-9381-a484d084937a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc2cc3a-nbl96" Nov 03 13:51:34.339522 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:34.339322 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f285fcb6-7753-4725-9381-a484d084937a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfc2cc3a-nbl96\" (UID: \"f285fcb6-7753-4725-9381-a484d084937a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc2cc3a-nbl96" Nov 03 13:51:34.339522 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:34.339462 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f285fcb6-7753-4725-9381-a484d084937a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfc2cc3a-nbl96\" (UID: \"f285fcb6-7753-4725-9381-a484d084937a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc2cc3a-nbl96" Nov 03 13:51:34.340052 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:34.340025 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f285fcb6-7753-4725-9381-a484d084937a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfc2cc3a-nbl96\" (UID: \"f285fcb6-7753-4725-9381-a484d084937a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc2cc3a-nbl96" Nov 03 13:51:34.341864 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:34.341837 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f285fcb6-7753-4725-9381-a484d084937a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfc2cc3a-nbl96\" (UID: \"f285fcb6-7753-4725-9381-a484d084937a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc2cc3a-nbl96" Nov 03 13:51:34.392797 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:34.392761 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9n9x\" (UniqueName: \"kubernetes.io/projected/f285fcb6-7753-4725-9381-a484d084937a-kube-api-access-m9n9x\") pod \"maas-api-revoke-sub-bfc2cc3a-nbl96\" (UID: \"f285fcb6-7753-4725-9381-a484d084937a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc2cc3a-nbl96" Nov 03 13:51:34.474168 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:34.474083 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc2cc3a-nbl96" Nov 03 13:51:34.858699 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:34.858618 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfc2cc3a-nbl96"] Nov 03 13:51:34.864728 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:51:34.864699 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf285fcb6_7753_4725_9381_a484d084937a.slice/crio-40d80086133f47366ef54f728d4afdcff0fdcf954ce8829d4561315b47131a76 WatchSource:0}: Error finding container 40d80086133f47366ef54f728d4afdcff0fdcf954ce8829d4561315b47131a76: Status 404 returned error can't find the container with id 40d80086133f47366ef54f728d4afdcff0fdcf954ce8829d4561315b47131a76 Nov 03 13:51:34.943503 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:34.943454 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc2cc3a-nbl96" event={"ID":"f285fcb6-7753-4725-9381-a484d084937a","Type":"ContainerStarted","Data":"40d80086133f47366ef54f728d4afdcff0fdcf954ce8829d4561315b47131a76"} Nov 03 13:51:35.953008 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:35.952975 5979 generic.go:356] "Generic (PLEG): container finished" podID="f285fcb6-7753-4725-9381-a484d084937a" containerID="ebded7ee370c2bb01b149c5cafc1a67f71dd4bf1b6bc07585d5e168c636f0660" exitCode=0 Nov 03 13:51:35.953500 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:35.953035 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc2cc3a-nbl96" event={"ID":"f285fcb6-7753-4725-9381-a484d084937a","Type":"ContainerDied","Data":"ebded7ee370c2bb01b149c5cafc1a67f71dd4bf1b6bc07585d5e168c636f0660"} Nov 03 13:51:36.810368 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:36.810327 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-60261a-76f8445748-w4cwv"] Nov 03 13:51:36.814601 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:36.814577 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-60261a-76f8445748-w4cwv" Nov 03 13:51:36.863433 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:36.863396 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-60261a-serving-cert\"" Nov 03 13:51:36.914912 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:36.914877 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-60261a-76f8445748-w4cwv"] Nov 03 13:51:36.982738 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:36.982699 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50674938-d53d-430b-a564-fa0bd1abed74-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-60261a-76f8445748-w4cwv\" (UID: \"50674938-d53d-430b-a564-fa0bd1abed74\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-60261a-76f8445748-w4cwv" Nov 03 13:51:36.983175 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:36.982761 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50674938-d53d-430b-a564-fa0bd1abed74-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-60261a-76f8445748-w4cwv\" (UID: \"50674938-d53d-430b-a564-fa0bd1abed74\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-60261a-76f8445748-w4cwv" Nov 03 13:51:36.983175 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:36.982923 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g6gg\" (UniqueName: \"kubernetes.io/projected/50674938-d53d-430b-a564-fa0bd1abed74-kube-api-access-7g6gg\") pod \"maas-api-e2e-shared-a-w1-60261a-76f8445748-w4cwv\" (UID: \"50674938-d53d-430b-a564-fa0bd1abed74\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-60261a-76f8445748-w4cwv" Nov 03 13:51:36.983175 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:36.983063 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50674938-d53d-430b-a564-fa0bd1abed74-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-60261a-76f8445748-w4cwv\" (UID: \"50674938-d53d-430b-a564-fa0bd1abed74\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-60261a-76f8445748-w4cwv" Nov 03 13:51:37.049609 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:37.049536 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03057985-ccfbz"] Nov 03 13:51:37.053648 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:37.053610 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03057985-ccfbz"] Nov 03 13:51:37.084425 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:37.084327 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50674938-d53d-430b-a564-fa0bd1abed74-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-60261a-76f8445748-w4cwv\" (UID: \"50674938-d53d-430b-a564-fa0bd1abed74\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-60261a-76f8445748-w4cwv" Nov 03 13:51:37.084425 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:37.084385 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50674938-d53d-430b-a564-fa0bd1abed74-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-60261a-76f8445748-w4cwv\" (UID: \"50674938-d53d-430b-a564-fa0bd1abed74\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-60261a-76f8445748-w4cwv" Nov 03 13:51:37.084717 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:37.084433 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g6gg\" (UniqueName: \"kubernetes.io/projected/50674938-d53d-430b-a564-fa0bd1abed74-kube-api-access-7g6gg\") pod \"maas-api-e2e-shared-a-w1-60261a-76f8445748-w4cwv\" (UID: \"50674938-d53d-430b-a564-fa0bd1abed74\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-60261a-76f8445748-w4cwv" Nov 03 13:51:37.084717 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:37.084536 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50674938-d53d-430b-a564-fa0bd1abed74-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-60261a-76f8445748-w4cwv\" (UID: \"50674938-d53d-430b-a564-fa0bd1abed74\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-60261a-76f8445748-w4cwv" Nov 03 13:51:37.084717 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:51:37.084615 5979 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-60261a-serving-cert: secret "maas-api-e2e-shared-a-w1-60261a-serving-cert" not found Nov 03 13:51:37.084717 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:51:37.084692 5979 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50674938-d53d-430b-a564-fa0bd1abed74-maas-api-tls podName:50674938-d53d-430b-a564-fa0bd1abed74 nodeName:}" failed. No retries permitted until 2026-11-03 13:51:37.584670493 +0000 UTC m=+2779.595358717 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/50674938-d53d-430b-a564-fa0bd1abed74-maas-api-tls") pod "maas-api-e2e-shared-a-w1-60261a-76f8445748-w4cwv" (UID: "50674938-d53d-430b-a564-fa0bd1abed74") : secret "maas-api-e2e-shared-a-w1-60261a-serving-cert" not found Nov 03 13:51:37.085493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:37.085171 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50674938-d53d-430b-a564-fa0bd1abed74-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-60261a-76f8445748-w4cwv\" (UID: \"50674938-d53d-430b-a564-fa0bd1abed74\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-60261a-76f8445748-w4cwv" Nov 03 13:51:37.087079 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:37.087058 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50674938-d53d-430b-a564-fa0bd1abed74-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-60261a-76f8445748-w4cwv\" (UID: \"50674938-d53d-430b-a564-fa0bd1abed74\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-60261a-76f8445748-w4cwv" Nov 03 13:51:37.162509 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:37.162336 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g6gg\" (UniqueName: \"kubernetes.io/projected/50674938-d53d-430b-a564-fa0bd1abed74-kube-api-access-7g6gg\") pod \"maas-api-e2e-shared-a-w1-60261a-76f8445748-w4cwv\" (UID: \"50674938-d53d-430b-a564-fa0bd1abed74\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-60261a-76f8445748-w4cwv" Nov 03 13:51:37.190457 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:37.190429 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc2cc3a-nbl96" Nov 03 13:51:37.234484 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:51:37.234437 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:51:37.391988 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:37.391938 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f285fcb6-7753-4725-9381-a484d084937a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f285fcb6-7753-4725-9381-a484d084937a-maas-api-xks-ca\") pod \"f285fcb6-7753-4725-9381-a484d084937a\" (UID: \"f285fcb6-7753-4725-9381-a484d084937a\") " Nov 03 13:51:37.392252 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:37.392234 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f285fcb6-7753-4725-9381-a484d084937a-kube-api-access-m9n9x\" (UniqueName: \"kubernetes.io/projected/f285fcb6-7753-4725-9381-a484d084937a-kube-api-access-m9n9x\") pod \"f285fcb6-7753-4725-9381-a484d084937a\" (UID: \"f285fcb6-7753-4725-9381-a484d084937a\") " Nov 03 13:51:37.392554 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:37.392533 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f285fcb6-7753-4725-9381-a484d084937a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f285fcb6-7753-4725-9381-a484d084937a-maas-api-openshift-ca\") pod \"f285fcb6-7753-4725-9381-a484d084937a\" (UID: \"f285fcb6-7753-4725-9381-a484d084937a\") " Nov 03 13:51:37.393769 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:37.393745 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f285fcb6-7753-4725-9381-a484d084937a-maas-api-openshift-ca" pod "f285fcb6-7753-4725-9381-a484d084937a" (UID: "f285fcb6-7753-4725-9381-a484d084937a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:51:37.401790 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:37.401761 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f285fcb6-7753-4725-9381-a484d084937a-maas-api-xks-ca" pod "f285fcb6-7753-4725-9381-a484d084937a" (UID: "f285fcb6-7753-4725-9381-a484d084937a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:51:37.402246 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:37.402219 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f285fcb6-7753-4725-9381-a484d084937a-kube-api-access-m9n9x" pod "f285fcb6-7753-4725-9381-a484d084937a" (UID: "f285fcb6-7753-4725-9381-a484d084937a"). InnerVolumeSpecName "kube-api-access-m9n9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:51:37.495106 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:37.495070 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f285fcb6-7753-4725-9381-a484d084937a-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:51:37.495106 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:37.495101 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f285fcb6-7753-4725-9381-a484d084937a-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:51:37.495106 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:37.495112 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9n9x\" (UniqueName: \"kubernetes.io/projected/f285fcb6-7753-4725-9381-a484d084937a-kube-api-access-m9n9x\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:51:37.596585 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:37.596540 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50674938-d53d-430b-a564-fa0bd1abed74-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-60261a-76f8445748-w4cwv\" (UID: \"50674938-d53d-430b-a564-fa0bd1abed74\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-60261a-76f8445748-w4cwv" Nov 03 13:51:37.599531 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:37.599435 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50674938-d53d-430b-a564-fa0bd1abed74-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-60261a-76f8445748-w4cwv\" (UID: \"50674938-d53d-430b-a564-fa0bd1abed74\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-60261a-76f8445748-w4cwv" Nov 03 13:51:37.728790 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:37.728697 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-60261a-76f8445748-w4cwv" Nov 03 13:51:37.968172 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:37.968131 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc2cc3a-nbl96" event={"ID":"f285fcb6-7753-4725-9381-a484d084937a","Type":"ContainerDied","Data":"40d80086133f47366ef54f728d4afdcff0fdcf954ce8829d4561315b47131a76"} Nov 03 13:51:37.968172 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:37.968176 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfc2cc3a-nbl96" Nov 03 13:51:37.968387 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:37.968221 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40d80086133f47366ef54f728d4afdcff0fdcf954ce8829d4561315b47131a76" Nov 03 13:51:38.103053 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:38.102960 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-60261a-76f8445748-w4cwv"] Nov 03 13:51:38.111051 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:51:38.111018 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50674938_d53d_430b_a564_fa0bd1abed74.slice/crio-e3cc3696798825dd25cff30a69caf2a5f370b6c431288da754a9e803c0082fc3 WatchSource:0}: Error finding container e3cc3696798825dd25cff30a69caf2a5f370b6c431288da754a9e803c0082fc3: Status 404 returned error can't find the container with id e3cc3696798825dd25cff30a69caf2a5f370b6c431288da754a9e803c0082fc3 Nov 03 13:51:38.119162 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:38.119132 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00e7d6b5-17dc-40e1-b6e0-ea2cdb359c79" path="/var/lib/kubelet/pods/00e7d6b5-17dc-40e1-b6e0-ea2cdb359c79/volumes" Nov 03 13:51:38.976535 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:38.976500 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-60261a-76f8445748-w4cwv" event={"ID":"50674938-d53d-430b-a564-fa0bd1abed74","Type":"ContainerStarted","Data":"e3cc3696798825dd25cff30a69caf2a5f370b6c431288da754a9e803c0082fc3"} Nov 03 13:51:39.986363 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:39.986318 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-60261a-76f8445748-w4cwv" event={"ID":"50674938-d53d-430b-a564-fa0bd1abed74","Type":"ContainerStarted","Data":"f337d4b4f7c390ef4d07325d4a4b22158b4fe6d1b2f359c6acf00b99d81b6b6a"} Nov 03 13:51:39.986833 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:39.986416 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-60261a-76f8445748-w4cwv" Nov 03 13:51:40.099371 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:40.099309 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-60261a-76f8445748-w4cwv" podStartSLOduration=3.188421373 podStartE2EDuration="4.099291148s" totalImagesPullingTime="910.869775ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:51:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:51:40.096092336 +0000 UTC m=+2782.106780579" watchObservedRunningTime="2026-11-03 13:51:40.099291148 +0000 UTC m=+2782.109979389" Nov 03 13:51:45.998335 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:45.998305 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-60261a-76f8445748-w4cwv" Nov 03 13:51:54.345757 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:54.345713 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w"] Nov 03 13:51:54.346228 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:54.345973 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w" podUID="2fe2e1d4-9fac-456e-ba2b-e221b6c59e98" containerName="istio-proxy" containerID="cri-o://1a57ea83cab3b1e78bf6133cd4eaf65965e89db715e28489cfdcb325932997c7" gracePeriod=30 Nov 03 13:51:55.043924 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:55.043885 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e44b9eec-dwxj2"] Nov 03 13:51:55.090277 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:55.090234 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e44b9eec-dwxj2"] Nov 03 13:51:56.165514 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:56.162709 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39663db1-4bd2-43b0-9537-6a9cdd4223c1" path="/var/lib/kubelet/pods/39663db1-4bd2-43b0-9537-6a9cdd4223c1/volumes" Nov 03 13:51:58.053943 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:58.053903 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57f8a8e4-gnp6p"] Nov 03 13:51:58.091309 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:58.091273 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57f8a8e4-gnp6p"] Nov 03 13:51:58.124517 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:58.117828 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18ec454b-720d-499d-9884-4b47962d35b7" path="/var/lib/kubelet/pods/18ec454b-720d-499d-9884-4b47962d35b7/volumes" Nov 03 13:51:59.717197 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:59.717163 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w" Nov 03 13:51:59.828748 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:59.828700 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-istio-envoy\") pod \"2fe2e1d4-9fac-456e-ba2b-e221b6c59e98\" (UID: \"2fe2e1d4-9fac-456e-ba2b-e221b6c59e98\") " Nov 03 13:51:59.828973 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:59.828796 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-istiod-ca-cert\") pod \"2fe2e1d4-9fac-456e-ba2b-e221b6c59e98\" (UID: \"2fe2e1d4-9fac-456e-ba2b-e221b6c59e98\") " Nov 03 13:51:59.828973 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:59.828839 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-istio-token\" (UniqueName: \"kubernetes.io/projected/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-istio-token\") pod \"2fe2e1d4-9fac-456e-ba2b-e221b6c59e98\" (UID: \"2fe2e1d4-9fac-456e-ba2b-e221b6c59e98\") " Nov 03 13:51:59.828973 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:59.828863 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-kube-api-access-4ftqk\" (UniqueName: \"kubernetes.io/projected/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-kube-api-access-4ftqk\") pod \"2fe2e1d4-9fac-456e-ba2b-e221b6c59e98\" (UID: \"2fe2e1d4-9fac-456e-ba2b-e221b6c59e98\") " Nov 03 13:51:59.828973 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:59.828911 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-workload-socket\") pod \"2fe2e1d4-9fac-456e-ba2b-e221b6c59e98\" (UID: \"2fe2e1d4-9fac-456e-ba2b-e221b6c59e98\") " Nov 03 13:51:59.828973 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:59.828946 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-istio-podinfo\") pod \"2fe2e1d4-9fac-456e-ba2b-e221b6c59e98\" (UID: \"2fe2e1d4-9fac-456e-ba2b-e221b6c59e98\") " Nov 03 13:51:59.828973 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:59.828975 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-credential-socket\") pod \"2fe2e1d4-9fac-456e-ba2b-e221b6c59e98\" (UID: \"2fe2e1d4-9fac-456e-ba2b-e221b6c59e98\") " Nov 03 13:51:59.829324 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:59.829023 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-istio-data\") pod \"2fe2e1d4-9fac-456e-ba2b-e221b6c59e98\" (UID: \"2fe2e1d4-9fac-456e-ba2b-e221b6c59e98\") " Nov 03 13:51:59.829324 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:59.829147 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-workload-certs\") pod \"2fe2e1d4-9fac-456e-ba2b-e221b6c59e98\" (UID: \"2fe2e1d4-9fac-456e-ba2b-e221b6c59e98\") " Nov 03 13:51:59.829441 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:59.829371 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-workload-socket" pod "2fe2e1d4-9fac-456e-ba2b-e221b6c59e98" (UID: "2fe2e1d4-9fac-456e-ba2b-e221b6c59e98"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:51:59.829441 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:59.829389 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-credential-socket" pod "2fe2e1d4-9fac-456e-ba2b-e221b6c59e98" (UID: "2fe2e1d4-9fac-456e-ba2b-e221b6c59e98"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:51:59.829584 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:59.829457 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-istiod-ca-cert" pod "2fe2e1d4-9fac-456e-ba2b-e221b6c59e98" (UID: "2fe2e1d4-9fac-456e-ba2b-e221b6c59e98"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:51:59.829584 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:59.829496 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-istio-data" pod "2fe2e1d4-9fac-456e-ba2b-e221b6c59e98" (UID: "2fe2e1d4-9fac-456e-ba2b-e221b6c59e98"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:51:59.829707 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:59.829583 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-workload-certs" pod "2fe2e1d4-9fac-456e-ba2b-e221b6c59e98" (UID: "2fe2e1d4-9fac-456e-ba2b-e221b6c59e98"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:51:59.829793 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:59.829777 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:51:59.829859 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:59.829794 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:51:59.829859 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:59.829804 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:51:59.829859 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:59.829813 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:51:59.829859 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:59.829822 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:51:59.831111 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:59.831085 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-istio-envoy" pod "2fe2e1d4-9fac-456e-ba2b-e221b6c59e98" (UID: "2fe2e1d4-9fac-456e-ba2b-e221b6c59e98"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:51:59.831611 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:59.831583 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-kube-api-access-4ftqk" pod "2fe2e1d4-9fac-456e-ba2b-e221b6c59e98" (UID: "2fe2e1d4-9fac-456e-ba2b-e221b6c59e98"). InnerVolumeSpecName "kube-api-access-4ftqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:51:59.831710 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:59.831617 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-istio-podinfo" pod "2fe2e1d4-9fac-456e-ba2b-e221b6c59e98" (UID: "2fe2e1d4-9fac-456e-ba2b-e221b6c59e98"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:51:59.831710 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:59.831683 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-istio-token" pod "2fe2e1d4-9fac-456e-ba2b-e221b6c59e98" (UID: "2fe2e1d4-9fac-456e-ba2b-e221b6c59e98"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:51:59.931847 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:59.931803 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:51:59.931847 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:59.931843 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:51:59.932081 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:59.931859 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ftqk\" (UniqueName: \"kubernetes.io/projected/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-kube-api-access-4ftqk\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:51:59.932081 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:51:59.931876 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:52:00.275484 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:00.275433 5979 generic.go:356] "Generic (PLEG): container finished" podID="2fe2e1d4-9fac-456e-ba2b-e221b6c59e98" containerID="1a57ea83cab3b1e78bf6133cd4eaf65965e89db715e28489cfdcb325932997c7" exitCode=0 Nov 03 13:52:00.275642 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:00.275502 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w" Nov 03 13:52:00.275642 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:00.275506 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w" event={"ID":"2fe2e1d4-9fac-456e-ba2b-e221b6c59e98","Type":"ContainerDied","Data":"1a57ea83cab3b1e78bf6133cd4eaf65965e89db715e28489cfdcb325932997c7"} Nov 03 13:52:00.275642 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:00.275564 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w" event={"ID":"2fe2e1d4-9fac-456e-ba2b-e221b6c59e98","Type":"ContainerDied","Data":"bdad44da43f400a6ce5fb88b2e93068cc43ac3565ef917f56a5554cf7cf93eb1"} Nov 03 13:52:00.275642 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:00.275588 5979 scope.go:122] "RemoveContainer" containerID="1a57ea83cab3b1e78bf6133cd4eaf65965e89db715e28489cfdcb325932997c7" Nov 03 13:52:00.285609 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:00.285589 5979 scope.go:122] "RemoveContainer" containerID="e99df3ac5df64c94c69193b5bce556f78818c051cae6de0dc0d96f4a98bef169" Nov 03 13:52:00.297168 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:00.297141 5979 scope.go:122] "RemoveContainer" containerID="1a57ea83cab3b1e78bf6133cd4eaf65965e89db715e28489cfdcb325932997c7" Nov 03 13:52:00.297492 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:52:00.297463 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a57ea83cab3b1e78bf6133cd4eaf65965e89db715e28489cfdcb325932997c7\": container with ID starting with 1a57ea83cab3b1e78bf6133cd4eaf65965e89db715e28489cfdcb325932997c7 not found: ID does not exist" containerID="1a57ea83cab3b1e78bf6133cd4eaf65965e89db715e28489cfdcb325932997c7" Nov 03 13:52:00.297580 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:00.297502 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a57ea83cab3b1e78bf6133cd4eaf65965e89db715e28489cfdcb325932997c7"} err="failed to get container status \"1a57ea83cab3b1e78bf6133cd4eaf65965e89db715e28489cfdcb325932997c7\": rpc error: code = NotFound desc = could not find container \"1a57ea83cab3b1e78bf6133cd4eaf65965e89db715e28489cfdcb325932997c7\": container with ID starting with 1a57ea83cab3b1e78bf6133cd4eaf65965e89db715e28489cfdcb325932997c7 not found: ID does not exist" Nov 03 13:52:00.297580 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:00.297523 5979 scope.go:122] "RemoveContainer" containerID="e99df3ac5df64c94c69193b5bce556f78818c051cae6de0dc0d96f4a98bef169" Nov 03 13:52:00.297830 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:52:00.297811 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e99df3ac5df64c94c69193b5bce556f78818c051cae6de0dc0d96f4a98bef169\": container with ID starting with e99df3ac5df64c94c69193b5bce556f78818c051cae6de0dc0d96f4a98bef169 not found: ID does not exist" containerID="e99df3ac5df64c94c69193b5bce556f78818c051cae6de0dc0d96f4a98bef169" Nov 03 13:52:00.297886 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:00.297835 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e99df3ac5df64c94c69193b5bce556f78818c051cae6de0dc0d96f4a98bef169"} err="failed to get container status \"e99df3ac5df64c94c69193b5bce556f78818c051cae6de0dc0d96f4a98bef169\": rpc error: code = NotFound desc = could not find container \"e99df3ac5df64c94c69193b5bce556f78818c051cae6de0dc0d96f4a98bef169\": container with ID starting with e99df3ac5df64c94c69193b5bce556f78818c051cae6de0dc0d96f4a98bef169 not found: ID does not exist" Nov 03 13:52:00.430098 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:00.430064 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w"] Nov 03 13:52:00.476918 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:00.476884 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-d83c93-openshift-default-7dd4b8dcdf-6wj8w"] Nov 03 13:52:02.117900 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:02.117865 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fe2e1d4-9fac-456e-ba2b-e221b6c59e98" path="/var/lib/kubelet/pods/2fe2e1d4-9fac-456e-ba2b-e221b6c59e98/volumes" Nov 03 13:52:02.710159 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:02.710124 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm"] Nov 03 13:52:02.710887 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:02.710863 5979 cpu_manager.go:414] "containerMap: removing container" podUID="2fe2e1d4-9fac-456e-ba2b-e221b6c59e98" containerName="istio-proxy" Nov 03 13:52:02.710887 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:02.710884 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fe2e1d4-9fac-456e-ba2b-e221b6c59e98" containerName="istio-proxy" Nov 03 13:52:02.711078 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:02.710920 5979 cpu_manager.go:414] "containerMap: removing container" podUID="f285fcb6-7753-4725-9381-a484d084937a" containerName="revoke-keys" Nov 03 13:52:02.711078 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:02.710929 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f285fcb6-7753-4725-9381-a484d084937a" containerName="revoke-keys" Nov 03 13:52:02.711078 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:02.710946 5979 cpu_manager.go:414] "containerMap: removing container" podUID="2fe2e1d4-9fac-456e-ba2b-e221b6c59e98" containerName="istio-proxy" Nov 03 13:52:02.711078 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:02.710955 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fe2e1d4-9fac-456e-ba2b-e221b6c59e98" containerName="istio-proxy" Nov 03 13:52:02.711248 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:02.711103 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fe2e1d4-9fac-456e-ba2b-e221b6c59e98" containerName="istio-proxy" Nov 03 13:52:02.711248 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:02.711120 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="f285fcb6-7753-4725-9381-a484d084937a" containerName="revoke-keys" Nov 03 13:52:02.716715 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:02.716694 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm" Nov 03 13:52:02.795001 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:02.794966 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-credential-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm\" (UID: \"c58e28cd-87a4-4a44-abb3-bc1ea9b60449\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm" Nov 03 13:52:02.795205 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:02.795098 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-istio-data\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm\" (UID: \"c58e28cd-87a4-4a44-abb3-bc1ea9b60449\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm" Nov 03 13:52:02.795205 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:02.795125 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-workload-certs\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm\" (UID: \"c58e28cd-87a4-4a44-abb3-bc1ea9b60449\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm" Nov 03 13:52:02.795205 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:02.795143 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-istio-podinfo\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm\" (UID: \"c58e28cd-87a4-4a44-abb3-bc1ea9b60449\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm" Nov 03 13:52:02.795328 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:02.795248 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-istiod-ca-cert\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm\" (UID: \"c58e28cd-87a4-4a44-abb3-bc1ea9b60449\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm" Nov 03 13:52:02.795328 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:02.795275 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-istio-token\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm\" (UID: \"c58e28cd-87a4-4a44-abb3-bc1ea9b60449\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm" Nov 03 13:52:02.795328 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:02.795317 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2h8d\" (UniqueName: \"kubernetes.io/projected/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-kube-api-access-h2h8d\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm\" (UID: \"c58e28cd-87a4-4a44-abb3-bc1ea9b60449\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm" Nov 03 13:52:02.795429 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:02.795421 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-istio-envoy\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm\" (UID: \"c58e28cd-87a4-4a44-abb3-bc1ea9b60449\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm" Nov 03 13:52:02.795483 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:02.795461 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-workload-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm\" (UID: \"c58e28cd-87a4-4a44-abb3-bc1ea9b60449\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm" Nov 03 13:52:02.813580 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:02.812978 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-3c5dbc-openshift-default-dockercfg-hfbq4\"" Nov 03 13:52:02.817742 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:02.817706 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm"] Nov 03 13:52:02.896711 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:02.896674 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-istiod-ca-cert\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm\" (UID: \"c58e28cd-87a4-4a44-abb3-bc1ea9b60449\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm" Nov 03 13:52:02.896903 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:02.896726 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-istio-token\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm\" (UID: \"c58e28cd-87a4-4a44-abb3-bc1ea9b60449\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm" Nov 03 13:52:02.896903 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:02.896795 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2h8d\" (UniqueName: \"kubernetes.io/projected/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-kube-api-access-h2h8d\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm\" (UID: \"c58e28cd-87a4-4a44-abb3-bc1ea9b60449\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm" Nov 03 13:52:02.896903 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:02.896835 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-istio-envoy\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm\" (UID: \"c58e28cd-87a4-4a44-abb3-bc1ea9b60449\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm" Nov 03 13:52:02.896903 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:02.896892 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-workload-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm\" (UID: \"c58e28cd-87a4-4a44-abb3-bc1ea9b60449\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm" Nov 03 13:52:02.897160 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:02.897045 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-credential-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm\" (UID: \"c58e28cd-87a4-4a44-abb3-bc1ea9b60449\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm" Nov 03 13:52:02.897249 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:02.897227 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-istio-data\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm\" (UID: \"c58e28cd-87a4-4a44-abb3-bc1ea9b60449\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm" Nov 03 13:52:02.897321 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:02.897269 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-workload-certs\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm\" (UID: \"c58e28cd-87a4-4a44-abb3-bc1ea9b60449\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm" Nov 03 13:52:02.897321 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:02.897295 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-workload-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm\" (UID: \"c58e28cd-87a4-4a44-abb3-bc1ea9b60449\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm" Nov 03 13:52:02.897438 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:02.897297 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-istio-podinfo\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm\" (UID: \"c58e28cd-87a4-4a44-abb3-bc1ea9b60449\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm" Nov 03 13:52:02.897438 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:02.897382 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-credential-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm\" (UID: \"c58e28cd-87a4-4a44-abb3-bc1ea9b60449\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm" Nov 03 13:52:02.897609 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:02.897586 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-istio-data\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm\" (UID: \"c58e28cd-87a4-4a44-abb3-bc1ea9b60449\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm" Nov 03 13:52:02.897687 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:02.897611 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-istiod-ca-cert\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm\" (UID: \"c58e28cd-87a4-4a44-abb3-bc1ea9b60449\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm" Nov 03 13:52:02.897687 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:02.897611 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-workload-certs\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm\" (UID: \"c58e28cd-87a4-4a44-abb3-bc1ea9b60449\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm" Nov 03 13:52:02.899297 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:02.899275 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-istio-envoy\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm\" (UID: \"c58e28cd-87a4-4a44-abb3-bc1ea9b60449\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm" Nov 03 13:52:02.899788 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:02.899759 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-istio-podinfo\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm\" (UID: \"c58e28cd-87a4-4a44-abb3-bc1ea9b60449\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm" Nov 03 13:52:02.956349 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:02.956309 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-istio-token\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm\" (UID: \"c58e28cd-87a4-4a44-abb3-bc1ea9b60449\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm" Nov 03 13:52:02.956557 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:02.956437 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2h8d\" (UniqueName: \"kubernetes.io/projected/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-kube-api-access-h2h8d\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm\" (UID: \"c58e28cd-87a4-4a44-abb3-bc1ea9b60449\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm" Nov 03 13:52:03.032547 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:03.032452 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm" Nov 03 13:52:03.466986 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:03.466944 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm"] Nov 03 13:52:03.473586 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:52:03.473544 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc58e28cd_87a4_4a44_abb3_bc1ea9b60449.slice/crio-2378d98905941c342b440a6a60523fb6f8e07357e14388286a3308ed673e6889 WatchSource:0}: Error finding container 2378d98905941c342b440a6a60523fb6f8e07357e14388286a3308ed673e6889: Status 404 returned error can't find the container with id 2378d98905941c342b440a6a60523fb6f8e07357e14388286a3308ed673e6889 Nov 03 13:52:03.476381 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:03.476339 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:52:03.476516 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:03.476429 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:52:03.476516 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:03.476482 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:52:04.310858 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:04.310819 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm" event={"ID":"c58e28cd-87a4-4a44-abb3-bc1ea9b60449","Type":"ContainerStarted","Data":"2203629e33a905e7d77a21f6ecfbb94bd82472c523ba907becd57b632c96009c"} Nov 03 13:52:04.311071 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:04.310890 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm" event={"ID":"c58e28cd-87a4-4a44-abb3-bc1ea9b60449","Type":"ContainerStarted","Data":"2378d98905941c342b440a6a60523fb6f8e07357e14388286a3308ed673e6889"} Nov 03 13:52:04.480482 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:04.480406 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm" podStartSLOduration=2.480386813 podStartE2EDuration="2.480386813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:52:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:52:04.477187358 +0000 UTC m=+2806.487875602" watchObservedRunningTime="2026-11-03 13:52:04.480386813 +0000 UTC m=+2806.491075056" Nov 03 13:52:05.033316 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:05.033275 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm" Nov 03 13:52:05.035226 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:05.035197 5979 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.23:15021/healthz/ready\": dial tcp 10.134.1.23:15021: connect: connection refused" start-of-body= Nov 03 13:52:05.035360 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:05.035261 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm" podUID="c58e28cd-87a4-4a44-abb3-bc1ea9b60449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.23:15021/healthz/ready\": dial tcp 10.134.1.23:15021: connect: connection refused" Nov 03 13:52:05.774086 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:05.774041 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d0231d-6c8bcc59bb-z4jqb"] Nov 03 13:52:05.775687 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:05.775230 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fe2e1d4-9fac-456e-ba2b-e221b6c59e98" containerName="istio-proxy" Nov 03 13:52:05.778180 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:05.778153 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d0231d-6c8bcc59bb-z4jqb" Nov 03 13:52:05.875658 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:05.875620 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-d0231d-serving-cert\"" Nov 03 13:52:05.879481 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:05.879427 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d0231d-6c8bcc59bb-z4jqb"] Nov 03 13:52:05.879761 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:05.879735 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e63fc009-1095-4586-96b7-ddae424d47ec-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-d0231d-6c8bcc59bb-z4jqb\" (UID: \"e63fc009-1095-4586-96b7-ddae424d47ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d0231d-6c8bcc59bb-z4jqb" Nov 03 13:52:05.879849 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:05.879832 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e63fc009-1095-4586-96b7-ddae424d47ec-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-d0231d-6c8bcc59bb-z4jqb\" (UID: \"e63fc009-1095-4586-96b7-ddae424d47ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d0231d-6c8bcc59bb-z4jqb" Nov 03 13:52:05.879915 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:05.879859 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e63fc009-1095-4586-96b7-ddae424d47ec-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-d0231d-6c8bcc59bb-z4jqb\" (UID: \"e63fc009-1095-4586-96b7-ddae424d47ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d0231d-6c8bcc59bb-z4jqb" Nov 03 13:52:05.879985 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:05.879946 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56fjv\" (UniqueName: \"kubernetes.io/projected/e63fc009-1095-4586-96b7-ddae424d47ec-kube-api-access-56fjv\") pod \"maas-api-e2e-shared-b-w0-d0231d-6c8bcc59bb-z4jqb\" (UID: \"e63fc009-1095-4586-96b7-ddae424d47ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d0231d-6c8bcc59bb-z4jqb" Nov 03 13:52:05.981782 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:05.981692 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e63fc009-1095-4586-96b7-ddae424d47ec-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-d0231d-6c8bcc59bb-z4jqb\" (UID: \"e63fc009-1095-4586-96b7-ddae424d47ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d0231d-6c8bcc59bb-z4jqb" Nov 03 13:52:05.981782 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:05.981730 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e63fc009-1095-4586-96b7-ddae424d47ec-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-d0231d-6c8bcc59bb-z4jqb\" (UID: \"e63fc009-1095-4586-96b7-ddae424d47ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d0231d-6c8bcc59bb-z4jqb" Nov 03 13:52:05.982025 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:05.981830 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56fjv\" (UniqueName: \"kubernetes.io/projected/e63fc009-1095-4586-96b7-ddae424d47ec-kube-api-access-56fjv\") pod \"maas-api-e2e-shared-b-w0-d0231d-6c8bcc59bb-z4jqb\" (UID: \"e63fc009-1095-4586-96b7-ddae424d47ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d0231d-6c8bcc59bb-z4jqb" Nov 03 13:52:05.982025 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:05.981914 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e63fc009-1095-4586-96b7-ddae424d47ec-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-d0231d-6c8bcc59bb-z4jqb\" (UID: \"e63fc009-1095-4586-96b7-ddae424d47ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d0231d-6c8bcc59bb-z4jqb" Nov 03 13:52:05.982150 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:52:05.982048 5979 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d0231d-serving-cert: secret "maas-api-e2e-shared-b-w0-d0231d-serving-cert" not found Nov 03 13:52:05.982150 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:52:05.982105 5979 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e63fc009-1095-4586-96b7-ddae424d47ec-maas-api-tls podName:e63fc009-1095-4586-96b7-ddae424d47ec nodeName:}" failed. No retries permitted until 2026-11-03 13:52:06.482090409 +0000 UTC m=+2808.492778629 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e63fc009-1095-4586-96b7-ddae424d47ec-maas-api-tls") pod "maas-api-e2e-shared-b-w0-d0231d-6c8bcc59bb-z4jqb" (UID: "e63fc009-1095-4586-96b7-ddae424d47ec") : secret "maas-api-e2e-shared-b-w0-d0231d-serving-cert" not found Nov 03 13:52:05.983148 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:05.983121 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e63fc009-1095-4586-96b7-ddae424d47ec-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-d0231d-6c8bcc59bb-z4jqb\" (UID: \"e63fc009-1095-4586-96b7-ddae424d47ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d0231d-6c8bcc59bb-z4jqb" Nov 03 13:52:05.984880 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:05.984854 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e63fc009-1095-4586-96b7-ddae424d47ec-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-d0231d-6c8bcc59bb-z4jqb\" (UID: \"e63fc009-1095-4586-96b7-ddae424d47ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d0231d-6c8bcc59bb-z4jqb" Nov 03 13:52:06.033558 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:06.033460 5979 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.23:15021/healthz/ready\": dial tcp 10.134.1.23:15021: connect: connection refused" start-of-body= Nov 03 13:52:06.033792 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:06.033765 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm" podUID="c58e28cd-87a4-4a44-abb3-bc1ea9b60449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.23:15021/healthz/ready\": dial tcp 10.134.1.23:15021: connect: connection refused" Nov 03 13:52:06.036446 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:06.036408 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56fjv\" (UniqueName: \"kubernetes.io/projected/e63fc009-1095-4586-96b7-ddae424d47ec-kube-api-access-56fjv\") pod \"maas-api-e2e-shared-b-w0-d0231d-6c8bcc59bb-z4jqb\" (UID: \"e63fc009-1095-4586-96b7-ddae424d47ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d0231d-6c8bcc59bb-z4jqb" Nov 03 13:52:06.327203 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:06.327110 5979 generic.go:356] "Generic (PLEG): container finished" podID="4723366a-b53b-40f7-a27a-d5cde461525d" containerID="bbccee5e13015c2853354a6cbc64fccec9b6655e2cd856a2b03a7bfb19300918" exitCode=0 Nov 03 13:52:06.327203 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:06.327184 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj" event={"ID":"4723366a-b53b-40f7-a27a-d5cde461525d","Type":"ContainerDied","Data":"bbccee5e13015c2853354a6cbc64fccec9b6655e2cd856a2b03a7bfb19300918"} Nov 03 13:52:06.327858 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:06.327839 5979 scope.go:122] "RemoveContainer" containerID="bbccee5e13015c2853354a6cbc64fccec9b6655e2cd856a2b03a7bfb19300918" Nov 03 13:52:06.329300 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:06.329275 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:52:06.329366 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:06.329339 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:52:06.329402 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:06.329372 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:52:06.530125 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:06.501186 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e63fc009-1095-4586-96b7-ddae424d47ec-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-d0231d-6c8bcc59bb-z4jqb\" (UID: \"e63fc009-1095-4586-96b7-ddae424d47ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d0231d-6c8bcc59bb-z4jqb" Nov 03 13:52:06.628028 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:06.618724 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e63fc009-1095-4586-96b7-ddae424d47ec-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-d0231d-6c8bcc59bb-z4jqb\" (UID: \"e63fc009-1095-4586-96b7-ddae424d47ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d0231d-6c8bcc59bb-z4jqb" Nov 03 13:52:06.695131 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:06.693851 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d0231d-6c8bcc59bb-z4jqb" Nov 03 13:52:07.033193 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:07.033147 5979 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.23:15021/healthz/ready\": dial tcp 10.134.1.23:15021: connect: connection refused" start-of-body= Nov 03 13:52:07.033712 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:07.033217 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm" podUID="c58e28cd-87a4-4a44-abb3-bc1ea9b60449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.23:15021/healthz/ready\": dial tcp 10.134.1.23:15021: connect: connection refused" Nov 03 13:52:07.110088 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:07.110047 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d0231d-6c8bcc59bb-z4jqb"] Nov 03 13:52:07.117412 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:52:07.117376 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode63fc009_1095_4586_96b7_ddae424d47ec.slice/crio-8f735aca9243b8bb4aee45db995ee57c3d609c9272fbacb5fdc5b15e808f674f WatchSource:0}: Error finding container 8f735aca9243b8bb4aee45db995ee57c3d609c9272fbacb5fdc5b15e808f674f: Status 404 returned error can't find the container with id 8f735aca9243b8bb4aee45db995ee57c3d609c9272fbacb5fdc5b15e808f674f Nov 03 13:52:07.181833 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:07.181792 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj" Nov 03 13:52:07.234695 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:52:07.234659 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:52:07.337146 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:07.337051 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj" event={"ID":"4723366a-b53b-40f7-a27a-d5cde461525d","Type":"ContainerStarted","Data":"78f537c5e25fcc7de464419ca6c3694c2f5921b25b7f21e362614923ccf286bc"} Nov 03 13:52:07.339785 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:07.339755 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d0231d-6c8bcc59bb-z4jqb" event={"ID":"e63fc009-1095-4586-96b7-ddae424d47ec","Type":"ContainerStarted","Data":"8f735aca9243b8bb4aee45db995ee57c3d609c9272fbacb5fdc5b15e808f674f"} Nov 03 13:52:07.574901 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:07.574865 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj" Nov 03 13:52:07.578630 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:07.578593 5979 patch_prober.go:29] interesting pod/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" start-of-body= Nov 03 13:52:07.578903 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:07.578875 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj" podUID="4723366a-b53b-40f7-a27a-d5cde461525d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" Nov 03 13:52:08.033440 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:08.033402 5979 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.23:15021/healthz/ready\": dial tcp 10.134.1.23:15021: connect: connection refused" start-of-body= Nov 03 13:52:08.034092 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:08.034061 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm" podUID="c58e28cd-87a4-4a44-abb3-bc1ea9b60449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.23:15021/healthz/ready\": dial tcp 10.134.1.23:15021: connect: connection refused" Nov 03 13:52:08.363256 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:08.363162 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d0231d-6c8bcc59bb-z4jqb" event={"ID":"e63fc009-1095-4586-96b7-ddae424d47ec","Type":"ContainerStarted","Data":"cb460a419f06f2ccd18a152c2691970e17efbfaffbd77029609b65472b5c59c4"} Nov 03 13:52:08.363554 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:08.363533 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d0231d-6c8bcc59bb-z4jqb" Nov 03 13:52:08.497456 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:08.496527 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d0231d-6c8bcc59bb-z4jqb" podStartSLOduration=2.548958923 podStartE2EDuration="3.49650525s" totalImagesPullingTime="947.546327ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:52:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:52:08.48378415 +0000 UTC m=+2810.494472393" watchObservedRunningTime="2026-11-03 13:52:08.49650525 +0000 UTC m=+2810.507193494" Nov 03 13:52:08.577494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:08.576864 5979 patch_prober.go:29] interesting pod/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" start-of-body= Nov 03 13:52:08.577494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:08.576928 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj" podUID="4723366a-b53b-40f7-a27a-d5cde461525d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" Nov 03 13:52:09.032890 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:09.032855 5979 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.23:15021/healthz/ready\": dial tcp 10.134.1.23:15021: connect: connection refused" start-of-body= Nov 03 13:52:09.033069 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:09.032902 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm" podUID="c58e28cd-87a4-4a44-abb3-bc1ea9b60449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.23:15021/healthz/ready\": dial tcp 10.134.1.23:15021: connect: connection refused" Nov 03 13:52:09.066575 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:09.066525 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef5ed956-pc2gz"] Nov 03 13:52:09.089311 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:09.089259 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef5ed956-pc2gz"] Nov 03 13:52:09.592496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:09.591488 5979 patch_prober.go:29] interesting pod/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" start-of-body= Nov 03 13:52:09.592496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:09.591555 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj" podUID="4723366a-b53b-40f7-a27a-d5cde461525d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" Nov 03 13:52:10.063884 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:10.063847 5979 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.23:15021/healthz/ready\": dial tcp 10.134.1.23:15021: connect: connection refused" start-of-body= Nov 03 13:52:10.064329 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:10.064283 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm" podUID="c58e28cd-87a4-4a44-abb3-bc1ea9b60449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.23:15021/healthz/ready\": dial tcp 10.134.1.23:15021: connect: connection refused" Nov 03 13:52:10.116567 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:10.116537 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b26f627f-b967-45b9-80d3-847cc513bdfe" path="/var/lib/kubelet/pods/b26f627f-b967-45b9-80d3-847cc513bdfe/volumes" Nov 03 13:52:10.576089 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:10.574912 5979 patch_prober.go:29] interesting pod/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" start-of-body= Nov 03 13:52:10.576089 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:10.574980 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj" podUID="4723366a-b53b-40f7-a27a-d5cde461525d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" Nov 03 13:52:11.305277 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:11.305236 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb4ca0-7cb4bfb5f4-j84xg"] Nov 03 13:52:11.306119 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:11.306083 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb4ca0-7cb4bfb5f4-j84xg" podUID="40b93c39-a07f-4e43-a1cc-66cfb1469e51" containerName="maas-api" containerID="cri-o://cfaa1145263e37611741492bccb4f61df847a434caa4d3ac8766db3bf01fe1c7" gracePeriod=30 Nov 03 13:52:11.583652 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:11.582950 5979 patch_prober.go:29] interesting pod/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" start-of-body= Nov 03 13:52:11.583652 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:11.583009 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj" podUID="4723366a-b53b-40f7-a27a-d5cde461525d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" Nov 03 13:52:11.895905 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:11.895872 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm" Nov 03 13:52:11.897179 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:11.897155 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm" Nov 03 13:52:11.898503 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:11.898430 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm" Nov 03 13:52:11.904717 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:11.904695 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb4ca0-7cb4bfb5f4-j84xg" Nov 03 13:52:11.931352 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:11.931321 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40b93c39-a07f-4e43-a1cc-66cfb1469e51-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40b93c39-a07f-4e43-a1cc-66cfb1469e51-openshift-service-ca\") pod \"40b93c39-a07f-4e43-a1cc-66cfb1469e51\" (UID: \"40b93c39-a07f-4e43-a1cc-66cfb1469e51\") " Nov 03 13:52:11.931587 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:11.931503 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40b93c39-a07f-4e43-a1cc-66cfb1469e51-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40b93c39-a07f-4e43-a1cc-66cfb1469e51-maas-api-tls\") pod \"40b93c39-a07f-4e43-a1cc-66cfb1469e51\" (UID: \"40b93c39-a07f-4e43-a1cc-66cfb1469e51\") " Nov 03 13:52:11.931587 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:11.931533 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40b93c39-a07f-4e43-a1cc-66cfb1469e51-metrics-certs\" (UniqueName: \"kubernetes.io/secret/40b93c39-a07f-4e43-a1cc-66cfb1469e51-metrics-certs\") pod \"40b93c39-a07f-4e43-a1cc-66cfb1469e51\" (UID: \"40b93c39-a07f-4e43-a1cc-66cfb1469e51\") " Nov 03 13:52:11.931587 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:11.931553 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40b93c39-a07f-4e43-a1cc-66cfb1469e51-kube-api-access-9hhh8\" (UniqueName: \"kubernetes.io/projected/40b93c39-a07f-4e43-a1cc-66cfb1469e51-kube-api-access-9hhh8\") pod \"40b93c39-a07f-4e43-a1cc-66cfb1469e51\" (UID: \"40b93c39-a07f-4e43-a1cc-66cfb1469e51\") " Nov 03 13:52:11.931795 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:11.931662 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40b93c39-a07f-4e43-a1cc-66cfb1469e51-openshift-service-ca" pod "40b93c39-a07f-4e43-a1cc-66cfb1469e51" (UID: "40b93c39-a07f-4e43-a1cc-66cfb1469e51"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:52:11.932265 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:11.932014 5979 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40b93c39-a07f-4e43-a1cc-66cfb1469e51-openshift-service-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:52:11.934246 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:11.934220 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40b93c39-a07f-4e43-a1cc-66cfb1469e51-metrics-certs" pod "40b93c39-a07f-4e43-a1cc-66cfb1469e51" (UID: "40b93c39-a07f-4e43-a1cc-66cfb1469e51"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:52:11.934366 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:11.934318 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40b93c39-a07f-4e43-a1cc-66cfb1469e51-maas-api-tls" pod "40b93c39-a07f-4e43-a1cc-66cfb1469e51" (UID: "40b93c39-a07f-4e43-a1cc-66cfb1469e51"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:52:11.934871 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:11.934842 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40b93c39-a07f-4e43-a1cc-66cfb1469e51-kube-api-access-9hhh8" pod "40b93c39-a07f-4e43-a1cc-66cfb1469e51" (UID: "40b93c39-a07f-4e43-a1cc-66cfb1469e51"). InnerVolumeSpecName "kube-api-access-9hhh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:52:12.033419 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:12.033383 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40b93c39-a07f-4e43-a1cc-66cfb1469e51-maas-api-tls\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:52:12.033419 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:12.033415 5979 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40b93c39-a07f-4e43-a1cc-66cfb1469e51-metrics-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:52:12.033419 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:12.033425 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hhh8\" (UniqueName: \"kubernetes.io/projected/40b93c39-a07f-4e43-a1cc-66cfb1469e51-kube-api-access-9hhh8\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:52:12.068652 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:12.054317 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5606cf0b-fkwr8"] Nov 03 13:52:12.069290 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:12.069251 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5606cf0b-fkwr8"] Nov 03 13:52:12.117185 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:12.117110 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb80b6d0-af59-468e-93a8-7ed4b40454ce" path="/var/lib/kubelet/pods/bb80b6d0-af59-468e-93a8-7ed4b40454ce/volumes" Nov 03 13:52:12.510508 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:12.508813 5979 generic.go:356] "Generic (PLEG): container finished" podID="40b93c39-a07f-4e43-a1cc-66cfb1469e51" containerID="cfaa1145263e37611741492bccb4f61df847a434caa4d3ac8766db3bf01fe1c7" exitCode=0 Nov 03 13:52:12.510508 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:12.509213 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb4ca0-7cb4bfb5f4-j84xg" Nov 03 13:52:12.511613 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:12.511095 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb4ca0-7cb4bfb5f4-j84xg" event={"ID":"40b93c39-a07f-4e43-a1cc-66cfb1469e51","Type":"ContainerDied","Data":"cfaa1145263e37611741492bccb4f61df847a434caa4d3ac8766db3bf01fe1c7"} Nov 03 13:52:12.511613 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:12.511172 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb4ca0-7cb4bfb5f4-j84xg" event={"ID":"40b93c39-a07f-4e43-a1cc-66cfb1469e51","Type":"ContainerDied","Data":"ff7bfb41dce1de4cac875262de99426af4f2df2dce660f9143e72570a3411fa5"} Nov 03 13:52:12.511613 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:12.511208 5979 scope.go:122] "RemoveContainer" containerID="cfaa1145263e37611741492bccb4f61df847a434caa4d3ac8766db3bf01fe1c7" Nov 03 13:52:12.540543 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:12.540422 5979 scope.go:122] "RemoveContainer" containerID="cfaa1145263e37611741492bccb4f61df847a434caa4d3ac8766db3bf01fe1c7" Nov 03 13:52:12.541194 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:52:12.541097 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfaa1145263e37611741492bccb4f61df847a434caa4d3ac8766db3bf01fe1c7\": container with ID starting with cfaa1145263e37611741492bccb4f61df847a434caa4d3ac8766db3bf01fe1c7 not found: ID does not exist" containerID="cfaa1145263e37611741492bccb4f61df847a434caa4d3ac8766db3bf01fe1c7" Nov 03 13:52:12.541194 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:12.541131 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfaa1145263e37611741492bccb4f61df847a434caa4d3ac8766db3bf01fe1c7"} err="failed to get container status \"cfaa1145263e37611741492bccb4f61df847a434caa4d3ac8766db3bf01fe1c7\": rpc error: code = NotFound desc = could not find container \"cfaa1145263e37611741492bccb4f61df847a434caa4d3ac8766db3bf01fe1c7\": container with ID starting with cfaa1145263e37611741492bccb4f61df847a434caa4d3ac8766db3bf01fe1c7 not found: ID does not exist" Nov 03 13:52:12.577044 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:12.577008 5979 patch_prober.go:29] interesting pod/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" start-of-body= Nov 03 13:52:12.577246 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:12.577060 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj" podUID="4723366a-b53b-40f7-a27a-d5cde461525d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.15:15021/healthz/ready\": dial tcp 10.134.1.15:15021: connect: connection refused" Nov 03 13:52:12.677171 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:12.677137 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb4ca0-7cb4bfb5f4-j84xg"] Nov 03 13:52:12.681568 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:12.681542 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-fb4ca0-7cb4bfb5f4-j84xg"] Nov 03 13:52:13.577702 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:13.577674 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj" Nov 03 13:52:13.578150 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:13.578133 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj" Nov 03 13:52:13.578919 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:13.578897 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj" Nov 03 13:52:14.116087 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:14.116048 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40b93c39-a07f-4e43-a1cc-66cfb1469e51" path="/var/lib/kubelet/pods/40b93c39-a07f-4e43-a1cc-66cfb1469e51/volumes" Nov 03 13:52:15.415582 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:15.415552 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d0231d-6c8bcc59bb-z4jqb" Nov 03 13:52:20.161700 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:20.161666 5979 scope.go:122] "RemoveContainer" containerID="fec1c6f4e0ecfa6773e04e38a58d359063d2f97ca0b942ebc15d8d66be00452d" Nov 03 13:52:20.171518 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:20.171497 5979 scope.go:122] "RemoveContainer" containerID="ac8d23199a6d4a6e218ace7c9dacea099f609d11bba9f98df1be885c06265b4f" Nov 03 13:52:20.180293 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:20.180273 5979 scope.go:122] "RemoveContainer" containerID="d0b90f0abe0d317f0b2afc11ad78963bdd4e47c1d24d010a4b9cb5fbfe8bd149" Nov 03 13:52:20.189016 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:20.188994 5979 scope.go:122] "RemoveContainer" containerID="7beb8632bdc9719249d00780113ce44c483d1eb4cb2720ea1404e37ddea59be9" Nov 03 13:52:20.197338 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:20.197319 5979 scope.go:122] "RemoveContainer" containerID="34fa15be788878a8d267de7acb38e62fc962ae0f975b81a8ba2c826e9b0008f3" Nov 03 13:52:20.206322 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:20.206302 5979 scope.go:122] "RemoveContainer" containerID="a6915fe74a323f051267790257d49c2ac8b94cc70c3cf26cad9128c19341c4cd" Nov 03 13:52:23.601546 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:23.601516 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:52:32.379637 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:32.379587 5979 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 13:52:32.380155 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:32.379659 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc" podUID="8df65c37-2cc2-4bd1-9d05-4576dfaec9a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 13:52:34.724795 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:34.724760 5979 generic.go:356] "Generic (PLEG): container finished" podID="8df65c37-2cc2-4bd1-9d05-4576dfaec9a6" containerID="285bb6e9355b6d8fdcaf8c55057073e92fb5e1074e1af38f22f3036fa5464d1a" exitCode=0 Nov 03 13:52:34.725229 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:34.724834 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc" event={"ID":"8df65c37-2cc2-4bd1-9d05-4576dfaec9a6","Type":"ContainerDied","Data":"285bb6e9355b6d8fdcaf8c55057073e92fb5e1074e1af38f22f3036fa5464d1a"} Nov 03 13:52:34.725278 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:34.725229 5979 scope.go:122] "RemoveContainer" containerID="285bb6e9355b6d8fdcaf8c55057073e92fb5e1074e1af38f22f3036fa5464d1a" Nov 03 13:52:34.726798 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:34.726769 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:52:34.726889 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:34.726837 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:52:34.726934 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:34.726891 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:52:35.047657 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:35.047563 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a94fa233-wl58w"] Nov 03 13:52:35.052250 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:35.052223 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a94fa233-wl58w"] Nov 03 13:52:35.732999 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:35.732963 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc" event={"ID":"8df65c37-2cc2-4bd1-9d05-4576dfaec9a6","Type":"ContainerStarted","Data":"7e16e97308b61bf8f320f624ea933515c3d2663ac782cbd17a7417458f9d65dd"} Nov 03 13:52:36.115942 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:36.115866 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecf690af-2ffa-482c-883c-7f4df7da17e9" path="/var/lib/kubelet/pods/ecf690af-2ffa-482c-883c-7f4df7da17e9/volumes" Nov 03 13:52:36.700362 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:36.700328 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc" Nov 03 13:52:36.700588 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:36.700498 5979 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" start-of-body= Nov 03 13:52:36.700588 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:36.700551 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc" podUID="8df65c37-2cc2-4bd1-9d05-4576dfaec9a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" Nov 03 13:52:37.234893 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:52:37.234848 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:52:37.699871 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:37.699835 5979 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" start-of-body= Nov 03 13:52:37.700064 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:37.699896 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc" podUID="8df65c37-2cc2-4bd1-9d05-4576dfaec9a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" Nov 03 13:52:38.700445 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:38.700411 5979 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" start-of-body= Nov 03 13:52:38.700878 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:38.700463 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc" podUID="8df65c37-2cc2-4bd1-9d05-4576dfaec9a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" Nov 03 13:52:39.699769 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:39.699729 5979 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" start-of-body= Nov 03 13:52:39.699973 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:39.699787 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc" podUID="8df65c37-2cc2-4bd1-9d05-4576dfaec9a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" Nov 03 13:52:40.699827 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:40.699794 5979 patch_prober.go:29] interesting pod/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" start-of-body= Nov 03 13:52:40.700432 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:40.699844 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc" podUID="8df65c37-2cc2-4bd1-9d05-4576dfaec9a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.16:15021/healthz/ready\": dial tcp 10.134.1.16:15021: connect: connection refused" Nov 03 13:52:41.703579 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:41.703542 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc" Nov 03 13:52:41.703990 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:41.703968 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc" Nov 03 13:52:41.704677 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:41.704658 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc" Nov 03 13:52:48.044444 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:48.044408 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82e848cb-n96lb"] Nov 03 13:52:48.090465 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:48.090432 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82e848cb-n96lb"] Nov 03 13:52:48.126664 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:48.126626 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abcb30e6-16a3-48a2-ab22-2c524ba95084" path="/var/lib/kubelet/pods/abcb30e6-16a3-48a2-ab22-2c524ba95084/volumes" Nov 03 13:52:51.875223 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:51.875194 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld"] Nov 03 13:52:51.875838 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:51.875818 5979 cpu_manager.go:414] "containerMap: removing container" podUID="40b93c39-a07f-4e43-a1cc-66cfb1469e51" containerName="maas-api" Nov 03 13:52:51.875838 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:51.875834 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40b93c39-a07f-4e43-a1cc-66cfb1469e51" containerName="maas-api" Nov 03 13:52:51.875960 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:51.875945 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="40b93c39-a07f-4e43-a1cc-66cfb1469e51" containerName="maas-api" Nov 03 13:52:51.878687 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:51.878669 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld" Nov 03 13:52:51.933159 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:51.933111 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-367381-openshift-default-dockercfg-gcp9z\"" Nov 03 13:52:51.967016 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:51.966982 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f76845b8-e625-4425-ad28-74612d33985c-workload-socket\") pod \"e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld\" (UID: \"f76845b8-e625-4425-ad28-74612d33985c\") " pod="openshift-ingress/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld" Nov 03 13:52:51.967016 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:51.967017 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f76845b8-e625-4425-ad28-74612d33985c-istio-envoy\") pod \"e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld\" (UID: \"f76845b8-e625-4425-ad28-74612d33985c\") " pod="openshift-ingress/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld" Nov 03 13:52:51.967253 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:51.967042 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f76845b8-e625-4425-ad28-74612d33985c-workload-certs\") pod \"e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld\" (UID: \"f76845b8-e625-4425-ad28-74612d33985c\") " pod="openshift-ingress/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld" Nov 03 13:52:51.967253 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:51.967195 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f76845b8-e625-4425-ad28-74612d33985c-istio-podinfo\") pod \"e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld\" (UID: \"f76845b8-e625-4425-ad28-74612d33985c\") " pod="openshift-ingress/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld" Nov 03 13:52:51.967253 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:51.967239 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f76845b8-e625-4425-ad28-74612d33985c-istiod-ca-cert\") pod \"e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld\" (UID: \"f76845b8-e625-4425-ad28-74612d33985c\") " pod="openshift-ingress/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld" Nov 03 13:52:51.967388 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:51.967261 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f76845b8-e625-4425-ad28-74612d33985c-istio-data\") pod \"e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld\" (UID: \"f76845b8-e625-4425-ad28-74612d33985c\") " pod="openshift-ingress/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld" Nov 03 13:52:51.967388 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:51.967360 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f76845b8-e625-4425-ad28-74612d33985c-credential-socket\") pod \"e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld\" (UID: \"f76845b8-e625-4425-ad28-74612d33985c\") " pod="openshift-ingress/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld" Nov 03 13:52:51.967502 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:51.967444 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cwfx\" (UniqueName: \"kubernetes.io/projected/f76845b8-e625-4425-ad28-74612d33985c-kube-api-access-9cwfx\") pod \"e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld\" (UID: \"f76845b8-e625-4425-ad28-74612d33985c\") " pod="openshift-ingress/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld" Nov 03 13:52:51.967551 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:51.967507 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f76845b8-e625-4425-ad28-74612d33985c-istio-token\") pod \"e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld\" (UID: \"f76845b8-e625-4425-ad28-74612d33985c\") " pod="openshift-ingress/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld" Nov 03 13:52:51.977953 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:51.977924 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld"] Nov 03 13:52:52.078495 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:52.075286 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f76845b8-e625-4425-ad28-74612d33985c-istio-podinfo\") pod \"e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld\" (UID: \"f76845b8-e625-4425-ad28-74612d33985c\") " pod="openshift-ingress/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld" Nov 03 13:52:52.078495 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:52.075365 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f76845b8-e625-4425-ad28-74612d33985c-istiod-ca-cert\") pod \"e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld\" (UID: \"f76845b8-e625-4425-ad28-74612d33985c\") " pod="openshift-ingress/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld" Nov 03 13:52:52.078495 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:52.075399 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f76845b8-e625-4425-ad28-74612d33985c-istio-data\") pod \"e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld\" (UID: \"f76845b8-e625-4425-ad28-74612d33985c\") " pod="openshift-ingress/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld" Nov 03 13:52:52.078495 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:52.075431 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f76845b8-e625-4425-ad28-74612d33985c-credential-socket\") pod \"e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld\" (UID: \"f76845b8-e625-4425-ad28-74612d33985c\") " pod="openshift-ingress/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld" Nov 03 13:52:52.078495 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:52.075539 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cwfx\" (UniqueName: \"kubernetes.io/projected/f76845b8-e625-4425-ad28-74612d33985c-kube-api-access-9cwfx\") pod \"e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld\" (UID: \"f76845b8-e625-4425-ad28-74612d33985c\") " pod="openshift-ingress/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld" Nov 03 13:52:52.078495 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:52.075603 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f76845b8-e625-4425-ad28-74612d33985c-istio-token\") pod \"e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld\" (UID: \"f76845b8-e625-4425-ad28-74612d33985c\") " pod="openshift-ingress/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld" Nov 03 13:52:52.078495 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:52.075731 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f76845b8-e625-4425-ad28-74612d33985c-workload-socket\") pod \"e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld\" (UID: \"f76845b8-e625-4425-ad28-74612d33985c\") " pod="openshift-ingress/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld" Nov 03 13:52:52.078495 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:52.075758 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f76845b8-e625-4425-ad28-74612d33985c-istio-envoy\") pod \"e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld\" (UID: \"f76845b8-e625-4425-ad28-74612d33985c\") " pod="openshift-ingress/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld" Nov 03 13:52:52.078495 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:52.075795 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f76845b8-e625-4425-ad28-74612d33985c-workload-certs\") pod \"e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld\" (UID: \"f76845b8-e625-4425-ad28-74612d33985c\") " pod="openshift-ingress/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld" Nov 03 13:52:52.078495 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:52.076287 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f76845b8-e625-4425-ad28-74612d33985c-workload-certs\") pod \"e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld\" (UID: \"f76845b8-e625-4425-ad28-74612d33985c\") " pod="openshift-ingress/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld" Nov 03 13:52:52.084423 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:52.084383 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f76845b8-e625-4425-ad28-74612d33985c-istio-podinfo\") pod \"e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld\" (UID: \"f76845b8-e625-4425-ad28-74612d33985c\") " pod="openshift-ingress/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld" Nov 03 13:52:52.085486 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:52.085265 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f76845b8-e625-4425-ad28-74612d33985c-istiod-ca-cert\") pod \"e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld\" (UID: \"f76845b8-e625-4425-ad28-74612d33985c\") " pod="openshift-ingress/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld" Nov 03 13:52:52.085935 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:52.085904 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f76845b8-e625-4425-ad28-74612d33985c-istio-data\") pod \"e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld\" (UID: \"f76845b8-e625-4425-ad28-74612d33985c\") " pod="openshift-ingress/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld" Nov 03 13:52:52.086287 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:52.086269 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f76845b8-e625-4425-ad28-74612d33985c-credential-socket\") pod \"e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld\" (UID: \"f76845b8-e625-4425-ad28-74612d33985c\") " pod="openshift-ingress/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld" Nov 03 13:52:52.087153 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:52.087129 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f76845b8-e625-4425-ad28-74612d33985c-workload-socket\") pod \"e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld\" (UID: \"f76845b8-e625-4425-ad28-74612d33985c\") " pod="openshift-ingress/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld" Nov 03 13:52:52.089992 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:52.089947 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f76845b8-e625-4425-ad28-74612d33985c-istio-envoy\") pod \"e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld\" (UID: \"f76845b8-e625-4425-ad28-74612d33985c\") " pod="openshift-ingress/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld" Nov 03 13:52:52.149023 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:52.148955 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f76845b8-e625-4425-ad28-74612d33985c-istio-token\") pod \"e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld\" (UID: \"f76845b8-e625-4425-ad28-74612d33985c\") " pod="openshift-ingress/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld" Nov 03 13:52:52.152284 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:52.152253 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cwfx\" (UniqueName: \"kubernetes.io/projected/f76845b8-e625-4425-ad28-74612d33985c-kube-api-access-9cwfx\") pod \"e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld\" (UID: \"f76845b8-e625-4425-ad28-74612d33985c\") " pod="openshift-ingress/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld" Nov 03 13:52:52.191589 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:52.191557 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld" Nov 03 13:52:52.655849 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:52.655811 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld"] Nov 03 13:52:52.663246 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:52:52.663217 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf76845b8_e625_4425_ad28_74612d33985c.slice/crio-a5d5d299592355bcf8c4c61f54b30c30349c5cf61b1bee75a61bc67a85b45300 WatchSource:0}: Error finding container a5d5d299592355bcf8c4c61f54b30c30349c5cf61b1bee75a61bc67a85b45300: Status 404 returned error can't find the container with id a5d5d299592355bcf8c4c61f54b30c30349c5cf61b1bee75a61bc67a85b45300 Nov 03 13:52:52.666366 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:52.666324 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:52:52.666487 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:52.666418 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:52:52.666487 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:52.666456 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:52:52.879520 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:52.879448 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld" event={"ID":"f76845b8-e625-4425-ad28-74612d33985c","Type":"ContainerStarted","Data":"a7cf676f1fa259a460515dd1a4b7a3c48b4d7a44850101fcad73a470b97e9673"} Nov 03 13:52:52.879937 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:52.879548 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld" event={"ID":"f76845b8-e625-4425-ad28-74612d33985c","Type":"ContainerStarted","Data":"a5d5d299592355bcf8c4c61f54b30c30349c5cf61b1bee75a61bc67a85b45300"} Nov 03 13:52:52.999286 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:52.999150 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld" podStartSLOduration=1.999131694 podStartE2EDuration="1.999131694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:52:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:52:52.989438435 +0000 UTC m=+2855.000126678" watchObservedRunningTime="2026-11-03 13:52:52.999131694 +0000 UTC m=+2855.009819941" Nov 03 13:52:53.191761 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:53.191730 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld" Nov 03 13:52:53.197117 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:53.197094 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld" Nov 03 13:52:53.891380 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:53.891347 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld" Nov 03 13:52:53.892375 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:53.892341 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld" Nov 03 13:52:54.046848 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:54.046815 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae12be2d-sgcvn"] Nov 03 13:52:54.049680 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:54.049655 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae12be2d-sgcvn"] Nov 03 13:52:54.119444 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:52:54.119402 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44b0445d-b9fe-460f-968b-9e4a950ff35a" path="/var/lib/kubelet/pods/44b0445d-b9fe-460f-968b-9e4a950ff35a/volumes" Nov 03 13:53:03.055970 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:03.055925 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd59e770-b2pvj"] Nov 03 13:53:03.102206 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:03.102169 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd59e770-b2pvj"] Nov 03 13:53:04.116457 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:04.116421 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="effdd48b-ffed-472e-8e08-b6214d3de071" path="/var/lib/kubelet/pods/effdd48b-ffed-472e-8e08-b6214d3de071/volumes" Nov 03 13:53:07.234203 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:53:07.234164 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:53:09.031507 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:09.031459 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9"] Nov 03 13:53:09.032161 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:09.031729 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9" podUID="77989bc1-5b8d-4939-9fd9-383865b9484d" containerName="istio-proxy" containerID="cri-o://ceeaee78fe0d459f4a1496aa067c3406667e6139f71287a7654613a676672289" gracePeriod=30 Nov 03 13:53:09.077774 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:09.077739 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz"] Nov 03 13:53:09.077968 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:09.077934 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz" podUID="0b95b7f2-8b31-487f-ba23-b551e9406cc8" containerName="istio-proxy" containerID="cri-o://f22122c5534f610328f6f44d3bf0d6b4649d8e91dee4b81e038142277ffb2409" gracePeriod=30 Nov 03 13:53:09.080632 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:09.080606 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m"] Nov 03 13:53:09.080961 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:09.080921 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m" podUID="79ff3314-693d-415c-a02d-b249b9a4748c" containerName="istio-proxy" containerID="cri-o://10ed8dfb524cd3c7fc0f1152890c143cbfa4391bc0a4eea1b19c1441278fd45b" gracePeriod=30 Nov 03 13:53:09.124204 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:09.124177 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz"] Nov 03 13:53:09.124364 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:09.124341 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz" podUID="70ed2a89-8231-4e71-862a-02968573d4fe" containerName="istio-proxy" containerID="cri-o://ff2997e72260c1176d52a45923ef02aabeb3a6cc98d253876911eab57073374b" gracePeriod=30 Nov 03 13:53:13.043885 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:13.043852 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24a93d8e-q9rpd"] Nov 03 13:53:13.046655 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:13.046627 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24a93d8e-q9rpd"] Nov 03 13:53:13.481279 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:13.481244 5979 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 13:53:13.481467 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:13.481296 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9" podUID="77989bc1-5b8d-4939-9fd9-383865b9484d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 13:53:13.985662 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:13.985620 5979 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 13:53:13.985838 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:13.985691 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz" podUID="70ed2a89-8231-4e71-862a-02968573d4fe" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 13:53:14.116290 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.116256 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3becf35-a2d0-4144-b9d8-89f69321b056" path="/var/lib/kubelet/pods/c3becf35-a2d0-4144-b9d8-89f69321b056/volumes" Nov 03 13:53:14.372553 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.372520 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz" Nov 03 13:53:14.405171 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.405140 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9" Nov 03 13:53:14.411246 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.411225 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz" Nov 03 13:53:14.453370 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.453344 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m" Nov 03 13:53:14.548142 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.548051 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70ed2a89-8231-4e71-862a-02968573d4fe-kube-api-access-sjg2b\" (UniqueName: \"kubernetes.io/projected/70ed2a89-8231-4e71-862a-02968573d4fe-kube-api-access-sjg2b\") pod \"70ed2a89-8231-4e71-862a-02968573d4fe\" (UID: \"70ed2a89-8231-4e71-862a-02968573d4fe\") " Nov 03 13:53:14.548142 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.548104 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b95b7f2-8b31-487f-ba23-b551e9406cc8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b95b7f2-8b31-487f-ba23-b551e9406cc8-istiod-ca-cert\") pod \"0b95b7f2-8b31-487f-ba23-b551e9406cc8\" (UID: \"0b95b7f2-8b31-487f-ba23-b551e9406cc8\") " Nov 03 13:53:14.548384 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.548154 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/77989bc1-5b8d-4939-9fd9-383865b9484d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77989bc1-5b8d-4939-9fd9-383865b9484d-istio-podinfo\") pod \"77989bc1-5b8d-4939-9fd9-383865b9484d\" (UID: \"77989bc1-5b8d-4939-9fd9-383865b9484d\") " Nov 03 13:53:14.548384 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.548193 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/70ed2a89-8231-4e71-862a-02968573d4fe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70ed2a89-8231-4e71-862a-02968573d4fe-istio-podinfo\") pod \"70ed2a89-8231-4e71-862a-02968573d4fe\" (UID: \"70ed2a89-8231-4e71-862a-02968573d4fe\") " Nov 03 13:53:14.548384 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.548223 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77989bc1-5b8d-4939-9fd9-383865b9484d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77989bc1-5b8d-4939-9fd9-383865b9484d-istiod-ca-cert\") pod \"77989bc1-5b8d-4939-9fd9-383865b9484d\" (UID: \"77989bc1-5b8d-4939-9fd9-383865b9484d\") " Nov 03 13:53:14.548384 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.548253 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70ed2a89-8231-4e71-862a-02968573d4fe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70ed2a89-8231-4e71-862a-02968573d4fe-istio-envoy\") pod \"70ed2a89-8231-4e71-862a-02968573d4fe\" (UID: \"70ed2a89-8231-4e71-862a-02968573d4fe\") " Nov 03 13:53:14.548384 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.548280 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0b95b7f2-8b31-487f-ba23-b551e9406cc8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b95b7f2-8b31-487f-ba23-b551e9406cc8-istio-podinfo\") pod \"0b95b7f2-8b31-487f-ba23-b551e9406cc8\" (UID: \"0b95b7f2-8b31-487f-ba23-b551e9406cc8\") " Nov 03 13:53:14.548384 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.548327 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70ed2a89-8231-4e71-862a-02968573d4fe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70ed2a89-8231-4e71-862a-02968573d4fe-istio-data\") pod \"70ed2a89-8231-4e71-862a-02968573d4fe\" (UID: \"70ed2a89-8231-4e71-862a-02968573d4fe\") " Nov 03 13:53:14.548384 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.548354 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79ff3314-693d-415c-a02d-b249b9a4748c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79ff3314-693d-415c-a02d-b249b9a4748c-istiod-ca-cert\") pod \"79ff3314-693d-415c-a02d-b249b9a4748c\" (UID: \"79ff3314-693d-415c-a02d-b249b9a4748c\") " Nov 03 13:53:14.548766 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.548386 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70ed2a89-8231-4e71-862a-02968573d4fe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70ed2a89-8231-4e71-862a-02968573d4fe-workload-certs\") pod \"70ed2a89-8231-4e71-862a-02968573d4fe\" (UID: \"70ed2a89-8231-4e71-862a-02968573d4fe\") " Nov 03 13:53:14.548766 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.548423 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b95b7f2-8b31-487f-ba23-b551e9406cc8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b95b7f2-8b31-487f-ba23-b551e9406cc8-credential-socket\") pod \"0b95b7f2-8b31-487f-ba23-b551e9406cc8\" (UID: \"0b95b7f2-8b31-487f-ba23-b551e9406cc8\") " Nov 03 13:53:14.548766 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.548458 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79ff3314-693d-415c-a02d-b249b9a4748c-kube-api-access-g6g4n\" (UniqueName: \"kubernetes.io/projected/79ff3314-693d-415c-a02d-b249b9a4748c-kube-api-access-g6g4n\") pod \"79ff3314-693d-415c-a02d-b249b9a4748c\" (UID: \"79ff3314-693d-415c-a02d-b249b9a4748c\") " Nov 03 13:53:14.548766 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.548506 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70ed2a89-8231-4e71-862a-02968573d4fe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70ed2a89-8231-4e71-862a-02968573d4fe-credential-socket\") pod \"70ed2a89-8231-4e71-862a-02968573d4fe\" (UID: \"70ed2a89-8231-4e71-862a-02968573d4fe\") " Nov 03 13:53:14.548766 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.548533 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b95b7f2-8b31-487f-ba23-b551e9406cc8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b95b7f2-8b31-487f-ba23-b551e9406cc8-istio-envoy\") pod \"0b95b7f2-8b31-487f-ba23-b551e9406cc8\" (UID: \"0b95b7f2-8b31-487f-ba23-b551e9406cc8\") " Nov 03 13:53:14.548766 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.548548 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b95b7f2-8b31-487f-ba23-b551e9406cc8-istiod-ca-cert" pod "0b95b7f2-8b31-487f-ba23-b551e9406cc8" (UID: "0b95b7f2-8b31-487f-ba23-b551e9406cc8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:53:14.548766 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.548581 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79ff3314-693d-415c-a02d-b249b9a4748c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ff3314-693d-415c-a02d-b249b9a4748c-workload-socket\") pod \"79ff3314-693d-415c-a02d-b249b9a4748c\" (UID: \"79ff3314-693d-415c-a02d-b249b9a4748c\") " Nov 03 13:53:14.548766 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.548663 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79ff3314-693d-415c-a02d-b249b9a4748c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ff3314-693d-415c-a02d-b249b9a4748c-credential-socket\") pod \"79ff3314-693d-415c-a02d-b249b9a4748c\" (UID: \"79ff3314-693d-415c-a02d-b249b9a4748c\") " Nov 03 13:53:14.548766 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.548702 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77989bc1-5b8d-4939-9fd9-383865b9484d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77989bc1-5b8d-4939-9fd9-383865b9484d-workload-certs\") pod \"77989bc1-5b8d-4939-9fd9-383865b9484d\" (UID: \"77989bc1-5b8d-4939-9fd9-383865b9484d\") " Nov 03 13:53:14.548766 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.548733 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b95b7f2-8b31-487f-ba23-b551e9406cc8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b95b7f2-8b31-487f-ba23-b551e9406cc8-workload-socket\") pod \"0b95b7f2-8b31-487f-ba23-b551e9406cc8\" (UID: \"0b95b7f2-8b31-487f-ba23-b551e9406cc8\") " Nov 03 13:53:14.549270 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.548789 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79ff3314-693d-415c-a02d-b249b9a4748c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79ff3314-693d-415c-a02d-b249b9a4748c-istio-podinfo\") pod \"79ff3314-693d-415c-a02d-b249b9a4748c\" (UID: \"79ff3314-693d-415c-a02d-b249b9a4748c\") " Nov 03 13:53:14.549270 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.548813 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b95b7f2-8b31-487f-ba23-b551e9406cc8-credential-socket" pod "0b95b7f2-8b31-487f-ba23-b551e9406cc8" (UID: "0b95b7f2-8b31-487f-ba23-b551e9406cc8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:53:14.549270 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.548823 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b95b7f2-8b31-487f-ba23-b551e9406cc8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b95b7f2-8b31-487f-ba23-b551e9406cc8-workload-certs\") pod \"0b95b7f2-8b31-487f-ba23-b551e9406cc8\" (UID: \"0b95b7f2-8b31-487f-ba23-b551e9406cc8\") " Nov 03 13:53:14.549270 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.548868 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79ff3314-693d-415c-a02d-b249b9a4748c-istio-token\" (UniqueName: \"kubernetes.io/projected/79ff3314-693d-415c-a02d-b249b9a4748c-istio-token\") pod \"79ff3314-693d-415c-a02d-b249b9a4748c\" (UID: \"79ff3314-693d-415c-a02d-b249b9a4748c\") " Nov 03 13:53:14.549270 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.548900 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b95b7f2-8b31-487f-ba23-b551e9406cc8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b95b7f2-8b31-487f-ba23-b551e9406cc8-istio-data\") pod \"0b95b7f2-8b31-487f-ba23-b551e9406cc8\" (UID: \"0b95b7f2-8b31-487f-ba23-b551e9406cc8\") " Nov 03 13:53:14.549270 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.548934 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79ff3314-693d-415c-a02d-b249b9a4748c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79ff3314-693d-415c-a02d-b249b9a4748c-istio-envoy\") pod \"79ff3314-693d-415c-a02d-b249b9a4748c\" (UID: \"79ff3314-693d-415c-a02d-b249b9a4748c\") " Nov 03 13:53:14.549270 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.548993 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70ed2a89-8231-4e71-862a-02968573d4fe-istio-data" pod "70ed2a89-8231-4e71-862a-02968573d4fe" (UID: "70ed2a89-8231-4e71-862a-02968573d4fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:53:14.549270 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.548995 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70ed2a89-8231-4e71-862a-02968573d4fe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70ed2a89-8231-4e71-862a-02968573d4fe-istiod-ca-cert\") pod \"70ed2a89-8231-4e71-862a-02968573d4fe\" (UID: \"70ed2a89-8231-4e71-862a-02968573d4fe\") " Nov 03 13:53:14.549270 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.549051 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77989bc1-5b8d-4939-9fd9-383865b9484d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77989bc1-5b8d-4939-9fd9-383865b9484d-istio-envoy\") pod \"77989bc1-5b8d-4939-9fd9-383865b9484d\" (UID: \"77989bc1-5b8d-4939-9fd9-383865b9484d\") " Nov 03 13:53:14.549270 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.549115 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77989bc1-5b8d-4939-9fd9-383865b9484d-istio-token\" (UniqueName: \"kubernetes.io/projected/77989bc1-5b8d-4939-9fd9-383865b9484d-istio-token\") pod \"77989bc1-5b8d-4939-9fd9-383865b9484d\" (UID: \"77989bc1-5b8d-4939-9fd9-383865b9484d\") " Nov 03 13:53:14.549270 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.549142 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79ff3314-693d-415c-a02d-b249b9a4748c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79ff3314-693d-415c-a02d-b249b9a4748c-istio-data\") pod \"79ff3314-693d-415c-a02d-b249b9a4748c\" (UID: \"79ff3314-693d-415c-a02d-b249b9a4748c\") " Nov 03 13:53:14.549270 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.549185 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70ed2a89-8231-4e71-862a-02968573d4fe-istio-token\" (UniqueName: \"kubernetes.io/projected/70ed2a89-8231-4e71-862a-02968573d4fe-istio-token\") pod \"70ed2a89-8231-4e71-862a-02968573d4fe\" (UID: \"70ed2a89-8231-4e71-862a-02968573d4fe\") " Nov 03 13:53:14.549270 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.549257 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70ed2a89-8231-4e71-862a-02968573d4fe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70ed2a89-8231-4e71-862a-02968573d4fe-workload-socket\") pod \"70ed2a89-8231-4e71-862a-02968573d4fe\" (UID: \"70ed2a89-8231-4e71-862a-02968573d4fe\") " Nov 03 13:53:14.551179 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.549305 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77989bc1-5b8d-4939-9fd9-383865b9484d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77989bc1-5b8d-4939-9fd9-383865b9484d-istio-data\") pod \"77989bc1-5b8d-4939-9fd9-383865b9484d\" (UID: \"77989bc1-5b8d-4939-9fd9-383865b9484d\") " Nov 03 13:53:14.551179 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.549340 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70ed2a89-8231-4e71-862a-02968573d4fe-istiod-ca-cert" pod "70ed2a89-8231-4e71-862a-02968573d4fe" (UID: "70ed2a89-8231-4e71-862a-02968573d4fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:53:14.551179 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.550237 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b95b7f2-8b31-487f-ba23-b551e9406cc8-workload-socket" pod "0b95b7f2-8b31-487f-ba23-b551e9406cc8" (UID: "0b95b7f2-8b31-487f-ba23-b551e9406cc8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:53:14.551179 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.550363 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77989bc1-5b8d-4939-9fd9-383865b9484d-istiod-ca-cert" pod "77989bc1-5b8d-4939-9fd9-383865b9484d" (UID: "77989bc1-5b8d-4939-9fd9-383865b9484d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:53:14.551179 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.550454 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70ed2a89-8231-4e71-862a-02968573d4fe-credential-socket" pod "70ed2a89-8231-4e71-862a-02968573d4fe" (UID: "70ed2a89-8231-4e71-862a-02968573d4fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:53:14.551179 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.551152 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79ff3314-693d-415c-a02d-b249b9a4748c-credential-socket" pod "79ff3314-693d-415c-a02d-b249b9a4748c" (UID: "79ff3314-693d-415c-a02d-b249b9a4748c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:53:14.551553 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.551357 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79ff3314-693d-415c-a02d-b249b9a4748c-workload-socket" pod "79ff3314-693d-415c-a02d-b249b9a4748c" (UID: "79ff3314-693d-415c-a02d-b249b9a4748c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:53:14.551553 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.551421 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b95b7f2-8b31-487f-ba23-b551e9406cc8-istio-podinfo" pod "0b95b7f2-8b31-487f-ba23-b551e9406cc8" (UID: "0b95b7f2-8b31-487f-ba23-b551e9406cc8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:53:14.551682 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.551651 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70ed2a89-8231-4e71-862a-02968573d4fe-workload-certs" pod "70ed2a89-8231-4e71-862a-02968573d4fe" (UID: "70ed2a89-8231-4e71-862a-02968573d4fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:53:14.551748 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.551727 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77989bc1-5b8d-4939-9fd9-383865b9484d-kube-api-access-hmwl4\" (UniqueName: \"kubernetes.io/projected/77989bc1-5b8d-4939-9fd9-383865b9484d-kube-api-access-hmwl4\") pod \"77989bc1-5b8d-4939-9fd9-383865b9484d\" (UID: \"77989bc1-5b8d-4939-9fd9-383865b9484d\") " Nov 03 13:53:14.551824 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.551787 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79ff3314-693d-415c-a02d-b249b9a4748c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79ff3314-693d-415c-a02d-b249b9a4748c-workload-certs\") pod \"79ff3314-693d-415c-a02d-b249b9a4748c\" (UID: \"79ff3314-693d-415c-a02d-b249b9a4748c\") " Nov 03 13:53:14.551892 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.551857 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77989bc1-5b8d-4939-9fd9-383865b9484d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77989bc1-5b8d-4939-9fd9-383865b9484d-workload-socket\") pod \"77989bc1-5b8d-4939-9fd9-383865b9484d\" (UID: \"77989bc1-5b8d-4939-9fd9-383865b9484d\") " Nov 03 13:53:14.551965 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.551942 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79ff3314-693d-415c-a02d-b249b9a4748c-istiod-ca-cert" pod "79ff3314-693d-415c-a02d-b249b9a4748c" (UID: "79ff3314-693d-415c-a02d-b249b9a4748c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:53:14.552169 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.551964 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77989bc1-5b8d-4939-9fd9-383865b9484d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77989bc1-5b8d-4939-9fd9-383865b9484d-credential-socket\") pod \"77989bc1-5b8d-4939-9fd9-383865b9484d\" (UID: \"77989bc1-5b8d-4939-9fd9-383865b9484d\") " Nov 03 13:53:14.552169 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.552059 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b95b7f2-8b31-487f-ba23-b551e9406cc8-istio-token\" (UniqueName: \"kubernetes.io/projected/0b95b7f2-8b31-487f-ba23-b551e9406cc8-istio-token\") pod \"0b95b7f2-8b31-487f-ba23-b551e9406cc8\" (UID: \"0b95b7f2-8b31-487f-ba23-b551e9406cc8\") " Nov 03 13:53:14.552169 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.552123 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b95b7f2-8b31-487f-ba23-b551e9406cc8-kube-api-access-7dm58\" (UniqueName: \"kubernetes.io/projected/0b95b7f2-8b31-487f-ba23-b551e9406cc8-kube-api-access-7dm58\") pod \"0b95b7f2-8b31-487f-ba23-b551e9406cc8\" (UID: \"0b95b7f2-8b31-487f-ba23-b551e9406cc8\") " Nov 03 13:53:14.552401 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.552288 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b95b7f2-8b31-487f-ba23-b551e9406cc8-workload-certs" pod "0b95b7f2-8b31-487f-ba23-b551e9406cc8" (UID: "0b95b7f2-8b31-487f-ba23-b551e9406cc8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:53:14.552401 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.552209 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79ff3314-693d-415c-a02d-b249b9a4748c-kube-api-access-g6g4n" pod "79ff3314-693d-415c-a02d-b249b9a4748c" (UID: "79ff3314-693d-415c-a02d-b249b9a4748c"). InnerVolumeSpecName "kube-api-access-g6g4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:53:14.552548 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.552493 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70ed2a89-8231-4e71-862a-02968573d4fe-kube-api-access-sjg2b" pod "70ed2a89-8231-4e71-862a-02968573d4fe" (UID: "70ed2a89-8231-4e71-862a-02968573d4fe"). InnerVolumeSpecName "kube-api-access-sjg2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:53:14.552964 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.552898 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjg2b\" (UniqueName: \"kubernetes.io/projected/70ed2a89-8231-4e71-862a-02968573d4fe-kube-api-access-sjg2b\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:14.553142 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.552991 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b95b7f2-8b31-487f-ba23-b551e9406cc8-istio-data" pod "0b95b7f2-8b31-487f-ba23-b551e9406cc8" (UID: "0b95b7f2-8b31-487f-ba23-b551e9406cc8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:53:14.553737 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.552926 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b95b7f2-8b31-487f-ba23-b551e9406cc8-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:14.553737 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.553258 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77989bc1-5b8d-4939-9fd9-383865b9484d-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:14.553737 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.553273 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b95b7f2-8b31-487f-ba23-b551e9406cc8-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:14.553737 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.553289 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70ed2a89-8231-4e71-862a-02968573d4fe-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:14.553737 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.553303 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79ff3314-693d-415c-a02d-b249b9a4748c-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:14.553737 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.553317 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70ed2a89-8231-4e71-862a-02968573d4fe-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:14.553737 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.553332 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b95b7f2-8b31-487f-ba23-b551e9406cc8-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:14.553737 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.553345 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6g4n\" (UniqueName: \"kubernetes.io/projected/79ff3314-693d-415c-a02d-b249b9a4748c-kube-api-access-g6g4n\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:14.553737 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.553359 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70ed2a89-8231-4e71-862a-02968573d4fe-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:14.553737 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.553373 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ff3314-693d-415c-a02d-b249b9a4748c-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:14.553737 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.553387 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ff3314-693d-415c-a02d-b249b9a4748c-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:14.553737 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.553401 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b95b7f2-8b31-487f-ba23-b551e9406cc8-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:14.553737 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.553415 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b95b7f2-8b31-487f-ba23-b551e9406cc8-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:14.553737 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.553430 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70ed2a89-8231-4e71-862a-02968573d4fe-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:14.554645 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.554177 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77989bc1-5b8d-4939-9fd9-383865b9484d-workload-certs" pod "77989bc1-5b8d-4939-9fd9-383865b9484d" (UID: "77989bc1-5b8d-4939-9fd9-383865b9484d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:53:14.554645 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.554427 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77989bc1-5b8d-4939-9fd9-383865b9484d-workload-socket" pod "77989bc1-5b8d-4939-9fd9-383865b9484d" (UID: "77989bc1-5b8d-4939-9fd9-383865b9484d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:53:14.554773 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.554648 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79ff3314-693d-415c-a02d-b249b9a4748c-workload-certs" pod "79ff3314-693d-415c-a02d-b249b9a4748c" (UID: "79ff3314-693d-415c-a02d-b249b9a4748c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:53:14.554835 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.554819 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70ed2a89-8231-4e71-862a-02968573d4fe-workload-socket" pod "70ed2a89-8231-4e71-862a-02968573d4fe" (UID: "70ed2a89-8231-4e71-862a-02968573d4fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:53:14.555066 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.555042 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77989bc1-5b8d-4939-9fd9-383865b9484d-istio-data" pod "77989bc1-5b8d-4939-9fd9-383865b9484d" (UID: "77989bc1-5b8d-4939-9fd9-383865b9484d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:53:14.555565 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.555531 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79ff3314-693d-415c-a02d-b249b9a4748c-istio-data" pod "79ff3314-693d-415c-a02d-b249b9a4748c" (UID: "79ff3314-693d-415c-a02d-b249b9a4748c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:53:14.555669 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.555575 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77989bc1-5b8d-4939-9fd9-383865b9484d-credential-socket" pod "77989bc1-5b8d-4939-9fd9-383865b9484d" (UID: "77989bc1-5b8d-4939-9fd9-383865b9484d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:53:14.563606 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.563563 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77989bc1-5b8d-4939-9fd9-383865b9484d-istio-podinfo" pod "77989bc1-5b8d-4939-9fd9-383865b9484d" (UID: "77989bc1-5b8d-4939-9fd9-383865b9484d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:53:14.564361 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.564335 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79ff3314-693d-415c-a02d-b249b9a4748c-istio-envoy" pod "79ff3314-693d-415c-a02d-b249b9a4748c" (UID: "79ff3314-693d-415c-a02d-b249b9a4748c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:53:14.564968 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.564677 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70ed2a89-8231-4e71-862a-02968573d4fe-istio-token" pod "70ed2a89-8231-4e71-862a-02968573d4fe" (UID: "70ed2a89-8231-4e71-862a-02968573d4fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:53:14.564968 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.564932 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b95b7f2-8b31-487f-ba23-b551e9406cc8-istio-token" pod "0b95b7f2-8b31-487f-ba23-b551e9406cc8" (UID: "0b95b7f2-8b31-487f-ba23-b551e9406cc8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:53:14.566341 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.566251 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79ff3314-693d-415c-a02d-b249b9a4748c-istio-podinfo" pod "79ff3314-693d-415c-a02d-b249b9a4748c" (UID: "79ff3314-693d-415c-a02d-b249b9a4748c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:53:14.566341 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.566273 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/70ed2a89-8231-4e71-862a-02968573d4fe-istio-podinfo" pod "70ed2a89-8231-4e71-862a-02968573d4fe" (UID: "70ed2a89-8231-4e71-862a-02968573d4fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:53:14.566341 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.566278 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b95b7f2-8b31-487f-ba23-b551e9406cc8-istio-envoy" pod "0b95b7f2-8b31-487f-ba23-b551e9406cc8" (UID: "0b95b7f2-8b31-487f-ba23-b551e9406cc8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:53:14.566341 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.566327 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79ff3314-693d-415c-a02d-b249b9a4748c-istio-token" pod "79ff3314-693d-415c-a02d-b249b9a4748c" (UID: "79ff3314-693d-415c-a02d-b249b9a4748c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:53:14.566341 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.566329 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70ed2a89-8231-4e71-862a-02968573d4fe-istio-envoy" pod "70ed2a89-8231-4e71-862a-02968573d4fe" (UID: "70ed2a89-8231-4e71-862a-02968573d4fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:53:14.566647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.566422 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77989bc1-5b8d-4939-9fd9-383865b9484d-istio-token" pod "77989bc1-5b8d-4939-9fd9-383865b9484d" (UID: "77989bc1-5b8d-4939-9fd9-383865b9484d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:53:14.566647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.566466 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77989bc1-5b8d-4939-9fd9-383865b9484d-istio-envoy" pod "77989bc1-5b8d-4939-9fd9-383865b9484d" (UID: "77989bc1-5b8d-4939-9fd9-383865b9484d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:53:14.566647 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.566616 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77989bc1-5b8d-4939-9fd9-383865b9484d-kube-api-access-hmwl4" pod "77989bc1-5b8d-4939-9fd9-383865b9484d" (UID: "77989bc1-5b8d-4939-9fd9-383865b9484d"). InnerVolumeSpecName "kube-api-access-hmwl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:53:14.568001 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.567982 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b95b7f2-8b31-487f-ba23-b551e9406cc8-kube-api-access-7dm58" pod "0b95b7f2-8b31-487f-ba23-b551e9406cc8" (UID: "0b95b7f2-8b31-487f-ba23-b551e9406cc8"). InnerVolumeSpecName "kube-api-access-7dm58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:53:14.654936 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.654903 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77989bc1-5b8d-4939-9fd9-383865b9484d-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:14.654936 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.654935 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70ed2a89-8231-4e71-862a-02968573d4fe-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:14.654936 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.654944 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70ed2a89-8231-4e71-862a-02968573d4fe-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:14.655231 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.654953 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b95b7f2-8b31-487f-ba23-b551e9406cc8-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:14.655231 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.654962 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77989bc1-5b8d-4939-9fd9-383865b9484d-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:14.655231 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.654974 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79ff3314-693d-415c-a02d-b249b9a4748c-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:14.655231 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.654987 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79ff3314-693d-415c-a02d-b249b9a4748c-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:14.655231 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.654999 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b95b7f2-8b31-487f-ba23-b551e9406cc8-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:14.655231 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.655012 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79ff3314-693d-415c-a02d-b249b9a4748c-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:14.655231 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.655021 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77989bc1-5b8d-4939-9fd9-383865b9484d-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:14.655231 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.655032 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77989bc1-5b8d-4939-9fd9-383865b9484d-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:14.655231 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.655047 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79ff3314-693d-415c-a02d-b249b9a4748c-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:14.655231 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.655060 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70ed2a89-8231-4e71-862a-02968573d4fe-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:14.655231 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.655074 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70ed2a89-8231-4e71-862a-02968573d4fe-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:14.655231 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.655086 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77989bc1-5b8d-4939-9fd9-383865b9484d-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:14.655231 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.655099 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmwl4\" (UniqueName: \"kubernetes.io/projected/77989bc1-5b8d-4939-9fd9-383865b9484d-kube-api-access-hmwl4\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:14.655231 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.655113 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79ff3314-693d-415c-a02d-b249b9a4748c-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:14.655231 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.655126 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77989bc1-5b8d-4939-9fd9-383865b9484d-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:14.655231 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.655140 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77989bc1-5b8d-4939-9fd9-383865b9484d-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:14.655231 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.655156 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b95b7f2-8b31-487f-ba23-b551e9406cc8-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:14.655231 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:14.655169 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dm58\" (UniqueName: \"kubernetes.io/projected/0b95b7f2-8b31-487f-ba23-b551e9406cc8-kube-api-access-7dm58\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:15.069496 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:15.069453 5979 generic.go:356] "Generic (PLEG): container finished" podID="79ff3314-693d-415c-a02d-b249b9a4748c" containerID="10ed8dfb524cd3c7fc0f1152890c143cbfa4391bc0a4eea1b19c1441278fd45b" exitCode=0 Nov 03 13:53:15.069626 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:15.069510 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m" Nov 03 13:53:15.069626 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:15.069514 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m" event={"ID":"79ff3314-693d-415c-a02d-b249b9a4748c","Type":"ContainerDied","Data":"10ed8dfb524cd3c7fc0f1152890c143cbfa4391bc0a4eea1b19c1441278fd45b"} Nov 03 13:53:15.069626 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:15.069572 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m" event={"ID":"79ff3314-693d-415c-a02d-b249b9a4748c","Type":"ContainerDied","Data":"b0ea5c9c8037065247b73345063d0c99212009ab966bbf7c9d84f1b074471c81"} Nov 03 13:53:15.069626 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:15.069601 5979 scope.go:122] "RemoveContainer" containerID="10ed8dfb524cd3c7fc0f1152890c143cbfa4391bc0a4eea1b19c1441278fd45b" Nov 03 13:53:15.071399 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:15.071375 5979 generic.go:356] "Generic (PLEG): container finished" podID="70ed2a89-8231-4e71-862a-02968573d4fe" containerID="ff2997e72260c1176d52a45923ef02aabeb3a6cc98d253876911eab57073374b" exitCode=0 Nov 03 13:53:15.071610 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:15.071420 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz" Nov 03 13:53:15.071828 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:15.071792 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz" event={"ID":"70ed2a89-8231-4e71-862a-02968573d4fe","Type":"ContainerDied","Data":"ff2997e72260c1176d52a45923ef02aabeb3a6cc98d253876911eab57073374b"} Nov 03 13:53:15.072042 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:15.072021 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz" event={"ID":"70ed2a89-8231-4e71-862a-02968573d4fe","Type":"ContainerDied","Data":"cf9c4ab20df8907ce397820809e836ee8cd31d574e198c5d79cd6cf37e472308"} Nov 03 13:53:15.073287 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:15.073265 5979 generic.go:356] "Generic (PLEG): container finished" podID="0b95b7f2-8b31-487f-ba23-b551e9406cc8" containerID="f22122c5534f610328f6f44d3bf0d6b4649d8e91dee4b81e038142277ffb2409" exitCode=0 Nov 03 13:53:15.073409 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:15.073323 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz" Nov 03 13:53:15.073409 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:15.073397 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz" event={"ID":"0b95b7f2-8b31-487f-ba23-b551e9406cc8","Type":"ContainerDied","Data":"f22122c5534f610328f6f44d3bf0d6b4649d8e91dee4b81e038142277ffb2409"} Nov 03 13:53:15.073560 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:15.073438 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz" event={"ID":"0b95b7f2-8b31-487f-ba23-b551e9406cc8","Type":"ContainerDied","Data":"fafe846d5d1cee428d221f58dc5929f8415cf9cbb5acbb75d917c2ef22b83939"} Nov 03 13:53:15.075438 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:15.075417 5979 generic.go:356] "Generic (PLEG): container finished" podID="77989bc1-5b8d-4939-9fd9-383865b9484d" containerID="ceeaee78fe0d459f4a1496aa067c3406667e6139f71287a7654613a676672289" exitCode=0 Nov 03 13:53:15.075560 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:15.075447 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9" event={"ID":"77989bc1-5b8d-4939-9fd9-383865b9484d","Type":"ContainerDied","Data":"ceeaee78fe0d459f4a1496aa067c3406667e6139f71287a7654613a676672289"} Nov 03 13:53:15.075560 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:15.075510 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9" event={"ID":"77989bc1-5b8d-4939-9fd9-383865b9484d","Type":"ContainerDied","Data":"c8d8aea498a6a74d203b369a495501d76d69d1dd03894df563003fa81e8a1287"} Nov 03 13:53:15.075560 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:15.075525 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9" Nov 03 13:53:15.090967 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:15.090950 5979 scope.go:122] "RemoveContainer" containerID="10ed8dfb524cd3c7fc0f1152890c143cbfa4391bc0a4eea1b19c1441278fd45b" Nov 03 13:53:15.091254 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:53:15.091226 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10ed8dfb524cd3c7fc0f1152890c143cbfa4391bc0a4eea1b19c1441278fd45b\": container with ID starting with 10ed8dfb524cd3c7fc0f1152890c143cbfa4391bc0a4eea1b19c1441278fd45b not found: ID does not exist" containerID="10ed8dfb524cd3c7fc0f1152890c143cbfa4391bc0a4eea1b19c1441278fd45b" Nov 03 13:53:15.091340 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:15.091265 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10ed8dfb524cd3c7fc0f1152890c143cbfa4391bc0a4eea1b19c1441278fd45b"} err="failed to get container status \"10ed8dfb524cd3c7fc0f1152890c143cbfa4391bc0a4eea1b19c1441278fd45b\": rpc error: code = NotFound desc = could not find container \"10ed8dfb524cd3c7fc0f1152890c143cbfa4391bc0a4eea1b19c1441278fd45b\": container with ID starting with 10ed8dfb524cd3c7fc0f1152890c143cbfa4391bc0a4eea1b19c1441278fd45b not found: ID does not exist" Nov 03 13:53:15.091340 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:15.091285 5979 scope.go:122] "RemoveContainer" containerID="ff2997e72260c1176d52a45923ef02aabeb3a6cc98d253876911eab57073374b" Nov 03 13:53:15.100544 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:15.100526 5979 scope.go:122] "RemoveContainer" containerID="ff2997e72260c1176d52a45923ef02aabeb3a6cc98d253876911eab57073374b" Nov 03 13:53:15.100809 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:53:15.100786 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff2997e72260c1176d52a45923ef02aabeb3a6cc98d253876911eab57073374b\": container with ID starting with ff2997e72260c1176d52a45923ef02aabeb3a6cc98d253876911eab57073374b not found: ID does not exist" containerID="ff2997e72260c1176d52a45923ef02aabeb3a6cc98d253876911eab57073374b" Nov 03 13:53:15.100861 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:15.100816 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff2997e72260c1176d52a45923ef02aabeb3a6cc98d253876911eab57073374b"} err="failed to get container status \"ff2997e72260c1176d52a45923ef02aabeb3a6cc98d253876911eab57073374b\": rpc error: code = NotFound desc = could not find container \"ff2997e72260c1176d52a45923ef02aabeb3a6cc98d253876911eab57073374b\": container with ID starting with ff2997e72260c1176d52a45923ef02aabeb3a6cc98d253876911eab57073374b not found: ID does not exist" Nov 03 13:53:15.100861 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:15.100834 5979 scope.go:122] "RemoveContainer" containerID="f22122c5534f610328f6f44d3bf0d6b4649d8e91dee4b81e038142277ffb2409" Nov 03 13:53:15.109989 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:15.109963 5979 scope.go:122] "RemoveContainer" containerID="f22122c5534f610328f6f44d3bf0d6b4649d8e91dee4b81e038142277ffb2409" Nov 03 13:53:15.110245 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:53:15.110227 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f22122c5534f610328f6f44d3bf0d6b4649d8e91dee4b81e038142277ffb2409\": container with ID starting with f22122c5534f610328f6f44d3bf0d6b4649d8e91dee4b81e038142277ffb2409 not found: ID does not exist" containerID="f22122c5534f610328f6f44d3bf0d6b4649d8e91dee4b81e038142277ffb2409" Nov 03 13:53:15.110295 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:15.110251 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f22122c5534f610328f6f44d3bf0d6b4649d8e91dee4b81e038142277ffb2409"} err="failed to get container status \"f22122c5534f610328f6f44d3bf0d6b4649d8e91dee4b81e038142277ffb2409\": rpc error: code = NotFound desc = could not find container \"f22122c5534f610328f6f44d3bf0d6b4649d8e91dee4b81e038142277ffb2409\": container with ID starting with f22122c5534f610328f6f44d3bf0d6b4649d8e91dee4b81e038142277ffb2409 not found: ID does not exist" Nov 03 13:53:15.110295 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:15.110265 5979 scope.go:122] "RemoveContainer" containerID="ceeaee78fe0d459f4a1496aa067c3406667e6139f71287a7654613a676672289" Nov 03 13:53:15.119376 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:15.119346 5979 scope.go:122] "RemoveContainer" containerID="ceeaee78fe0d459f4a1496aa067c3406667e6139f71287a7654613a676672289" Nov 03 13:53:15.119731 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:53:15.119710 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ceeaee78fe0d459f4a1496aa067c3406667e6139f71287a7654613a676672289\": container with ID starting with ceeaee78fe0d459f4a1496aa067c3406667e6139f71287a7654613a676672289 not found: ID does not exist" containerID="ceeaee78fe0d459f4a1496aa067c3406667e6139f71287a7654613a676672289" Nov 03 13:53:15.119791 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:15.119743 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ceeaee78fe0d459f4a1496aa067c3406667e6139f71287a7654613a676672289"} err="failed to get container status \"ceeaee78fe0d459f4a1496aa067c3406667e6139f71287a7654613a676672289\": rpc error: code = NotFound desc = could not find container \"ceeaee78fe0d459f4a1496aa067c3406667e6139f71287a7654613a676672289\": container with ID starting with ceeaee78fe0d459f4a1496aa067c3406667e6139f71287a7654613a676672289 not found: ID does not exist" Nov 03 13:53:15.225913 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:15.225837 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz"] Nov 03 13:53:15.229341 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:15.229315 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-7lrqz"] Nov 03 13:53:15.331692 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:15.331607 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz"] Nov 03 13:53:15.336785 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:15.336756 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-zrzfz"] Nov 03 13:53:15.439002 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:15.438968 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9"] Nov 03 13:53:15.445738 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:15.445703 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-lnct9"] Nov 03 13:53:15.547746 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:15.547703 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m"] Nov 03 13:53:15.551907 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:15.551880 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-95s2m"] Nov 03 13:53:16.115551 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:16.115523 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b95b7f2-8b31-487f-ba23-b551e9406cc8" path="/var/lib/kubelet/pods/0b95b7f2-8b31-487f-ba23-b551e9406cc8/volumes" Nov 03 13:53:16.116021 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:16.116007 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70ed2a89-8231-4e71-862a-02968573d4fe" path="/var/lib/kubelet/pods/70ed2a89-8231-4e71-862a-02968573d4fe/volumes" Nov 03 13:53:16.116440 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:16.116428 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77989bc1-5b8d-4939-9fd9-383865b9484d" path="/var/lib/kubelet/pods/77989bc1-5b8d-4939-9fd9-383865b9484d/volumes" Nov 03 13:53:16.117183 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:16.117166 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79ff3314-693d-415c-a02d-b249b9a4748c" path="/var/lib/kubelet/pods/79ff3314-693d-415c-a02d-b249b9a4748c/volumes" Nov 03 13:53:17.046109 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:17.046079 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-261647f7-7286h"] Nov 03 13:53:17.092390 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:17.092358 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-261647f7-7286h"] Nov 03 13:53:18.117225 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:18.117192 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54920681-626a-41cc-8a46-d0b3c13237be" path="/var/lib/kubelet/pods/54920681-626a-41cc-8a46-d0b3c13237be/volumes" Nov 03 13:53:20.278582 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:20.278547 5979 scope.go:122] "RemoveContainer" containerID="7af69ad90395a627d7c34ae8e167449836759f918f045317c6f117efba44edfb" Nov 03 13:53:20.288305 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:20.288282 5979 scope.go:122] "RemoveContainer" containerID="96b70c6dbf0a8a6ef489e62e555fb0ab4454210809c0dd94be7540845cd9382b" Nov 03 13:53:20.297263 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:20.297238 5979 scope.go:122] "RemoveContainer" containerID="75e7ea3cb7180751c87665b504b7eff8d8d831b85e38dbf2e33bbc8bd1b350e9" Nov 03 13:53:20.306349 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:20.306325 5979 scope.go:122] "RemoveContainer" containerID="1f883628bc15b1215fad464d415fd5be2996c1dd7548e09417a424026d9aa331" Nov 03 13:53:20.314837 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:20.314810 5979 scope.go:122] "RemoveContainer" containerID="a5058b5b5c261d8040f2bda6f9b8f8bf59483bde1ec1afb4bb753ffbbd3ce830" Nov 03 13:53:21.057881 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:21.057850 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t"] Nov 03 13:53:21.058140 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:21.058101 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t" podUID="7512d231-097f-4192-99f7-44bb0f98bdda" containerName="istio-proxy" containerID="cri-o://154c304c370c17c8c81c1e288e693e21a4591eb3d5015f2087e536d9b015952d" gracePeriod=30 Nov 03 13:53:26.344402 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:26.344373 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t" Nov 03 13:53:26.451048 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:26.451013 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7512d231-097f-4192-99f7-44bb0f98bdda-istio-token\" (UniqueName: \"kubernetes.io/projected/7512d231-097f-4192-99f7-44bb0f98bdda-istio-token\") pod \"7512d231-097f-4192-99f7-44bb0f98bdda\" (UID: \"7512d231-097f-4192-99f7-44bb0f98bdda\") " Nov 03 13:53:26.451211 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:26.451064 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7512d231-097f-4192-99f7-44bb0f98bdda-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7512d231-097f-4192-99f7-44bb0f98bdda-credential-socket\") pod \"7512d231-097f-4192-99f7-44bb0f98bdda\" (UID: \"7512d231-097f-4192-99f7-44bb0f98bdda\") " Nov 03 13:53:26.451211 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:26.451089 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7512d231-097f-4192-99f7-44bb0f98bdda-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7512d231-097f-4192-99f7-44bb0f98bdda-workload-socket\") pod \"7512d231-097f-4192-99f7-44bb0f98bdda\" (UID: \"7512d231-097f-4192-99f7-44bb0f98bdda\") " Nov 03 13:53:26.451211 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:26.451157 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7512d231-097f-4192-99f7-44bb0f98bdda-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7512d231-097f-4192-99f7-44bb0f98bdda-istio-envoy\") pod \"7512d231-097f-4192-99f7-44bb0f98bdda\" (UID: \"7512d231-097f-4192-99f7-44bb0f98bdda\") " Nov 03 13:53:26.451211 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:26.451208 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7512d231-097f-4192-99f7-44bb0f98bdda-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7512d231-097f-4192-99f7-44bb0f98bdda-workload-certs\") pod \"7512d231-097f-4192-99f7-44bb0f98bdda\" (UID: \"7512d231-097f-4192-99f7-44bb0f98bdda\") " Nov 03 13:53:26.451457 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:26.451276 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7512d231-097f-4192-99f7-44bb0f98bdda-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7512d231-097f-4192-99f7-44bb0f98bdda-istiod-ca-cert\") pod \"7512d231-097f-4192-99f7-44bb0f98bdda\" (UID: \"7512d231-097f-4192-99f7-44bb0f98bdda\") " Nov 03 13:53:26.451457 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:26.451294 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7512d231-097f-4192-99f7-44bb0f98bdda-kube-api-access-q6wk5\" (UniqueName: \"kubernetes.io/projected/7512d231-097f-4192-99f7-44bb0f98bdda-kube-api-access-q6wk5\") pod \"7512d231-097f-4192-99f7-44bb0f98bdda\" (UID: \"7512d231-097f-4192-99f7-44bb0f98bdda\") " Nov 03 13:53:26.451457 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:26.451338 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7512d231-097f-4192-99f7-44bb0f98bdda-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7512d231-097f-4192-99f7-44bb0f98bdda-istio-data\") pod \"7512d231-097f-4192-99f7-44bb0f98bdda\" (UID: \"7512d231-097f-4192-99f7-44bb0f98bdda\") " Nov 03 13:53:26.451457 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:26.451371 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7512d231-097f-4192-99f7-44bb0f98bdda-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7512d231-097f-4192-99f7-44bb0f98bdda-istio-podinfo\") pod \"7512d231-097f-4192-99f7-44bb0f98bdda\" (UID: \"7512d231-097f-4192-99f7-44bb0f98bdda\") " Nov 03 13:53:26.451457 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:26.451436 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7512d231-097f-4192-99f7-44bb0f98bdda-credential-socket" pod "7512d231-097f-4192-99f7-44bb0f98bdda" (UID: "7512d231-097f-4192-99f7-44bb0f98bdda"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:53:26.451795 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:26.451539 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7512d231-097f-4192-99f7-44bb0f98bdda-workload-socket" pod "7512d231-097f-4192-99f7-44bb0f98bdda" (UID: "7512d231-097f-4192-99f7-44bb0f98bdda"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:53:26.451795 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:26.451667 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7512d231-097f-4192-99f7-44bb0f98bdda-workload-certs" pod "7512d231-097f-4192-99f7-44bb0f98bdda" (UID: "7512d231-097f-4192-99f7-44bb0f98bdda"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:53:26.451795 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:26.451778 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7512d231-097f-4192-99f7-44bb0f98bdda-istio-data" pod "7512d231-097f-4192-99f7-44bb0f98bdda" (UID: "7512d231-097f-4192-99f7-44bb0f98bdda"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:53:26.451991 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:26.451860 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7512d231-097f-4192-99f7-44bb0f98bdda-istiod-ca-cert" pod "7512d231-097f-4192-99f7-44bb0f98bdda" (UID: "7512d231-097f-4192-99f7-44bb0f98bdda"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:53:26.452110 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:26.452086 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7512d231-097f-4192-99f7-44bb0f98bdda-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:26.452110 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:26.452109 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7512d231-097f-4192-99f7-44bb0f98bdda-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:26.452327 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:26.452118 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7512d231-097f-4192-99f7-44bb0f98bdda-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:26.452327 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:26.452127 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7512d231-097f-4192-99f7-44bb0f98bdda-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:26.452327 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:26.452137 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7512d231-097f-4192-99f7-44bb0f98bdda-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:26.453439 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:26.453414 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7512d231-097f-4192-99f7-44bb0f98bdda-istio-token" pod "7512d231-097f-4192-99f7-44bb0f98bdda" (UID: "7512d231-097f-4192-99f7-44bb0f98bdda"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:53:26.453568 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:26.453465 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7512d231-097f-4192-99f7-44bb0f98bdda-istio-envoy" pod "7512d231-097f-4192-99f7-44bb0f98bdda" (UID: "7512d231-097f-4192-99f7-44bb0f98bdda"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:53:26.453738 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:26.453716 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7512d231-097f-4192-99f7-44bb0f98bdda-kube-api-access-q6wk5" pod "7512d231-097f-4192-99f7-44bb0f98bdda" (UID: "7512d231-097f-4192-99f7-44bb0f98bdda"). InnerVolumeSpecName "kube-api-access-q6wk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:53:26.453831 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:26.453723 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7512d231-097f-4192-99f7-44bb0f98bdda-istio-podinfo" pod "7512d231-097f-4192-99f7-44bb0f98bdda" (UID: "7512d231-097f-4192-99f7-44bb0f98bdda"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:53:26.553032 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:26.552997 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7512d231-097f-4192-99f7-44bb0f98bdda-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:26.553032 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:26.553035 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6wk5\" (UniqueName: \"kubernetes.io/projected/7512d231-097f-4192-99f7-44bb0f98bdda-kube-api-access-q6wk5\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:26.553236 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:26.553052 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7512d231-097f-4192-99f7-44bb0f98bdda-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:26.553236 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:26.553067 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7512d231-097f-4192-99f7-44bb0f98bdda-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:27.159344 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:27.159298 5979 generic.go:356] "Generic (PLEG): container finished" podID="7512d231-097f-4192-99f7-44bb0f98bdda" containerID="154c304c370c17c8c81c1e288e693e21a4591eb3d5015f2087e536d9b015952d" exitCode=0 Nov 03 13:53:27.159561 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:27.159358 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t" Nov 03 13:53:27.159561 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:27.159380 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t" event={"ID":"7512d231-097f-4192-99f7-44bb0f98bdda","Type":"ContainerDied","Data":"154c304c370c17c8c81c1e288e693e21a4591eb3d5015f2087e536d9b015952d"} Nov 03 13:53:27.159561 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:27.159446 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t" event={"ID":"7512d231-097f-4192-99f7-44bb0f98bdda","Type":"ContainerDied","Data":"76148efece7439fa0aa6651c91fd8bfa2d5ca4a9e20f7f5126d5c2dc8089a35c"} Nov 03 13:53:27.159561 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:27.159506 5979 scope.go:122] "RemoveContainer" containerID="154c304c370c17c8c81c1e288e693e21a4591eb3d5015f2087e536d9b015952d" Nov 03 13:53:27.173880 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:27.173857 5979 scope.go:122] "RemoveContainer" containerID="599f67b0f10ae1ba88ca3a8777bf623a8dfae54ca11b20e5c4a82fc28b79a7cc" Nov 03 13:53:27.183442 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:27.183282 5979 scope.go:122] "RemoveContainer" containerID="154c304c370c17c8c81c1e288e693e21a4591eb3d5015f2087e536d9b015952d" Nov 03 13:53:27.183578 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:53:27.183559 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"154c304c370c17c8c81c1e288e693e21a4591eb3d5015f2087e536d9b015952d\": container with ID starting with 154c304c370c17c8c81c1e288e693e21a4591eb3d5015f2087e536d9b015952d not found: ID does not exist" containerID="154c304c370c17c8c81c1e288e693e21a4591eb3d5015f2087e536d9b015952d" Nov 03 13:53:27.183620 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:27.183587 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"154c304c370c17c8c81c1e288e693e21a4591eb3d5015f2087e536d9b015952d"} err="failed to get container status \"154c304c370c17c8c81c1e288e693e21a4591eb3d5015f2087e536d9b015952d\": rpc error: code = NotFound desc = could not find container \"154c304c370c17c8c81c1e288e693e21a4591eb3d5015f2087e536d9b015952d\": container with ID starting with 154c304c370c17c8c81c1e288e693e21a4591eb3d5015f2087e536d9b015952d not found: ID does not exist" Nov 03 13:53:27.183620 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:27.183604 5979 scope.go:122] "RemoveContainer" containerID="599f67b0f10ae1ba88ca3a8777bf623a8dfae54ca11b20e5c4a82fc28b79a7cc" Nov 03 13:53:27.183841 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:53:27.183822 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"599f67b0f10ae1ba88ca3a8777bf623a8dfae54ca11b20e5c4a82fc28b79a7cc\": container with ID starting with 599f67b0f10ae1ba88ca3a8777bf623a8dfae54ca11b20e5c4a82fc28b79a7cc not found: ID does not exist" containerID="599f67b0f10ae1ba88ca3a8777bf623a8dfae54ca11b20e5c4a82fc28b79a7cc" Nov 03 13:53:27.183879 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:27.183850 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"599f67b0f10ae1ba88ca3a8777bf623a8dfae54ca11b20e5c4a82fc28b79a7cc"} err="failed to get container status \"599f67b0f10ae1ba88ca3a8777bf623a8dfae54ca11b20e5c4a82fc28b79a7cc\": rpc error: code = NotFound desc = could not find container \"599f67b0f10ae1ba88ca3a8777bf623a8dfae54ca11b20e5c4a82fc28b79a7cc\": container with ID starting with 599f67b0f10ae1ba88ca3a8777bf623a8dfae54ca11b20e5c4a82fc28b79a7cc not found: ID does not exist" Nov 03 13:53:27.314897 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:27.314862 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t"] Nov 03 13:53:27.361889 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:27.361856 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-fb4ca0-openshift-default-656977bdcc-v9h2t"] Nov 03 13:53:28.115901 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:28.115867 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7512d231-097f-4192-99f7-44bb0f98bdda" path="/var/lib/kubelet/pods/7512d231-097f-4192-99f7-44bb0f98bdda/volumes" Nov 03 13:53:32.464905 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:32.464866 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d6af7fa-gwchs"] Nov 03 13:53:32.465440 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:32.465424 5979 cpu_manager.go:414] "containerMap: removing container" podUID="7512d231-097f-4192-99f7-44bb0f98bdda" containerName="istio-proxy" Nov 03 13:53:32.465510 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:32.465442 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7512d231-097f-4192-99f7-44bb0f98bdda" containerName="istio-proxy" Nov 03 13:53:32.465510 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:32.465480 5979 cpu_manager.go:414] "containerMap: removing container" podUID="70ed2a89-8231-4e71-862a-02968573d4fe" containerName="istio-proxy" Nov 03 13:53:32.465510 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:32.465485 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70ed2a89-8231-4e71-862a-02968573d4fe" containerName="istio-proxy" Nov 03 13:53:32.465510 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:32.465495 5979 cpu_manager.go:414] "containerMap: removing container" podUID="79ff3314-693d-415c-a02d-b249b9a4748c" containerName="istio-proxy" Nov 03 13:53:32.465510 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:32.465501 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79ff3314-693d-415c-a02d-b249b9a4748c" containerName="istio-proxy" Nov 03 13:53:32.465682 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:32.465518 5979 cpu_manager.go:414] "containerMap: removing container" podUID="0b95b7f2-8b31-487f-ba23-b551e9406cc8" containerName="istio-proxy" Nov 03 13:53:32.465682 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:32.465523 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b95b7f2-8b31-487f-ba23-b551e9406cc8" containerName="istio-proxy" Nov 03 13:53:32.465682 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:32.465536 5979 cpu_manager.go:414] "containerMap: removing container" podUID="77989bc1-5b8d-4939-9fd9-383865b9484d" containerName="istio-proxy" Nov 03 13:53:32.465682 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:32.465542 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77989bc1-5b8d-4939-9fd9-383865b9484d" containerName="istio-proxy" Nov 03 13:53:32.465682 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:32.465628 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="7512d231-097f-4192-99f7-44bb0f98bdda" containerName="istio-proxy" Nov 03 13:53:32.465682 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:32.465639 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="70ed2a89-8231-4e71-862a-02968573d4fe" containerName="istio-proxy" Nov 03 13:53:32.465682 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:32.465646 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="79ff3314-693d-415c-a02d-b249b9a4748c" containerName="istio-proxy" Nov 03 13:53:32.465682 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:32.465655 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b95b7f2-8b31-487f-ba23-b551e9406cc8" containerName="istio-proxy" Nov 03 13:53:32.465682 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:32.465663 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="7512d231-097f-4192-99f7-44bb0f98bdda" containerName="istio-proxy" Nov 03 13:53:32.465682 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:32.465669 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="77989bc1-5b8d-4939-9fd9-383865b9484d" containerName="istio-proxy" Nov 03 13:53:32.470725 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:32.470703 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d6af7fa-gwchs" Nov 03 13:53:32.521151 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:32.521116 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d6af7fa-gwchs"] Nov 03 13:53:32.564555 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:32.564521 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 13:53:32.564722 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:32.564604 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 13:53:32.628104 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:32.628069 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2be45447-2758-432d-b668-2da835855c14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d6af7fa-gwchs\" (UID: \"2be45447-2758-432d-b668-2da835855c14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d6af7fa-gwchs" Nov 03 13:53:32.628293 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:32.628263 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2be45447-2758-432d-b668-2da835855c14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d6af7fa-gwchs\" (UID: \"2be45447-2758-432d-b668-2da835855c14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d6af7fa-gwchs" Nov 03 13:53:32.628432 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:32.628318 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pvsc\" (UniqueName: \"kubernetes.io/projected/2be45447-2758-432d-b668-2da835855c14-kube-api-access-6pvsc\") pod \"maas-api-revoke-sub-0d6af7fa-gwchs\" (UID: \"2be45447-2758-432d-b668-2da835855c14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d6af7fa-gwchs" Nov 03 13:53:32.730044 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:32.729939 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2be45447-2758-432d-b668-2da835855c14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d6af7fa-gwchs\" (UID: \"2be45447-2758-432d-b668-2da835855c14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d6af7fa-gwchs" Nov 03 13:53:32.730044 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:32.730010 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pvsc\" (UniqueName: \"kubernetes.io/projected/2be45447-2758-432d-b668-2da835855c14-kube-api-access-6pvsc\") pod \"maas-api-revoke-sub-0d6af7fa-gwchs\" (UID: \"2be45447-2758-432d-b668-2da835855c14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d6af7fa-gwchs" Nov 03 13:53:32.730268 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:32.730112 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2be45447-2758-432d-b668-2da835855c14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d6af7fa-gwchs\" (UID: \"2be45447-2758-432d-b668-2da835855c14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d6af7fa-gwchs" Nov 03 13:53:32.730638 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:32.730617 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2be45447-2758-432d-b668-2da835855c14-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d6af7fa-gwchs\" (UID: \"2be45447-2758-432d-b668-2da835855c14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d6af7fa-gwchs" Nov 03 13:53:32.732520 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:32.732502 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2be45447-2758-432d-b668-2da835855c14-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d6af7fa-gwchs\" (UID: \"2be45447-2758-432d-b668-2da835855c14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d6af7fa-gwchs" Nov 03 13:53:32.800447 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:32.800411 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pvsc\" (UniqueName: \"kubernetes.io/projected/2be45447-2758-432d-b668-2da835855c14-kube-api-access-6pvsc\") pod \"maas-api-revoke-sub-0d6af7fa-gwchs\" (UID: \"2be45447-2758-432d-b668-2da835855c14\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d6af7fa-gwchs" Nov 03 13:53:33.014706 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:33.014619 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:53:33.046503 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:33.046452 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78202144-lppdj"] Nov 03 13:53:33.051516 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:33.051446 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-78202144-lppdj"] Nov 03 13:53:33.084341 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:33.084313 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d6af7fa-gwchs" Nov 03 13:53:33.459115 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:33.459076 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d6af7fa-gwchs"] Nov 03 13:53:33.465761 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:53:33.465727 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2be45447_2758_432d_b668_2da835855c14.slice/crio-5210a1b4124b96cb3a355a8fc53661cc3cb6ab25d979296438d2009c144f56aa WatchSource:0}: Error finding container 5210a1b4124b96cb3a355a8fc53661cc3cb6ab25d979296438d2009c144f56aa: Status 404 returned error can't find the container with id 5210a1b4124b96cb3a355a8fc53661cc3cb6ab25d979296438d2009c144f56aa Nov 03 13:53:34.119820 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:34.119746 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06ffc052-27e6-4de3-a4de-002f346c9f02" path="/var/lib/kubelet/pods/06ffc052-27e6-4de3-a4de-002f346c9f02/volumes" Nov 03 13:53:34.282238 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:34.282201 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d6af7fa-gwchs" event={"ID":"2be45447-2758-432d-b668-2da835855c14","Type":"ContainerStarted","Data":"a39ccd5f2a1dd3eb6be9eeb32470e5dfc2ec0a59661f102acd7626dc686212be"} Nov 03 13:53:34.282426 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:34.282278 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d6af7fa-gwchs" event={"ID":"2be45447-2758-432d-b668-2da835855c14","Type":"ContainerStarted","Data":"5210a1b4124b96cb3a355a8fc53661cc3cb6ab25d979296438d2009c144f56aa"} Nov 03 13:53:34.392869 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:34.392817 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d6af7fa-gwchs" podStartSLOduration=2.392803294 podStartE2EDuration="2.392803294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:53:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:53:34.390291049 +0000 UTC m=+2896.400979282" watchObservedRunningTime="2026-11-03 13:53:34.392803294 +0000 UTC m=+2896.403491536" Nov 03 13:53:35.290032 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:35.289989 5979 generic.go:356] "Generic (PLEG): container finished" podID="2be45447-2758-432d-b668-2da835855c14" containerID="a39ccd5f2a1dd3eb6be9eeb32470e5dfc2ec0a59661f102acd7626dc686212be" exitCode=0 Nov 03 13:53:35.290442 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:35.290072 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d6af7fa-gwchs" event={"ID":"2be45447-2758-432d-b668-2da835855c14","Type":"ContainerDied","Data":"a39ccd5f2a1dd3eb6be9eeb32470e5dfc2ec0a59661f102acd7626dc686212be"} Nov 03 13:53:36.437672 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:36.437645 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d6af7fa-gwchs" Nov 03 13:53:36.590832 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:36.590749 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2be45447-2758-432d-b668-2da835855c14-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2be45447-2758-432d-b668-2da835855c14-maas-api-openshift-ca\") pod \"2be45447-2758-432d-b668-2da835855c14\" (UID: \"2be45447-2758-432d-b668-2da835855c14\") " Nov 03 13:53:36.590996 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:36.590895 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2be45447-2758-432d-b668-2da835855c14-kube-api-access-6pvsc\" (UniqueName: \"kubernetes.io/projected/2be45447-2758-432d-b668-2da835855c14-kube-api-access-6pvsc\") pod \"2be45447-2758-432d-b668-2da835855c14\" (UID: \"2be45447-2758-432d-b668-2da835855c14\") " Nov 03 13:53:36.591073 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:36.591000 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2be45447-2758-432d-b668-2da835855c14-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2be45447-2758-432d-b668-2da835855c14-maas-api-xks-ca\") pod \"2be45447-2758-432d-b668-2da835855c14\" (UID: \"2be45447-2758-432d-b668-2da835855c14\") " Nov 03 13:53:36.591140 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:36.591088 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2be45447-2758-432d-b668-2da835855c14-maas-api-openshift-ca" pod "2be45447-2758-432d-b668-2da835855c14" (UID: "2be45447-2758-432d-b668-2da835855c14"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:53:36.591341 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:36.591323 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2be45447-2758-432d-b668-2da835855c14-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:36.593050 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:36.593017 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2be45447-2758-432d-b668-2da835855c14-kube-api-access-6pvsc" pod "2be45447-2758-432d-b668-2da835855c14" (UID: "2be45447-2758-432d-b668-2da835855c14"). InnerVolumeSpecName "kube-api-access-6pvsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:53:36.593163 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:36.593075 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2be45447-2758-432d-b668-2da835855c14-maas-api-xks-ca" pod "2be45447-2758-432d-b668-2da835855c14" (UID: "2be45447-2758-432d-b668-2da835855c14"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:53:36.692232 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:36.692194 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pvsc\" (UniqueName: \"kubernetes.io/projected/2be45447-2758-432d-b668-2da835855c14-kube-api-access-6pvsc\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:36.692232 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:36.692224 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2be45447-2758-432d-b668-2da835855c14-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:53:37.234918 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:53:37.234874 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:53:37.304402 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:37.304369 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d6af7fa-gwchs" event={"ID":"2be45447-2758-432d-b668-2da835855c14","Type":"ContainerDied","Data":"5210a1b4124b96cb3a355a8fc53661cc3cb6ab25d979296438d2009c144f56aa"} Nov 03 13:53:37.304609 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:37.304417 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5210a1b4124b96cb3a355a8fc53661cc3cb6ab25d979296438d2009c144f56aa" Nov 03 13:53:37.304609 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:37.304385 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d6af7fa-gwchs" Nov 03 13:53:45.245738 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:45.245701 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m"] Nov 03 13:53:45.246361 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:45.246343 5979 cpu_manager.go:414] "containerMap: removing container" podUID="7512d231-097f-4192-99f7-44bb0f98bdda" containerName="istio-proxy" Nov 03 13:53:45.246361 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:45.246360 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7512d231-097f-4192-99f7-44bb0f98bdda" containerName="istio-proxy" Nov 03 13:53:45.246539 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:45.246403 5979 cpu_manager.go:414] "containerMap: removing container" podUID="2be45447-2758-432d-b668-2da835855c14" containerName="revoke-keys" Nov 03 13:53:45.246539 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:45.246409 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2be45447-2758-432d-b668-2da835855c14" containerName="revoke-keys" Nov 03 13:53:45.246539 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:45.246538 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="2be45447-2758-432d-b668-2da835855c14" containerName="revoke-keys" Nov 03 13:53:45.250117 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:45.250099 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m" Nov 03 13:53:45.292255 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:45.292214 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a1133dc-605d-4871-8190-7714aca1ce00-istio-podinfo\") pod \"e2e-explicit-92546f-openshift-default-6ff5974d9-s845m\" (UID: \"8a1133dc-605d-4871-8190-7714aca1ce00\") " pod="openshift-ingress/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m" Nov 03 13:53:45.292435 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:45.292273 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a1133dc-605d-4871-8190-7714aca1ce00-istio-envoy\") pod \"e2e-explicit-92546f-openshift-default-6ff5974d9-s845m\" (UID: \"8a1133dc-605d-4871-8190-7714aca1ce00\") " pod="openshift-ingress/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m" Nov 03 13:53:45.292435 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:45.292399 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a1133dc-605d-4871-8190-7714aca1ce00-istio-data\") pod \"e2e-explicit-92546f-openshift-default-6ff5974d9-s845m\" (UID: \"8a1133dc-605d-4871-8190-7714aca1ce00\") " pod="openshift-ingress/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m" Nov 03 13:53:45.292597 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:45.292434 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a1133dc-605d-4871-8190-7714aca1ce00-workload-certs\") pod \"e2e-explicit-92546f-openshift-default-6ff5974d9-s845m\" (UID: \"8a1133dc-605d-4871-8190-7714aca1ce00\") " pod="openshift-ingress/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m" Nov 03 13:53:45.292597 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:45.292456 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a1133dc-605d-4871-8190-7714aca1ce00-workload-socket\") pod \"e2e-explicit-92546f-openshift-default-6ff5974d9-s845m\" (UID: \"8a1133dc-605d-4871-8190-7714aca1ce00\") " pod="openshift-ingress/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m" Nov 03 13:53:45.292597 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:45.292512 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a1133dc-605d-4871-8190-7714aca1ce00-credential-socket\") pod \"e2e-explicit-92546f-openshift-default-6ff5974d9-s845m\" (UID: \"8a1133dc-605d-4871-8190-7714aca1ce00\") " pod="openshift-ingress/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m" Nov 03 13:53:45.292775 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:45.292621 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a1133dc-605d-4871-8190-7714aca1ce00-istio-token\") pod \"e2e-explicit-92546f-openshift-default-6ff5974d9-s845m\" (UID: \"8a1133dc-605d-4871-8190-7714aca1ce00\") " pod="openshift-ingress/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m" Nov 03 13:53:45.292775 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:45.292638 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9nl7\" (UniqueName: \"kubernetes.io/projected/8a1133dc-605d-4871-8190-7714aca1ce00-kube-api-access-z9nl7\") pod \"e2e-explicit-92546f-openshift-default-6ff5974d9-s845m\" (UID: \"8a1133dc-605d-4871-8190-7714aca1ce00\") " pod="openshift-ingress/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m" Nov 03 13:53:45.292775 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:45.292661 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a1133dc-605d-4871-8190-7714aca1ce00-istiod-ca-cert\") pod \"e2e-explicit-92546f-openshift-default-6ff5974d9-s845m\" (UID: \"8a1133dc-605d-4871-8190-7714aca1ce00\") " pod="openshift-ingress/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m" Nov 03 13:53:45.299680 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:45.299655 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-92546f-openshift-default-dockercfg-7fxlm\"" Nov 03 13:53:45.348156 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:45.348117 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m"] Nov 03 13:53:45.396394 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:45.396357 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a1133dc-605d-4871-8190-7714aca1ce00-istio-data\") pod \"e2e-explicit-92546f-openshift-default-6ff5974d9-s845m\" (UID: \"8a1133dc-605d-4871-8190-7714aca1ce00\") " pod="openshift-ingress/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m" Nov 03 13:53:45.396679 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:45.396652 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a1133dc-605d-4871-8190-7714aca1ce00-workload-certs\") pod \"e2e-explicit-92546f-openshift-default-6ff5974d9-s845m\" (UID: \"8a1133dc-605d-4871-8190-7714aca1ce00\") " pod="openshift-ingress/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m" Nov 03 13:53:45.396817 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:45.396799 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a1133dc-605d-4871-8190-7714aca1ce00-workload-socket\") pod \"e2e-explicit-92546f-openshift-default-6ff5974d9-s845m\" (UID: \"8a1133dc-605d-4871-8190-7714aca1ce00\") " pod="openshift-ingress/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m" Nov 03 13:53:45.397729 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:45.397655 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a1133dc-605d-4871-8190-7714aca1ce00-credential-socket\") pod \"e2e-explicit-92546f-openshift-default-6ff5974d9-s845m\" (UID: \"8a1133dc-605d-4871-8190-7714aca1ce00\") " pod="openshift-ingress/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m" Nov 03 13:53:45.398126 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:45.398104 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a1133dc-605d-4871-8190-7714aca1ce00-istio-token\") pod \"e2e-explicit-92546f-openshift-default-6ff5974d9-s845m\" (UID: \"8a1133dc-605d-4871-8190-7714aca1ce00\") " pod="openshift-ingress/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m" Nov 03 13:53:45.398268 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:45.398250 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9nl7\" (UniqueName: \"kubernetes.io/projected/8a1133dc-605d-4871-8190-7714aca1ce00-kube-api-access-z9nl7\") pod \"e2e-explicit-92546f-openshift-default-6ff5974d9-s845m\" (UID: \"8a1133dc-605d-4871-8190-7714aca1ce00\") " pod="openshift-ingress/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m" Nov 03 13:53:45.398377 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:45.398363 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a1133dc-605d-4871-8190-7714aca1ce00-istiod-ca-cert\") pod \"e2e-explicit-92546f-openshift-default-6ff5974d9-s845m\" (UID: \"8a1133dc-605d-4871-8190-7714aca1ce00\") " pod="openshift-ingress/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m" Nov 03 13:53:45.398638 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:45.398614 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a1133dc-605d-4871-8190-7714aca1ce00-istio-podinfo\") pod \"e2e-explicit-92546f-openshift-default-6ff5974d9-s845m\" (UID: \"8a1133dc-605d-4871-8190-7714aca1ce00\") " pod="openshift-ingress/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m" Nov 03 13:53:45.398790 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:45.398775 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a1133dc-605d-4871-8190-7714aca1ce00-istio-envoy\") pod \"e2e-explicit-92546f-openshift-default-6ff5974d9-s845m\" (UID: \"8a1133dc-605d-4871-8190-7714aca1ce00\") " pod="openshift-ingress/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m" Nov 03 13:53:45.399394 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:45.399368 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a1133dc-605d-4871-8190-7714aca1ce00-workload-certs\") pod \"e2e-explicit-92546f-openshift-default-6ff5974d9-s845m\" (UID: \"8a1133dc-605d-4871-8190-7714aca1ce00\") " pod="openshift-ingress/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m" Nov 03 13:53:45.453247 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:45.399655 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a1133dc-605d-4871-8190-7714aca1ce00-credential-socket\") pod \"e2e-explicit-92546f-openshift-default-6ff5974d9-s845m\" (UID: \"8a1133dc-605d-4871-8190-7714aca1ce00\") " pod="openshift-ingress/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m" Nov 03 13:53:45.455751 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:45.397287 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a1133dc-605d-4871-8190-7714aca1ce00-istio-data\") pod \"e2e-explicit-92546f-openshift-default-6ff5974d9-s845m\" (UID: \"8a1133dc-605d-4871-8190-7714aca1ce00\") " pod="openshift-ingress/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m" Nov 03 13:53:45.455969 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:45.411341 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a1133dc-605d-4871-8190-7714aca1ce00-istiod-ca-cert\") pod \"e2e-explicit-92546f-openshift-default-6ff5974d9-s845m\" (UID: \"8a1133dc-605d-4871-8190-7714aca1ce00\") " pod="openshift-ingress/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m" Nov 03 13:53:45.456090 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:45.415247 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a1133dc-605d-4871-8190-7714aca1ce00-workload-socket\") pod \"e2e-explicit-92546f-openshift-default-6ff5974d9-s845m\" (UID: \"8a1133dc-605d-4871-8190-7714aca1ce00\") " pod="openshift-ingress/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m" Nov 03 13:53:45.456195 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:45.447163 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a1133dc-605d-4871-8190-7714aca1ce00-istio-podinfo\") pod \"e2e-explicit-92546f-openshift-default-6ff5974d9-s845m\" (UID: \"8a1133dc-605d-4871-8190-7714aca1ce00\") " pod="openshift-ingress/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m" Nov 03 13:53:45.476447 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:45.476046 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9nl7\" (UniqueName: \"kubernetes.io/projected/8a1133dc-605d-4871-8190-7714aca1ce00-kube-api-access-z9nl7\") pod \"e2e-explicit-92546f-openshift-default-6ff5974d9-s845m\" (UID: \"8a1133dc-605d-4871-8190-7714aca1ce00\") " pod="openshift-ingress/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m" Nov 03 13:53:45.496186 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:45.496071 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a1133dc-605d-4871-8190-7714aca1ce00-istio-token\") pod \"e2e-explicit-92546f-openshift-default-6ff5974d9-s845m\" (UID: \"8a1133dc-605d-4871-8190-7714aca1ce00\") " pod="openshift-ingress/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m" Nov 03 13:53:45.499120 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:45.499066 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a1133dc-605d-4871-8190-7714aca1ce00-istio-envoy\") pod \"e2e-explicit-92546f-openshift-default-6ff5974d9-s845m\" (UID: \"8a1133dc-605d-4871-8190-7714aca1ce00\") " pod="openshift-ingress/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m" Nov 03 13:53:45.562895 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:45.562854 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m" Nov 03 13:53:45.992845 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:45.992813 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m"] Nov 03 13:53:45.999526 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:53:45.999496 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a1133dc_605d_4871_8190_7714aca1ce00.slice/crio-7c557d49b6134c2d03f8c2b7e31a5679a5c139467de28d1825c9095b639d4491 WatchSource:0}: Error finding container 7c557d49b6134c2d03f8c2b7e31a5679a5c139467de28d1825c9095b639d4491: Status 404 returned error can't find the container with id 7c557d49b6134c2d03f8c2b7e31a5679a5c139467de28d1825c9095b639d4491 Nov 03 13:53:46.001935 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:46.001901 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:53:46.002034 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:46.001968 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:53:46.002034 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:46.001999 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:53:46.364715 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:46.364624 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m" event={"ID":"8a1133dc-605d-4871-8190-7714aca1ce00","Type":"ContainerStarted","Data":"808a83b27db3fc13e449f2a22c80e386dab0cadb6aecd4ceaa90c7a860866190"} Nov 03 13:53:46.364715 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:46.364689 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m" event={"ID":"8a1133dc-605d-4871-8190-7714aca1ce00","Type":"ContainerStarted","Data":"7c557d49b6134c2d03f8c2b7e31a5679a5c139467de28d1825c9095b639d4491"} Nov 03 13:53:46.478914 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:46.478862 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m" podStartSLOduration=1.478847764 podStartE2EDuration="1.478847764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:53:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:53:46.475242063 +0000 UTC m=+2908.485930351" watchObservedRunningTime="2026-11-03 13:53:46.478847764 +0000 UTC m=+2908.489536005" Nov 03 13:53:46.563754 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:46.563724 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m" Nov 03 13:53:47.569956 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:47.569928 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m" Nov 03 13:53:48.377057 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:48.377027 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m" Nov 03 13:53:48.377977 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:48.377956 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m" Nov 03 13:53:53.027206 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:53.027173 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-92546f-6b56868cc5-k4nsh"] Nov 03 13:53:53.031393 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:53.031371 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-92546f-6b56868cc5-k4nsh" Nov 03 13:53:53.038330 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:53.038308 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4c48\" (UniqueName: \"kubernetes.io/projected/bf056db6-6323-4966-bb10-0e82540a3273-kube-api-access-f4c48\") pod \"payload-pre-processing-e2e-explicit-92546f-6b56868cc5-k4nsh\" (UID: \"bf056db6-6323-4966-bb10-0e82540a3273\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-92546f-6b56868cc5-k4nsh" Nov 03 13:53:53.038432 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:53.038336 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf056db6-6323-4966-bb10-0e82540a3273-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-92546f-6b56868cc5-k4nsh\" (UID: \"bf056db6-6323-4966-bb10-0e82540a3273\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-92546f-6b56868cc5-k4nsh" Nov 03 13:53:53.081812 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:53.081781 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-92546f\"" Nov 03 13:53:53.085395 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:53.085363 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-92546f-6b56868cc5-k4nsh"] Nov 03 13:53:53.161498 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:53.157073 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-92546f-dockercfg-x78nb\"" Nov 03 13:53:53.195975 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:53.195933 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4c48\" (UniqueName: \"kubernetes.io/projected/bf056db6-6323-4966-bb10-0e82540a3273-kube-api-access-f4c48\") pod \"payload-pre-processing-e2e-explicit-92546f-6b56868cc5-k4nsh\" (UID: \"bf056db6-6323-4966-bb10-0e82540a3273\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-92546f-6b56868cc5-k4nsh" Nov 03 13:53:53.196270 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:53.196249 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf056db6-6323-4966-bb10-0e82540a3273-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-92546f-6b56868cc5-k4nsh\" (UID: \"bf056db6-6323-4966-bb10-0e82540a3273\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-92546f-6b56868cc5-k4nsh" Nov 03 13:53:53.197996 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:53.197244 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf056db6-6323-4966-bb10-0e82540a3273-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-92546f-6b56868cc5-k4nsh\" (UID: \"bf056db6-6323-4966-bb10-0e82540a3273\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-92546f-6b56868cc5-k4nsh" Nov 03 13:53:53.212756 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:53.212004 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-92546f-7f46f65c8f-97r54"] Nov 03 13:53:53.218351 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:53.218323 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-92546f-7f46f65c8f-97r54" Nov 03 13:53:53.231928 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:53.231902 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4c48\" (UniqueName: \"kubernetes.io/projected/bf056db6-6323-4966-bb10-0e82540a3273-kube-api-access-f4c48\") pod \"payload-pre-processing-e2e-explicit-92546f-6b56868cc5-k4nsh\" (UID: \"bf056db6-6323-4966-bb10-0e82540a3273\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-92546f-6b56868cc5-k4nsh" Nov 03 13:53:53.250340 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:53.250310 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-92546f-7f46f65c8f-97r54"] Nov 03 13:53:53.299210 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:53.299132 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw694\" (UniqueName: \"kubernetes.io/projected/21e41083-d6b8-45ad-9cf3-02eb692596eb-kube-api-access-mw694\") pod \"payload-processing-e2e-explicit-92546f-7f46f65c8f-97r54\" (UID: \"21e41083-d6b8-45ad-9cf3-02eb692596eb\") " pod="openshift-ingress/payload-processing-e2e-explicit-92546f-7f46f65c8f-97r54" Nov 03 13:53:53.299375 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:53.299353 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21e41083-d6b8-45ad-9cf3-02eb692596eb-plugins-config-volume\") pod \"payload-processing-e2e-explicit-92546f-7f46f65c8f-97r54\" (UID: \"21e41083-d6b8-45ad-9cf3-02eb692596eb\") " pod="openshift-ingress/payload-processing-e2e-explicit-92546f-7f46f65c8f-97r54" Nov 03 13:53:53.345650 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:53.345623 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-92546f-6b56868cc5-k4nsh" Nov 03 13:53:53.400973 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:53.400938 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21e41083-d6b8-45ad-9cf3-02eb692596eb-plugins-config-volume\") pod \"payload-processing-e2e-explicit-92546f-7f46f65c8f-97r54\" (UID: \"21e41083-d6b8-45ad-9cf3-02eb692596eb\") " pod="openshift-ingress/payload-processing-e2e-explicit-92546f-7f46f65c8f-97r54" Nov 03 13:53:53.401117 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:53.401100 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw694\" (UniqueName: \"kubernetes.io/projected/21e41083-d6b8-45ad-9cf3-02eb692596eb-kube-api-access-mw694\") pod \"payload-processing-e2e-explicit-92546f-7f46f65c8f-97r54\" (UID: \"21e41083-d6b8-45ad-9cf3-02eb692596eb\") " pod="openshift-ingress/payload-processing-e2e-explicit-92546f-7f46f65c8f-97r54" Nov 03 13:53:53.402212 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:53.402183 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21e41083-d6b8-45ad-9cf3-02eb692596eb-plugins-config-volume\") pod \"payload-processing-e2e-explicit-92546f-7f46f65c8f-97r54\" (UID: \"21e41083-d6b8-45ad-9cf3-02eb692596eb\") " pod="openshift-ingress/payload-processing-e2e-explicit-92546f-7f46f65c8f-97r54" Nov 03 13:53:53.454928 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:53.454894 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw694\" (UniqueName: \"kubernetes.io/projected/21e41083-d6b8-45ad-9cf3-02eb692596eb-kube-api-access-mw694\") pod \"payload-processing-e2e-explicit-92546f-7f46f65c8f-97r54\" (UID: \"21e41083-d6b8-45ad-9cf3-02eb692596eb\") " pod="openshift-ingress/payload-processing-e2e-explicit-92546f-7f46f65c8f-97r54" Nov 03 13:53:53.550381 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:53.550295 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-92546f-7f46f65c8f-97r54" Nov 03 13:53:53.723593 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:53.723552 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-92546f-6b56868cc5-k4nsh"] Nov 03 13:53:53.730166 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:53:53.730135 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf056db6_6323_4966_bb10_0e82540a3273.slice/crio-3c7c18fe2ba7a45ec9a3f38665afa79ca9217fe01072bc66e4eae21f3510f011 WatchSource:0}: Error finding container 3c7c18fe2ba7a45ec9a3f38665afa79ca9217fe01072bc66e4eae21f3510f011: Status 404 returned error can't find the container with id 3c7c18fe2ba7a45ec9a3f38665afa79ca9217fe01072bc66e4eae21f3510f011 Nov 03 13:53:53.925063 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:53.925032 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-92546f-7f46f65c8f-97r54"] Nov 03 13:53:53.932022 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:53:53.931994 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21e41083_d6b8_45ad_9cf3_02eb692596eb.slice/crio-49e3da0aff972117bd1f7f4a06c5568701eb80cca65364db35046715942997e3 WatchSource:0}: Error finding container 49e3da0aff972117bd1f7f4a06c5568701eb80cca65364db35046715942997e3: Status 404 returned error can't find the container with id 49e3da0aff972117bd1f7f4a06c5568701eb80cca65364db35046715942997e3 Nov 03 13:53:54.423311 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:54.423272 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-92546f-7f46f65c8f-97r54" event={"ID":"21e41083-d6b8-45ad-9cf3-02eb692596eb","Type":"ContainerStarted","Data":"bf04f35ba2d24136fc880b16dc26ccb3a75ad4b40ce8a42efb917451c3fb71f3"} Nov 03 13:53:54.423824 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:54.423346 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-92546f-7f46f65c8f-97r54" Nov 03 13:53:54.423824 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:54.423382 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-92546f-7f46f65c8f-97r54" event={"ID":"21e41083-d6b8-45ad-9cf3-02eb692596eb","Type":"ContainerStarted","Data":"49e3da0aff972117bd1f7f4a06c5568701eb80cca65364db35046715942997e3"} Nov 03 13:53:54.425051 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:54.425017 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-92546f-6b56868cc5-k4nsh" event={"ID":"bf056db6-6323-4966-bb10-0e82540a3273","Type":"ContainerStarted","Data":"bb9701d682745993a41b28e39413b081ea3224e4079b7a02b5ce9f7e34525fdb"} Nov 03 13:53:54.425187 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:54.425073 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-92546f-6b56868cc5-k4nsh" event={"ID":"bf056db6-6323-4966-bb10-0e82540a3273","Type":"ContainerStarted","Data":"3c7c18fe2ba7a45ec9a3f38665afa79ca9217fe01072bc66e4eae21f3510f011"} Nov 03 13:53:54.425187 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:54.425102 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-92546f-6b56868cc5-k4nsh" Nov 03 13:53:54.534518 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:54.533699 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-92546f-7f46f65c8f-97r54" podStartSLOduration=1.533679859 podStartE2EDuration="1.533679859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:53:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:53:54.531675938 +0000 UTC m=+2916.542364181" watchObservedRunningTime="2026-11-03 13:53:54.533679859 +0000 UTC m=+2916.544368103" Nov 03 13:53:54.640069 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:53:54.639912 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-92546f-6b56868cc5-k4nsh" podStartSLOduration=2.639895735 podStartE2EDuration="2.639895735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:53:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:53:54.639291893 +0000 UTC m=+2916.649980135" watchObservedRunningTime="2026-11-03 13:53:54.639895735 +0000 UTC m=+2916.650583978" Nov 03 13:54:00.611611 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:00.611574 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-367381-55bcdb6c57-6kxd4"] Nov 03 13:54:00.616630 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:00.616608 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-367381-55bcdb6c57-6kxd4" Nov 03 13:54:00.666078 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:00.666044 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-367381-serving-cert\"" Nov 03 13:54:00.694380 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:00.693223 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80a05ef5-573e-4b3a-aaaf-6c91d6cba523-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-367381-55bcdb6c57-6kxd4\" (UID: \"80a05ef5-573e-4b3a-aaaf-6c91d6cba523\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-367381-55bcdb6c57-6kxd4" Nov 03 13:54:00.694380 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:00.693316 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80a05ef5-573e-4b3a-aaaf-6c91d6cba523-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-367381-55bcdb6c57-6kxd4\" (UID: \"80a05ef5-573e-4b3a-aaaf-6c91d6cba523\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-367381-55bcdb6c57-6kxd4" Nov 03 13:54:00.694380 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:00.693505 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kjmx\" (UniqueName: \"kubernetes.io/projected/80a05ef5-573e-4b3a-aaaf-6c91d6cba523-kube-api-access-2kjmx\") pod \"maas-api-e2e-shared-b-w1-367381-55bcdb6c57-6kxd4\" (UID: \"80a05ef5-573e-4b3a-aaaf-6c91d6cba523\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-367381-55bcdb6c57-6kxd4" Nov 03 13:54:00.694380 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:00.693579 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80a05ef5-573e-4b3a-aaaf-6c91d6cba523-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-367381-55bcdb6c57-6kxd4\" (UID: \"80a05ef5-573e-4b3a-aaaf-6c91d6cba523\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-367381-55bcdb6c57-6kxd4" Nov 03 13:54:00.715335 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:00.715293 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-367381-55bcdb6c57-6kxd4"] Nov 03 13:54:00.794795 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:00.794755 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80a05ef5-573e-4b3a-aaaf-6c91d6cba523-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-367381-55bcdb6c57-6kxd4\" (UID: \"80a05ef5-573e-4b3a-aaaf-6c91d6cba523\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-367381-55bcdb6c57-6kxd4" Nov 03 13:54:00.794977 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:00.794857 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80a05ef5-573e-4b3a-aaaf-6c91d6cba523-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-367381-55bcdb6c57-6kxd4\" (UID: \"80a05ef5-573e-4b3a-aaaf-6c91d6cba523\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-367381-55bcdb6c57-6kxd4" Nov 03 13:54:00.794977 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:00.794876 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80a05ef5-573e-4b3a-aaaf-6c91d6cba523-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-367381-55bcdb6c57-6kxd4\" (UID: \"80a05ef5-573e-4b3a-aaaf-6c91d6cba523\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-367381-55bcdb6c57-6kxd4" Nov 03 13:54:00.794977 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:54:00.794912 5979 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-367381-serving-cert: secret "maas-api-e2e-shared-b-w1-367381-serving-cert" not found Nov 03 13:54:00.794977 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:00.794967 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kjmx\" (UniqueName: \"kubernetes.io/projected/80a05ef5-573e-4b3a-aaaf-6c91d6cba523-kube-api-access-2kjmx\") pod \"maas-api-e2e-shared-b-w1-367381-55bcdb6c57-6kxd4\" (UID: \"80a05ef5-573e-4b3a-aaaf-6c91d6cba523\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-367381-55bcdb6c57-6kxd4" Nov 03 13:54:00.795131 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:54:00.795004 5979 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80a05ef5-573e-4b3a-aaaf-6c91d6cba523-maas-api-tls podName:80a05ef5-573e-4b3a-aaaf-6c91d6cba523 nodeName:}" failed. No retries permitted until 2026-11-03 13:54:01.294981789 +0000 UTC m=+2923.305670011 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/80a05ef5-573e-4b3a-aaaf-6c91d6cba523-maas-api-tls") pod "maas-api-e2e-shared-b-w1-367381-55bcdb6c57-6kxd4" (UID: "80a05ef5-573e-4b3a-aaaf-6c91d6cba523") : secret "maas-api-e2e-shared-b-w1-367381-serving-cert" not found Nov 03 13:54:00.795587 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:00.795561 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80a05ef5-573e-4b3a-aaaf-6c91d6cba523-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-367381-55bcdb6c57-6kxd4\" (UID: \"80a05ef5-573e-4b3a-aaaf-6c91d6cba523\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-367381-55bcdb6c57-6kxd4" Nov 03 13:54:00.797346 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:00.797329 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80a05ef5-573e-4b3a-aaaf-6c91d6cba523-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-367381-55bcdb6c57-6kxd4\" (UID: \"80a05ef5-573e-4b3a-aaaf-6c91d6cba523\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-367381-55bcdb6c57-6kxd4" Nov 03 13:54:00.849235 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:00.849203 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kjmx\" (UniqueName: \"kubernetes.io/projected/80a05ef5-573e-4b3a-aaaf-6c91d6cba523-kube-api-access-2kjmx\") pod \"maas-api-e2e-shared-b-w1-367381-55bcdb6c57-6kxd4\" (UID: \"80a05ef5-573e-4b3a-aaaf-6c91d6cba523\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-367381-55bcdb6c57-6kxd4" Nov 03 13:54:01.302885 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:01.302848 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80a05ef5-573e-4b3a-aaaf-6c91d6cba523-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-367381-55bcdb6c57-6kxd4\" (UID: \"80a05ef5-573e-4b3a-aaaf-6c91d6cba523\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-367381-55bcdb6c57-6kxd4" Nov 03 13:54:01.305346 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:01.305320 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80a05ef5-573e-4b3a-aaaf-6c91d6cba523-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-367381-55bcdb6c57-6kxd4\" (UID: \"80a05ef5-573e-4b3a-aaaf-6c91d6cba523\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-367381-55bcdb6c57-6kxd4" Nov 03 13:54:01.531011 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:01.530978 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-367381-55bcdb6c57-6kxd4" Nov 03 13:54:01.907202 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:01.907156 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-367381-55bcdb6c57-6kxd4"] Nov 03 13:54:01.915277 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:54:01.915243 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80a05ef5_573e_4b3a_aaaf_6c91d6cba523.slice/crio-f307cc8d760340800ac1121e250f8ef74718bc6ce4a7d06ba458447e1fbec878 WatchSource:0}: Error finding container f307cc8d760340800ac1121e250f8ef74718bc6ce4a7d06ba458447e1fbec878: Status 404 returned error can't find the container with id f307cc8d760340800ac1121e250f8ef74718bc6ce4a7d06ba458447e1fbec878 Nov 03 13:54:02.477251 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:02.477217 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-367381-55bcdb6c57-6kxd4" event={"ID":"80a05ef5-573e-4b3a-aaaf-6c91d6cba523","Type":"ContainerStarted","Data":"f307cc8d760340800ac1121e250f8ef74718bc6ce4a7d06ba458447e1fbec878"} Nov 03 13:54:03.485234 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:03.485196 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-367381-55bcdb6c57-6kxd4" event={"ID":"80a05ef5-573e-4b3a-aaaf-6c91d6cba523","Type":"ContainerStarted","Data":"d8bcca5fb8b1879744bfebdaea6f6157f5967f83b66df8718ecbd647e5b44bb2"} Nov 03 13:54:03.485752 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:03.485333 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-367381-55bcdb6c57-6kxd4" Nov 03 13:54:03.598028 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:03.597971 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-367381-55bcdb6c57-6kxd4" podStartSLOduration=2.676442301 podStartE2EDuration="3.597957345s" totalImagesPullingTime="921.515044ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:54:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:54:03.59775478 +0000 UTC m=+2925.608443026" watchObservedRunningTime="2026-11-03 13:54:03.597957345 +0000 UTC m=+2925.608645586" Nov 03 13:54:05.435399 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:05.435370 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-92546f-6b56868cc5-k4nsh" Nov 03 13:54:05.435939 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:05.435440 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-92546f-7f46f65c8f-97r54" Nov 03 13:54:07.234358 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:54:07.234321 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:54:09.468867 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:09.468805 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6"] Nov 03 13:54:09.469367 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:09.469111 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6" podUID="0c837757-208a-4cbe-a6b7-2b0014ae29bf" containerName="istio-proxy" containerID="cri-o://ff314bcb937698a6a1bd2547d8fa021aecb6b13d446d0683c6e3b4b3a76faa94" gracePeriod=30 Nov 03 13:54:09.498188 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:09.498163 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-367381-55bcdb6c57-6kxd4" Nov 03 13:54:14.725236 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:14.725208 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6" Nov 03 13:54:14.789431 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:14.789347 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c837757-208a-4cbe-a6b7-2b0014ae29bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c837757-208a-4cbe-a6b7-2b0014ae29bf-istio-podinfo\") pod \"0c837757-208a-4cbe-a6b7-2b0014ae29bf\" (UID: \"0c837757-208a-4cbe-a6b7-2b0014ae29bf\") " Nov 03 13:54:14.789431 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:14.789407 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c837757-208a-4cbe-a6b7-2b0014ae29bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c837757-208a-4cbe-a6b7-2b0014ae29bf-istio-envoy\") pod \"0c837757-208a-4cbe-a6b7-2b0014ae29bf\" (UID: \"0c837757-208a-4cbe-a6b7-2b0014ae29bf\") " Nov 03 13:54:14.789700 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:14.789463 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c837757-208a-4cbe-a6b7-2b0014ae29bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c837757-208a-4cbe-a6b7-2b0014ae29bf-workload-socket\") pod \"0c837757-208a-4cbe-a6b7-2b0014ae29bf\" (UID: \"0c837757-208a-4cbe-a6b7-2b0014ae29bf\") " Nov 03 13:54:14.789700 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:14.789509 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c837757-208a-4cbe-a6b7-2b0014ae29bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c837757-208a-4cbe-a6b7-2b0014ae29bf-istiod-ca-cert\") pod \"0c837757-208a-4cbe-a6b7-2b0014ae29bf\" (UID: \"0c837757-208a-4cbe-a6b7-2b0014ae29bf\") " Nov 03 13:54:14.789700 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:14.789545 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c837757-208a-4cbe-a6b7-2b0014ae29bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c837757-208a-4cbe-a6b7-2b0014ae29bf-workload-certs\") pod \"0c837757-208a-4cbe-a6b7-2b0014ae29bf\" (UID: \"0c837757-208a-4cbe-a6b7-2b0014ae29bf\") " Nov 03 13:54:14.789700 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:14.789580 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c837757-208a-4cbe-a6b7-2b0014ae29bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c837757-208a-4cbe-a6b7-2b0014ae29bf-credential-socket\") pod \"0c837757-208a-4cbe-a6b7-2b0014ae29bf\" (UID: \"0c837757-208a-4cbe-a6b7-2b0014ae29bf\") " Nov 03 13:54:14.789700 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:14.789622 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c837757-208a-4cbe-a6b7-2b0014ae29bf-kube-api-access-bdcf9\" (UniqueName: \"kubernetes.io/projected/0c837757-208a-4cbe-a6b7-2b0014ae29bf-kube-api-access-bdcf9\") pod \"0c837757-208a-4cbe-a6b7-2b0014ae29bf\" (UID: \"0c837757-208a-4cbe-a6b7-2b0014ae29bf\") " Nov 03 13:54:14.789700 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:14.789663 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c837757-208a-4cbe-a6b7-2b0014ae29bf-istio-token\" (UniqueName: \"kubernetes.io/projected/0c837757-208a-4cbe-a6b7-2b0014ae29bf-istio-token\") pod \"0c837757-208a-4cbe-a6b7-2b0014ae29bf\" (UID: \"0c837757-208a-4cbe-a6b7-2b0014ae29bf\") " Nov 03 13:54:14.790067 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:14.789758 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c837757-208a-4cbe-a6b7-2b0014ae29bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c837757-208a-4cbe-a6b7-2b0014ae29bf-istio-data\") pod \"0c837757-208a-4cbe-a6b7-2b0014ae29bf\" (UID: \"0c837757-208a-4cbe-a6b7-2b0014ae29bf\") " Nov 03 13:54:14.790067 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:14.789759 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c837757-208a-4cbe-a6b7-2b0014ae29bf-workload-socket" pod "0c837757-208a-4cbe-a6b7-2b0014ae29bf" (UID: "0c837757-208a-4cbe-a6b7-2b0014ae29bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:54:14.790067 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:14.789850 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c837757-208a-4cbe-a6b7-2b0014ae29bf-workload-certs" pod "0c837757-208a-4cbe-a6b7-2b0014ae29bf" (UID: "0c837757-208a-4cbe-a6b7-2b0014ae29bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:54:14.790067 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:14.789945 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c837757-208a-4cbe-a6b7-2b0014ae29bf-istiod-ca-cert" pod "0c837757-208a-4cbe-a6b7-2b0014ae29bf" (UID: "0c837757-208a-4cbe-a6b7-2b0014ae29bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:54:14.790307 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:14.790195 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c837757-208a-4cbe-a6b7-2b0014ae29bf-credential-socket" pod "0c837757-208a-4cbe-a6b7-2b0014ae29bf" (UID: "0c837757-208a-4cbe-a6b7-2b0014ae29bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:54:14.790372 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:14.790293 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c837757-208a-4cbe-a6b7-2b0014ae29bf-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:54:14.790372 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:14.790339 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c837757-208a-4cbe-a6b7-2b0014ae29bf-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:54:14.790372 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:14.790354 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c837757-208a-4cbe-a6b7-2b0014ae29bf-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:54:14.790372 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:14.790368 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c837757-208a-4cbe-a6b7-2b0014ae29bf-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:54:14.790625 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:14.790366 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c837757-208a-4cbe-a6b7-2b0014ae29bf-istio-data" pod "0c837757-208a-4cbe-a6b7-2b0014ae29bf" (UID: "0c837757-208a-4cbe-a6b7-2b0014ae29bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:54:14.791764 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:14.791741 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c837757-208a-4cbe-a6b7-2b0014ae29bf-istio-podinfo" pod "0c837757-208a-4cbe-a6b7-2b0014ae29bf" (UID: "0c837757-208a-4cbe-a6b7-2b0014ae29bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:54:14.792099 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:14.792048 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c837757-208a-4cbe-a6b7-2b0014ae29bf-istio-token" pod "0c837757-208a-4cbe-a6b7-2b0014ae29bf" (UID: "0c837757-208a-4cbe-a6b7-2b0014ae29bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:54:14.792099 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:14.792091 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c837757-208a-4cbe-a6b7-2b0014ae29bf-istio-envoy" pod "0c837757-208a-4cbe-a6b7-2b0014ae29bf" (UID: "0c837757-208a-4cbe-a6b7-2b0014ae29bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:54:14.792273 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:14.792157 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c837757-208a-4cbe-a6b7-2b0014ae29bf-kube-api-access-bdcf9" pod "0c837757-208a-4cbe-a6b7-2b0014ae29bf" (UID: "0c837757-208a-4cbe-a6b7-2b0014ae29bf"). InnerVolumeSpecName "kube-api-access-bdcf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:54:14.892431 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:14.892389 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdcf9\" (UniqueName: \"kubernetes.io/projected/0c837757-208a-4cbe-a6b7-2b0014ae29bf-kube-api-access-bdcf9\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:54:14.892431 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:14.892431 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c837757-208a-4cbe-a6b7-2b0014ae29bf-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:54:14.892659 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:14.892446 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c837757-208a-4cbe-a6b7-2b0014ae29bf-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:54:14.892659 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:14.892459 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c837757-208a-4cbe-a6b7-2b0014ae29bf-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:54:14.892659 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:14.892506 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c837757-208a-4cbe-a6b7-2b0014ae29bf-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:54:15.565895 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:15.565860 5979 generic.go:356] "Generic (PLEG): container finished" podID="0c837757-208a-4cbe-a6b7-2b0014ae29bf" containerID="ff314bcb937698a6a1bd2547d8fa021aecb6b13d446d0683c6e3b4b3a76faa94" exitCode=0 Nov 03 13:54:15.566112 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:15.565919 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6" Nov 03 13:54:15.566112 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:15.565952 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6" event={"ID":"0c837757-208a-4cbe-a6b7-2b0014ae29bf","Type":"ContainerDied","Data":"ff314bcb937698a6a1bd2547d8fa021aecb6b13d446d0683c6e3b4b3a76faa94"} Nov 03 13:54:15.566112 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:15.566006 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6" event={"ID":"0c837757-208a-4cbe-a6b7-2b0014ae29bf","Type":"ContainerDied","Data":"328b7ec9408595fc407dc9a23a5d4395f3921514e49fc92f1a250383a4e1920d"} Nov 03 13:54:15.566112 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:15.566025 5979 scope.go:122] "RemoveContainer" containerID="ff314bcb937698a6a1bd2547d8fa021aecb6b13d446d0683c6e3b4b3a76faa94" Nov 03 13:54:15.576740 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:15.576721 5979 scope.go:122] "RemoveContainer" containerID="ff314bcb937698a6a1bd2547d8fa021aecb6b13d446d0683c6e3b4b3a76faa94" Nov 03 13:54:15.576993 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:54:15.576977 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff314bcb937698a6a1bd2547d8fa021aecb6b13d446d0683c6e3b4b3a76faa94\": container with ID starting with ff314bcb937698a6a1bd2547d8fa021aecb6b13d446d0683c6e3b4b3a76faa94 not found: ID does not exist" containerID="ff314bcb937698a6a1bd2547d8fa021aecb6b13d446d0683c6e3b4b3a76faa94" Nov 03 13:54:15.577060 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:15.577003 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff314bcb937698a6a1bd2547d8fa021aecb6b13d446d0683c6e3b4b3a76faa94"} err="failed to get container status \"ff314bcb937698a6a1bd2547d8fa021aecb6b13d446d0683c6e3b4b3a76faa94\": rpc error: code = NotFound desc = could not find container \"ff314bcb937698a6a1bd2547d8fa021aecb6b13d446d0683c6e3b4b3a76faa94\": container with ID starting with ff314bcb937698a6a1bd2547d8fa021aecb6b13d446d0683c6e3b4b3a76faa94 not found: ID does not exist" Nov 03 13:54:15.721145 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:15.721113 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6"] Nov 03 13:54:15.769340 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:15.769300 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-xs5z6"] Nov 03 13:54:16.115961 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:16.115925 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c837757-208a-4cbe-a6b7-2b0014ae29bf" path="/var/lib/kubelet/pods/0c837757-208a-4cbe-a6b7-2b0014ae29bf/volumes" Nov 03 13:54:20.378172 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:20.378142 5979 scope.go:122] "RemoveContainer" containerID="a4fd78ebdfe8de9cfb24a8769e5439855d154c6e9379144ddaec022448864fa4" Nov 03 13:54:20.387781 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:20.387761 5979 scope.go:122] "RemoveContainer" containerID="ed97c789227603f89237b448cbdc85a8d1ee25668242f86236a7aa9d6dbf14c8" Nov 03 13:54:20.396178 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:20.396161 5979 scope.go:122] "RemoveContainer" containerID="7d14c6b7f64f14d72f770a740d55185df80ecfa51d22161f3fd54f0d3de9b735" Nov 03 13:54:20.404062 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:20.404041 5979 scope.go:122] "RemoveContainer" containerID="be7ff0f1196ea421b2441046129310b141803d33079d3759406bfed84aaf4fea" Nov 03 13:54:20.412142 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:20.412112 5979 scope.go:122] "RemoveContainer" containerID="f8209fab595619d9b798f11adbb4023946a28f69c8c9cfc88cc0a67620aeda42" Nov 03 13:54:20.420553 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:20.420460 5979 scope.go:122] "RemoveContainer" containerID="90caf5f74e76d1591e7778c86c252e408a5da8cb5694cefc9d013050886782a8" Nov 03 13:54:22.044503 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:22.044458 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd6096f9-dbjmz"] Nov 03 13:54:22.090408 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:22.090378 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd6096f9-dbjmz"] Nov 03 13:54:22.115862 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:22.115832 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb1baabd-140e-4516-8044-ea044b02afab" path="/var/lib/kubelet/pods/cb1baabd-140e-4516-8044-ea044b02afab/volumes" Nov 03 13:54:25.060871 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:25.060833 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:54:26.339333 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:26.339293 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv"] Nov 03 13:54:26.339902 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:26.339886 5979 cpu_manager.go:414] "containerMap: removing container" podUID="0c837757-208a-4cbe-a6b7-2b0014ae29bf" containerName="istio-proxy" Nov 03 13:54:26.339980 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:26.339903 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c837757-208a-4cbe-a6b7-2b0014ae29bf" containerName="istio-proxy" Nov 03 13:54:26.340034 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:26.340024 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c837757-208a-4cbe-a6b7-2b0014ae29bf" containerName="istio-proxy" Nov 03 13:54:26.361831 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:26.361797 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv" Nov 03 13:54:26.413559 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:26.413517 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-efbdec-openshift-default-dockercfg-vkvxx\"" Nov 03 13:54:26.439422 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:26.439327 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e73bb972-7cdd-4a6a-aae0-4345d0630989-istiod-ca-cert\") pod \"e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv\" (UID: \"e73bb972-7cdd-4a6a-aae0-4345d0630989\") " pod="openshift-ingress/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv" Nov 03 13:54:26.439641 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:26.439431 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e73bb972-7cdd-4a6a-aae0-4345d0630989-workload-certs\") pod \"e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv\" (UID: \"e73bb972-7cdd-4a6a-aae0-4345d0630989\") " pod="openshift-ingress/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv" Nov 03 13:54:26.439641 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:26.439504 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e73bb972-7cdd-4a6a-aae0-4345d0630989-istio-envoy\") pod \"e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv\" (UID: \"e73bb972-7cdd-4a6a-aae0-4345d0630989\") " pod="openshift-ingress/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv" Nov 03 13:54:26.439641 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:26.439552 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e73bb972-7cdd-4a6a-aae0-4345d0630989-istio-podinfo\") pod \"e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv\" (UID: \"e73bb972-7cdd-4a6a-aae0-4345d0630989\") " pod="openshift-ingress/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv" Nov 03 13:54:26.439641 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:26.439610 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6vkv\" (UniqueName: \"kubernetes.io/projected/e73bb972-7cdd-4a6a-aae0-4345d0630989-kube-api-access-p6vkv\") pod \"e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv\" (UID: \"e73bb972-7cdd-4a6a-aae0-4345d0630989\") " pod="openshift-ingress/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv" Nov 03 13:54:26.439882 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:26.439714 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e73bb972-7cdd-4a6a-aae0-4345d0630989-credential-socket\") pod \"e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv\" (UID: \"e73bb972-7cdd-4a6a-aae0-4345d0630989\") " pod="openshift-ingress/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv" Nov 03 13:54:26.439882 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:26.439801 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e73bb972-7cdd-4a6a-aae0-4345d0630989-istio-token\") pod \"e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv\" (UID: \"e73bb972-7cdd-4a6a-aae0-4345d0630989\") " pod="openshift-ingress/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv" Nov 03 13:54:26.439882 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:26.439840 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e73bb972-7cdd-4a6a-aae0-4345d0630989-istio-data\") pod \"e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv\" (UID: \"e73bb972-7cdd-4a6a-aae0-4345d0630989\") " pod="openshift-ingress/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv" Nov 03 13:54:26.439882 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:26.439876 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e73bb972-7cdd-4a6a-aae0-4345d0630989-workload-socket\") pod \"e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv\" (UID: \"e73bb972-7cdd-4a6a-aae0-4345d0630989\") " pod="openshift-ingress/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv" Nov 03 13:54:26.566495 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:26.543523 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e73bb972-7cdd-4a6a-aae0-4345d0630989-istio-token\") pod \"e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv\" (UID: \"e73bb972-7cdd-4a6a-aae0-4345d0630989\") " pod="openshift-ingress/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv" Nov 03 13:54:26.582498 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:26.566796 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e73bb972-7cdd-4a6a-aae0-4345d0630989-istio-data\") pod \"e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv\" (UID: \"e73bb972-7cdd-4a6a-aae0-4345d0630989\") " pod="openshift-ingress/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv" Nov 03 13:54:26.582498 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:26.580598 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e73bb972-7cdd-4a6a-aae0-4345d0630989-workload-socket\") pod \"e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv\" (UID: \"e73bb972-7cdd-4a6a-aae0-4345d0630989\") " pod="openshift-ingress/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv" Nov 03 13:54:26.582498 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:26.580673 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e73bb972-7cdd-4a6a-aae0-4345d0630989-istiod-ca-cert\") pod \"e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv\" (UID: \"e73bb972-7cdd-4a6a-aae0-4345d0630989\") " pod="openshift-ingress/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv" Nov 03 13:54:26.582498 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:26.580761 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e73bb972-7cdd-4a6a-aae0-4345d0630989-workload-certs\") pod \"e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv\" (UID: \"e73bb972-7cdd-4a6a-aae0-4345d0630989\") " pod="openshift-ingress/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv" Nov 03 13:54:26.582498 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:26.580823 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e73bb972-7cdd-4a6a-aae0-4345d0630989-istio-envoy\") pod \"e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv\" (UID: \"e73bb972-7cdd-4a6a-aae0-4345d0630989\") " pod="openshift-ingress/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv" Nov 03 13:54:26.582498 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:26.580876 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e73bb972-7cdd-4a6a-aae0-4345d0630989-istio-podinfo\") pod \"e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv\" (UID: \"e73bb972-7cdd-4a6a-aae0-4345d0630989\") " pod="openshift-ingress/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv" Nov 03 13:54:26.582498 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:26.580934 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6vkv\" (UniqueName: \"kubernetes.io/projected/e73bb972-7cdd-4a6a-aae0-4345d0630989-kube-api-access-p6vkv\") pod \"e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv\" (UID: \"e73bb972-7cdd-4a6a-aae0-4345d0630989\") " pod="openshift-ingress/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv" Nov 03 13:54:26.582498 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:26.581116 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e73bb972-7cdd-4a6a-aae0-4345d0630989-credential-socket\") pod \"e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv\" (UID: \"e73bb972-7cdd-4a6a-aae0-4345d0630989\") " pod="openshift-ingress/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv" Nov 03 13:54:26.582498 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:26.581643 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e73bb972-7cdd-4a6a-aae0-4345d0630989-credential-socket\") pod \"e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv\" (UID: \"e73bb972-7cdd-4a6a-aae0-4345d0630989\") " pod="openshift-ingress/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv" Nov 03 13:54:26.582498 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:26.562057 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv"] Nov 03 13:54:26.582498 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:26.581970 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e73bb972-7cdd-4a6a-aae0-4345d0630989-workload-socket\") pod \"e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv\" (UID: \"e73bb972-7cdd-4a6a-aae0-4345d0630989\") " pod="openshift-ingress/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv" Nov 03 13:54:26.582498 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:26.568513 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e73bb972-7cdd-4a6a-aae0-4345d0630989-istio-data\") pod \"e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv\" (UID: \"e73bb972-7cdd-4a6a-aae0-4345d0630989\") " pod="openshift-ingress/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv" Nov 03 13:54:26.583304 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:26.582629 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e73bb972-7cdd-4a6a-aae0-4345d0630989-istiod-ca-cert\") pod \"e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv\" (UID: \"e73bb972-7cdd-4a6a-aae0-4345d0630989\") " pod="openshift-ingress/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv" Nov 03 13:54:26.583304 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:26.582868 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e73bb972-7cdd-4a6a-aae0-4345d0630989-workload-certs\") pod \"e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv\" (UID: \"e73bb972-7cdd-4a6a-aae0-4345d0630989\") " pod="openshift-ingress/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv" Nov 03 13:54:26.591143 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:26.590907 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e73bb972-7cdd-4a6a-aae0-4345d0630989-istio-podinfo\") pod \"e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv\" (UID: \"e73bb972-7cdd-4a6a-aae0-4345d0630989\") " pod="openshift-ingress/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv" Nov 03 13:54:26.601818 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:26.599683 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e73bb972-7cdd-4a6a-aae0-4345d0630989-istio-envoy\") pod \"e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv\" (UID: \"e73bb972-7cdd-4a6a-aae0-4345d0630989\") " pod="openshift-ingress/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv" Nov 03 13:54:26.608701 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:26.608533 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e73bb972-7cdd-4a6a-aae0-4345d0630989-istio-token\") pod \"e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv\" (UID: \"e73bb972-7cdd-4a6a-aae0-4345d0630989\") " pod="openshift-ingress/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv" Nov 03 13:54:26.684111 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:26.684075 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6vkv\" (UniqueName: \"kubernetes.io/projected/e73bb972-7cdd-4a6a-aae0-4345d0630989-kube-api-access-p6vkv\") pod \"e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv\" (UID: \"e73bb972-7cdd-4a6a-aae0-4345d0630989\") " pod="openshift-ingress/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv" Nov 03 13:54:26.688708 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:26.688683 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv" Nov 03 13:54:27.164568 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:27.160456 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv"] Nov 03 13:54:27.177575 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:54:27.177450 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode73bb972_7cdd_4a6a_aae0_4345d0630989.slice/crio-55c0615996ff2de819668b93628c591919491fd5d6e37ae5b77334af1723ed25 WatchSource:0}: Error finding container 55c0615996ff2de819668b93628c591919491fd5d6e37ae5b77334af1723ed25: Status 404 returned error can't find the container with id 55c0615996ff2de819668b93628c591919491fd5d6e37ae5b77334af1723ed25 Nov 03 13:54:27.181965 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:27.181718 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:54:27.181965 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:27.181818 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:54:27.181965 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:27.181876 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:54:27.697085 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:27.697052 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv" event={"ID":"e73bb972-7cdd-4a6a-aae0-4345d0630989","Type":"ContainerStarted","Data":"34bffb888a3da97cb7456853f1a0f47fa4c8a1f0126f74a9075af4538585d433"} Nov 03 13:54:27.697560 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:27.697102 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv" event={"ID":"e73bb972-7cdd-4a6a-aae0-4345d0630989","Type":"ContainerStarted","Data":"55c0615996ff2de819668b93628c591919491fd5d6e37ae5b77334af1723ed25"} Nov 03 13:54:27.809543 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:27.809495 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv" podStartSLOduration=1.8094657079999998 podStartE2EDuration="1.809465708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:54:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:54:27.806446469 +0000 UTC m=+2949.817134710" watchObservedRunningTime="2026-11-03 13:54:27.809465708 +0000 UTC m=+2949.820153949" Nov 03 13:54:28.689362 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:28.689327 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv" Nov 03 13:54:28.695289 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:28.695260 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv" Nov 03 13:54:28.702876 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:28.702855 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv" Nov 03 13:54:28.703581 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:28.703563 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv" Nov 03 13:54:34.565493 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:34.558368 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-efbdec-7c85dd48cc-8s447"] Nov 03 13:54:34.582026 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:34.581995 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-efbdec-7c85dd48cc-8s447" Nov 03 13:54:34.598859 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:34.598822 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-efbdec-7c85dd48cc-8s447"] Nov 03 13:54:34.633969 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:34.633886 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-efbdec\"" Nov 03 13:54:34.637282 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:34.637258 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-efbdec-dockercfg-rnp9m\"" Nov 03 13:54:34.706904 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:34.706872 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-efbdec-84dbbbd855-n2fds"] Nov 03 13:54:34.711360 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:34.711336 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-efbdec-84dbbbd855-n2fds" Nov 03 13:54:34.765063 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:34.765030 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-efbdec-84dbbbd855-n2fds"] Nov 03 13:54:34.782801 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:34.782765 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzsdr\" (UniqueName: \"kubernetes.io/projected/609e1b10-a933-48b3-b224-1476116d3e66-kube-api-access-bzsdr\") pod \"payload-processing-e2e-shared-b-w4-efbdec-7c85dd48cc-8s447\" (UID: \"609e1b10-a933-48b3-b224-1476116d3e66\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-efbdec-7c85dd48cc-8s447" Nov 03 13:54:34.782988 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:34.782898 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/609e1b10-a933-48b3-b224-1476116d3e66-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-efbdec-7c85dd48cc-8s447\" (UID: \"609e1b10-a933-48b3-b224-1476116d3e66\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-efbdec-7c85dd48cc-8s447" Nov 03 13:54:34.783146 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:34.783031 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e0e9df1-0191-446b-95d9-6e08b67f5ed2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-efbdec-84dbbbd855-n2fds\" (UID: \"6e0e9df1-0191-446b-95d9-6e08b67f5ed2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-efbdec-84dbbbd855-n2fds" Nov 03 13:54:34.783146 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:34.783078 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gkjv\" (UniqueName: \"kubernetes.io/projected/6e0e9df1-0191-446b-95d9-6e08b67f5ed2-kube-api-access-4gkjv\") pod \"payload-pre-processing-e2e-shared-b-w4-efbdec-84dbbbd855-n2fds\" (UID: \"6e0e9df1-0191-446b-95d9-6e08b67f5ed2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-efbdec-84dbbbd855-n2fds" Nov 03 13:54:34.884098 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:34.884057 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gkjv\" (UniqueName: \"kubernetes.io/projected/6e0e9df1-0191-446b-95d9-6e08b67f5ed2-kube-api-access-4gkjv\") pod \"payload-pre-processing-e2e-shared-b-w4-efbdec-84dbbbd855-n2fds\" (UID: \"6e0e9df1-0191-446b-95d9-6e08b67f5ed2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-efbdec-84dbbbd855-n2fds" Nov 03 13:54:34.884278 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:34.884152 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzsdr\" (UniqueName: \"kubernetes.io/projected/609e1b10-a933-48b3-b224-1476116d3e66-kube-api-access-bzsdr\") pod \"payload-processing-e2e-shared-b-w4-efbdec-7c85dd48cc-8s447\" (UID: \"609e1b10-a933-48b3-b224-1476116d3e66\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-efbdec-7c85dd48cc-8s447" Nov 03 13:54:34.884278 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:34.884249 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/609e1b10-a933-48b3-b224-1476116d3e66-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-efbdec-7c85dd48cc-8s447\" (UID: \"609e1b10-a933-48b3-b224-1476116d3e66\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-efbdec-7c85dd48cc-8s447" Nov 03 13:54:34.884456 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:34.884388 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e0e9df1-0191-446b-95d9-6e08b67f5ed2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-efbdec-84dbbbd855-n2fds\" (UID: \"6e0e9df1-0191-446b-95d9-6e08b67f5ed2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-efbdec-84dbbbd855-n2fds" Nov 03 13:54:34.885179 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:34.885155 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/609e1b10-a933-48b3-b224-1476116d3e66-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-efbdec-7c85dd48cc-8s447\" (UID: \"609e1b10-a933-48b3-b224-1476116d3e66\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-efbdec-7c85dd48cc-8s447" Nov 03 13:54:34.885256 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:34.885175 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e0e9df1-0191-446b-95d9-6e08b67f5ed2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-efbdec-84dbbbd855-n2fds\" (UID: \"6e0e9df1-0191-446b-95d9-6e08b67f5ed2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-efbdec-84dbbbd855-n2fds" Nov 03 13:54:34.937325 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:34.937300 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzsdr\" (UniqueName: \"kubernetes.io/projected/609e1b10-a933-48b3-b224-1476116d3e66-kube-api-access-bzsdr\") pod \"payload-processing-e2e-shared-b-w4-efbdec-7c85dd48cc-8s447\" (UID: \"609e1b10-a933-48b3-b224-1476116d3e66\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-efbdec-7c85dd48cc-8s447" Nov 03 13:54:34.937510 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:34.937346 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gkjv\" (UniqueName: \"kubernetes.io/projected/6e0e9df1-0191-446b-95d9-6e08b67f5ed2-kube-api-access-4gkjv\") pod \"payload-pre-processing-e2e-shared-b-w4-efbdec-84dbbbd855-n2fds\" (UID: \"6e0e9df1-0191-446b-95d9-6e08b67f5ed2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-efbdec-84dbbbd855-n2fds" Nov 03 13:54:34.944278 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:34.944238 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-efbdec-7c85dd48cc-8s447" Nov 03 13:54:35.026053 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:35.026022 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-efbdec-84dbbbd855-n2fds" Nov 03 13:54:35.320758 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:35.320730 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-efbdec-7c85dd48cc-8s447"] Nov 03 13:54:35.329889 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:54:35.327532 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod609e1b10_a933_48b3_b224_1476116d3e66.slice/crio-5887f8d4a903bce0280e63e2ea4e38c51c8427eb57fca79d031098f958e943ef WatchSource:0}: Error finding container 5887f8d4a903bce0280e63e2ea4e38c51c8427eb57fca79d031098f958e943ef: Status 404 returned error can't find the container with id 5887f8d4a903bce0280e63e2ea4e38c51c8427eb57fca79d031098f958e943ef Nov 03 13:54:35.434260 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:35.434234 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-efbdec-84dbbbd855-n2fds"] Nov 03 13:54:35.486533 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:54:35.486501 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e0e9df1_0191_446b_95d9_6e08b67f5ed2.slice/crio-9861d275c9d6d2d14cd256e6a1064e15abe0f881b70d6d902038eb41695d87e0 WatchSource:0}: Error finding container 9861d275c9d6d2d14cd256e6a1064e15abe0f881b70d6d902038eb41695d87e0: Status 404 returned error can't find the container with id 9861d275c9d6d2d14cd256e6a1064e15abe0f881b70d6d902038eb41695d87e0 Nov 03 13:54:35.749073 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:35.749035 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-efbdec-84dbbbd855-n2fds" event={"ID":"6e0e9df1-0191-446b-95d9-6e08b67f5ed2","Type":"ContainerStarted","Data":"febdaa01b20192744f094880b240e87a3ccd5a07cbe3562e0004f7c62a7ccd3b"} Nov 03 13:54:35.749573 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:35.749101 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-efbdec-84dbbbd855-n2fds" event={"ID":"6e0e9df1-0191-446b-95d9-6e08b67f5ed2","Type":"ContainerStarted","Data":"9861d275c9d6d2d14cd256e6a1064e15abe0f881b70d6d902038eb41695d87e0"} Nov 03 13:54:35.749573 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:35.749241 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-efbdec-84dbbbd855-n2fds" Nov 03 13:54:35.750820 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:35.750789 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-efbdec-7c85dd48cc-8s447" event={"ID":"609e1b10-a933-48b3-b224-1476116d3e66","Type":"ContainerStarted","Data":"adf9b3084e6d79ee34c230c3792704efbdd328feb6bf71ad7ca3b8f75d455a0f"} Nov 03 13:54:35.750949 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:35.750843 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-efbdec-7c85dd48cc-8s447" event={"ID":"609e1b10-a933-48b3-b224-1476116d3e66","Type":"ContainerStarted","Data":"5887f8d4a903bce0280e63e2ea4e38c51c8427eb57fca79d031098f958e943ef"} Nov 03 13:54:35.750949 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:35.750871 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-efbdec-7c85dd48cc-8s447" Nov 03 13:54:35.752443 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:35.752420 5979 generic.go:356] "Generic (PLEG): container finished" podID="f76845b8-e625-4425-ad28-74612d33985c" containerID="a7cf676f1fa259a460515dd1a4b7a3c48b4d7a44850101fcad73a470b97e9673" exitCode=0 Nov 03 13:54:35.752538 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:35.752460 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld" event={"ID":"f76845b8-e625-4425-ad28-74612d33985c","Type":"ContainerDied","Data":"a7cf676f1fa259a460515dd1a4b7a3c48b4d7a44850101fcad73a470b97e9673"} Nov 03 13:54:35.752786 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:35.752773 5979 scope.go:122] "RemoveContainer" containerID="a7cf676f1fa259a460515dd1a4b7a3c48b4d7a44850101fcad73a470b97e9673" Nov 03 13:54:35.754209 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:35.754176 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:54:35.754317 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:35.754261 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:54:35.754317 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:35.754311 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:54:35.856219 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:35.856170 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-efbdec-84dbbbd855-n2fds" podStartSLOduration=1.8561571460000001 podStartE2EDuration="1.856157146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:54:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:54:35.855848868 +0000 UTC m=+2957.866537110" watchObservedRunningTime="2026-11-03 13:54:35.856157146 +0000 UTC m=+2957.866845388" Nov 03 13:54:35.962728 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:35.962665 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-efbdec-7c85dd48cc-8s447" podStartSLOduration=1.9626464129999999 podStartE2EDuration="1.962646413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:54:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:54:35.961664849 +0000 UTC m=+2957.972353092" watchObservedRunningTime="2026-11-03 13:54:35.962646413 +0000 UTC m=+2957.973334656" Nov 03 13:54:36.774927 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:36.774882 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld" event={"ID":"f76845b8-e625-4425-ad28-74612d33985c","Type":"ContainerStarted","Data":"f2d4e8845a5e7f053b9899cc345a5e90e1d95ca83cbf5184e38dcf144c002eb7"} Nov 03 13:54:37.191782 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:37.191741 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld" Nov 03 13:54:37.192996 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:37.192967 5979 patch_prober.go:29] interesting pod/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" start-of-body= Nov 03 13:54:37.193126 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:37.193023 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld" podUID="f76845b8-e625-4425-ad28-74612d33985c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" Nov 03 13:54:37.235068 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:54:37.235026 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:54:38.191905 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:38.191863 5979 patch_prober.go:29] interesting pod/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" start-of-body= Nov 03 13:54:38.192337 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:38.191916 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld" podUID="f76845b8-e625-4425-ad28-74612d33985c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" Nov 03 13:54:38.892071 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:38.892034 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld" Nov 03 13:54:39.192751 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:39.192657 5979 patch_prober.go:29] interesting pod/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" start-of-body= Nov 03 13:54:39.192751 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:39.192719 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld" podUID="f76845b8-e625-4425-ad28-74612d33985c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" Nov 03 13:54:40.192803 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:40.192766 5979 patch_prober.go:29] interesting pod/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" start-of-body= Nov 03 13:54:40.193210 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:40.192819 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld" podUID="f76845b8-e625-4425-ad28-74612d33985c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" Nov 03 13:54:41.192177 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:41.192136 5979 patch_prober.go:29] interesting pod/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" start-of-body= Nov 03 13:54:41.192374 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:41.192196 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld" podUID="f76845b8-e625-4425-ad28-74612d33985c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.25:15021/healthz/ready\": dial tcp 10.134.1.25:15021: connect: connection refused" Nov 03 13:54:42.197311 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:42.197282 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld" Nov 03 13:54:42.198348 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:42.198329 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld" Nov 03 13:54:46.778638 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:46.778606 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-efbdec-84dbbbd855-n2fds" Nov 03 13:54:46.779102 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:46.778837 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-efbdec-7c85dd48cc-8s447" Nov 03 13:54:58.328565 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:58.328533 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-92546f-6ddf84c8fb-dnftl"] Nov 03 13:54:58.331769 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:58.331748 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-92546f-6ddf84c8fb-dnftl" Nov 03 13:54:58.430304 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:58.430271 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-92546f-serving-cert\"" Nov 03 13:54:58.433411 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:58.433388 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-92546f-6ddf84c8fb-dnftl"] Nov 03 13:54:58.504964 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:58.504929 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08240bd7-0d0e-485e-ae5b-481d8bde3d4a-maas-api-tls\") pod \"maas-api-e2e-explicit-92546f-6ddf84c8fb-dnftl\" (UID: \"08240bd7-0d0e-485e-ae5b-481d8bde3d4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-92546f-6ddf84c8fb-dnftl" Nov 03 13:54:58.505157 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:58.505137 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmh65\" (UniqueName: \"kubernetes.io/projected/08240bd7-0d0e-485e-ae5b-481d8bde3d4a-kube-api-access-qmh65\") pod \"maas-api-e2e-explicit-92546f-6ddf84c8fb-dnftl\" (UID: \"08240bd7-0d0e-485e-ae5b-481d8bde3d4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-92546f-6ddf84c8fb-dnftl" Nov 03 13:54:58.505250 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:58.505236 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08240bd7-0d0e-485e-ae5b-481d8bde3d4a-openshift-service-ca\") pod \"maas-api-e2e-explicit-92546f-6ddf84c8fb-dnftl\" (UID: \"08240bd7-0d0e-485e-ae5b-481d8bde3d4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-92546f-6ddf84c8fb-dnftl" Nov 03 13:54:58.505311 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:58.505295 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08240bd7-0d0e-485e-ae5b-481d8bde3d4a-metrics-certs\") pod \"maas-api-e2e-explicit-92546f-6ddf84c8fb-dnftl\" (UID: \"08240bd7-0d0e-485e-ae5b-481d8bde3d4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-92546f-6ddf84c8fb-dnftl" Nov 03 13:54:58.613109 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:58.610362 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmh65\" (UniqueName: \"kubernetes.io/projected/08240bd7-0d0e-485e-ae5b-481d8bde3d4a-kube-api-access-qmh65\") pod \"maas-api-e2e-explicit-92546f-6ddf84c8fb-dnftl\" (UID: \"08240bd7-0d0e-485e-ae5b-481d8bde3d4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-92546f-6ddf84c8fb-dnftl" Nov 03 13:54:58.613524 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:58.613454 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08240bd7-0d0e-485e-ae5b-481d8bde3d4a-openshift-service-ca\") pod \"maas-api-e2e-explicit-92546f-6ddf84c8fb-dnftl\" (UID: \"08240bd7-0d0e-485e-ae5b-481d8bde3d4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-92546f-6ddf84c8fb-dnftl" Nov 03 13:54:58.613693 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:58.613542 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08240bd7-0d0e-485e-ae5b-481d8bde3d4a-metrics-certs\") pod \"maas-api-e2e-explicit-92546f-6ddf84c8fb-dnftl\" (UID: \"08240bd7-0d0e-485e-ae5b-481d8bde3d4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-92546f-6ddf84c8fb-dnftl" Nov 03 13:54:58.613693 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:58.613668 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08240bd7-0d0e-485e-ae5b-481d8bde3d4a-maas-api-tls\") pod \"maas-api-e2e-explicit-92546f-6ddf84c8fb-dnftl\" (UID: \"08240bd7-0d0e-485e-ae5b-481d8bde3d4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-92546f-6ddf84c8fb-dnftl" Nov 03 13:54:58.614251 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:58.614226 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08240bd7-0d0e-485e-ae5b-481d8bde3d4a-openshift-service-ca\") pod \"maas-api-e2e-explicit-92546f-6ddf84c8fb-dnftl\" (UID: \"08240bd7-0d0e-485e-ae5b-481d8bde3d4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-92546f-6ddf84c8fb-dnftl" Nov 03 13:54:58.620238 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:58.620208 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08240bd7-0d0e-485e-ae5b-481d8bde3d4a-metrics-certs\") pod \"maas-api-e2e-explicit-92546f-6ddf84c8fb-dnftl\" (UID: \"08240bd7-0d0e-485e-ae5b-481d8bde3d4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-92546f-6ddf84c8fb-dnftl" Nov 03 13:54:58.621499 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:54:58.620923 5979 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-92546f-serving-cert: secret "maas-api-e2e-explicit-92546f-serving-cert" not found Nov 03 13:54:58.621499 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:54:58.620996 5979 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08240bd7-0d0e-485e-ae5b-481d8bde3d4a-maas-api-tls podName:08240bd7-0d0e-485e-ae5b-481d8bde3d4a nodeName:}" failed. No retries permitted until 2026-11-03 13:54:59.120975789 +0000 UTC m=+2981.131664015 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/08240bd7-0d0e-485e-ae5b-481d8bde3d4a-maas-api-tls") pod "maas-api-e2e-explicit-92546f-6ddf84c8fb-dnftl" (UID: "08240bd7-0d0e-485e-ae5b-481d8bde3d4a") : secret "maas-api-e2e-explicit-92546f-serving-cert" not found Nov 03 13:54:58.664110 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:58.664069 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmh65\" (UniqueName: \"kubernetes.io/projected/08240bd7-0d0e-485e-ae5b-481d8bde3d4a-kube-api-access-qmh65\") pod \"maas-api-e2e-explicit-92546f-6ddf84c8fb-dnftl\" (UID: \"08240bd7-0d0e-485e-ae5b-481d8bde3d4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-92546f-6ddf84c8fb-dnftl" Nov 03 13:54:59.122099 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:59.122061 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08240bd7-0d0e-485e-ae5b-481d8bde3d4a-maas-api-tls\") pod \"maas-api-e2e-explicit-92546f-6ddf84c8fb-dnftl\" (UID: \"08240bd7-0d0e-485e-ae5b-481d8bde3d4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-92546f-6ddf84c8fb-dnftl" Nov 03 13:54:59.124579 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:59.124551 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08240bd7-0d0e-485e-ae5b-481d8bde3d4a-maas-api-tls\") pod \"maas-api-e2e-explicit-92546f-6ddf84c8fb-dnftl\" (UID: \"08240bd7-0d0e-485e-ae5b-481d8bde3d4a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-92546f-6ddf84c8fb-dnftl" Nov 03 13:54:59.244427 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:59.244387 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-92546f-6ddf84c8fb-dnftl" Nov 03 13:54:59.618776 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:59.618738 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-92546f-6ddf84c8fb-dnftl"] Nov 03 13:54:59.626076 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:54:59.626050 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08240bd7_0d0e_485e_ae5b_481d8bde3d4a.slice/crio-10a5afcb61f04ad5fd992cd918fefb25875d0445cec233e7b7290b1e4f46fbac WatchSource:0}: Error finding container 10a5afcb61f04ad5fd992cd918fefb25875d0445cec233e7b7290b1e4f46fbac: Status 404 returned error can't find the container with id 10a5afcb61f04ad5fd992cd918fefb25875d0445cec233e7b7290b1e4f46fbac Nov 03 13:54:59.628164 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:59.628143 5979 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 13:54:59.921935 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:54:59.921897 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-92546f-6ddf84c8fb-dnftl" event={"ID":"08240bd7-0d0e-485e-ae5b-481d8bde3d4a","Type":"ContainerStarted","Data":"10a5afcb61f04ad5fd992cd918fefb25875d0445cec233e7b7290b1e4f46fbac"} Nov 03 13:55:00.931380 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:55:00.931343 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-92546f-6ddf84c8fb-dnftl" event={"ID":"08240bd7-0d0e-485e-ae5b-481d8bde3d4a","Type":"ContainerStarted","Data":"85883f15a4a4698dbdaad9637527aa0a4702c31a5e9f74cbdeb671f6be04ca26"} Nov 03 13:55:00.931828 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:55:00.931501 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-92546f-6ddf84c8fb-dnftl" Nov 03 13:55:01.042058 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:55:01.042004 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-92546f-6ddf84c8fb-dnftl" podStartSLOduration=2.084856872 podStartE2EDuration="3.041989061s" totalImagesPullingTime="957.132189ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:54:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:55:01.041213296 +0000 UTC m=+2983.051901550" watchObservedRunningTime="2026-11-03 13:55:01.041989061 +0000 UTC m=+2983.052677303" Nov 03 13:55:06.944988 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:55:06.944957 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-92546f-6ddf84c8fb-dnftl" Nov 03 13:55:07.234847 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:55:07.234759 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:55:15.227313 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:55:15.227275 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:55:20.480360 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:55:20.480331 5979 scope.go:122] "RemoveContainer" containerID="c84e56fbead790292df2cf6619c1195b77e24295d8e89f50a5058f5df5fc942c" Nov 03 13:55:20.489867 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:55:20.489843 5979 scope.go:122] "RemoveContainer" containerID="4186b6866a3ddff825ba1618ba27a4245046474dc8c67a7e89b6775fd4efa34b" Nov 03 13:55:26.162975 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:55:26.162895 5979 generic.go:356] "Generic (PLEG): container finished" podID="8a1133dc-605d-4871-8190-7714aca1ce00" containerID="808a83b27db3fc13e449f2a22c80e386dab0cadb6aecd4ceaa90c7a860866190" exitCode=0 Nov 03 13:55:26.163396 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:55:26.162967 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m" event={"ID":"8a1133dc-605d-4871-8190-7714aca1ce00","Type":"ContainerDied","Data":"808a83b27db3fc13e449f2a22c80e386dab0cadb6aecd4ceaa90c7a860866190"} Nov 03 13:55:26.163439 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:55:26.163412 5979 scope.go:122] "RemoveContainer" containerID="808a83b27db3fc13e449f2a22c80e386dab0cadb6aecd4ceaa90c7a860866190" Nov 03 13:55:26.165130 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:55:26.165105 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:55:26.165222 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:55:26.165170 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:55:26.165222 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:55:26.165202 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:55:27.175193 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:55:27.175153 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m" event={"ID":"8a1133dc-605d-4871-8190-7714aca1ce00","Type":"ContainerStarted","Data":"cd3e320ba158fcb7fe6ac78b1ed06eff5950bc64d9ce9dd40f4d72152a7a5b9e"} Nov 03 13:55:27.567413 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:55:27.566878 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m" Nov 03 13:55:27.567413 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:55:27.567009 5979 patch_prober.go:29] interesting pod/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" start-of-body= Nov 03 13:55:27.567413 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:55:27.567066 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m" podUID="8a1133dc-605d-4871-8190-7714aca1ce00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" Nov 03 13:55:28.568736 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:55:28.568698 5979 patch_prober.go:29] interesting pod/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" start-of-body= Nov 03 13:55:28.569316 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:55:28.569285 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m" podUID="8a1133dc-605d-4871-8190-7714aca1ce00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" Nov 03 13:55:29.573521 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:55:29.573258 5979 patch_prober.go:29] interesting pod/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" start-of-body= Nov 03 13:55:29.573521 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:55:29.573317 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m" podUID="8a1133dc-605d-4871-8190-7714aca1ce00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" Nov 03 13:55:30.566494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:55:30.565623 5979 patch_prober.go:29] interesting pod/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" start-of-body= Nov 03 13:55:30.566494 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:55:30.565697 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m" podUID="8a1133dc-605d-4871-8190-7714aca1ce00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" Nov 03 13:55:31.568940 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:55:31.568906 5979 patch_prober.go:29] interesting pod/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" start-of-body= Nov 03 13:55:31.569426 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:55:31.569404 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m" podUID="8a1133dc-605d-4871-8190-7714aca1ce00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" Nov 03 13:55:32.568167 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:55:32.568131 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m" Nov 03 13:55:32.568602 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:55:32.568582 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m" Nov 03 13:55:32.569289 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:55:32.569268 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m" Nov 03 13:55:37.236980 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:55:37.236943 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:55:44.753545 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:55:44.753507 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:56:07.234602 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:56:07.234566 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:56:19.555927 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:56:19.555889 5979 generic.go:356] "Generic (PLEG): container finished" podID="e73bb972-7cdd-4a6a-aae0-4345d0630989" containerID="34bffb888a3da97cb7456853f1a0f47fa4c8a1f0126f74a9075af4538585d433" exitCode=0 Nov 03 13:56:19.556435 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:56:19.555962 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv" event={"ID":"e73bb972-7cdd-4a6a-aae0-4345d0630989","Type":"ContainerDied","Data":"34bffb888a3da97cb7456853f1a0f47fa4c8a1f0126f74a9075af4538585d433"} Nov 03 13:56:19.556574 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:56:19.556555 5979 scope.go:122] "RemoveContainer" containerID="34bffb888a3da97cb7456853f1a0f47fa4c8a1f0126f74a9075af4538585d433" Nov 03 13:56:19.558280 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:56:19.558252 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:56:19.558376 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:56:19.558322 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:56:19.558376 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:56:19.558353 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:56:20.576813 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:56:20.576776 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv" event={"ID":"e73bb972-7cdd-4a6a-aae0-4345d0630989","Type":"ContainerStarted","Data":"1a5c1ff335b0329062bdb928f4584e35d2e0842f6f869d033edff69eabc799d8"} Nov 03 13:56:20.689545 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:56:20.689511 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv" Nov 03 13:56:20.689759 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:56:20.689727 5979 patch_prober.go:29] interesting pod/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.31:15021/healthz/ready\": dial tcp 10.134.1.31:15021: connect: connection refused" start-of-body= Nov 03 13:56:20.689827 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:56:20.689770 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv" podUID="e73bb972-7cdd-4a6a-aae0-4345d0630989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.31:15021/healthz/ready\": dial tcp 10.134.1.31:15021: connect: connection refused" Nov 03 13:56:21.689624 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:56:21.689592 5979 patch_prober.go:29] interesting pod/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.31:15021/healthz/ready\": dial tcp 10.134.1.31:15021: connect: connection refused" start-of-body= Nov 03 13:56:21.690111 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:56:21.689642 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv" podUID="e73bb972-7cdd-4a6a-aae0-4345d0630989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.31:15021/healthz/ready\": dial tcp 10.134.1.31:15021: connect: connection refused" Nov 03 13:56:22.689558 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:56:22.689517 5979 patch_prober.go:29] interesting pod/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.31:15021/healthz/ready\": dial tcp 10.134.1.31:15021: connect: connection refused" start-of-body= Nov 03 13:56:22.689779 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:56:22.689572 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv" podUID="e73bb972-7cdd-4a6a-aae0-4345d0630989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.31:15021/healthz/ready\": dial tcp 10.134.1.31:15021: connect: connection refused" Nov 03 13:56:23.689808 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:56:23.689766 5979 patch_prober.go:29] interesting pod/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.31:15021/healthz/ready\": dial tcp 10.134.1.31:15021: connect: connection refused" start-of-body= Nov 03 13:56:23.690217 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:56:23.689842 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv" podUID="e73bb972-7cdd-4a6a-aae0-4345d0630989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.31:15021/healthz/ready\": dial tcp 10.134.1.31:15021: connect: connection refused" Nov 03 13:56:24.690019 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:56:24.689979 5979 patch_prober.go:29] interesting pod/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.31:15021/healthz/ready\": dial tcp 10.134.1.31:15021: connect: connection refused" start-of-body= Nov 03 13:56:24.690550 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:56:24.690056 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv" podUID="e73bb972-7cdd-4a6a-aae0-4345d0630989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.31:15021/healthz/ready\": dial tcp 10.134.1.31:15021: connect: connection refused" Nov 03 13:56:25.689508 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:56:25.689448 5979 patch_prober.go:29] interesting pod/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.31:15021/healthz/ready\": dial tcp 10.134.1.31:15021: connect: connection refused" start-of-body= Nov 03 13:56:25.689709 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:56:25.689540 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv" podUID="e73bb972-7cdd-4a6a-aae0-4345d0630989" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.31:15021/healthz/ready\": dial tcp 10.134.1.31:15021: connect: connection refused" Nov 03 13:56:26.699238 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:56:26.699206 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv" Nov 03 13:56:26.699719 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:56:26.699634 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv" Nov 03 13:56:26.700324 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:56:26.700305 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv" Nov 03 13:56:37.234120 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:56:37.234077 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:56:38.045904 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:56:38.045859 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfc2cc3a-nbl96"] Nov 03 13:56:38.092370 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:56:38.092333 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfc2cc3a-nbl96"] Nov 03 13:56:38.117197 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:56:38.117166 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f285fcb6-7753-4725-9381-a484d084937a" path="/var/lib/kubelet/pods/f285fcb6-7753-4725-9381-a484d084937a/volumes" Nov 03 13:56:40.339261 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:56:40.339226 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:57:07.234855 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:57:07.234806 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:57:19.011606 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:19.011572 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:57:37.235133 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:57:37.235087 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:57:41.113377 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:41.113341 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj"] Nov 03 13:57:41.113987 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:41.113589 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj" podUID="210b9bed-6df7-4f71-982e-ce2f72287b17" containerName="istio-proxy" containerID="cri-o://a2db1f88c60565bd7643e74b1123e391ad271b0cd09eed4cd8718876b6c26f87" gracePeriod=30 Nov 03 13:57:45.270058 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:45.270021 5979 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 13:57:45.270453 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:45.270076 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj" podUID="210b9bed-6df7-4f71-982e-ce2f72287b17" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 13:57:46.371677 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:46.371649 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj" Nov 03 13:57:46.392364 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:46.392336 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/210b9bed-6df7-4f71-982e-ce2f72287b17-istio-token\" (UniqueName: \"kubernetes.io/projected/210b9bed-6df7-4f71-982e-ce2f72287b17-istio-token\") pod \"210b9bed-6df7-4f71-982e-ce2f72287b17\" (UID: \"210b9bed-6df7-4f71-982e-ce2f72287b17\") " Nov 03 13:57:46.392516 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:46.392396 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/210b9bed-6df7-4f71-982e-ce2f72287b17-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/210b9bed-6df7-4f71-982e-ce2f72287b17-istio-envoy\") pod \"210b9bed-6df7-4f71-982e-ce2f72287b17\" (UID: \"210b9bed-6df7-4f71-982e-ce2f72287b17\") " Nov 03 13:57:46.392516 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:46.392492 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/210b9bed-6df7-4f71-982e-ce2f72287b17-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/210b9bed-6df7-4f71-982e-ce2f72287b17-workload-socket\") pod \"210b9bed-6df7-4f71-982e-ce2f72287b17\" (UID: \"210b9bed-6df7-4f71-982e-ce2f72287b17\") " Nov 03 13:57:46.392660 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:46.392529 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/210b9bed-6df7-4f71-982e-ce2f72287b17-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/210b9bed-6df7-4f71-982e-ce2f72287b17-credential-socket\") pod \"210b9bed-6df7-4f71-982e-ce2f72287b17\" (UID: \"210b9bed-6df7-4f71-982e-ce2f72287b17\") " Nov 03 13:57:46.392660 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:46.392569 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/210b9bed-6df7-4f71-982e-ce2f72287b17-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/210b9bed-6df7-4f71-982e-ce2f72287b17-istio-data\") pod \"210b9bed-6df7-4f71-982e-ce2f72287b17\" (UID: \"210b9bed-6df7-4f71-982e-ce2f72287b17\") " Nov 03 13:57:46.392660 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:46.392603 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/210b9bed-6df7-4f71-982e-ce2f72287b17-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/210b9bed-6df7-4f71-982e-ce2f72287b17-workload-certs\") pod \"210b9bed-6df7-4f71-982e-ce2f72287b17\" (UID: \"210b9bed-6df7-4f71-982e-ce2f72287b17\") " Nov 03 13:57:46.392838 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:46.392729 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/210b9bed-6df7-4f71-982e-ce2f72287b17-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/210b9bed-6df7-4f71-982e-ce2f72287b17-istio-podinfo\") pod \"210b9bed-6df7-4f71-982e-ce2f72287b17\" (UID: \"210b9bed-6df7-4f71-982e-ce2f72287b17\") " Nov 03 13:57:46.392838 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:46.392769 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/210b9bed-6df7-4f71-982e-ce2f72287b17-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/210b9bed-6df7-4f71-982e-ce2f72287b17-istiod-ca-cert\") pod \"210b9bed-6df7-4f71-982e-ce2f72287b17\" (UID: \"210b9bed-6df7-4f71-982e-ce2f72287b17\") " Nov 03 13:57:46.392838 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:46.392774 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/210b9bed-6df7-4f71-982e-ce2f72287b17-workload-socket" pod "210b9bed-6df7-4f71-982e-ce2f72287b17" (UID: "210b9bed-6df7-4f71-982e-ce2f72287b17"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:57:46.392838 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:46.392818 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/210b9bed-6df7-4f71-982e-ce2f72287b17-credential-socket" pod "210b9bed-6df7-4f71-982e-ce2f72287b17" (UID: "210b9bed-6df7-4f71-982e-ce2f72287b17"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:57:46.393060 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:46.392895 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/210b9bed-6df7-4f71-982e-ce2f72287b17-istio-data" pod "210b9bed-6df7-4f71-982e-ce2f72287b17" (UID: "210b9bed-6df7-4f71-982e-ce2f72287b17"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:57:46.393060 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:46.392917 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/210b9bed-6df7-4f71-982e-ce2f72287b17-kube-api-access-fm725\" (UniqueName: \"kubernetes.io/projected/210b9bed-6df7-4f71-982e-ce2f72287b17-kube-api-access-fm725\") pod \"210b9bed-6df7-4f71-982e-ce2f72287b17\" (UID: \"210b9bed-6df7-4f71-982e-ce2f72287b17\") " Nov 03 13:57:46.393060 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:46.393004 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/210b9bed-6df7-4f71-982e-ce2f72287b17-workload-certs" pod "210b9bed-6df7-4f71-982e-ce2f72287b17" (UID: "210b9bed-6df7-4f71-982e-ce2f72287b17"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:57:46.393268 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:46.393173 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/210b9bed-6df7-4f71-982e-ce2f72287b17-istiod-ca-cert" pod "210b9bed-6df7-4f71-982e-ce2f72287b17" (UID: "210b9bed-6df7-4f71-982e-ce2f72287b17"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:57:46.393456 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:46.393437 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/210b9bed-6df7-4f71-982e-ce2f72287b17-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:57:46.393566 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:46.393462 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/210b9bed-6df7-4f71-982e-ce2f72287b17-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:57:46.393566 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:46.393490 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/210b9bed-6df7-4f71-982e-ce2f72287b17-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:57:46.393566 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:46.393503 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/210b9bed-6df7-4f71-982e-ce2f72287b17-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:57:46.393566 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:46.393516 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/210b9bed-6df7-4f71-982e-ce2f72287b17-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:57:46.394742 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:46.394722 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/210b9bed-6df7-4f71-982e-ce2f72287b17-istio-podinfo" pod "210b9bed-6df7-4f71-982e-ce2f72287b17" (UID: "210b9bed-6df7-4f71-982e-ce2f72287b17"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:57:46.394816 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:46.394763 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/210b9bed-6df7-4f71-982e-ce2f72287b17-istio-envoy" pod "210b9bed-6df7-4f71-982e-ce2f72287b17" (UID: "210b9bed-6df7-4f71-982e-ce2f72287b17"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:57:46.395070 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:46.395042 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/210b9bed-6df7-4f71-982e-ce2f72287b17-istio-token" pod "210b9bed-6df7-4f71-982e-ce2f72287b17" (UID: "210b9bed-6df7-4f71-982e-ce2f72287b17"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:57:46.395183 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:46.395167 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/210b9bed-6df7-4f71-982e-ce2f72287b17-kube-api-access-fm725" pod "210b9bed-6df7-4f71-982e-ce2f72287b17" (UID: "210b9bed-6df7-4f71-982e-ce2f72287b17"). InnerVolumeSpecName "kube-api-access-fm725". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:57:46.494938 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:46.494902 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/210b9bed-6df7-4f71-982e-ce2f72287b17-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:57:46.494938 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:46.494931 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/210b9bed-6df7-4f71-982e-ce2f72287b17-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:57:46.494938 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:46.494942 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/210b9bed-6df7-4f71-982e-ce2f72287b17-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:57:46.495194 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:46.494953 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm725\" (UniqueName: \"kubernetes.io/projected/210b9bed-6df7-4f71-982e-ce2f72287b17-kube-api-access-fm725\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:57:47.143216 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:47.143180 5979 generic.go:356] "Generic (PLEG): container finished" podID="210b9bed-6df7-4f71-982e-ce2f72287b17" containerID="a2db1f88c60565bd7643e74b1123e391ad271b0cd09eed4cd8718876b6c26f87" exitCode=0 Nov 03 13:57:47.143418 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:47.143230 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj" Nov 03 13:57:47.143418 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:47.143260 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj" event={"ID":"210b9bed-6df7-4f71-982e-ce2f72287b17","Type":"ContainerDied","Data":"a2db1f88c60565bd7643e74b1123e391ad271b0cd09eed4cd8718876b6c26f87"} Nov 03 13:57:47.143418 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:47.143319 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj" event={"ID":"210b9bed-6df7-4f71-982e-ce2f72287b17","Type":"ContainerDied","Data":"9419e77ab6fdb1bb18a6cf93fd5748c40916bd44d01795dce8b7bd95f45a5b3d"} Nov 03 13:57:47.143418 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:47.143351 5979 scope.go:122] "RemoveContainer" containerID="a2db1f88c60565bd7643e74b1123e391ad271b0cd09eed4cd8718876b6c26f87" Nov 03 13:57:47.158349 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:47.158326 5979 scope.go:122] "RemoveContainer" containerID="a2db1f88c60565bd7643e74b1123e391ad271b0cd09eed4cd8718876b6c26f87" Nov 03 13:57:47.158635 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:57:47.158609 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2db1f88c60565bd7643e74b1123e391ad271b0cd09eed4cd8718876b6c26f87\": container with ID starting with a2db1f88c60565bd7643e74b1123e391ad271b0cd09eed4cd8718876b6c26f87 not found: ID does not exist" containerID="a2db1f88c60565bd7643e74b1123e391ad271b0cd09eed4cd8718876b6c26f87" Nov 03 13:57:47.158731 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:47.158644 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2db1f88c60565bd7643e74b1123e391ad271b0cd09eed4cd8718876b6c26f87"} err="failed to get container status \"a2db1f88c60565bd7643e74b1123e391ad271b0cd09eed4cd8718876b6c26f87\": rpc error: code = NotFound desc = could not find container \"a2db1f88c60565bd7643e74b1123e391ad271b0cd09eed4cd8718876b6c26f87\": container with ID starting with a2db1f88c60565bd7643e74b1123e391ad271b0cd09eed4cd8718876b6c26f87 not found: ID does not exist" Nov 03 13:57:47.300162 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:47.300123 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj"] Nov 03 13:57:47.347669 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:47.347627 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-ch9rj"] Nov 03 13:57:48.115703 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:48.115670 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="210b9bed-6df7-4f71-982e-ce2f72287b17" path="/var/lib/kubelet/pods/210b9bed-6df7-4f71-982e-ce2f72287b17/volumes" Nov 03 13:57:50.406969 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:50.406927 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm"] Nov 03 13:57:50.407457 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:50.407183 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm" podUID="c58e28cd-87a4-4a44-abb3-bc1ea9b60449" containerName="istio-proxy" containerID="cri-o://2203629e33a905e7d77a21f6ecfbb94bd82472c523ba907becd57b632c96009c" gracePeriod=30 Nov 03 13:57:55.686714 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:55.686657 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm" Nov 03 13:57:55.811265 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:55.811224 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-istiod-ca-cert\") pod \"c58e28cd-87a4-4a44-abb3-bc1ea9b60449\" (UID: \"c58e28cd-87a4-4a44-abb3-bc1ea9b60449\") " Nov 03 13:57:55.811265 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:55.811267 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-workload-certs\") pod \"c58e28cd-87a4-4a44-abb3-bc1ea9b60449\" (UID: \"c58e28cd-87a4-4a44-abb3-bc1ea9b60449\") " Nov 03 13:57:55.811566 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:55.811311 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-workload-socket\") pod \"c58e28cd-87a4-4a44-abb3-bc1ea9b60449\" (UID: \"c58e28cd-87a4-4a44-abb3-bc1ea9b60449\") " Nov 03 13:57:55.811566 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:55.811349 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-istio-envoy\") pod \"c58e28cd-87a4-4a44-abb3-bc1ea9b60449\" (UID: \"c58e28cd-87a4-4a44-abb3-bc1ea9b60449\") " Nov 03 13:57:55.811566 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:55.811391 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-istio-podinfo\") pod \"c58e28cd-87a4-4a44-abb3-bc1ea9b60449\" (UID: \"c58e28cd-87a4-4a44-abb3-bc1ea9b60449\") " Nov 03 13:57:55.811566 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:55.811437 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-kube-api-access-h2h8d\" (UniqueName: \"kubernetes.io/projected/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-kube-api-access-h2h8d\") pod \"c58e28cd-87a4-4a44-abb3-bc1ea9b60449\" (UID: \"c58e28cd-87a4-4a44-abb3-bc1ea9b60449\") " Nov 03 13:57:55.811799 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:55.811615 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-credential-socket\") pod \"c58e28cd-87a4-4a44-abb3-bc1ea9b60449\" (UID: \"c58e28cd-87a4-4a44-abb3-bc1ea9b60449\") " Nov 03 13:57:55.811799 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:55.811679 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-workload-certs" pod "c58e28cd-87a4-4a44-abb3-bc1ea9b60449" (UID: "c58e28cd-87a4-4a44-abb3-bc1ea9b60449"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:57:55.811799 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:55.811686 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-istiod-ca-cert" pod "c58e28cd-87a4-4a44-abb3-bc1ea9b60449" (UID: "c58e28cd-87a4-4a44-abb3-bc1ea9b60449"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:57:55.811799 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:55.811692 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-workload-socket" pod "c58e28cd-87a4-4a44-abb3-bc1ea9b60449" (UID: "c58e28cd-87a4-4a44-abb3-bc1ea9b60449"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:57:55.811799 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:55.811725 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-istio-token\" (UniqueName: \"kubernetes.io/projected/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-istio-token\") pod \"c58e28cd-87a4-4a44-abb3-bc1ea9b60449\" (UID: \"c58e28cd-87a4-4a44-abb3-bc1ea9b60449\") " Nov 03 13:57:55.811799 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:55.811782 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-istio-data\") pod \"c58e28cd-87a4-4a44-abb3-bc1ea9b60449\" (UID: \"c58e28cd-87a4-4a44-abb3-bc1ea9b60449\") " Nov 03 13:57:55.812136 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:55.811909 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-credential-socket" pod "c58e28cd-87a4-4a44-abb3-bc1ea9b60449" (UID: "c58e28cd-87a4-4a44-abb3-bc1ea9b60449"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:57:55.812136 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:55.812072 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-istio-data" pod "c58e28cd-87a4-4a44-abb3-bc1ea9b60449" (UID: "c58e28cd-87a4-4a44-abb3-bc1ea9b60449"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:57:55.812419 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:55.812398 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:57:55.812465 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:55.812429 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:57:55.812465 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:55.812447 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:57:55.812567 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:55.812464 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:57:55.812567 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:55.812508 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:57:55.813820 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:55.813799 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-istio-podinfo" pod "c58e28cd-87a4-4a44-abb3-bc1ea9b60449" (UID: "c58e28cd-87a4-4a44-abb3-bc1ea9b60449"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:57:55.813932 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:55.813845 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-istio-token" pod "c58e28cd-87a4-4a44-abb3-bc1ea9b60449" (UID: "c58e28cd-87a4-4a44-abb3-bc1ea9b60449"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:57:55.813997 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:55.813973 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-istio-envoy" pod "c58e28cd-87a4-4a44-abb3-bc1ea9b60449" (UID: "c58e28cd-87a4-4a44-abb3-bc1ea9b60449"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:57:55.814052 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:55.814025 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-kube-api-access-h2h8d" pod "c58e28cd-87a4-4a44-abb3-bc1ea9b60449" (UID: "c58e28cd-87a4-4a44-abb3-bc1ea9b60449"). InnerVolumeSpecName "kube-api-access-h2h8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:57:55.913796 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:55.913751 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:57:55.913796 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:55.913789 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:57:55.914057 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:55.913806 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2h8d\" (UniqueName: \"kubernetes.io/projected/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-kube-api-access-h2h8d\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:57:55.914057 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:55.913820 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c58e28cd-87a4-4a44-abb3-bc1ea9b60449-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:57:56.202409 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:56.202368 5979 generic.go:356] "Generic (PLEG): container finished" podID="c58e28cd-87a4-4a44-abb3-bc1ea9b60449" containerID="2203629e33a905e7d77a21f6ecfbb94bd82472c523ba907becd57b632c96009c" exitCode=0 Nov 03 13:57:56.202600 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:56.202425 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm" Nov 03 13:57:56.202600 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:56.202449 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm" event={"ID":"c58e28cd-87a4-4a44-abb3-bc1ea9b60449","Type":"ContainerDied","Data":"2203629e33a905e7d77a21f6ecfbb94bd82472c523ba907becd57b632c96009c"} Nov 03 13:57:56.202600 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:56.202517 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm" event={"ID":"c58e28cd-87a4-4a44-abb3-bc1ea9b60449","Type":"ContainerDied","Data":"2378d98905941c342b440a6a60523fb6f8e07357e14388286a3308ed673e6889"} Nov 03 13:57:56.202600 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:56.202540 5979 scope.go:122] "RemoveContainer" containerID="2203629e33a905e7d77a21f6ecfbb94bd82472c523ba907becd57b632c96009c" Nov 03 13:57:56.213164 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:56.213145 5979 scope.go:122] "RemoveContainer" containerID="2203629e33a905e7d77a21f6ecfbb94bd82472c523ba907becd57b632c96009c" Nov 03 13:57:56.213441 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:57:56.213418 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2203629e33a905e7d77a21f6ecfbb94bd82472c523ba907becd57b632c96009c\": container with ID starting with 2203629e33a905e7d77a21f6ecfbb94bd82472c523ba907becd57b632c96009c not found: ID does not exist" containerID="2203629e33a905e7d77a21f6ecfbb94bd82472c523ba907becd57b632c96009c" Nov 03 13:57:56.213535 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:56.213451 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2203629e33a905e7d77a21f6ecfbb94bd82472c523ba907becd57b632c96009c"} err="failed to get container status \"2203629e33a905e7d77a21f6ecfbb94bd82472c523ba907becd57b632c96009c\": rpc error: code = NotFound desc = could not find container \"2203629e33a905e7d77a21f6ecfbb94bd82472c523ba907becd57b632c96009c\": container with ID starting with 2203629e33a905e7d77a21f6ecfbb94bd82472c523ba907becd57b632c96009c not found: ID does not exist" Nov 03 13:57:56.356938 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:56.356906 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm"] Nov 03 13:57:56.362172 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:56.362144 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-5lfgm"] Nov 03 13:57:57.685033 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:57.685000 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:57:58.117353 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:57:58.117259 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c58e28cd-87a4-4a44-abb3-bc1ea9b60449" path="/var/lib/kubelet/pods/c58e28cd-87a4-4a44-abb3-bc1ea9b60449/volumes" Nov 03 13:58:03.682361 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:03.682272 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-92546f-7f46f65c8f-97r54"] Nov 03 13:58:03.789721 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:03.789680 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-92546f-6b56868cc5-k4nsh"] Nov 03 13:58:07.233491 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:58:07.233437 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:58:08.683349 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:08.683277 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-92546f-7f46f65c8f-97r54" podUID="21e41083-d6b8-45ad-9cf3-02eb692596eb" containerName="payload-processing" containerID="cri-o://bf04f35ba2d24136fc880b16dc26ccb3a75ad4b40ce8a42efb917451c3fb71f3" gracePeriod=25 Nov 03 13:58:08.790524 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:08.790484 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-92546f-6b56868cc5-k4nsh" podUID="bf056db6-6323-4966-bb10-0e82540a3273" containerName="payload-pre-processing" containerID="cri-o://bb9701d682745993a41b28e39413b081ea3224e4079b7a02b5ce9f7e34525fdb" gracePeriod=25 Nov 03 13:58:09.001154 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:09.001120 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-92546f-7f46f65c8f-97r54" Nov 03 13:58:09.053685 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:09.053661 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-92546f-6b56868cc5-k4nsh" Nov 03 13:58:09.097780 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:09.097754 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21e41083-d6b8-45ad-9cf3-02eb692596eb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21e41083-d6b8-45ad-9cf3-02eb692596eb-plugins-config-volume\") pod \"21e41083-d6b8-45ad-9cf3-02eb692596eb\" (UID: \"21e41083-d6b8-45ad-9cf3-02eb692596eb\") " Nov 03 13:58:09.097889 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:09.097869 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21e41083-d6b8-45ad-9cf3-02eb692596eb-kube-api-access-mw694\" (UniqueName: \"kubernetes.io/projected/21e41083-d6b8-45ad-9cf3-02eb692596eb-kube-api-access-mw694\") pod \"21e41083-d6b8-45ad-9cf3-02eb692596eb\" (UID: \"21e41083-d6b8-45ad-9cf3-02eb692596eb\") " Nov 03 13:58:09.098117 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:09.098092 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21e41083-d6b8-45ad-9cf3-02eb692596eb-plugins-config-volume" pod "21e41083-d6b8-45ad-9cf3-02eb692596eb" (UID: "21e41083-d6b8-45ad-9cf3-02eb692596eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:58:09.098228 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:09.098216 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21e41083-d6b8-45ad-9cf3-02eb692596eb-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:58:09.099826 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:09.099805 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21e41083-d6b8-45ad-9cf3-02eb692596eb-kube-api-access-mw694" pod "21e41083-d6b8-45ad-9cf3-02eb692596eb" (UID: "21e41083-d6b8-45ad-9cf3-02eb692596eb"). InnerVolumeSpecName "kube-api-access-mw694". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:58:09.199684 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:09.199651 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf056db6-6323-4966-bb10-0e82540a3273-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf056db6-6323-4966-bb10-0e82540a3273-plugins-config-volume\") pod \"bf056db6-6323-4966-bb10-0e82540a3273\" (UID: \"bf056db6-6323-4966-bb10-0e82540a3273\") " Nov 03 13:58:09.199886 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:09.199707 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf056db6-6323-4966-bb10-0e82540a3273-kube-api-access-f4c48\" (UniqueName: \"kubernetes.io/projected/bf056db6-6323-4966-bb10-0e82540a3273-kube-api-access-f4c48\") pod \"bf056db6-6323-4966-bb10-0e82540a3273\" (UID: \"bf056db6-6323-4966-bb10-0e82540a3273\") " Nov 03 13:58:09.200025 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:09.200010 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mw694\" (UniqueName: \"kubernetes.io/projected/21e41083-d6b8-45ad-9cf3-02eb692596eb-kube-api-access-mw694\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:58:09.200117 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:09.200091 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf056db6-6323-4966-bb10-0e82540a3273-plugins-config-volume" pod "bf056db6-6323-4966-bb10-0e82540a3273" (UID: "bf056db6-6323-4966-bb10-0e82540a3273"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:58:09.201798 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:09.201778 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf056db6-6323-4966-bb10-0e82540a3273-kube-api-access-f4c48" pod "bf056db6-6323-4966-bb10-0e82540a3273" (UID: "bf056db6-6323-4966-bb10-0e82540a3273"). InnerVolumeSpecName "kube-api-access-f4c48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:58:09.300947 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:09.300864 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf056db6-6323-4966-bb10-0e82540a3273-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:58:09.300947 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:09.300896 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4c48\" (UniqueName: \"kubernetes.io/projected/bf056db6-6323-4966-bb10-0e82540a3273-kube-api-access-f4c48\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:58:09.301156 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:09.300977 5979 generic.go:356] "Generic (PLEG): container finished" podID="21e41083-d6b8-45ad-9cf3-02eb692596eb" containerID="bf04f35ba2d24136fc880b16dc26ccb3a75ad4b40ce8a42efb917451c3fb71f3" exitCode=0 Nov 03 13:58:09.301156 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:09.301035 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-92546f-7f46f65c8f-97r54" Nov 03 13:58:09.301156 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:09.301064 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-92546f-7f46f65c8f-97r54" event={"ID":"21e41083-d6b8-45ad-9cf3-02eb692596eb","Type":"ContainerDied","Data":"bf04f35ba2d24136fc880b16dc26ccb3a75ad4b40ce8a42efb917451c3fb71f3"} Nov 03 13:58:09.301156 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:09.301113 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-92546f-7f46f65c8f-97r54" event={"ID":"21e41083-d6b8-45ad-9cf3-02eb692596eb","Type":"ContainerDied","Data":"49e3da0aff972117bd1f7f4a06c5568701eb80cca65364db35046715942997e3"} Nov 03 13:58:09.301156 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:09.301140 5979 scope.go:122] "RemoveContainer" containerID="bf04f35ba2d24136fc880b16dc26ccb3a75ad4b40ce8a42efb917451c3fb71f3" Nov 03 13:58:09.302763 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:09.302743 5979 generic.go:356] "Generic (PLEG): container finished" podID="bf056db6-6323-4966-bb10-0e82540a3273" containerID="bb9701d682745993a41b28e39413b081ea3224e4079b7a02b5ce9f7e34525fdb" exitCode=0 Nov 03 13:58:09.302878 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:09.302833 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-92546f-6b56868cc5-k4nsh" Nov 03 13:58:09.302936 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:09.302834 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-92546f-6b56868cc5-k4nsh" event={"ID":"bf056db6-6323-4966-bb10-0e82540a3273","Type":"ContainerDied","Data":"bb9701d682745993a41b28e39413b081ea3224e4079b7a02b5ce9f7e34525fdb"} Nov 03 13:58:09.302979 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:09.302953 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-92546f-6b56868cc5-k4nsh" event={"ID":"bf056db6-6323-4966-bb10-0e82540a3273","Type":"ContainerDied","Data":"3c7c18fe2ba7a45ec9a3f38665afa79ca9217fe01072bc66e4eae21f3510f011"} Nov 03 13:58:09.313048 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:09.313010 5979 scope.go:122] "RemoveContainer" containerID="bf04f35ba2d24136fc880b16dc26ccb3a75ad4b40ce8a42efb917451c3fb71f3" Nov 03 13:58:09.313344 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:58:09.313318 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf04f35ba2d24136fc880b16dc26ccb3a75ad4b40ce8a42efb917451c3fb71f3\": container with ID starting with bf04f35ba2d24136fc880b16dc26ccb3a75ad4b40ce8a42efb917451c3fb71f3 not found: ID does not exist" containerID="bf04f35ba2d24136fc880b16dc26ccb3a75ad4b40ce8a42efb917451c3fb71f3" Nov 03 13:58:09.313438 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:09.313353 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf04f35ba2d24136fc880b16dc26ccb3a75ad4b40ce8a42efb917451c3fb71f3"} err="failed to get container status \"bf04f35ba2d24136fc880b16dc26ccb3a75ad4b40ce8a42efb917451c3fb71f3\": rpc error: code = NotFound desc = could not find container \"bf04f35ba2d24136fc880b16dc26ccb3a75ad4b40ce8a42efb917451c3fb71f3\": container with ID starting with bf04f35ba2d24136fc880b16dc26ccb3a75ad4b40ce8a42efb917451c3fb71f3 not found: ID does not exist" Nov 03 13:58:09.313438 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:09.313378 5979 scope.go:122] "RemoveContainer" containerID="bb9701d682745993a41b28e39413b081ea3224e4079b7a02b5ce9f7e34525fdb" Nov 03 13:58:09.322276 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:09.322256 5979 scope.go:122] "RemoveContainer" containerID="bb9701d682745993a41b28e39413b081ea3224e4079b7a02b5ce9f7e34525fdb" Nov 03 13:58:09.322549 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:58:09.322533 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb9701d682745993a41b28e39413b081ea3224e4079b7a02b5ce9f7e34525fdb\": container with ID starting with bb9701d682745993a41b28e39413b081ea3224e4079b7a02b5ce9f7e34525fdb not found: ID does not exist" containerID="bb9701d682745993a41b28e39413b081ea3224e4079b7a02b5ce9f7e34525fdb" Nov 03 13:58:09.322595 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:09.322555 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb9701d682745993a41b28e39413b081ea3224e4079b7a02b5ce9f7e34525fdb"} err="failed to get container status \"bb9701d682745993a41b28e39413b081ea3224e4079b7a02b5ce9f7e34525fdb\": rpc error: code = NotFound desc = could not find container \"bb9701d682745993a41b28e39413b081ea3224e4079b7a02b5ce9f7e34525fdb\": container with ID starting with bb9701d682745993a41b28e39413b081ea3224e4079b7a02b5ce9f7e34525fdb not found: ID does not exist" Nov 03 13:58:09.455280 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:09.455246 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-92546f-7f46f65c8f-97r54"] Nov 03 13:58:09.459688 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:09.459655 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-92546f-7f46f65c8f-97r54"] Nov 03 13:58:09.561654 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:09.561570 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-92546f-6b56868cc5-k4nsh"] Nov 03 13:58:09.566030 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:09.566004 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-92546f-6b56868cc5-k4nsh"] Nov 03 13:58:10.116045 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:10.116006 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21e41083-d6b8-45ad-9cf3-02eb692596eb" path="/var/lib/kubelet/pods/21e41083-d6b8-45ad-9cf3-02eb692596eb/volumes" Nov 03 13:58:10.116456 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:10.116444 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf056db6-6323-4966-bb10-0e82540a3273" path="/var/lib/kubelet/pods/bf056db6-6323-4966-bb10-0e82540a3273/volumes" Nov 03 13:58:20.536298 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:20.536263 5979 scope.go:122] "RemoveContainer" containerID="ebded7ee370c2bb01b149c5cafc1a67f71dd4bf1b6bc07585d5e168c636f0660" Nov 03 13:58:37.045929 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:37.045893 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d6af7fa-gwchs"] Nov 03 13:58:37.092379 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:37.092343 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d6af7fa-gwchs"] Nov 03 13:58:37.233801 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:58:37.233766 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:58:38.116352 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:38.116317 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2be45447-2758-432d-b668-2da835855c14" path="/var/lib/kubelet/pods/2be45447-2758-432d-b668-2da835855c14/volumes" Nov 03 13:58:41.668592 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:41.668556 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:58:48.880623 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:48.880588 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-92546f-6ddf84c8fb-dnftl"] Nov 03 13:58:48.881128 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:48.880849 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-92546f-6ddf84c8fb-dnftl" podUID="08240bd7-0d0e-485e-ae5b-481d8bde3d4a" containerName="maas-api" containerID="cri-o://85883f15a4a4698dbdaad9637527aa0a4702c31a5e9f74cbdeb671f6be04ca26" gracePeriod=30 Nov 03 13:58:49.168794 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:49.168767 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-92546f-6ddf84c8fb-dnftl" Nov 03 13:58:49.222894 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:49.222861 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08240bd7-0d0e-485e-ae5b-481d8bde3d4a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08240bd7-0d0e-485e-ae5b-481d8bde3d4a-openshift-service-ca\") pod \"08240bd7-0d0e-485e-ae5b-481d8bde3d4a\" (UID: \"08240bd7-0d0e-485e-ae5b-481d8bde3d4a\") " Nov 03 13:58:49.223082 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:49.222910 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08240bd7-0d0e-485e-ae5b-481d8bde3d4a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08240bd7-0d0e-485e-ae5b-481d8bde3d4a-maas-api-tls\") pod \"08240bd7-0d0e-485e-ae5b-481d8bde3d4a\" (UID: \"08240bd7-0d0e-485e-ae5b-481d8bde3d4a\") " Nov 03 13:58:49.223082 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:49.222947 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08240bd7-0d0e-485e-ae5b-481d8bde3d4a-kube-api-access-qmh65\" (UniqueName: \"kubernetes.io/projected/08240bd7-0d0e-485e-ae5b-481d8bde3d4a-kube-api-access-qmh65\") pod \"08240bd7-0d0e-485e-ae5b-481d8bde3d4a\" (UID: \"08240bd7-0d0e-485e-ae5b-481d8bde3d4a\") " Nov 03 13:58:49.223082 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:49.223000 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08240bd7-0d0e-485e-ae5b-481d8bde3d4a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/08240bd7-0d0e-485e-ae5b-481d8bde3d4a-metrics-certs\") pod \"08240bd7-0d0e-485e-ae5b-481d8bde3d4a\" (UID: \"08240bd7-0d0e-485e-ae5b-481d8bde3d4a\") " Nov 03 13:58:49.223333 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:49.223307 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08240bd7-0d0e-485e-ae5b-481d8bde3d4a-openshift-service-ca" pod "08240bd7-0d0e-485e-ae5b-481d8bde3d4a" (UID: "08240bd7-0d0e-485e-ae5b-481d8bde3d4a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:58:49.225261 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:49.225229 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08240bd7-0d0e-485e-ae5b-481d8bde3d4a-kube-api-access-qmh65" pod "08240bd7-0d0e-485e-ae5b-481d8bde3d4a" (UID: "08240bd7-0d0e-485e-ae5b-481d8bde3d4a"). InnerVolumeSpecName "kube-api-access-qmh65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:58:49.225261 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:49.225235 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08240bd7-0d0e-485e-ae5b-481d8bde3d4a-metrics-certs" pod "08240bd7-0d0e-485e-ae5b-481d8bde3d4a" (UID: "08240bd7-0d0e-485e-ae5b-481d8bde3d4a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:58:49.225421 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:49.225289 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08240bd7-0d0e-485e-ae5b-481d8bde3d4a-maas-api-tls" pod "08240bd7-0d0e-485e-ae5b-481d8bde3d4a" (UID: "08240bd7-0d0e-485e-ae5b-481d8bde3d4a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 13:58:49.327251 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:49.327153 5979 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08240bd7-0d0e-485e-ae5b-481d8bde3d4a-openshift-service-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:58:49.327251 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:49.327189 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08240bd7-0d0e-485e-ae5b-481d8bde3d4a-maas-api-tls\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:58:49.327251 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:49.327200 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmh65\" (UniqueName: \"kubernetes.io/projected/08240bd7-0d0e-485e-ae5b-481d8bde3d4a-kube-api-access-qmh65\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:58:49.327251 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:49.327209 5979 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08240bd7-0d0e-485e-ae5b-481d8bde3d4a-metrics-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:58:49.552733 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:49.552636 5979 generic.go:356] "Generic (PLEG): container finished" podID="08240bd7-0d0e-485e-ae5b-481d8bde3d4a" containerID="85883f15a4a4698dbdaad9637527aa0a4702c31a5e9f74cbdeb671f6be04ca26" exitCode=0 Nov 03 13:58:49.552733 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:49.552691 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-92546f-6ddf84c8fb-dnftl" Nov 03 13:58:49.552972 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:49.552732 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-92546f-6ddf84c8fb-dnftl" event={"ID":"08240bd7-0d0e-485e-ae5b-481d8bde3d4a","Type":"ContainerDied","Data":"85883f15a4a4698dbdaad9637527aa0a4702c31a5e9f74cbdeb671f6be04ca26"} Nov 03 13:58:49.552972 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:49.552785 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-92546f-6ddf84c8fb-dnftl" event={"ID":"08240bd7-0d0e-485e-ae5b-481d8bde3d4a","Type":"ContainerDied","Data":"10a5afcb61f04ad5fd992cd918fefb25875d0445cec233e7b7290b1e4f46fbac"} Nov 03 13:58:49.552972 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:49.552808 5979 scope.go:122] "RemoveContainer" containerID="85883f15a4a4698dbdaad9637527aa0a4702c31a5e9f74cbdeb671f6be04ca26" Nov 03 13:58:49.563446 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:49.563425 5979 scope.go:122] "RemoveContainer" containerID="85883f15a4a4698dbdaad9637527aa0a4702c31a5e9f74cbdeb671f6be04ca26" Nov 03 13:58:49.563771 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:58:49.563751 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85883f15a4a4698dbdaad9637527aa0a4702c31a5e9f74cbdeb671f6be04ca26\": container with ID starting with 85883f15a4a4698dbdaad9637527aa0a4702c31a5e9f74cbdeb671f6be04ca26 not found: ID does not exist" containerID="85883f15a4a4698dbdaad9637527aa0a4702c31a5e9f74cbdeb671f6be04ca26" Nov 03 13:58:49.563840 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:49.563778 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85883f15a4a4698dbdaad9637527aa0a4702c31a5e9f74cbdeb671f6be04ca26"} err="failed to get container status \"85883f15a4a4698dbdaad9637527aa0a4702c31a5e9f74cbdeb671f6be04ca26\": rpc error: code = NotFound desc = could not find container \"85883f15a4a4698dbdaad9637527aa0a4702c31a5e9f74cbdeb671f6be04ca26\": container with ID starting with 85883f15a4a4698dbdaad9637527aa0a4702c31a5e9f74cbdeb671f6be04ca26 not found: ID does not exist" Nov 03 13:58:49.707596 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:49.707565 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-92546f-6ddf84c8fb-dnftl"] Nov 03 13:58:49.714187 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:49.714161 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-92546f-6ddf84c8fb-dnftl"] Nov 03 13:58:50.115588 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:50.115558 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08240bd7-0d0e-485e-ae5b-481d8bde3d4a" path="/var/lib/kubelet/pods/08240bd7-0d0e-485e-ae5b-481d8bde3d4a/volumes" Nov 03 13:58:53.311927 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:53.311890 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m"] Nov 03 13:58:53.312430 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:53.312123 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m" podUID="8a1133dc-605d-4871-8190-7714aca1ce00" containerName="istio-proxy" containerID="cri-o://cd3e320ba158fcb7fe6ac78b1ed06eff5950bc64d9ce9dd40f4d72152a7a5b9e" gracePeriod=30 Nov 03 13:58:54.480233 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:54.480196 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-921466-openshift-default-5fc5946897-pc6bx"] Nov 03 13:58:54.480775 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:54.480760 5979 cpu_manager.go:414] "containerMap: removing container" podUID="08240bd7-0d0e-485e-ae5b-481d8bde3d4a" containerName="maas-api" Nov 03 13:58:54.480841 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:54.480775 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08240bd7-0d0e-485e-ae5b-481d8bde3d4a" containerName="maas-api" Nov 03 13:58:54.480841 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:54.480795 5979 cpu_manager.go:414] "containerMap: removing container" podUID="bf056db6-6323-4966-bb10-0e82540a3273" containerName="payload-pre-processing" Nov 03 13:58:54.480841 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:54.480802 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf056db6-6323-4966-bb10-0e82540a3273" containerName="payload-pre-processing" Nov 03 13:58:54.480841 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:54.480813 5979 cpu_manager.go:414] "containerMap: removing container" podUID="21e41083-d6b8-45ad-9cf3-02eb692596eb" containerName="payload-processing" Nov 03 13:58:54.480841 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:54.480818 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21e41083-d6b8-45ad-9cf3-02eb692596eb" containerName="payload-processing" Nov 03 13:58:54.481029 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:54.480848 5979 cpu_manager.go:414] "containerMap: removing container" podUID="c58e28cd-87a4-4a44-abb3-bc1ea9b60449" containerName="istio-proxy" Nov 03 13:58:54.481029 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:54.480853 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c58e28cd-87a4-4a44-abb3-bc1ea9b60449" containerName="istio-proxy" Nov 03 13:58:54.481029 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:54.480888 5979 cpu_manager.go:414] "containerMap: removing container" podUID="210b9bed-6df7-4f71-982e-ce2f72287b17" containerName="istio-proxy" Nov 03 13:58:54.481029 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:54.480894 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="210b9bed-6df7-4f71-982e-ce2f72287b17" containerName="istio-proxy" Nov 03 13:58:54.481029 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:54.480967 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="c58e28cd-87a4-4a44-abb3-bc1ea9b60449" containerName="istio-proxy" Nov 03 13:58:54.481029 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:54.480979 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="210b9bed-6df7-4f71-982e-ce2f72287b17" containerName="istio-proxy" Nov 03 13:58:54.481029 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:54.480986 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="08240bd7-0d0e-485e-ae5b-481d8bde3d4a" containerName="maas-api" Nov 03 13:58:54.481029 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:54.481020 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf056db6-6323-4966-bb10-0e82540a3273" containerName="payload-pre-processing" Nov 03 13:58:54.481322 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:54.481032 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="21e41083-d6b8-45ad-9cf3-02eb692596eb" containerName="payload-processing" Nov 03 13:58:54.486045 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:54.486025 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-921466-openshift-default-5fc5946897-pc6bx" Nov 03 13:58:54.596265 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:54.596222 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/313666bc-2334-42a4-8dfe-ab7054ef13dd-istio-envoy\") pod \"e2e-noten-921466-openshift-default-5fc5946897-pc6bx\" (UID: \"313666bc-2334-42a4-8dfe-ab7054ef13dd\") " pod="openshift-ingress/e2e-noten-921466-openshift-default-5fc5946897-pc6bx" Nov 03 13:58:54.596507 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:54.596300 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/313666bc-2334-42a4-8dfe-ab7054ef13dd-workload-certs\") pod \"e2e-noten-921466-openshift-default-5fc5946897-pc6bx\" (UID: \"313666bc-2334-42a4-8dfe-ab7054ef13dd\") " pod="openshift-ingress/e2e-noten-921466-openshift-default-5fc5946897-pc6bx" Nov 03 13:58:54.596507 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:54.596422 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/313666bc-2334-42a4-8dfe-ab7054ef13dd-istio-data\") pod \"e2e-noten-921466-openshift-default-5fc5946897-pc6bx\" (UID: \"313666bc-2334-42a4-8dfe-ab7054ef13dd\") " pod="openshift-ingress/e2e-noten-921466-openshift-default-5fc5946897-pc6bx" Nov 03 13:58:54.596507 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:54.596488 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/313666bc-2334-42a4-8dfe-ab7054ef13dd-istio-token\") pod \"e2e-noten-921466-openshift-default-5fc5946897-pc6bx\" (UID: \"313666bc-2334-42a4-8dfe-ab7054ef13dd\") " pod="openshift-ingress/e2e-noten-921466-openshift-default-5fc5946897-pc6bx" Nov 03 13:58:54.596723 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:54.596520 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr97b\" (UniqueName: \"kubernetes.io/projected/313666bc-2334-42a4-8dfe-ab7054ef13dd-kube-api-access-nr97b\") pod \"e2e-noten-921466-openshift-default-5fc5946897-pc6bx\" (UID: \"313666bc-2334-42a4-8dfe-ab7054ef13dd\") " pod="openshift-ingress/e2e-noten-921466-openshift-default-5fc5946897-pc6bx" Nov 03 13:58:54.596723 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:54.596562 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/313666bc-2334-42a4-8dfe-ab7054ef13dd-credential-socket\") pod \"e2e-noten-921466-openshift-default-5fc5946897-pc6bx\" (UID: \"313666bc-2334-42a4-8dfe-ab7054ef13dd\") " pod="openshift-ingress/e2e-noten-921466-openshift-default-5fc5946897-pc6bx" Nov 03 13:58:54.596723 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:54.596585 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/313666bc-2334-42a4-8dfe-ab7054ef13dd-istiod-ca-cert\") pod \"e2e-noten-921466-openshift-default-5fc5946897-pc6bx\" (UID: \"313666bc-2334-42a4-8dfe-ab7054ef13dd\") " pod="openshift-ingress/e2e-noten-921466-openshift-default-5fc5946897-pc6bx" Nov 03 13:58:54.596723 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:54.596657 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/313666bc-2334-42a4-8dfe-ab7054ef13dd-workload-socket\") pod \"e2e-noten-921466-openshift-default-5fc5946897-pc6bx\" (UID: \"313666bc-2334-42a4-8dfe-ab7054ef13dd\") " pod="openshift-ingress/e2e-noten-921466-openshift-default-5fc5946897-pc6bx" Nov 03 13:58:54.596723 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:54.596706 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/313666bc-2334-42a4-8dfe-ab7054ef13dd-istio-podinfo\") pod \"e2e-noten-921466-openshift-default-5fc5946897-pc6bx\" (UID: \"313666bc-2334-42a4-8dfe-ab7054ef13dd\") " pod="openshift-ingress/e2e-noten-921466-openshift-default-5fc5946897-pc6bx" Nov 03 13:58:54.619306 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:54.619274 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-921466-openshift-default-dockercfg-4qtq2\"" Nov 03 13:58:54.630009 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:54.626556 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-921466-openshift-default-5fc5946897-pc6bx"] Nov 03 13:58:54.698290 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:54.698252 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/313666bc-2334-42a4-8dfe-ab7054ef13dd-istio-data\") pod \"e2e-noten-921466-openshift-default-5fc5946897-pc6bx\" (UID: \"313666bc-2334-42a4-8dfe-ab7054ef13dd\") " pod="openshift-ingress/e2e-noten-921466-openshift-default-5fc5946897-pc6bx" Nov 03 13:58:54.698516 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:54.698297 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/313666bc-2334-42a4-8dfe-ab7054ef13dd-istio-token\") pod \"e2e-noten-921466-openshift-default-5fc5946897-pc6bx\" (UID: \"313666bc-2334-42a4-8dfe-ab7054ef13dd\") " pod="openshift-ingress/e2e-noten-921466-openshift-default-5fc5946897-pc6bx" Nov 03 13:58:54.698516 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:54.698320 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr97b\" (UniqueName: \"kubernetes.io/projected/313666bc-2334-42a4-8dfe-ab7054ef13dd-kube-api-access-nr97b\") pod \"e2e-noten-921466-openshift-default-5fc5946897-pc6bx\" (UID: \"313666bc-2334-42a4-8dfe-ab7054ef13dd\") " pod="openshift-ingress/e2e-noten-921466-openshift-default-5fc5946897-pc6bx" Nov 03 13:58:54.698516 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:54.698355 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/313666bc-2334-42a4-8dfe-ab7054ef13dd-credential-socket\") pod \"e2e-noten-921466-openshift-default-5fc5946897-pc6bx\" (UID: \"313666bc-2334-42a4-8dfe-ab7054ef13dd\") " pod="openshift-ingress/e2e-noten-921466-openshift-default-5fc5946897-pc6bx" Nov 03 13:58:54.698516 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:54.698371 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/313666bc-2334-42a4-8dfe-ab7054ef13dd-istiod-ca-cert\") pod \"e2e-noten-921466-openshift-default-5fc5946897-pc6bx\" (UID: \"313666bc-2334-42a4-8dfe-ab7054ef13dd\") " pod="openshift-ingress/e2e-noten-921466-openshift-default-5fc5946897-pc6bx" Nov 03 13:58:54.698516 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:54.698424 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/313666bc-2334-42a4-8dfe-ab7054ef13dd-workload-socket\") pod \"e2e-noten-921466-openshift-default-5fc5946897-pc6bx\" (UID: \"313666bc-2334-42a4-8dfe-ab7054ef13dd\") " pod="openshift-ingress/e2e-noten-921466-openshift-default-5fc5946897-pc6bx" Nov 03 13:58:54.698863 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:54.698604 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/313666bc-2334-42a4-8dfe-ab7054ef13dd-istio-podinfo\") pod \"e2e-noten-921466-openshift-default-5fc5946897-pc6bx\" (UID: \"313666bc-2334-42a4-8dfe-ab7054ef13dd\") " pod="openshift-ingress/e2e-noten-921466-openshift-default-5fc5946897-pc6bx" Nov 03 13:58:54.698863 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:54.698711 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/313666bc-2334-42a4-8dfe-ab7054ef13dd-istio-envoy\") pod \"e2e-noten-921466-openshift-default-5fc5946897-pc6bx\" (UID: \"313666bc-2334-42a4-8dfe-ab7054ef13dd\") " pod="openshift-ingress/e2e-noten-921466-openshift-default-5fc5946897-pc6bx" Nov 03 13:58:54.698863 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:54.698728 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/313666bc-2334-42a4-8dfe-ab7054ef13dd-istio-data\") pod \"e2e-noten-921466-openshift-default-5fc5946897-pc6bx\" (UID: \"313666bc-2334-42a4-8dfe-ab7054ef13dd\") " pod="openshift-ingress/e2e-noten-921466-openshift-default-5fc5946897-pc6bx" Nov 03 13:58:54.698863 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:54.698770 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/313666bc-2334-42a4-8dfe-ab7054ef13dd-workload-certs\") pod \"e2e-noten-921466-openshift-default-5fc5946897-pc6bx\" (UID: \"313666bc-2334-42a4-8dfe-ab7054ef13dd\") " pod="openshift-ingress/e2e-noten-921466-openshift-default-5fc5946897-pc6bx" Nov 03 13:58:54.699038 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:54.698945 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/313666bc-2334-42a4-8dfe-ab7054ef13dd-credential-socket\") pod \"e2e-noten-921466-openshift-default-5fc5946897-pc6bx\" (UID: \"313666bc-2334-42a4-8dfe-ab7054ef13dd\") " pod="openshift-ingress/e2e-noten-921466-openshift-default-5fc5946897-pc6bx" Nov 03 13:58:54.699038 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:54.698958 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/313666bc-2334-42a4-8dfe-ab7054ef13dd-workload-socket\") pod \"e2e-noten-921466-openshift-default-5fc5946897-pc6bx\" (UID: \"313666bc-2334-42a4-8dfe-ab7054ef13dd\") " pod="openshift-ingress/e2e-noten-921466-openshift-default-5fc5946897-pc6bx" Nov 03 13:58:54.699160 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:54.699139 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/313666bc-2334-42a4-8dfe-ab7054ef13dd-workload-certs\") pod \"e2e-noten-921466-openshift-default-5fc5946897-pc6bx\" (UID: \"313666bc-2334-42a4-8dfe-ab7054ef13dd\") " pod="openshift-ingress/e2e-noten-921466-openshift-default-5fc5946897-pc6bx" Nov 03 13:58:54.699416 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:54.699396 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/313666bc-2334-42a4-8dfe-ab7054ef13dd-istiod-ca-cert\") pod \"e2e-noten-921466-openshift-default-5fc5946897-pc6bx\" (UID: \"313666bc-2334-42a4-8dfe-ab7054ef13dd\") " pod="openshift-ingress/e2e-noten-921466-openshift-default-5fc5946897-pc6bx" Nov 03 13:58:54.701774 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:54.701748 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/313666bc-2334-42a4-8dfe-ab7054ef13dd-istio-envoy\") pod \"e2e-noten-921466-openshift-default-5fc5946897-pc6bx\" (UID: \"313666bc-2334-42a4-8dfe-ab7054ef13dd\") " pod="openshift-ingress/e2e-noten-921466-openshift-default-5fc5946897-pc6bx" Nov 03 13:58:54.701939 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:54.701921 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/313666bc-2334-42a4-8dfe-ab7054ef13dd-istio-podinfo\") pod \"e2e-noten-921466-openshift-default-5fc5946897-pc6bx\" (UID: \"313666bc-2334-42a4-8dfe-ab7054ef13dd\") " pod="openshift-ingress/e2e-noten-921466-openshift-default-5fc5946897-pc6bx" Nov 03 13:58:54.753495 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:54.753409 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/313666bc-2334-42a4-8dfe-ab7054ef13dd-istio-token\") pod \"e2e-noten-921466-openshift-default-5fc5946897-pc6bx\" (UID: \"313666bc-2334-42a4-8dfe-ab7054ef13dd\") " pod="openshift-ingress/e2e-noten-921466-openshift-default-5fc5946897-pc6bx" Nov 03 13:58:54.753917 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:54.753887 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr97b\" (UniqueName: \"kubernetes.io/projected/313666bc-2334-42a4-8dfe-ab7054ef13dd-kube-api-access-nr97b\") pod \"e2e-noten-921466-openshift-default-5fc5946897-pc6bx\" (UID: \"313666bc-2334-42a4-8dfe-ab7054ef13dd\") " pod="openshift-ingress/e2e-noten-921466-openshift-default-5fc5946897-pc6bx" Nov 03 13:58:54.800133 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:54.800102 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-921466-openshift-default-5fc5946897-pc6bx" Nov 03 13:58:55.227783 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:55.227744 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-921466-openshift-default-5fc5946897-pc6bx"] Nov 03 13:58:55.234370 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:58:55.234343 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod313666bc_2334_42a4_8dfe_ab7054ef13dd.slice/crio-dca67d2a23e0899d9e8642881b4ad04d4b875d5212585527e47aca2b74934c00 WatchSource:0}: Error finding container dca67d2a23e0899d9e8642881b4ad04d4b875d5212585527e47aca2b74934c00: Status 404 returned error can't find the container with id dca67d2a23e0899d9e8642881b4ad04d4b875d5212585527e47aca2b74934c00 Nov 03 13:58:55.236935 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:55.236903 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:58:55.237011 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:55.236970 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:58:55.237011 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:55.236997 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:58:55.638635 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:55.638594 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-921466-openshift-default-5fc5946897-pc6bx" event={"ID":"313666bc-2334-42a4-8dfe-ab7054ef13dd","Type":"ContainerStarted","Data":"de623b561483fa5529aecfc623031856aee6898b8dfc28f5f01f8c5e04470973"} Nov 03 13:58:55.639076 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:55.638657 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-921466-openshift-default-5fc5946897-pc6bx" event={"ID":"313666bc-2334-42a4-8dfe-ab7054ef13dd","Type":"ContainerStarted","Data":"dca67d2a23e0899d9e8642881b4ad04d4b875d5212585527e47aca2b74934c00"} Nov 03 13:58:55.751314 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:55.751265 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-921466-openshift-default-5fc5946897-pc6bx" podStartSLOduration=1.7512515629999998 podStartE2EDuration="1.751251563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:58:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:58:55.750358974 +0000 UTC m=+3217.761047314" watchObservedRunningTime="2026-11-03 13:58:55.751251563 +0000 UTC m=+3217.761939805" Nov 03 13:58:55.800889 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:55.800855 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-921466-openshift-default-5fc5946897-pc6bx" Nov 03 13:58:56.807067 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:56.807036 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-921466-openshift-default-5fc5946897-pc6bx" Nov 03 13:58:57.653354 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:57.653322 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-921466-openshift-default-5fc5946897-pc6bx" Nov 03 13:58:57.654251 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:57.654232 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-921466-openshift-default-5fc5946897-pc6bx" Nov 03 13:58:58.589923 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:58.589854 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m" Nov 03 13:58:58.653216 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:58.653175 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a1133dc-605d-4871-8190-7714aca1ce00-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a1133dc-605d-4871-8190-7714aca1ce00-istio-data\") pod \"8a1133dc-605d-4871-8190-7714aca1ce00\" (UID: \"8a1133dc-605d-4871-8190-7714aca1ce00\") " Nov 03 13:58:58.653449 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:58.653241 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a1133dc-605d-4871-8190-7714aca1ce00-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a1133dc-605d-4871-8190-7714aca1ce00-workload-socket\") pod \"8a1133dc-605d-4871-8190-7714aca1ce00\" (UID: \"8a1133dc-605d-4871-8190-7714aca1ce00\") " Nov 03 13:58:58.653449 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:58.653324 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a1133dc-605d-4871-8190-7714aca1ce00-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a1133dc-605d-4871-8190-7714aca1ce00-istiod-ca-cert\") pod \"8a1133dc-605d-4871-8190-7714aca1ce00\" (UID: \"8a1133dc-605d-4871-8190-7714aca1ce00\") " Nov 03 13:58:58.653449 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:58.653430 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a1133dc-605d-4871-8190-7714aca1ce00-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a1133dc-605d-4871-8190-7714aca1ce00-credential-socket\") pod \"8a1133dc-605d-4871-8190-7714aca1ce00\" (UID: \"8a1133dc-605d-4871-8190-7714aca1ce00\") " Nov 03 13:58:58.653686 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:58.653503 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a1133dc-605d-4871-8190-7714aca1ce00-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a1133dc-605d-4871-8190-7714aca1ce00-istio-podinfo\") pod \"8a1133dc-605d-4871-8190-7714aca1ce00\" (UID: \"8a1133dc-605d-4871-8190-7714aca1ce00\") " Nov 03 13:58:58.653686 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:58.653543 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a1133dc-605d-4871-8190-7714aca1ce00-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a1133dc-605d-4871-8190-7714aca1ce00-istio-envoy\") pod \"8a1133dc-605d-4871-8190-7714aca1ce00\" (UID: \"8a1133dc-605d-4871-8190-7714aca1ce00\") " Nov 03 13:58:58.653686 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:58.653592 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a1133dc-605d-4871-8190-7714aca1ce00-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a1133dc-605d-4871-8190-7714aca1ce00-workload-certs\") pod \"8a1133dc-605d-4871-8190-7714aca1ce00\" (UID: \"8a1133dc-605d-4871-8190-7714aca1ce00\") " Nov 03 13:58:58.653852 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:58.653761 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a1133dc-605d-4871-8190-7714aca1ce00-kube-api-access-z9nl7\" (UniqueName: \"kubernetes.io/projected/8a1133dc-605d-4871-8190-7714aca1ce00-kube-api-access-z9nl7\") pod \"8a1133dc-605d-4871-8190-7714aca1ce00\" (UID: \"8a1133dc-605d-4871-8190-7714aca1ce00\") " Nov 03 13:58:58.653852 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:58.653808 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a1133dc-605d-4871-8190-7714aca1ce00-istio-token\" (UniqueName: \"kubernetes.io/projected/8a1133dc-605d-4871-8190-7714aca1ce00-istio-token\") pod \"8a1133dc-605d-4871-8190-7714aca1ce00\" (UID: \"8a1133dc-605d-4871-8190-7714aca1ce00\") " Nov 03 13:58:58.653989 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:58.653455 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a1133dc-605d-4871-8190-7714aca1ce00-istio-data" pod "8a1133dc-605d-4871-8190-7714aca1ce00" (UID: "8a1133dc-605d-4871-8190-7714aca1ce00"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:58:58.653989 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:58.653542 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a1133dc-605d-4871-8190-7714aca1ce00-workload-socket" pod "8a1133dc-605d-4871-8190-7714aca1ce00" (UID: "8a1133dc-605d-4871-8190-7714aca1ce00"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:58:58.653989 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:58.653760 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a1133dc-605d-4871-8190-7714aca1ce00-credential-socket" pod "8a1133dc-605d-4871-8190-7714aca1ce00" (UID: "8a1133dc-605d-4871-8190-7714aca1ce00"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:58:58.654179 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:58.653828 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a1133dc-605d-4871-8190-7714aca1ce00-istiod-ca-cert" pod "8a1133dc-605d-4871-8190-7714aca1ce00" (UID: "8a1133dc-605d-4871-8190-7714aca1ce00"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 13:58:58.654438 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:58.654409 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a1133dc-605d-4871-8190-7714aca1ce00-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:58:58.654616 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:58.654440 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a1133dc-605d-4871-8190-7714aca1ce00-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:58:58.654616 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:58.654457 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a1133dc-605d-4871-8190-7714aca1ce00-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:58:58.654616 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:58.654484 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a1133dc-605d-4871-8190-7714aca1ce00-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:58:58.654616 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:58.654546 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a1133dc-605d-4871-8190-7714aca1ce00-workload-certs" pod "8a1133dc-605d-4871-8190-7714aca1ce00" (UID: "8a1133dc-605d-4871-8190-7714aca1ce00"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:58:58.656155 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:58.656130 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a1133dc-605d-4871-8190-7714aca1ce00-istio-envoy" pod "8a1133dc-605d-4871-8190-7714aca1ce00" (UID: "8a1133dc-605d-4871-8190-7714aca1ce00"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 13:58:58.656293 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:58.656269 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a1133dc-605d-4871-8190-7714aca1ce00-istio-token" pod "8a1133dc-605d-4871-8190-7714aca1ce00" (UID: "8a1133dc-605d-4871-8190-7714aca1ce00"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:58:58.656370 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:58.656342 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a1133dc-605d-4871-8190-7714aca1ce00-istio-podinfo" pod "8a1133dc-605d-4871-8190-7714aca1ce00" (UID: "8a1133dc-605d-4871-8190-7714aca1ce00"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 13:58:58.656422 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:58.656393 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a1133dc-605d-4871-8190-7714aca1ce00-kube-api-access-z9nl7" pod "8a1133dc-605d-4871-8190-7714aca1ce00" (UID: "8a1133dc-605d-4871-8190-7714aca1ce00"). InnerVolumeSpecName "kube-api-access-z9nl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 13:58:58.662578 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:58.662553 5979 generic.go:356] "Generic (PLEG): container finished" podID="8a1133dc-605d-4871-8190-7714aca1ce00" containerID="cd3e320ba158fcb7fe6ac78b1ed06eff5950bc64d9ce9dd40f4d72152a7a5b9e" exitCode=0 Nov 03 13:58:58.662696 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:58.662615 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m" Nov 03 13:58:58.662696 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:58.662634 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m" event={"ID":"8a1133dc-605d-4871-8190-7714aca1ce00","Type":"ContainerDied","Data":"cd3e320ba158fcb7fe6ac78b1ed06eff5950bc64d9ce9dd40f4d72152a7a5b9e"} Nov 03 13:58:58.662797 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:58.662715 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m" event={"ID":"8a1133dc-605d-4871-8190-7714aca1ce00","Type":"ContainerDied","Data":"7c557d49b6134c2d03f8c2b7e31a5679a5c139467de28d1825c9095b639d4491"} Nov 03 13:58:58.662797 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:58.662739 5979 scope.go:122] "RemoveContainer" containerID="cd3e320ba158fcb7fe6ac78b1ed06eff5950bc64d9ce9dd40f4d72152a7a5b9e" Nov 03 13:58:58.678027 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:58.678006 5979 scope.go:122] "RemoveContainer" containerID="808a83b27db3fc13e449f2a22c80e386dab0cadb6aecd4ceaa90c7a860866190" Nov 03 13:58:58.688882 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:58.688859 5979 scope.go:122] "RemoveContainer" containerID="cd3e320ba158fcb7fe6ac78b1ed06eff5950bc64d9ce9dd40f4d72152a7a5b9e" Nov 03 13:58:58.689161 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:58:58.689141 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd3e320ba158fcb7fe6ac78b1ed06eff5950bc64d9ce9dd40f4d72152a7a5b9e\": container with ID starting with cd3e320ba158fcb7fe6ac78b1ed06eff5950bc64d9ce9dd40f4d72152a7a5b9e not found: ID does not exist" containerID="cd3e320ba158fcb7fe6ac78b1ed06eff5950bc64d9ce9dd40f4d72152a7a5b9e" Nov 03 13:58:58.689227 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:58.689171 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd3e320ba158fcb7fe6ac78b1ed06eff5950bc64d9ce9dd40f4d72152a7a5b9e"} err="failed to get container status \"cd3e320ba158fcb7fe6ac78b1ed06eff5950bc64d9ce9dd40f4d72152a7a5b9e\": rpc error: code = NotFound desc = could not find container \"cd3e320ba158fcb7fe6ac78b1ed06eff5950bc64d9ce9dd40f4d72152a7a5b9e\": container with ID starting with cd3e320ba158fcb7fe6ac78b1ed06eff5950bc64d9ce9dd40f4d72152a7a5b9e not found: ID does not exist" Nov 03 13:58:58.689227 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:58.689188 5979 scope.go:122] "RemoveContainer" containerID="808a83b27db3fc13e449f2a22c80e386dab0cadb6aecd4ceaa90c7a860866190" Nov 03 13:58:58.689462 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:58:58.689448 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"808a83b27db3fc13e449f2a22c80e386dab0cadb6aecd4ceaa90c7a860866190\": container with ID starting with 808a83b27db3fc13e449f2a22c80e386dab0cadb6aecd4ceaa90c7a860866190 not found: ID does not exist" containerID="808a83b27db3fc13e449f2a22c80e386dab0cadb6aecd4ceaa90c7a860866190" Nov 03 13:58:58.689544 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:58.689494 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"808a83b27db3fc13e449f2a22c80e386dab0cadb6aecd4ceaa90c7a860866190"} err="failed to get container status \"808a83b27db3fc13e449f2a22c80e386dab0cadb6aecd4ceaa90c7a860866190\": rpc error: code = NotFound desc = could not find container \"808a83b27db3fc13e449f2a22c80e386dab0cadb6aecd4ceaa90c7a860866190\": container with ID starting with 808a83b27db3fc13e449f2a22c80e386dab0cadb6aecd4ceaa90c7a860866190 not found: ID does not exist" Nov 03 13:58:58.755591 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:58.755523 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a1133dc-605d-4871-8190-7714aca1ce00-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:58:58.755591 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:58.755558 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a1133dc-605d-4871-8190-7714aca1ce00-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:58:58.755591 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:58.755572 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a1133dc-605d-4871-8190-7714aca1ce00-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:58:58.755591 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:58.755586 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9nl7\" (UniqueName: \"kubernetes.io/projected/8a1133dc-605d-4871-8190-7714aca1ce00-kube-api-access-z9nl7\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:58:58.755887 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:58.755610 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a1133dc-605d-4871-8190-7714aca1ce00-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 13:58:58.852409 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:58.852326 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m"] Nov 03 13:58:58.857915 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:58:58.857888 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-92546f-openshift-default-6ff5974d9-s845m"] Nov 03 13:59:00.116486 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:00.116443 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a1133dc-605d-4871-8190-7714aca1ce00" path="/var/lib/kubelet/pods/8a1133dc-605d-4871-8190-7714aca1ce00/volumes" Nov 03 13:59:07.234038 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:59:07.233999 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:59:37.235440 ip-10-0-130-198 kubenswrapper[5979]: E1103 13:59:37.235400 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 13:59:38.583334 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:38.583301 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 13:59:41.437919 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:41.437874 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44"] Nov 03 13:59:41.438489 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:41.438462 5979 cpu_manager.go:414] "containerMap: removing container" podUID="8a1133dc-605d-4871-8190-7714aca1ce00" containerName="istio-proxy" Nov 03 13:59:41.438546 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:41.438491 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a1133dc-605d-4871-8190-7714aca1ce00" containerName="istio-proxy" Nov 03 13:59:41.438546 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:41.438522 5979 cpu_manager.go:414] "containerMap: removing container" podUID="8a1133dc-605d-4871-8190-7714aca1ce00" containerName="istio-proxy" Nov 03 13:59:41.438546 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:41.438528 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a1133dc-605d-4871-8190-7714aca1ce00" containerName="istio-proxy" Nov 03 13:59:41.438650 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:41.438616 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a1133dc-605d-4871-8190-7714aca1ce00" containerName="istio-proxy" Nov 03 13:59:41.438650 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:41.438631 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a1133dc-605d-4871-8190-7714aca1ce00" containerName="istio-proxy" Nov 03 13:59:41.443679 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:41.443657 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44" Nov 03 13:59:41.487396 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:41.487347 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0e4153d-37d8-4233-8744-7515f9878bdf-istio-podinfo\") pod \"e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44\" (UID: \"e0e4153d-37d8-4233-8744-7515f9878bdf\") " pod="openshift-ingress/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44" Nov 03 13:59:41.487593 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:41.487420 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0e4153d-37d8-4233-8744-7515f9878bdf-workload-socket\") pod \"e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44\" (UID: \"e0e4153d-37d8-4233-8744-7515f9878bdf\") " pod="openshift-ingress/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44" Nov 03 13:59:41.487593 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:41.487444 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0e4153d-37d8-4233-8744-7515f9878bdf-istiod-ca-cert\") pod \"e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44\" (UID: \"e0e4153d-37d8-4233-8744-7515f9878bdf\") " pod="openshift-ingress/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44" Nov 03 13:59:41.487593 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:41.487502 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tpl4\" (UniqueName: \"kubernetes.io/projected/e0e4153d-37d8-4233-8744-7515f9878bdf-kube-api-access-9tpl4\") pod \"e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44\" (UID: \"e0e4153d-37d8-4233-8744-7515f9878bdf\") " pod="openshift-ingress/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44" Nov 03 13:59:41.487593 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:41.487538 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0e4153d-37d8-4233-8744-7515f9878bdf-workload-certs\") pod \"e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44\" (UID: \"e0e4153d-37d8-4233-8744-7515f9878bdf\") " pod="openshift-ingress/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44" Nov 03 13:59:41.487796 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:41.487638 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0e4153d-37d8-4233-8744-7515f9878bdf-istio-token\") pod \"e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44\" (UID: \"e0e4153d-37d8-4233-8744-7515f9878bdf\") " pod="openshift-ingress/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44" Nov 03 13:59:41.487796 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:41.487725 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0e4153d-37d8-4233-8744-7515f9878bdf-istio-data\") pod \"e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44\" (UID: \"e0e4153d-37d8-4233-8744-7515f9878bdf\") " pod="openshift-ingress/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44" Nov 03 13:59:41.487796 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:41.487752 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0e4153d-37d8-4233-8744-7515f9878bdf-credential-socket\") pod \"e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44\" (UID: \"e0e4153d-37d8-4233-8744-7515f9878bdf\") " pod="openshift-ingress/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44" Nov 03 13:59:41.487973 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:41.487824 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0e4153d-37d8-4233-8744-7515f9878bdf-istio-envoy\") pod \"e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44\" (UID: \"e0e4153d-37d8-4233-8744-7515f9878bdf\") " pod="openshift-ingress/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44" Nov 03 13:59:41.492704 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:41.492663 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-ae248d-openshift-default-dockercfg-s4ldn\"" Nov 03 13:59:41.541578 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:41.541533 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44"] Nov 03 13:59:41.591296 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:41.589537 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0e4153d-37d8-4233-8744-7515f9878bdf-istio-envoy\") pod \"e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44\" (UID: \"e0e4153d-37d8-4233-8744-7515f9878bdf\") " pod="openshift-ingress/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44" Nov 03 13:59:41.591296 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:41.589613 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0e4153d-37d8-4233-8744-7515f9878bdf-istio-podinfo\") pod \"e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44\" (UID: \"e0e4153d-37d8-4233-8744-7515f9878bdf\") " pod="openshift-ingress/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44" Nov 03 13:59:41.591296 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:41.589665 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0e4153d-37d8-4233-8744-7515f9878bdf-workload-socket\") pod \"e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44\" (UID: \"e0e4153d-37d8-4233-8744-7515f9878bdf\") " pod="openshift-ingress/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44" Nov 03 13:59:41.591296 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:41.589693 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0e4153d-37d8-4233-8744-7515f9878bdf-istiod-ca-cert\") pod \"e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44\" (UID: \"e0e4153d-37d8-4233-8744-7515f9878bdf\") " pod="openshift-ingress/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44" Nov 03 13:59:41.591296 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:41.589731 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tpl4\" (UniqueName: \"kubernetes.io/projected/e0e4153d-37d8-4233-8744-7515f9878bdf-kube-api-access-9tpl4\") pod \"e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44\" (UID: \"e0e4153d-37d8-4233-8744-7515f9878bdf\") " pod="openshift-ingress/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44" Nov 03 13:59:41.591296 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:41.589769 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0e4153d-37d8-4233-8744-7515f9878bdf-workload-certs\") pod \"e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44\" (UID: \"e0e4153d-37d8-4233-8744-7515f9878bdf\") " pod="openshift-ingress/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44" Nov 03 13:59:41.591296 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:41.589894 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0e4153d-37d8-4233-8744-7515f9878bdf-istio-token\") pod \"e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44\" (UID: \"e0e4153d-37d8-4233-8744-7515f9878bdf\") " pod="openshift-ingress/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44" Nov 03 13:59:41.591296 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:41.590018 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0e4153d-37d8-4233-8744-7515f9878bdf-istio-data\") pod \"e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44\" (UID: \"e0e4153d-37d8-4233-8744-7515f9878bdf\") " pod="openshift-ingress/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44" Nov 03 13:59:41.591296 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:41.590054 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0e4153d-37d8-4233-8744-7515f9878bdf-credential-socket\") pod \"e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44\" (UID: \"e0e4153d-37d8-4233-8744-7515f9878bdf\") " pod="openshift-ingress/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44" Nov 03 13:59:41.591296 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:41.590567 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0e4153d-37d8-4233-8744-7515f9878bdf-credential-socket\") pod \"e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44\" (UID: \"e0e4153d-37d8-4233-8744-7515f9878bdf\") " pod="openshift-ingress/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44" Nov 03 13:59:41.592907 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:41.592879 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0e4153d-37d8-4233-8744-7515f9878bdf-workload-socket\") pod \"e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44\" (UID: \"e0e4153d-37d8-4233-8744-7515f9878bdf\") " pod="openshift-ingress/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44" Nov 03 13:59:41.593974 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:41.593952 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0e4153d-37d8-4233-8744-7515f9878bdf-workload-certs\") pod \"e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44\" (UID: \"e0e4153d-37d8-4233-8744-7515f9878bdf\") " pod="openshift-ingress/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44" Nov 03 13:59:41.604577 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:41.594305 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0e4153d-37d8-4233-8744-7515f9878bdf-istio-data\") pod \"e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44\" (UID: \"e0e4153d-37d8-4233-8744-7515f9878bdf\") " pod="openshift-ingress/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44" Nov 03 13:59:41.604577 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:41.594415 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0e4153d-37d8-4233-8744-7515f9878bdf-istiod-ca-cert\") pod \"e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44\" (UID: \"e0e4153d-37d8-4233-8744-7515f9878bdf\") " pod="openshift-ingress/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44" Nov 03 13:59:41.609923 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:41.609896 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0e4153d-37d8-4233-8744-7515f9878bdf-istio-envoy\") pod \"e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44\" (UID: \"e0e4153d-37d8-4233-8744-7515f9878bdf\") " pod="openshift-ingress/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44" Nov 03 13:59:41.622097 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:41.621989 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0e4153d-37d8-4233-8744-7515f9878bdf-istio-podinfo\") pod \"e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44\" (UID: \"e0e4153d-37d8-4233-8744-7515f9878bdf\") " pod="openshift-ingress/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44" Nov 03 13:59:41.655756 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:41.655719 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tpl4\" (UniqueName: \"kubernetes.io/projected/e0e4153d-37d8-4233-8744-7515f9878bdf-kube-api-access-9tpl4\") pod \"e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44\" (UID: \"e0e4153d-37d8-4233-8744-7515f9878bdf\") " pod="openshift-ingress/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44" Nov 03 13:59:41.664097 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:41.664065 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0e4153d-37d8-4233-8744-7515f9878bdf-istio-token\") pod \"e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44\" (UID: \"e0e4153d-37d8-4233-8744-7515f9878bdf\") " pod="openshift-ingress/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44" Nov 03 13:59:41.757429 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:41.757335 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44" Nov 03 13:59:42.244445 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:42.244266 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44"] Nov 03 13:59:42.250660 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:59:42.250628 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0e4153d_37d8_4233_8744_7515f9878bdf.slice/crio-6b965580cf5acf314b6709e30d9fda871ce94e52eab9bb375f7fda0b5fbb3daf WatchSource:0}: Error finding container 6b965580cf5acf314b6709e30d9fda871ce94e52eab9bb375f7fda0b5fbb3daf: Status 404 returned error can't find the container with id 6b965580cf5acf314b6709e30d9fda871ce94e52eab9bb375f7fda0b5fbb3daf Nov 03 13:59:42.253241 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:42.253207 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:59:42.253356 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:42.253293 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:59:42.253356 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:42.253340 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 13:59:42.926948 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:42.926911 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44" event={"ID":"e0e4153d-37d8-4233-8744-7515f9878bdf","Type":"ContainerStarted","Data":"8313cbab19ffb9001c91a3a40203b463da8cb6e7b820af64d57a964a311ed62c"} Nov 03 13:59:42.927443 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:42.926964 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44" event={"ID":"e0e4153d-37d8-4233-8744-7515f9878bdf","Type":"ContainerStarted","Data":"6b965580cf5acf314b6709e30d9fda871ce94e52eab9bb375f7fda0b5fbb3daf"} Nov 03 13:59:43.131322 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:43.131265 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44" podStartSLOduration=2.131250149 podStartE2EDuration="2.131250149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:59:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:59:43.082267214 +0000 UTC m=+3265.092955456" watchObservedRunningTime="2026-11-03 13:59:43.131250149 +0000 UTC m=+3265.141938391" Nov 03 13:59:43.758377 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:43.758349 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44" Nov 03 13:59:43.764039 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:43.764012 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44" Nov 03 13:59:43.933020 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:43.932968 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44" Nov 03 13:59:43.933813 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:43.933795 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44" Nov 03 13:59:49.596697 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:49.596659 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ae248d-7456846d9f-wntx2"] Nov 03 13:59:49.601108 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:49.601087 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ae248d-7456846d9f-wntx2" Nov 03 13:59:49.656747 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:49.656710 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ae248d-7456846d9f-wntx2"] Nov 03 13:59:49.699709 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:49.699671 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-ae248d\"" Nov 03 13:59:49.700161 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:49.700136 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-ae248d-dockercfg-s7clc\"" Nov 03 13:59:49.701220 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:49.701180 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hg2d\" (UniqueName: \"kubernetes.io/projected/acda8eb6-a116-4a80-8572-4fc538ddb5b6-kube-api-access-6hg2d\") pod \"payload-pre-processing-e2e-extmod-a-ae248d-7456846d9f-wntx2\" (UID: \"acda8eb6-a116-4a80-8572-4fc538ddb5b6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ae248d-7456846d9f-wntx2" Nov 03 13:59:49.701427 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:49.701410 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acda8eb6-a116-4a80-8572-4fc538ddb5b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-ae248d-7456846d9f-wntx2\" (UID: \"acda8eb6-a116-4a80-8572-4fc538ddb5b6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ae248d-7456846d9f-wntx2" Nov 03 13:59:49.750789 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:49.750703 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ae248d-6d79d84655-b88z9"] Nov 03 13:59:49.756038 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:49.756012 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ae248d-6d79d84655-b88z9" Nov 03 13:59:49.803241 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:49.803203 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hg2d\" (UniqueName: \"kubernetes.io/projected/acda8eb6-a116-4a80-8572-4fc538ddb5b6-kube-api-access-6hg2d\") pod \"payload-pre-processing-e2e-extmod-a-ae248d-7456846d9f-wntx2\" (UID: \"acda8eb6-a116-4a80-8572-4fc538ddb5b6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ae248d-7456846d9f-wntx2" Nov 03 13:59:49.803434 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:49.803253 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acda8eb6-a116-4a80-8572-4fc538ddb5b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-ae248d-7456846d9f-wntx2\" (UID: \"acda8eb6-a116-4a80-8572-4fc538ddb5b6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ae248d-7456846d9f-wntx2" Nov 03 13:59:49.804044 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:49.804019 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acda8eb6-a116-4a80-8572-4fc538ddb5b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-ae248d-7456846d9f-wntx2\" (UID: \"acda8eb6-a116-4a80-8572-4fc538ddb5b6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ae248d-7456846d9f-wntx2" Nov 03 13:59:49.807983 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:49.807948 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ae248d-6d79d84655-b88z9"] Nov 03 13:59:49.857932 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:49.857866 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hg2d\" (UniqueName: \"kubernetes.io/projected/acda8eb6-a116-4a80-8572-4fc538ddb5b6-kube-api-access-6hg2d\") pod \"payload-pre-processing-e2e-extmod-a-ae248d-7456846d9f-wntx2\" (UID: \"acda8eb6-a116-4a80-8572-4fc538ddb5b6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ae248d-7456846d9f-wntx2" Nov 03 13:59:49.904388 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:49.904352 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c5348d9-47e4-428f-b789-558b167b9ae5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-ae248d-6d79d84655-b88z9\" (UID: \"2c5348d9-47e4-428f-b789-558b167b9ae5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ae248d-6d79d84655-b88z9" Nov 03 13:59:49.904626 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:49.904532 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4cx9\" (UniqueName: \"kubernetes.io/projected/2c5348d9-47e4-428f-b789-558b167b9ae5-kube-api-access-j4cx9\") pod \"payload-processing-e2e-extmod-a-ae248d-6d79d84655-b88z9\" (UID: \"2c5348d9-47e4-428f-b789-558b167b9ae5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ae248d-6d79d84655-b88z9" Nov 03 13:59:49.916051 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:49.916030 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ae248d-7456846d9f-wntx2" Nov 03 13:59:50.005675 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:50.005643 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c5348d9-47e4-428f-b789-558b167b9ae5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-ae248d-6d79d84655-b88z9\" (UID: \"2c5348d9-47e4-428f-b789-558b167b9ae5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ae248d-6d79d84655-b88z9" Nov 03 13:59:50.005877 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:50.005759 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4cx9\" (UniqueName: \"kubernetes.io/projected/2c5348d9-47e4-428f-b789-558b167b9ae5-kube-api-access-j4cx9\") pod \"payload-processing-e2e-extmod-a-ae248d-6d79d84655-b88z9\" (UID: \"2c5348d9-47e4-428f-b789-558b167b9ae5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ae248d-6d79d84655-b88z9" Nov 03 13:59:50.006400 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:50.006379 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c5348d9-47e4-428f-b789-558b167b9ae5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-ae248d-6d79d84655-b88z9\" (UID: \"2c5348d9-47e4-428f-b789-558b167b9ae5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ae248d-6d79d84655-b88z9" Nov 03 13:59:50.059593 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:50.059556 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4cx9\" (UniqueName: \"kubernetes.io/projected/2c5348d9-47e4-428f-b789-558b167b9ae5-kube-api-access-j4cx9\") pod \"payload-processing-e2e-extmod-a-ae248d-6d79d84655-b88z9\" (UID: \"2c5348d9-47e4-428f-b789-558b167b9ae5\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ae248d-6d79d84655-b88z9" Nov 03 13:59:50.092252 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:50.092226 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ae248d-6d79d84655-b88z9" Nov 03 13:59:50.289202 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:50.289166 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ae248d-7456846d9f-wntx2"] Nov 03 13:59:50.297001 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:59:50.296968 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacda8eb6_a116_4a80_8572_4fc538ddb5b6.slice/crio-7fea12a3d43cc84fccfa80523ad17a2e8738967d70b49cdc17cc50a790d75f59 WatchSource:0}: Error finding container 7fea12a3d43cc84fccfa80523ad17a2e8738967d70b49cdc17cc50a790d75f59: Status 404 returned error can't find the container with id 7fea12a3d43cc84fccfa80523ad17a2e8738967d70b49cdc17cc50a790d75f59 Nov 03 13:59:50.469454 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:50.469367 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ae248d-6d79d84655-b88z9"] Nov 03 13:59:50.474311 ip-10-0-130-198 kubenswrapper[5979]: W1103 13:59:50.474285 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c5348d9_47e4_428f_b789_558b167b9ae5.slice/crio-8b58c1a5083f289f249bad68b2ca62474757fc4716fc0b1375463cbcf3114b8e WatchSource:0}: Error finding container 8b58c1a5083f289f249bad68b2ca62474757fc4716fc0b1375463cbcf3114b8e: Status 404 returned error can't find the container with id 8b58c1a5083f289f249bad68b2ca62474757fc4716fc0b1375463cbcf3114b8e Nov 03 13:59:50.979013 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:50.978979 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ae248d-7456846d9f-wntx2" event={"ID":"acda8eb6-a116-4a80-8572-4fc538ddb5b6","Type":"ContainerStarted","Data":"efff3607abe96de52392d0f3f00188c14a528e774c973de8306db829e2a122d1"} Nov 03 13:59:50.979533 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:50.979041 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ae248d-7456846d9f-wntx2" event={"ID":"acda8eb6-a116-4a80-8572-4fc538ddb5b6","Type":"ContainerStarted","Data":"7fea12a3d43cc84fccfa80523ad17a2e8738967d70b49cdc17cc50a790d75f59"} Nov 03 13:59:50.979533 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:50.979068 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ae248d-7456846d9f-wntx2" Nov 03 13:59:50.980672 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:50.980647 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ae248d-6d79d84655-b88z9" event={"ID":"2c5348d9-47e4-428f-b789-558b167b9ae5","Type":"ContainerStarted","Data":"362365845d0e40cbd118f4e94e761c771496f90d8c7a17b57c4ee7336314ce9e"} Nov 03 13:59:50.980792 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:50.980686 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ae248d-6d79d84655-b88z9" event={"ID":"2c5348d9-47e4-428f-b789-558b167b9ae5","Type":"ContainerStarted","Data":"8b58c1a5083f289f249bad68b2ca62474757fc4716fc0b1375463cbcf3114b8e"} Nov 03 13:59:50.980792 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:50.980734 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-ae248d-6d79d84655-b88z9" Nov 03 13:59:51.090639 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:51.090584 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ae248d-7456846d9f-wntx2" podStartSLOduration=2.090570165 podStartE2EDuration="2.090570165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:59:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:59:51.086667263 +0000 UTC m=+3273.097355505" watchObservedRunningTime="2026-11-03 13:59:51.090570165 +0000 UTC m=+3273.101258425" Nov 03 13:59:51.194400 ip-10-0-130-198 kubenswrapper[5979]: I1103 13:59:51.194328 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-ae248d-6d79d84655-b88z9" podStartSLOduration=2.194310315 podStartE2EDuration="2.194310315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 13:59:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 13:59:51.19356339 +0000 UTC m=+3273.204251634" watchObservedRunningTime="2026-11-03 13:59:51.194310315 +0000 UTC m=+3273.204998558" Nov 03 14:00:00.166526 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.166461 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895240-p7lk8"] Nov 03 14:00:00.171157 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.171132 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895240-x7b9t"] Nov 03 14:00:00.171308 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.171233 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895240-p7lk8" Nov 03 14:00:00.175347 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.175325 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895240-x7b9t" Nov 03 14:00:00.201373 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.201347 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895240-2wbbp"] Nov 03 14:00:00.205319 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.205300 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895240-2wbbp" Nov 03 14:00:00.213219 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.213198 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895240-xwhnj"] Nov 03 14:00:00.217021 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.217002 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895240-5qpqx"] Nov 03 14:00:00.217135 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.217108 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895240-xwhnj" Nov 03 14:00:00.221168 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.221151 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895240-5qpqx" Nov 03 14:00:00.224910 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.224890 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895240-p9n95"] Nov 03 14:00:00.228738 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.228719 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895240-p9n95" Nov 03 14:00:00.244821 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.244801 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 14:00:00.259928 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.259893 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895240-p7lk8"] Nov 03 14:00:00.262205 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.262185 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895240-x7b9t"] Nov 03 14:00:00.271398 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.271377 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c1c368a-b074-4c74-afe6-77fc73e982bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895240-p7lk8\" (UID: \"7c1c368a-b074-4c74-afe6-77fc73e982bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895240-p7lk8" Nov 03 14:00:00.271513 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.271413 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5fa16b1-200a-44b7-a5bf-4fa8305c7916-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895240-5qpqx\" (UID: \"c5fa16b1-200a-44b7-a5bf-4fa8305c7916\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895240-5qpqx" Nov 03 14:00:00.271513 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.271436 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33a20f02-e3b2-4f59-a01f-68b3d837c090-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895240-x7b9t\" (UID: \"33a20f02-e3b2-4f59-a01f-68b3d837c090\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895240-x7b9t" Nov 03 14:00:00.271602 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.271568 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z5s7\" (UniqueName: \"kubernetes.io/projected/6cd99276-aae5-4f57-bb7b-fde6e6821f20-kube-api-access-4z5s7\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895240-p9n95\" (UID: \"6cd99276-aae5-4f57-bb7b-fde6e6821f20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895240-p9n95" Nov 03 14:00:00.271648 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.271625 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61dbd591-8f07-493d-8a27-9475e5fc84a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895240-xwhnj\" (UID: \"61dbd591-8f07-493d-8a27-9475e5fc84a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895240-xwhnj" Nov 03 14:00:00.271727 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.271708 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phr2g\" (UniqueName: \"kubernetes.io/projected/c5fa16b1-200a-44b7-a5bf-4fa8305c7916-kube-api-access-phr2g\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895240-5qpqx\" (UID: \"c5fa16b1-200a-44b7-a5bf-4fa8305c7916\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895240-5qpqx" Nov 03 14:00:00.271761 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.271739 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mp2g\" (UniqueName: \"kubernetes.io/projected/7c1c368a-b074-4c74-afe6-77fc73e982bd-kube-api-access-5mp2g\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895240-p7lk8\" (UID: \"7c1c368a-b074-4c74-afe6-77fc73e982bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895240-p7lk8" Nov 03 14:00:00.271800 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.271769 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/582acff3-15b0-4430-9ff2-25188540fcab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895240-2wbbp\" (UID: \"582acff3-15b0-4430-9ff2-25188540fcab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895240-2wbbp" Nov 03 14:00:00.271839 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.271823 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv9b5\" (UniqueName: \"kubernetes.io/projected/61dbd591-8f07-493d-8a27-9475e5fc84a8-kube-api-access-nv9b5\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895240-xwhnj\" (UID: \"61dbd591-8f07-493d-8a27-9475e5fc84a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895240-xwhnj" Nov 03 14:00:00.271921 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.271908 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cd99276-aae5-4f57-bb7b-fde6e6821f20-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895240-p9n95\" (UID: \"6cd99276-aae5-4f57-bb7b-fde6e6821f20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895240-p9n95" Nov 03 14:00:00.271986 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.271974 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tq2w\" (UniqueName: \"kubernetes.io/projected/33a20f02-e3b2-4f59-a01f-68b3d837c090-kube-api-access-2tq2w\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895240-x7b9t\" (UID: \"33a20f02-e3b2-4f59-a01f-68b3d837c090\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895240-x7b9t" Nov 03 14:00:00.272025 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.271998 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6r55\" (UniqueName: \"kubernetes.io/projected/582acff3-15b0-4430-9ff2-25188540fcab-kube-api-access-x6r55\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895240-2wbbp\" (UID: \"582acff3-15b0-4430-9ff2-25188540fcab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895240-2wbbp" Nov 03 14:00:00.309172 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.309139 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895240-xwhnj"] Nov 03 14:00:00.313617 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.313582 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895240-2wbbp"] Nov 03 14:00:00.358954 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.358923 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895240-5qpqx"] Nov 03 14:00:00.371735 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.371698 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895240-p9n95"] Nov 03 14:00:00.373194 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.373163 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33a20f02-e3b2-4f59-a01f-68b3d837c090-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895240-x7b9t\" (UID: \"33a20f02-e3b2-4f59-a01f-68b3d837c090\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895240-x7b9t" Nov 03 14:00:00.374217 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.374190 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z5s7\" (UniqueName: \"kubernetes.io/projected/6cd99276-aae5-4f57-bb7b-fde6e6821f20-kube-api-access-4z5s7\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895240-p9n95\" (UID: \"6cd99276-aae5-4f57-bb7b-fde6e6821f20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895240-p9n95" Nov 03 14:00:00.374346 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.374096 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33a20f02-e3b2-4f59-a01f-68b3d837c090-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895240-x7b9t\" (UID: \"33a20f02-e3b2-4f59-a01f-68b3d837c090\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895240-x7b9t" Nov 03 14:00:00.374346 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.374279 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61dbd591-8f07-493d-8a27-9475e5fc84a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895240-xwhnj\" (UID: \"61dbd591-8f07-493d-8a27-9475e5fc84a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895240-xwhnj" Nov 03 14:00:00.374512 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.374382 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phr2g\" (UniqueName: \"kubernetes.io/projected/c5fa16b1-200a-44b7-a5bf-4fa8305c7916-kube-api-access-phr2g\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895240-5qpqx\" (UID: \"c5fa16b1-200a-44b7-a5bf-4fa8305c7916\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895240-5qpqx" Nov 03 14:00:00.374512 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.374439 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mp2g\" (UniqueName: \"kubernetes.io/projected/7c1c368a-b074-4c74-afe6-77fc73e982bd-kube-api-access-5mp2g\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895240-p7lk8\" (UID: \"7c1c368a-b074-4c74-afe6-77fc73e982bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895240-p7lk8" Nov 03 14:00:00.378515 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.374526 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/582acff3-15b0-4430-9ff2-25188540fcab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895240-2wbbp\" (UID: \"582acff3-15b0-4430-9ff2-25188540fcab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895240-2wbbp" Nov 03 14:00:00.378515 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.374560 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv9b5\" (UniqueName: \"kubernetes.io/projected/61dbd591-8f07-493d-8a27-9475e5fc84a8-kube-api-access-nv9b5\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895240-xwhnj\" (UID: \"61dbd591-8f07-493d-8a27-9475e5fc84a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895240-xwhnj" Nov 03 14:00:00.378515 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.374643 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cd99276-aae5-4f57-bb7b-fde6e6821f20-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895240-p9n95\" (UID: \"6cd99276-aae5-4f57-bb7b-fde6e6821f20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895240-p9n95" Nov 03 14:00:00.378515 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.374772 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tq2w\" (UniqueName: \"kubernetes.io/projected/33a20f02-e3b2-4f59-a01f-68b3d837c090-kube-api-access-2tq2w\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895240-x7b9t\" (UID: \"33a20f02-e3b2-4f59-a01f-68b3d837c090\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895240-x7b9t" Nov 03 14:00:00.378515 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.374806 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6r55\" (UniqueName: \"kubernetes.io/projected/582acff3-15b0-4430-9ff2-25188540fcab-kube-api-access-x6r55\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895240-2wbbp\" (UID: \"582acff3-15b0-4430-9ff2-25188540fcab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895240-2wbbp" Nov 03 14:00:00.378515 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.374902 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c1c368a-b074-4c74-afe6-77fc73e982bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895240-p7lk8\" (UID: \"7c1c368a-b074-4c74-afe6-77fc73e982bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895240-p7lk8" Nov 03 14:00:00.378515 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.374943 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5fa16b1-200a-44b7-a5bf-4fa8305c7916-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895240-5qpqx\" (UID: \"c5fa16b1-200a-44b7-a5bf-4fa8305c7916\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895240-5qpqx" Nov 03 14:00:00.378515 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.374955 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61dbd591-8f07-493d-8a27-9475e5fc84a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895240-xwhnj\" (UID: \"61dbd591-8f07-493d-8a27-9475e5fc84a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895240-xwhnj" Nov 03 14:00:00.378515 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.375504 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/582acff3-15b0-4430-9ff2-25188540fcab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895240-2wbbp\" (UID: \"582acff3-15b0-4430-9ff2-25188540fcab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895240-2wbbp" Nov 03 14:00:00.378515 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.375739 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c1c368a-b074-4c74-afe6-77fc73e982bd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895240-p7lk8\" (UID: \"7c1c368a-b074-4c74-afe6-77fc73e982bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895240-p7lk8" Nov 03 14:00:00.378515 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.376716 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cd99276-aae5-4f57-bb7b-fde6e6821f20-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895240-p9n95\" (UID: \"6cd99276-aae5-4f57-bb7b-fde6e6821f20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895240-p9n95" Nov 03 14:00:00.378515 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.377011 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5fa16b1-200a-44b7-a5bf-4fa8305c7916-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895240-5qpqx\" (UID: \"c5fa16b1-200a-44b7-a5bf-4fa8305c7916\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895240-5qpqx" Nov 03 14:00:00.431877 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.431796 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tq2w\" (UniqueName: \"kubernetes.io/projected/33a20f02-e3b2-4f59-a01f-68b3d837c090-kube-api-access-2tq2w\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895240-x7b9t\" (UID: \"33a20f02-e3b2-4f59-a01f-68b3d837c090\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895240-x7b9t" Nov 03 14:00:00.432026 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.431871 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mp2g\" (UniqueName: \"kubernetes.io/projected/7c1c368a-b074-4c74-afe6-77fc73e982bd-kube-api-access-5mp2g\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895240-p7lk8\" (UID: \"7c1c368a-b074-4c74-afe6-77fc73e982bd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895240-p7lk8" Nov 03 14:00:00.432085 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.432057 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z5s7\" (UniqueName: \"kubernetes.io/projected/6cd99276-aae5-4f57-bb7b-fde6e6821f20-kube-api-access-4z5s7\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895240-p9n95\" (UID: \"6cd99276-aae5-4f57-bb7b-fde6e6821f20\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895240-p9n95" Nov 03 14:00:00.432585 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.432559 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6r55\" (UniqueName: \"kubernetes.io/projected/582acff3-15b0-4430-9ff2-25188540fcab-kube-api-access-x6r55\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895240-2wbbp\" (UID: \"582acff3-15b0-4430-9ff2-25188540fcab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895240-2wbbp" Nov 03 14:00:00.432752 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.432731 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phr2g\" (UniqueName: \"kubernetes.io/projected/c5fa16b1-200a-44b7-a5bf-4fa8305c7916-kube-api-access-phr2g\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895240-5qpqx\" (UID: \"c5fa16b1-200a-44b7-a5bf-4fa8305c7916\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895240-5qpqx" Nov 03 14:00:00.432801 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.432731 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv9b5\" (UniqueName: \"kubernetes.io/projected/61dbd591-8f07-493d-8a27-9475e5fc84a8-kube-api-access-nv9b5\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895240-xwhnj\" (UID: \"61dbd591-8f07-493d-8a27-9475e5fc84a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895240-xwhnj" Nov 03 14:00:00.488011 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.487977 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895240-p7lk8" Nov 03 14:00:00.494256 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.494230 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895240-x7b9t" Nov 03 14:00:00.519322 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.519293 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895240-2wbbp" Nov 03 14:00:00.532362 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.532335 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895240-xwhnj" Nov 03 14:00:00.540881 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.540681 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895240-5qpqx" Nov 03 14:00:00.553519 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.545906 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895240-p9n95" Nov 03 14:00:00.888521 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.888490 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895240-p7lk8"] Nov 03 14:00:00.893540 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:00:00.893504 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c1c368a_b074_4c74_afe6_77fc73e982bd.slice/crio-7aebbc4e8d7f89d3570e381240e7814927a2a649f6a3154e77d2edc34dcdc5fb WatchSource:0}: Error finding container 7aebbc4e8d7f89d3570e381240e7814927a2a649f6a3154e77d2edc34dcdc5fb: Status 404 returned error can't find the container with id 7aebbc4e8d7f89d3570e381240e7814927a2a649f6a3154e77d2edc34dcdc5fb Nov 03 14:00:00.941918 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.941835 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895240-x7b9t"] Nov 03 14:00:00.945965 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:00.945942 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895240-2wbbp"] Nov 03 14:00:00.948619 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:00:00.948595 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33a20f02_e3b2_4f59_a01f_68b3d837c090.slice/crio-a69f9f28ef7569c31f4792f7b70d30c70db0d715f4b5e061d9f415697a3f4bca WatchSource:0}: Error finding container a69f9f28ef7569c31f4792f7b70d30c70db0d715f4b5e061d9f415697a3f4bca: Status 404 returned error can't find the container with id a69f9f28ef7569c31f4792f7b70d30c70db0d715f4b5e061d9f415697a3f4bca Nov 03 14:00:00.949966 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:00:00.949943 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod582acff3_15b0_4430_9ff2_25188540fcab.slice/crio-f30b1c75963b6aab0afbaa664ed159ed02cce99d290c1d58695421a1ee4cbb31 WatchSource:0}: Error finding container f30b1c75963b6aab0afbaa664ed159ed02cce99d290c1d58695421a1ee4cbb31: Status 404 returned error can't find the container with id f30b1c75963b6aab0afbaa664ed159ed02cce99d290c1d58695421a1ee4cbb31 Nov 03 14:00:01.000511 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:01.000482 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895240-p9n95"] Nov 03 14:00:01.006974 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:01.006926 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895240-xwhnj"] Nov 03 14:00:01.012007 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:01.011984 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895240-5qpqx"] Nov 03 14:00:01.046857 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:00:01.046824 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cd99276_aae5_4f57_bb7b_fde6e6821f20.slice/crio-ea8658630120da80f58f25880f9155c7f4cb8383120c318891c8d0a9438e10c9 WatchSource:0}: Error finding container ea8658630120da80f58f25880f9155c7f4cb8383120c318891c8d0a9438e10c9: Status 404 returned error can't find the container with id ea8658630120da80f58f25880f9155c7f4cb8383120c318891c8d0a9438e10c9 Nov 03 14:00:01.048207 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:01.048176 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895240-2wbbp" event={"ID":"582acff3-15b0-4430-9ff2-25188540fcab","Type":"ContainerStarted","Data":"f30b1c75963b6aab0afbaa664ed159ed02cce99d290c1d58695421a1ee4cbb31"} Nov 03 14:00:01.050072 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:01.050034 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895240-p7lk8" event={"ID":"7c1c368a-b074-4c74-afe6-77fc73e982bd","Type":"ContainerStarted","Data":"7aebbc4e8d7f89d3570e381240e7814927a2a649f6a3154e77d2edc34dcdc5fb"} Nov 03 14:00:01.051901 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:01.051867 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895240-x7b9t" event={"ID":"33a20f02-e3b2-4f59-a01f-68b3d837c090","Type":"ContainerStarted","Data":"a69f9f28ef7569c31f4792f7b70d30c70db0d715f4b5e061d9f415697a3f4bca"} Nov 03 14:00:01.053564 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:00:01.053524 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5fa16b1_200a_44b7_a5bf_4fa8305c7916.slice/crio-6d7893f4d4a05a8b73737992dec0776566d9fc01dc111aeeb430e6810b26d4fd WatchSource:0}: Error finding container 6d7893f4d4a05a8b73737992dec0776566d9fc01dc111aeeb430e6810b26d4fd: Status 404 returned error can't find the container with id 6d7893f4d4a05a8b73737992dec0776566d9fc01dc111aeeb430e6810b26d4fd Nov 03 14:00:01.769757 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:01.769721 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6"] Nov 03 14:00:01.774347 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:01.774325 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6" Nov 03 14:00:01.840280 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:01.840191 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6"] Nov 03 14:00:01.880466 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:01.880430 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-c4e398-openshift-default-dockercfg-d4vg5\"" Nov 03 14:00:01.900898 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:01.900860 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ab327a8-7012-4504-9557-6dd31ed7a365-credential-socket\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6\" (UID: \"5ab327a8-7012-4504-9557-6dd31ed7a365\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6" Nov 03 14:00:01.900898 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:01.900902 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ab327a8-7012-4504-9557-6dd31ed7a365-workload-socket\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6\" (UID: \"5ab327a8-7012-4504-9557-6dd31ed7a365\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6" Nov 03 14:00:01.901136 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:01.901025 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ab327a8-7012-4504-9557-6dd31ed7a365-istio-data\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6\" (UID: \"5ab327a8-7012-4504-9557-6dd31ed7a365\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6" Nov 03 14:00:01.901136 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:01.901106 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ab327a8-7012-4504-9557-6dd31ed7a365-istio-token\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6\" (UID: \"5ab327a8-7012-4504-9557-6dd31ed7a365\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6" Nov 03 14:00:01.901266 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:01.901162 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pvz2\" (UniqueName: \"kubernetes.io/projected/5ab327a8-7012-4504-9557-6dd31ed7a365-kube-api-access-5pvz2\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6\" (UID: \"5ab327a8-7012-4504-9557-6dd31ed7a365\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6" Nov 03 14:00:01.901355 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:01.901333 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ab327a8-7012-4504-9557-6dd31ed7a365-istio-podinfo\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6\" (UID: \"5ab327a8-7012-4504-9557-6dd31ed7a365\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6" Nov 03 14:00:01.901394 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:01.901383 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ab327a8-7012-4504-9557-6dd31ed7a365-istio-envoy\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6\" (UID: \"5ab327a8-7012-4504-9557-6dd31ed7a365\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6" Nov 03 14:00:01.901438 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:01.901401 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ab327a8-7012-4504-9557-6dd31ed7a365-workload-certs\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6\" (UID: \"5ab327a8-7012-4504-9557-6dd31ed7a365\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6" Nov 03 14:00:01.901511 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:01.901435 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ab327a8-7012-4504-9557-6dd31ed7a365-istiod-ca-cert\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6\" (UID: \"5ab327a8-7012-4504-9557-6dd31ed7a365\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6" Nov 03 14:00:01.990605 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:01.990575 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ae248d-7456846d9f-wntx2" Nov 03 14:00:01.990779 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:01.990647 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-ae248d-6d79d84655-b88z9" Nov 03 14:00:02.002764 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:02.002733 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ab327a8-7012-4504-9557-6dd31ed7a365-istio-envoy\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6\" (UID: \"5ab327a8-7012-4504-9557-6dd31ed7a365\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6" Nov 03 14:00:02.002764 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:02.002767 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ab327a8-7012-4504-9557-6dd31ed7a365-workload-certs\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6\" (UID: \"5ab327a8-7012-4504-9557-6dd31ed7a365\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6" Nov 03 14:00:02.003015 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:02.002785 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ab327a8-7012-4504-9557-6dd31ed7a365-istiod-ca-cert\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6\" (UID: \"5ab327a8-7012-4504-9557-6dd31ed7a365\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6" Nov 03 14:00:02.003015 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:02.002827 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ab327a8-7012-4504-9557-6dd31ed7a365-credential-socket\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6\" (UID: \"5ab327a8-7012-4504-9557-6dd31ed7a365\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6" Nov 03 14:00:02.003015 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:02.002843 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ab327a8-7012-4504-9557-6dd31ed7a365-workload-socket\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6\" (UID: \"5ab327a8-7012-4504-9557-6dd31ed7a365\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6" Nov 03 14:00:02.003015 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:02.002890 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ab327a8-7012-4504-9557-6dd31ed7a365-istio-data\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6\" (UID: \"5ab327a8-7012-4504-9557-6dd31ed7a365\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6" Nov 03 14:00:02.003015 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:02.002922 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ab327a8-7012-4504-9557-6dd31ed7a365-istio-token\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6\" (UID: \"5ab327a8-7012-4504-9557-6dd31ed7a365\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6" Nov 03 14:00:02.003015 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:02.002971 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pvz2\" (UniqueName: \"kubernetes.io/projected/5ab327a8-7012-4504-9557-6dd31ed7a365-kube-api-access-5pvz2\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6\" (UID: \"5ab327a8-7012-4504-9557-6dd31ed7a365\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6" Nov 03 14:00:02.003330 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:02.003071 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ab327a8-7012-4504-9557-6dd31ed7a365-istio-podinfo\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6\" (UID: \"5ab327a8-7012-4504-9557-6dd31ed7a365\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6" Nov 03 14:00:02.003330 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:02.003193 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ab327a8-7012-4504-9557-6dd31ed7a365-workload-certs\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6\" (UID: \"5ab327a8-7012-4504-9557-6dd31ed7a365\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6" Nov 03 14:00:02.003330 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:02.003270 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ab327a8-7012-4504-9557-6dd31ed7a365-credential-socket\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6\" (UID: \"5ab327a8-7012-4504-9557-6dd31ed7a365\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6" Nov 03 14:00:02.003594 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:02.003571 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ab327a8-7012-4504-9557-6dd31ed7a365-workload-socket\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6\" (UID: \"5ab327a8-7012-4504-9557-6dd31ed7a365\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6" Nov 03 14:00:02.003668 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:02.003593 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ab327a8-7012-4504-9557-6dd31ed7a365-istio-data\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6\" (UID: \"5ab327a8-7012-4504-9557-6dd31ed7a365\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6" Nov 03 14:00:02.003668 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:02.003623 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ab327a8-7012-4504-9557-6dd31ed7a365-istiod-ca-cert\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6\" (UID: \"5ab327a8-7012-4504-9557-6dd31ed7a365\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6" Nov 03 14:00:02.005123 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:02.005103 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ab327a8-7012-4504-9557-6dd31ed7a365-istio-envoy\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6\" (UID: \"5ab327a8-7012-4504-9557-6dd31ed7a365\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6" Nov 03 14:00:02.005369 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:02.005350 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ab327a8-7012-4504-9557-6dd31ed7a365-istio-podinfo\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6\" (UID: \"5ab327a8-7012-4504-9557-6dd31ed7a365\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6" Nov 03 14:00:02.058173 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:02.058141 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pvz2\" (UniqueName: \"kubernetes.io/projected/5ab327a8-7012-4504-9557-6dd31ed7a365-kube-api-access-5pvz2\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6\" (UID: \"5ab327a8-7012-4504-9557-6dd31ed7a365\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6" Nov 03 14:00:02.058696 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:02.058676 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ab327a8-7012-4504-9557-6dd31ed7a365-istio-token\") pod \"e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6\" (UID: \"5ab327a8-7012-4504-9557-6dd31ed7a365\") " pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6" Nov 03 14:00:02.063092 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:02.063059 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895240-p9n95" event={"ID":"6cd99276-aae5-4f57-bb7b-fde6e6821f20","Type":"ContainerStarted","Data":"6f91d2233f9bb4bd551b63128ef202793f96c6bc1bfa90daf966a16a6ded1af8"} Nov 03 14:00:02.063249 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:02.063127 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895240-p9n95" event={"ID":"6cd99276-aae5-4f57-bb7b-fde6e6821f20","Type":"ContainerStarted","Data":"ea8658630120da80f58f25880f9155c7f4cb8383120c318891c8d0a9438e10c9"} Nov 03 14:00:02.064827 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:02.064796 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895240-2wbbp" event={"ID":"582acff3-15b0-4430-9ff2-25188540fcab","Type":"ContainerStarted","Data":"160ed787f8126d40357e1625c684e0659863ff7665c747bc0f4922f5a75c504a"} Nov 03 14:00:02.066539 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:02.066514 5979 generic.go:356] "Generic (PLEG): container finished" podID="7c1c368a-b074-4c74-afe6-77fc73e982bd" containerID="cda24e45e9736840e7010cf5a326d61b113f5fed4b7c7cb84fcd50ea05b64a3e" exitCode=60 Nov 03 14:00:02.066672 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:02.066590 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895240-p7lk8" event={"ID":"7c1c368a-b074-4c74-afe6-77fc73e982bd","Type":"ContainerDied","Data":"cda24e45e9736840e7010cf5a326d61b113f5fed4b7c7cb84fcd50ea05b64a3e"} Nov 03 14:00:02.066805 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:02.066791 5979 scope.go:122] "RemoveContainer" containerID="cda24e45e9736840e7010cf5a326d61b113f5fed4b7c7cb84fcd50ea05b64a3e" Nov 03 14:00:02.068196 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:02.068178 5979 generic.go:356] "Generic (PLEG): container finished" podID="33a20f02-e3b2-4f59-a01f-68b3d837c090" containerID="e2cdc8d717d593e43412c43667518c45384431606d026283118284b6bfd5f3fe" exitCode=60 Nov 03 14:00:02.068291 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:02.068256 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895240-x7b9t" event={"ID":"33a20f02-e3b2-4f59-a01f-68b3d837c090","Type":"ContainerDied","Data":"e2cdc8d717d593e43412c43667518c45384431606d026283118284b6bfd5f3fe"} Nov 03 14:00:02.068487 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:02.068461 5979 scope.go:122] "RemoveContainer" containerID="e2cdc8d717d593e43412c43667518c45384431606d026283118284b6bfd5f3fe" Nov 03 14:00:02.078938 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:02.070121 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895240-5qpqx" event={"ID":"c5fa16b1-200a-44b7-a5bf-4fa8305c7916","Type":"ContainerStarted","Data":"388285ba849cbc136b0fc3670b0849f1f0031d5776d5a5ef83a9c03884691a38"} Nov 03 14:00:02.078938 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:02.070155 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895240-5qpqx" event={"ID":"c5fa16b1-200a-44b7-a5bf-4fa8305c7916","Type":"ContainerStarted","Data":"6d7893f4d4a05a8b73737992dec0776566d9fc01dc111aeeb430e6810b26d4fd"} Nov 03 14:00:02.078938 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:02.071750 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895240-xwhnj" event={"ID":"61dbd591-8f07-493d-8a27-9475e5fc84a8","Type":"ContainerStarted","Data":"f7fe8ecafe7bbe8e6766fb363f4569b4f66a546e313ee3fca493d63410ea1a9c"} Nov 03 14:00:02.078938 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:02.071783 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895240-xwhnj" event={"ID":"61dbd591-8f07-493d-8a27-9475e5fc84a8","Type":"ContainerStarted","Data":"e1058f08c8dd0a3266d16c2db21ba27e80e2d96a62b00f78896952584d457d3f"} Nov 03 14:00:02.089927 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:02.089899 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6" Nov 03 14:00:02.324220 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:02.324166 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895240-2wbbp" podStartSLOduration=2.324153718 podStartE2EDuration="2.324153718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:00:02.322336639 +0000 UTC m=+3284.333024881" watchObservedRunningTime="2026-11-03 14:00:02.324153718 +0000 UTC m=+3284.334841960" Nov 03 14:00:02.499560 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:02.499520 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6"] Nov 03 14:00:02.505693 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:00:02.505667 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ab327a8_7012_4504_9557_6dd31ed7a365.slice/crio-9d4abc6a4bdc7c7267a375e4d83d4e7d86ef70ae32a433c643dbf3e15e59c639 WatchSource:0}: Error finding container 9d4abc6a4bdc7c7267a375e4d83d4e7d86ef70ae32a433c643dbf3e15e59c639: Status 404 returned error can't find the container with id 9d4abc6a4bdc7c7267a375e4d83d4e7d86ef70ae32a433c643dbf3e15e59c639 Nov 03 14:00:02.508568 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:02.508530 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:00:02.508676 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:02.508622 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:00:02.508676 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:02.508665 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:00:02.652009 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:02.651945 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895240-5qpqx" podStartSLOduration=2.651926015 podStartE2EDuration="2.651926015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:00:02.64795494 +0000 UTC m=+3284.658643183" watchObservedRunningTime="2026-11-03 14:00:02.651926015 +0000 UTC m=+3284.662614270" Nov 03 14:00:02.756243 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:02.756182 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895240-p9n95" podStartSLOduration=2.756162782 podStartE2EDuration="2.756162782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:00:02.754376241 +0000 UTC m=+3284.765064483" watchObservedRunningTime="2026-11-03 14:00:02.756162782 +0000 UTC m=+3284.766851027" Nov 03 14:00:02.861695 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:02.861626 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895240-xwhnj" podStartSLOduration=2.861604787 podStartE2EDuration="2.861604787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:00:02.860109922 +0000 UTC m=+3284.870798165" watchObservedRunningTime="2026-11-03 14:00:02.861604787 +0000 UTC m=+3284.872293031" Nov 03 14:00:03.080403 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:03.080376 5979 generic.go:356] "Generic (PLEG): container finished" podID="c5fa16b1-200a-44b7-a5bf-4fa8305c7916" containerID="388285ba849cbc136b0fc3670b0849f1f0031d5776d5a5ef83a9c03884691a38" exitCode=60 Nov 03 14:00:03.080551 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:03.080442 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895240-5qpqx" event={"ID":"c5fa16b1-200a-44b7-a5bf-4fa8305c7916","Type":"ContainerDied","Data":"388285ba849cbc136b0fc3670b0849f1f0031d5776d5a5ef83a9c03884691a38"} Nov 03 14:00:03.080738 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:03.080720 5979 scope.go:122] "RemoveContainer" containerID="388285ba849cbc136b0fc3670b0849f1f0031d5776d5a5ef83a9c03884691a38" Nov 03 14:00:03.082188 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:03.082168 5979 generic.go:356] "Generic (PLEG): container finished" podID="61dbd591-8f07-493d-8a27-9475e5fc84a8" containerID="f7fe8ecafe7bbe8e6766fb363f4569b4f66a546e313ee3fca493d63410ea1a9c" exitCode=60 Nov 03 14:00:03.082313 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:03.082256 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895240-xwhnj" event={"ID":"61dbd591-8f07-493d-8a27-9475e5fc84a8","Type":"ContainerDied","Data":"f7fe8ecafe7bbe8e6766fb363f4569b4f66a546e313ee3fca493d63410ea1a9c"} Nov 03 14:00:03.082489 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:03.082430 5979 scope.go:122] "RemoveContainer" containerID="f7fe8ecafe7bbe8e6766fb363f4569b4f66a546e313ee3fca493d63410ea1a9c" Nov 03 14:00:03.084156 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:03.084075 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6" event={"ID":"5ab327a8-7012-4504-9557-6dd31ed7a365","Type":"ContainerStarted","Data":"b4f4f426888318cccd58a37b998208567c7222a935db9d014cfea1cb15ebf6bf"} Nov 03 14:00:03.084156 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:03.084115 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6" event={"ID":"5ab327a8-7012-4504-9557-6dd31ed7a365","Type":"ContainerStarted","Data":"9d4abc6a4bdc7c7267a375e4d83d4e7d86ef70ae32a433c643dbf3e15e59c639"} Nov 03 14:00:03.085784 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:03.085763 5979 generic.go:356] "Generic (PLEG): container finished" podID="6cd99276-aae5-4f57-bb7b-fde6e6821f20" containerID="6f91d2233f9bb4bd551b63128ef202793f96c6bc1bfa90daf966a16a6ded1af8" exitCode=60 Nov 03 14:00:03.085848 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:03.085833 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895240-p9n95" event={"ID":"6cd99276-aae5-4f57-bb7b-fde6e6821f20","Type":"ContainerDied","Data":"6f91d2233f9bb4bd551b63128ef202793f96c6bc1bfa90daf966a16a6ded1af8"} Nov 03 14:00:03.086016 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:03.085999 5979 scope.go:122] "RemoveContainer" containerID="6f91d2233f9bb4bd551b63128ef202793f96c6bc1bfa90daf966a16a6ded1af8" Nov 03 14:00:03.087523 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:03.087503 5979 generic.go:356] "Generic (PLEG): container finished" podID="582acff3-15b0-4430-9ff2-25188540fcab" containerID="160ed787f8126d40357e1625c684e0659863ff7665c747bc0f4922f5a75c504a" exitCode=60 Nov 03 14:00:03.087630 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:03.087589 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895240-2wbbp" event={"ID":"582acff3-15b0-4430-9ff2-25188540fcab","Type":"ContainerDied","Data":"160ed787f8126d40357e1625c684e0659863ff7665c747bc0f4922f5a75c504a"} Nov 03 14:00:03.087743 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:03.087727 5979 scope.go:122] "RemoveContainer" containerID="160ed787f8126d40357e1625c684e0659863ff7665c747bc0f4922f5a75c504a" Nov 03 14:00:03.089442 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:03.089397 5979 generic.go:356] "Generic (PLEG): container finished" podID="7c1c368a-b074-4c74-afe6-77fc73e982bd" containerID="eb73c14be3f82f51b7e726eb41a44e698e7c7a15ab00a977f8df0a17dd23a787" exitCode=60 Nov 03 14:00:03.089442 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:03.089427 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895240-p7lk8" event={"ID":"7c1c368a-b074-4c74-afe6-77fc73e982bd","Type":"ContainerDied","Data":"eb73c14be3f82f51b7e726eb41a44e698e7c7a15ab00a977f8df0a17dd23a787"} Nov 03 14:00:03.089628 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:03.089496 5979 scope.go:122] "RemoveContainer" containerID="cda24e45e9736840e7010cf5a326d61b113f5fed4b7c7cb84fcd50ea05b64a3e" Nov 03 14:00:03.089752 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:03.089732 5979 scope.go:122] "RemoveContainer" containerID="eb73c14be3f82f51b7e726eb41a44e698e7c7a15ab00a977f8df0a17dd23a787" Nov 03 14:00:03.089981 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:00:03.089956 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895240-p7lk8_odh-ai-gateway-infra(7c1c368a-b074-4c74-afe6-77fc73e982bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895240-p7lk8" podUID="7c1c368a-b074-4c74-afe6-77fc73e982bd" Nov 03 14:00:03.090073 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:03.090013 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6" Nov 03 14:00:03.091538 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:03.091514 5979 generic.go:356] "Generic (PLEG): container finished" podID="33a20f02-e3b2-4f59-a01f-68b3d837c090" containerID="ac8731bca64522c55db2ef5925597f2d0f4d4e21d3eac1b98bb0b48ba311a634" exitCode=60 Nov 03 14:00:03.091652 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:03.091569 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895240-x7b9t" event={"ID":"33a20f02-e3b2-4f59-a01f-68b3d837c090","Type":"ContainerDied","Data":"ac8731bca64522c55db2ef5925597f2d0f4d4e21d3eac1b98bb0b48ba311a634"} Nov 03 14:00:03.091894 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:03.091873 5979 scope.go:122] "RemoveContainer" containerID="ac8731bca64522c55db2ef5925597f2d0f4d4e21d3eac1b98bb0b48ba311a634" Nov 03 14:00:03.092156 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:00:03.092114 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895240-x7b9t_odh-ai-gateway-infra(33a20f02-e3b2-4f59-a01f-68b3d837c090)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895240-x7b9t" podUID="33a20f02-e3b2-4f59-a01f-68b3d837c090" Nov 03 14:00:03.096758 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:03.096734 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6" Nov 03 14:00:03.158373 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:03.158351 5979 scope.go:122] "RemoveContainer" containerID="e2cdc8d717d593e43412c43667518c45384431606d026283118284b6bfd5f3fe" Nov 03 14:00:03.844946 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:03.844878 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6" podStartSLOduration=2.844857566 podStartE2EDuration="2.844857566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:00:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:00:03.843226259 +0000 UTC m=+3285.853914500" watchObservedRunningTime="2026-11-03 14:00:03.844857566 +0000 UTC m=+3285.855545807" Nov 03 14:00:04.110576 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:04.110547 5979 scope.go:122] "RemoveContainer" containerID="eb73c14be3f82f51b7e726eb41a44e698e7c7a15ab00a977f8df0a17dd23a787" Nov 03 14:00:04.111029 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:00:04.110921 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895240-p7lk8_odh-ai-gateway-infra(7c1c368a-b074-4c74-afe6-77fc73e982bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895240-p7lk8" podUID="7c1c368a-b074-4c74-afe6-77fc73e982bd" Nov 03 14:00:04.112249 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:04.112225 5979 scope.go:122] "RemoveContainer" containerID="ac8731bca64522c55db2ef5925597f2d0f4d4e21d3eac1b98bb0b48ba311a634" Nov 03 14:00:04.112512 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:00:04.112484 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895240-x7b9t_odh-ai-gateway-infra(33a20f02-e3b2-4f59-a01f-68b3d837c090)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895240-x7b9t" podUID="33a20f02-e3b2-4f59-a01f-68b3d837c090" Nov 03 14:00:04.113968 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:04.113946 5979 generic.go:356] "Generic (PLEG): container finished" podID="c5fa16b1-200a-44b7-a5bf-4fa8305c7916" containerID="da4521631843c3439b0517fec85761e365b62ce4d266c0e114707fbef5cf3a0d" exitCode=60 Nov 03 14:00:04.114341 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:04.114321 5979 scope.go:122] "RemoveContainer" containerID="da4521631843c3439b0517fec85761e365b62ce4d266c0e114707fbef5cf3a0d" Nov 03 14:00:04.114609 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:00:04.114587 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895240-5qpqx_odh-ai-gateway-infra(c5fa16b1-200a-44b7-a5bf-4fa8305c7916)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895240-5qpqx" podUID="c5fa16b1-200a-44b7-a5bf-4fa8305c7916" Nov 03 14:00:04.116195 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:04.116173 5979 generic.go:356] "Generic (PLEG): container finished" podID="61dbd591-8f07-493d-8a27-9475e5fc84a8" containerID="a775edd4e063642d5f9e48c6d679b23b00d08868e68eb5cb8b3b66b2361a6f97" exitCode=60 Nov 03 14:00:04.116566 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:04.116548 5979 scope.go:122] "RemoveContainer" containerID="a775edd4e063642d5f9e48c6d679b23b00d08868e68eb5cb8b3b66b2361a6f97" Nov 03 14:00:04.116796 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:00:04.116772 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895240-xwhnj_odh-ai-gateway-infra(61dbd591-8f07-493d-8a27-9475e5fc84a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895240-xwhnj" podUID="61dbd591-8f07-493d-8a27-9475e5fc84a8" Nov 03 14:00:04.117625 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:04.117603 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895240-5qpqx" event={"ID":"c5fa16b1-200a-44b7-a5bf-4fa8305c7916","Type":"ContainerDied","Data":"da4521631843c3439b0517fec85761e365b62ce4d266c0e114707fbef5cf3a0d"} Nov 03 14:00:04.117707 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:04.117647 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895240-xwhnj" event={"ID":"61dbd591-8f07-493d-8a27-9475e5fc84a8","Type":"ContainerDied","Data":"a775edd4e063642d5f9e48c6d679b23b00d08868e68eb5cb8b3b66b2361a6f97"} Nov 03 14:00:04.117707 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:04.117680 5979 scope.go:122] "RemoveContainer" containerID="388285ba849cbc136b0fc3670b0849f1f0031d5776d5a5ef83a9c03884691a38" Nov 03 14:00:04.118247 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:04.118218 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895240-p9n95" event={"ID":"6cd99276-aae5-4f57-bb7b-fde6e6821f20","Type":"ContainerStarted","Data":"f35bb1f71d0e163ce92a51082bead4c2d19f6128d920a91f79ef6d4afe7b09a8"} Nov 03 14:00:04.120735 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:04.120705 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895240-2wbbp" event={"ID":"582acff3-15b0-4430-9ff2-25188540fcab","Type":"ContainerStarted","Data":"fa0628e6b8fef97f0d06062a9dd18f342b77aedf5944fecad23092875894a2fd"} Nov 03 14:00:04.120846 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:04.120828 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6" Nov 03 14:00:04.121753 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:04.121737 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6" Nov 03 14:00:04.133296 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:04.133278 5979 scope.go:122] "RemoveContainer" containerID="f7fe8ecafe7bbe8e6766fb363f4569b4f66a546e313ee3fca493d63410ea1a9c" Nov 03 14:00:05.130522 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:05.130462 5979 scope.go:122] "RemoveContainer" containerID="a775edd4e063642d5f9e48c6d679b23b00d08868e68eb5cb8b3b66b2361a6f97" Nov 03 14:00:05.130970 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:00:05.130821 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895240-xwhnj_odh-ai-gateway-infra(61dbd591-8f07-493d-8a27-9475e5fc84a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895240-xwhnj" podUID="61dbd591-8f07-493d-8a27-9475e5fc84a8" Nov 03 14:00:05.132138 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:05.132113 5979 generic.go:356] "Generic (PLEG): container finished" podID="6cd99276-aae5-4f57-bb7b-fde6e6821f20" containerID="f35bb1f71d0e163ce92a51082bead4c2d19f6128d920a91f79ef6d4afe7b09a8" exitCode=60 Nov 03 14:00:05.132274 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:05.132197 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895240-p9n95" event={"ID":"6cd99276-aae5-4f57-bb7b-fde6e6821f20","Type":"ContainerDied","Data":"f35bb1f71d0e163ce92a51082bead4c2d19f6128d920a91f79ef6d4afe7b09a8"} Nov 03 14:00:05.132274 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:05.132268 5979 scope.go:122] "RemoveContainer" containerID="6f91d2233f9bb4bd551b63128ef202793f96c6bc1bfa90daf966a16a6ded1af8" Nov 03 14:00:05.132421 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:05.132403 5979 scope.go:122] "RemoveContainer" containerID="f35bb1f71d0e163ce92a51082bead4c2d19f6128d920a91f79ef6d4afe7b09a8" Nov 03 14:00:05.132741 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:00:05.132718 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895240-p9n95_odh-ai-gateway-infra(6cd99276-aae5-4f57-bb7b-fde6e6821f20)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895240-p9n95" podUID="6cd99276-aae5-4f57-bb7b-fde6e6821f20" Nov 03 14:00:05.134120 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:05.134098 5979 generic.go:356] "Generic (PLEG): container finished" podID="582acff3-15b0-4430-9ff2-25188540fcab" containerID="fa0628e6b8fef97f0d06062a9dd18f342b77aedf5944fecad23092875894a2fd" exitCode=60 Nov 03 14:00:05.134221 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:05.134173 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895240-2wbbp" event={"ID":"582acff3-15b0-4430-9ff2-25188540fcab","Type":"ContainerDied","Data":"fa0628e6b8fef97f0d06062a9dd18f342b77aedf5944fecad23092875894a2fd"} Nov 03 14:00:05.134333 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:05.134317 5979 scope.go:122] "RemoveContainer" containerID="fa0628e6b8fef97f0d06062a9dd18f342b77aedf5944fecad23092875894a2fd" Nov 03 14:00:05.134540 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:00:05.134523 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895240-2wbbp_odh-ai-gateway-infra(582acff3-15b0-4430-9ff2-25188540fcab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895240-2wbbp" podUID="582acff3-15b0-4430-9ff2-25188540fcab" Nov 03 14:00:05.135862 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:05.135844 5979 scope.go:122] "RemoveContainer" containerID="da4521631843c3439b0517fec85761e365b62ce4d266c0e114707fbef5cf3a0d" Nov 03 14:00:05.136030 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:00:05.136013 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895240-5qpqx_odh-ai-gateway-infra(c5fa16b1-200a-44b7-a5bf-4fa8305c7916)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895240-5qpqx" podUID="c5fa16b1-200a-44b7-a5bf-4fa8305c7916" Nov 03 14:00:05.145225 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:05.145197 5979 scope.go:122] "RemoveContainer" containerID="160ed787f8126d40357e1625c684e0659863ff7665c747bc0f4922f5a75c504a" Nov 03 14:00:06.143812 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:06.143768 5979 scope.go:122] "RemoveContainer" containerID="f35bb1f71d0e163ce92a51082bead4c2d19f6128d920a91f79ef6d4afe7b09a8" Nov 03 14:00:06.144271 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:00:06.144140 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895240-p9n95_odh-ai-gateway-infra(6cd99276-aae5-4f57-bb7b-fde6e6821f20)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895240-p9n95" podUID="6cd99276-aae5-4f57-bb7b-fde6e6821f20" Nov 03 14:00:06.145422 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:06.145398 5979 scope.go:122] "RemoveContainer" containerID="fa0628e6b8fef97f0d06062a9dd18f342b77aedf5944fecad23092875894a2fd" Nov 03 14:00:06.145693 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:00:06.145665 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895240-2wbbp_odh-ai-gateway-infra(582acff3-15b0-4430-9ff2-25188540fcab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895240-2wbbp" podUID="582acff3-15b0-4430-9ff2-25188540fcab" Nov 03 14:00:07.239397 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:00:07.239356 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:00:10.679818 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:10.678249 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c4e398-5b4997669f-bxk4r"] Nov 03 14:00:10.702986 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:10.693997 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c4e398-5b4997669f-bxk4r" Nov 03 14:00:10.728262 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:10.728223 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c4e398-5b4997669f-bxk4r"] Nov 03 14:00:10.770378 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:10.770349 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-c4e398-dockercfg-77rnl\"" Nov 03 14:00:10.770552 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:10.770376 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-c4e398\"" Nov 03 14:00:10.830911 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:10.830872 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84b8df5b-8fd5-4de1-819f-8b543a53ebf2-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-c4e398-5b4997669f-bxk4r\" (UID: \"84b8df5b-8fd5-4de1-819f-8b543a53ebf2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c4e398-5b4997669f-bxk4r" Nov 03 14:00:10.831290 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:10.831269 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfrfq\" (UniqueName: \"kubernetes.io/projected/84b8df5b-8fd5-4de1-819f-8b543a53ebf2-kube-api-access-bfrfq\") pod \"payload-processing-e2e-extmod-a-c4e398-5b4997669f-bxk4r\" (UID: \"84b8df5b-8fd5-4de1-819f-8b543a53ebf2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c4e398-5b4997669f-bxk4r" Nov 03 14:00:10.933057 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:10.932974 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84b8df5b-8fd5-4de1-819f-8b543a53ebf2-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-c4e398-5b4997669f-bxk4r\" (UID: \"84b8df5b-8fd5-4de1-819f-8b543a53ebf2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c4e398-5b4997669f-bxk4r" Nov 03 14:00:10.933218 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:10.933082 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfrfq\" (UniqueName: \"kubernetes.io/projected/84b8df5b-8fd5-4de1-819f-8b543a53ebf2-kube-api-access-bfrfq\") pod \"payload-processing-e2e-extmod-a-c4e398-5b4997669f-bxk4r\" (UID: \"84b8df5b-8fd5-4de1-819f-8b543a53ebf2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c4e398-5b4997669f-bxk4r" Nov 03 14:00:10.933722 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:10.933697 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84b8df5b-8fd5-4de1-819f-8b543a53ebf2-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-c4e398-5b4997669f-bxk4r\" (UID: \"84b8df5b-8fd5-4de1-819f-8b543a53ebf2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c4e398-5b4997669f-bxk4r" Nov 03 14:00:10.988060 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:10.988031 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfrfq\" (UniqueName: \"kubernetes.io/projected/84b8df5b-8fd5-4de1-819f-8b543a53ebf2-kube-api-access-bfrfq\") pod \"payload-processing-e2e-extmod-a-c4e398-5b4997669f-bxk4r\" (UID: \"84b8df5b-8fd5-4de1-819f-8b543a53ebf2\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c4e398-5b4997669f-bxk4r" Nov 03 14:00:11.031596 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:11.031564 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c4e398-5b4997669f-bxk4r" Nov 03 14:00:11.405434 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:11.405400 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c4e398-5b4997669f-bxk4r"] Nov 03 14:00:11.411666 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:00:11.411641 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84b8df5b_8fd5_4de1_819f_8b543a53ebf2.slice/crio-13bb1a495b7b5e70dbf5d22c01fd44ad31d625dfefcc9ad4cc0db38c560ceaad WatchSource:0}: Error finding container 13bb1a495b7b5e70dbf5d22c01fd44ad31d625dfefcc9ad4cc0db38c560ceaad: Status 404 returned error can't find the container with id 13bb1a495b7b5e70dbf5d22c01fd44ad31d625dfefcc9ad4cc0db38c560ceaad Nov 03 14:00:12.185430 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:12.185381 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c4e398-5b4997669f-bxk4r" event={"ID":"84b8df5b-8fd5-4de1-819f-8b543a53ebf2","Type":"ContainerStarted","Data":"f61cd932758e41bbeac4c5ea3ac5fba52bca7eeb08f82504cf20314cfb9d3856"} Nov 03 14:00:12.185864 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:12.185449 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c4e398-5b4997669f-bxk4r" event={"ID":"84b8df5b-8fd5-4de1-819f-8b543a53ebf2","Type":"ContainerStarted","Data":"13bb1a495b7b5e70dbf5d22c01fd44ad31d625dfefcc9ad4cc0db38c560ceaad"} Nov 03 14:00:12.185864 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:12.185497 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-c4e398-5b4997669f-bxk4r" Nov 03 14:00:12.296372 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:12.296315 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-c4e398-5b4997669f-bxk4r" podStartSLOduration=2.296300864 podStartE2EDuration="2.296300864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:00:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:00:12.293788939 +0000 UTC m=+3294.304477175" watchObservedRunningTime="2026-11-03 14:00:12.296300864 +0000 UTC m=+3294.306989114" Nov 03 14:00:15.165495 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:15.145712 5979 scope.go:122] "RemoveContainer" containerID="a775edd4e063642d5f9e48c6d679b23b00d08868e68eb5cb8b3b66b2361a6f97" Nov 03 14:00:15.166656 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:15.166628 5979 scope.go:122] "RemoveContainer" containerID="da4521631843c3439b0517fec85761e365b62ce4d266c0e114707fbef5cf3a0d" Nov 03 14:00:16.108218 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:16.108182 5979 scope.go:122] "RemoveContainer" containerID="eb73c14be3f82f51b7e726eb41a44e698e7c7a15ab00a977f8df0a17dd23a787" Nov 03 14:00:16.108436 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:16.108327 5979 scope.go:122] "RemoveContainer" containerID="fa0628e6b8fef97f0d06062a9dd18f342b77aedf5944fecad23092875894a2fd" Nov 03 14:00:16.108547 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:16.108459 5979 scope.go:122] "RemoveContainer" containerID="ac8731bca64522c55db2ef5925597f2d0f4d4e21d3eac1b98bb0b48ba311a634" Nov 03 14:00:16.215777 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:16.215745 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895240-5qpqx" event={"ID":"c5fa16b1-200a-44b7-a5bf-4fa8305c7916","Type":"ContainerStarted","Data":"57d410c3a5ec48eae00fb4b661c331e2ab27209b5a0a616ce96366d5df1ae6a0"} Nov 03 14:00:16.217729 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:16.217701 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895240-xwhnj" event={"ID":"61dbd591-8f07-493d-8a27-9475e5fc84a8","Type":"ContainerStarted","Data":"c9c59520dd55478c792e637ce307bc39c5d4f6afeb7aba9df1d02073c7c45d64"} Nov 03 14:00:17.097677 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:00:17.097642 5979 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33a20f02_e3b2_4f59_a01f_68b3d837c090.slice/crio-a07401a4a53e790982340113888e3fb0972aad58f23725f688681e80bcee229f.scope\": RecentStats: unable to find data in memory cache]" Nov 03 14:00:17.227842 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:17.227813 5979 generic.go:356] "Generic (PLEG): container finished" podID="61dbd591-8f07-493d-8a27-9475e5fc84a8" containerID="c9c59520dd55478c792e637ce307bc39c5d4f6afeb7aba9df1d02073c7c45d64" exitCode=60 Nov 03 14:00:17.228214 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:17.227883 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895240-xwhnj" event={"ID":"61dbd591-8f07-493d-8a27-9475e5fc84a8","Type":"ContainerDied","Data":"c9c59520dd55478c792e637ce307bc39c5d4f6afeb7aba9df1d02073c7c45d64"} Nov 03 14:00:17.228214 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:17.227944 5979 scope.go:122] "RemoveContainer" containerID="a775edd4e063642d5f9e48c6d679b23b00d08868e68eb5cb8b3b66b2361a6f97" Nov 03 14:00:17.228367 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:17.228329 5979 scope.go:122] "RemoveContainer" containerID="c9c59520dd55478c792e637ce307bc39c5d4f6afeb7aba9df1d02073c7c45d64" Nov 03 14:00:17.228728 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:00:17.228691 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895240-xwhnj_odh-ai-gateway-infra(61dbd591-8f07-493d-8a27-9475e5fc84a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895240-xwhnj" podUID="61dbd591-8f07-493d-8a27-9475e5fc84a8" Nov 03 14:00:17.230106 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:17.229894 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895240-2wbbp" event={"ID":"582acff3-15b0-4430-9ff2-25188540fcab","Type":"ContainerStarted","Data":"f0571d54e4b2632f5628a401615a1dc28630e8995dde3838bde6b77b9a928f40"} Nov 03 14:00:17.231785 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:17.231761 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895240-p7lk8" event={"ID":"7c1c368a-b074-4c74-afe6-77fc73e982bd","Type":"ContainerStarted","Data":"3c826b7e6cc7ac1e440559a1c39264a4db4e4a0624b9a95169fb577db27bd24c"} Nov 03 14:00:17.233546 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:17.233525 5979 generic.go:356] "Generic (PLEG): container finished" podID="33a20f02-e3b2-4f59-a01f-68b3d837c090" containerID="a07401a4a53e790982340113888e3fb0972aad58f23725f688681e80bcee229f" exitCode=60 Nov 03 14:00:17.233641 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:17.233598 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895240-x7b9t" event={"ID":"33a20f02-e3b2-4f59-a01f-68b3d837c090","Type":"ContainerDied","Data":"a07401a4a53e790982340113888e3fb0972aad58f23725f688681e80bcee229f"} Nov 03 14:00:17.233883 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:17.233867 5979 scope.go:122] "RemoveContainer" containerID="a07401a4a53e790982340113888e3fb0972aad58f23725f688681e80bcee229f" Nov 03 14:00:17.234126 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:00:17.234108 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895240-x7b9t_odh-ai-gateway-infra(33a20f02-e3b2-4f59-a01f-68b3d837c090)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895240-x7b9t" podUID="33a20f02-e3b2-4f59-a01f-68b3d837c090" Nov 03 14:00:17.235299 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:17.235278 5979 generic.go:356] "Generic (PLEG): container finished" podID="c5fa16b1-200a-44b7-a5bf-4fa8305c7916" containerID="57d410c3a5ec48eae00fb4b661c331e2ab27209b5a0a616ce96366d5df1ae6a0" exitCode=60 Nov 03 14:00:17.235433 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:17.235318 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895240-5qpqx" event={"ID":"c5fa16b1-200a-44b7-a5bf-4fa8305c7916","Type":"ContainerDied","Data":"57d410c3a5ec48eae00fb4b661c331e2ab27209b5a0a616ce96366d5df1ae6a0"} Nov 03 14:00:17.235663 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:17.235647 5979 scope.go:122] "RemoveContainer" containerID="57d410c3a5ec48eae00fb4b661c331e2ab27209b5a0a616ce96366d5df1ae6a0" Nov 03 14:00:17.235910 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:00:17.235891 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895240-5qpqx_odh-ai-gateway-infra(c5fa16b1-200a-44b7-a5bf-4fa8305c7916)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895240-5qpqx" podUID="c5fa16b1-200a-44b7-a5bf-4fa8305c7916" Nov 03 14:00:17.328976 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:17.328954 5979 scope.go:122] "RemoveContainer" containerID="ac8731bca64522c55db2ef5925597f2d0f4d4e21d3eac1b98bb0b48ba311a634" Nov 03 14:00:17.334514 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:17.334464 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895240-5qpqx"] Nov 03 14:00:17.344568 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:17.344543 5979 scope.go:122] "RemoveContainer" containerID="da4521631843c3439b0517fec85761e365b62ce4d266c0e114707fbef5cf3a0d" Nov 03 14:00:17.445620 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:17.445584 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895240-xwhnj"] Nov 03 14:00:17.664603 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:17.664552 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895240-p7lk8" podStartSLOduration=17.664537541 podStartE2EDuration="17.664537541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:00:17.662932525 +0000 UTC m=+3299.673620767" watchObservedRunningTime="2026-11-03 14:00:17.664537541 +0000 UTC m=+3299.675225783" Nov 03 14:00:18.112803 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:18.112713 5979 scope.go:122] "RemoveContainer" containerID="f35bb1f71d0e163ce92a51082bead4c2d19f6128d920a91f79ef6d4afe7b09a8" Nov 03 14:00:18.244176 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:18.243441 5979 generic.go:356] "Generic (PLEG): container finished" podID="7c1c368a-b074-4c74-afe6-77fc73e982bd" containerID="3c826b7e6cc7ac1e440559a1c39264a4db4e4a0624b9a95169fb577db27bd24c" exitCode=60 Nov 03 14:00:18.244176 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:18.243499 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895240-p7lk8" event={"ID":"7c1c368a-b074-4c74-afe6-77fc73e982bd","Type":"ContainerDied","Data":"3c826b7e6cc7ac1e440559a1c39264a4db4e4a0624b9a95169fb577db27bd24c"} Nov 03 14:00:18.244176 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:18.243556 5979 scope.go:122] "RemoveContainer" containerID="eb73c14be3f82f51b7e726eb41a44e698e7c7a15ab00a977f8df0a17dd23a787" Nov 03 14:00:18.244176 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:18.243925 5979 scope.go:122] "RemoveContainer" containerID="3c826b7e6cc7ac1e440559a1c39264a4db4e4a0624b9a95169fb577db27bd24c" Nov 03 14:00:18.244756 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:00:18.244247 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895240-p7lk8_odh-ai-gateway-infra(7c1c368a-b074-4c74-afe6-77fc73e982bd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895240-p7lk8" podUID="7c1c368a-b074-4c74-afe6-77fc73e982bd" Nov 03 14:00:18.251239 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:18.251201 5979 generic.go:356] "Generic (PLEG): container finished" podID="582acff3-15b0-4430-9ff2-25188540fcab" containerID="f0571d54e4b2632f5628a401615a1dc28630e8995dde3838bde6b77b9a928f40" exitCode=60 Nov 03 14:00:18.251239 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:18.251241 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895240-2wbbp" event={"ID":"582acff3-15b0-4430-9ff2-25188540fcab","Type":"ContainerDied","Data":"f0571d54e4b2632f5628a401615a1dc28630e8995dde3838bde6b77b9a928f40"} Nov 03 14:00:18.251662 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:18.251644 5979 scope.go:122] "RemoveContainer" containerID="f0571d54e4b2632f5628a401615a1dc28630e8995dde3838bde6b77b9a928f40" Nov 03 14:00:18.252002 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:00:18.251975 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895240-2wbbp_odh-ai-gateway-infra(582acff3-15b0-4430-9ff2-25188540fcab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895240-2wbbp" podUID="582acff3-15b0-4430-9ff2-25188540fcab" Nov 03 14:00:18.259426 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:18.259404 5979 scope.go:122] "RemoveContainer" containerID="fa0628e6b8fef97f0d06062a9dd18f342b77aedf5944fecad23092875894a2fd" Nov 03 14:00:18.420252 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:18.420222 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895240-5qpqx" Nov 03 14:00:18.458601 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:18.458567 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895240-2wbbp"] Nov 03 14:00:18.475951 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:18.475920 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895240-xwhnj" Nov 03 14:00:18.484391 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:18.484357 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5fa16b1-200a-44b7-a5bf-4fa8305c7916-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5fa16b1-200a-44b7-a5bf-4fa8305c7916-maas-api-service-ca\") pod \"c5fa16b1-200a-44b7-a5bf-4fa8305c7916\" (UID: \"c5fa16b1-200a-44b7-a5bf-4fa8305c7916\") " Nov 03 14:00:18.484668 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:18.484641 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5fa16b1-200a-44b7-a5bf-4fa8305c7916-kube-api-access-phr2g\" (UniqueName: \"kubernetes.io/projected/c5fa16b1-200a-44b7-a5bf-4fa8305c7916-kube-api-access-phr2g\") pod \"c5fa16b1-200a-44b7-a5bf-4fa8305c7916\" (UID: \"c5fa16b1-200a-44b7-a5bf-4fa8305c7916\") " Nov 03 14:00:18.484723 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:18.484690 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5fa16b1-200a-44b7-a5bf-4fa8305c7916-maas-api-service-ca" pod "c5fa16b1-200a-44b7-a5bf-4fa8305c7916" (UID: "c5fa16b1-200a-44b7-a5bf-4fa8305c7916"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:00:18.485161 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:18.485136 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5fa16b1-200a-44b7-a5bf-4fa8305c7916-maas-api-service-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:00:18.486987 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:18.486963 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5fa16b1-200a-44b7-a5bf-4fa8305c7916-kube-api-access-phr2g" pod "c5fa16b1-200a-44b7-a5bf-4fa8305c7916" (UID: "c5fa16b1-200a-44b7-a5bf-4fa8305c7916"). InnerVolumeSpecName "kube-api-access-phr2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:00:18.570562 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:18.570526 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895240-x7b9t"] Nov 03 14:00:18.586459 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:18.586419 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61dbd591-8f07-493d-8a27-9475e5fc84a8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61dbd591-8f07-493d-8a27-9475e5fc84a8-maas-api-service-ca\") pod \"61dbd591-8f07-493d-8a27-9475e5fc84a8\" (UID: \"61dbd591-8f07-493d-8a27-9475e5fc84a8\") " Nov 03 14:00:18.586675 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:18.586529 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61dbd591-8f07-493d-8a27-9475e5fc84a8-kube-api-access-nv9b5\" (UniqueName: \"kubernetes.io/projected/61dbd591-8f07-493d-8a27-9475e5fc84a8-kube-api-access-nv9b5\") pod \"61dbd591-8f07-493d-8a27-9475e5fc84a8\" (UID: \"61dbd591-8f07-493d-8a27-9475e5fc84a8\") " Nov 03 14:00:18.586789 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:18.586760 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61dbd591-8f07-493d-8a27-9475e5fc84a8-maas-api-service-ca" pod "61dbd591-8f07-493d-8a27-9475e5fc84a8" (UID: "61dbd591-8f07-493d-8a27-9475e5fc84a8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:00:18.586979 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:18.586963 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phr2g\" (UniqueName: \"kubernetes.io/projected/c5fa16b1-200a-44b7-a5bf-4fa8305c7916-kube-api-access-phr2g\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:00:18.587044 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:18.586981 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61dbd591-8f07-493d-8a27-9475e5fc84a8-maas-api-service-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:00:18.588731 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:18.588702 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61dbd591-8f07-493d-8a27-9475e5fc84a8-kube-api-access-nv9b5" pod "61dbd591-8f07-493d-8a27-9475e5fc84a8" (UID: "61dbd591-8f07-493d-8a27-9475e5fc84a8"). InnerVolumeSpecName "kube-api-access-nv9b5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:00:18.677685 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:18.677648 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895240-p7lk8"] Nov 03 14:00:18.698800 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:18.688062 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv9b5\" (UniqueName: \"kubernetes.io/projected/61dbd591-8f07-493d-8a27-9475e5fc84a8-kube-api-access-nv9b5\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:00:18.734378 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:18.734353 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895240-x7b9t" Nov 03 14:00:18.789300 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:18.789266 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33a20f02-e3b2-4f59-a01f-68b3d837c090-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33a20f02-e3b2-4f59-a01f-68b3d837c090-maas-api-service-ca\") pod \"33a20f02-e3b2-4f59-a01f-68b3d837c090\" (UID: \"33a20f02-e3b2-4f59-a01f-68b3d837c090\") " Nov 03 14:00:18.789544 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:18.789346 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33a20f02-e3b2-4f59-a01f-68b3d837c090-kube-api-access-2tq2w\" (UniqueName: \"kubernetes.io/projected/33a20f02-e3b2-4f59-a01f-68b3d837c090-kube-api-access-2tq2w\") pod \"33a20f02-e3b2-4f59-a01f-68b3d837c090\" (UID: \"33a20f02-e3b2-4f59-a01f-68b3d837c090\") " Nov 03 14:00:18.789751 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:18.789727 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33a20f02-e3b2-4f59-a01f-68b3d837c090-maas-api-service-ca" pod "33a20f02-e3b2-4f59-a01f-68b3d837c090" (UID: "33a20f02-e3b2-4f59-a01f-68b3d837c090"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:00:18.790152 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:18.790128 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33a20f02-e3b2-4f59-a01f-68b3d837c090-maas-api-service-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:00:18.791628 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:18.791604 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33a20f02-e3b2-4f59-a01f-68b3d837c090-kube-api-access-2tq2w" pod "33a20f02-e3b2-4f59-a01f-68b3d837c090" (UID: "33a20f02-e3b2-4f59-a01f-68b3d837c090"). InnerVolumeSpecName "kube-api-access-2tq2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:00:18.891824 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:18.891787 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tq2w\" (UniqueName: \"kubernetes.io/projected/33a20f02-e3b2-4f59-a01f-68b3d837c090-kube-api-access-2tq2w\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:00:19.269661 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:19.269626 5979 generic.go:356] "Generic (PLEG): container finished" podID="6cd99276-aae5-4f57-bb7b-fde6e6821f20" containerID="8fc7b18459a542261055b1283f6d7f2c4590db0a7e8bd180626fe937018b5562" exitCode=60 Nov 03 14:00:19.270170 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:19.269708 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895240-p9n95" event={"ID":"6cd99276-aae5-4f57-bb7b-fde6e6821f20","Type":"ContainerDied","Data":"8fc7b18459a542261055b1283f6d7f2c4590db0a7e8bd180626fe937018b5562"} Nov 03 14:00:19.270170 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:19.269767 5979 scope.go:122] "RemoveContainer" containerID="f35bb1f71d0e163ce92a51082bead4c2d19f6128d920a91f79ef6d4afe7b09a8" Nov 03 14:00:19.270276 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:19.270250 5979 scope.go:122] "RemoveContainer" containerID="8fc7b18459a542261055b1283f6d7f2c4590db0a7e8bd180626fe937018b5562" Nov 03 14:00:19.270645 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:00:19.270620 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895240-p9n95_odh-ai-gateway-infra(6cd99276-aae5-4f57-bb7b-fde6e6821f20)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895240-p9n95" podUID="6cd99276-aae5-4f57-bb7b-fde6e6821f20" Nov 03 14:00:19.275157 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:19.275134 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895240-x7b9t" Nov 03 14:00:19.275255 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:19.275158 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895240-x7b9t" event={"ID":"33a20f02-e3b2-4f59-a01f-68b3d837c090","Type":"ContainerDied","Data":"a69f9f28ef7569c31f4792f7b70d30c70db0d715f4b5e061d9f415697a3f4bca"} Nov 03 14:00:19.276857 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:19.276835 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895240-5qpqx" Nov 03 14:00:19.276976 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:19.276830 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895240-5qpqx" event={"ID":"c5fa16b1-200a-44b7-a5bf-4fa8305c7916","Type":"ContainerDied","Data":"6d7893f4d4a05a8b73737992dec0776566d9fc01dc111aeeb430e6810b26d4fd"} Nov 03 14:00:19.279688 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:19.279665 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895240-xwhnj" Nov 03 14:00:19.279688 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:19.279684 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895240-xwhnj" event={"ID":"61dbd591-8f07-493d-8a27-9475e5fc84a8","Type":"ContainerDied","Data":"e1058f08c8dd0a3266d16c2db21ba27e80e2d96a62b00f78896952584d457d3f"} Nov 03 14:00:19.287486 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:19.287437 5979 scope.go:122] "RemoveContainer" containerID="a07401a4a53e790982340113888e3fb0972aad58f23725f688681e80bcee229f" Nov 03 14:00:19.300391 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:19.300300 5979 scope.go:122] "RemoveContainer" containerID="57d410c3a5ec48eae00fb4b661c331e2ab27209b5a0a616ce96366d5df1ae6a0" Nov 03 14:00:19.310569 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:19.310432 5979 scope.go:122] "RemoveContainer" containerID="c9c59520dd55478c792e637ce307bc39c5d4f6afeb7aba9df1d02073c7c45d64" Nov 03 14:00:19.449977 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:19.449947 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895240-2wbbp" Nov 03 14:00:19.467783 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:19.467755 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895240-p7lk8" Nov 03 14:00:19.499077 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:19.499045 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/582acff3-15b0-4430-9ff2-25188540fcab-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/582acff3-15b0-4430-9ff2-25188540fcab-maas-api-service-ca\") pod \"582acff3-15b0-4430-9ff2-25188540fcab\" (UID: \"582acff3-15b0-4430-9ff2-25188540fcab\") " Nov 03 14:00:19.499260 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:19.499134 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c1c368a-b074-4c74-afe6-77fc73e982bd-kube-api-access-5mp2g\" (UniqueName: \"kubernetes.io/projected/7c1c368a-b074-4c74-afe6-77fc73e982bd-kube-api-access-5mp2g\") pod \"7c1c368a-b074-4c74-afe6-77fc73e982bd\" (UID: \"7c1c368a-b074-4c74-afe6-77fc73e982bd\") " Nov 03 14:00:19.499343 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:19.499274 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/582acff3-15b0-4430-9ff2-25188540fcab-kube-api-access-x6r55\" (UniqueName: \"kubernetes.io/projected/582acff3-15b0-4430-9ff2-25188540fcab-kube-api-access-x6r55\") pod \"582acff3-15b0-4430-9ff2-25188540fcab\" (UID: \"582acff3-15b0-4430-9ff2-25188540fcab\") " Nov 03 14:00:19.499343 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:19.499306 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c1c368a-b074-4c74-afe6-77fc73e982bd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c1c368a-b074-4c74-afe6-77fc73e982bd-maas-api-service-ca\") pod \"7c1c368a-b074-4c74-afe6-77fc73e982bd\" (UID: \"7c1c368a-b074-4c74-afe6-77fc73e982bd\") " Nov 03 14:00:19.499453 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:19.499419 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/582acff3-15b0-4430-9ff2-25188540fcab-maas-api-service-ca" pod "582acff3-15b0-4430-9ff2-25188540fcab" (UID: "582acff3-15b0-4430-9ff2-25188540fcab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:00:19.499778 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:19.499749 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c1c368a-b074-4c74-afe6-77fc73e982bd-maas-api-service-ca" pod "7c1c368a-b074-4c74-afe6-77fc73e982bd" (UID: "7c1c368a-b074-4c74-afe6-77fc73e982bd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:00:19.499910 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:19.499875 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c1c368a-b074-4c74-afe6-77fc73e982bd-maas-api-service-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:00:19.499910 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:19.499891 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/582acff3-15b0-4430-9ff2-25188540fcab-maas-api-service-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:00:19.501313 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:19.501289 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c1c368a-b074-4c74-afe6-77fc73e982bd-kube-api-access-5mp2g" pod "7c1c368a-b074-4c74-afe6-77fc73e982bd" (UID: "7c1c368a-b074-4c74-afe6-77fc73e982bd"). InnerVolumeSpecName "kube-api-access-5mp2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:00:19.501682 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:19.501661 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/582acff3-15b0-4430-9ff2-25188540fcab-kube-api-access-x6r55" pod "582acff3-15b0-4430-9ff2-25188540fcab" (UID: "582acff3-15b0-4430-9ff2-25188540fcab"). InnerVolumeSpecName "kube-api-access-x6r55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:00:19.502038 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:19.502020 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895240-xwhnj"] Nov 03 14:00:19.506721 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:19.506700 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-60261a-29895240-xwhnj"] Nov 03 14:00:19.601321 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:19.601285 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6r55\" (UniqueName: \"kubernetes.io/projected/582acff3-15b0-4430-9ff2-25188540fcab-kube-api-access-x6r55\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:00:19.601321 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:19.601320 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mp2g\" (UniqueName: \"kubernetes.io/projected/7c1c368a-b074-4c74-afe6-77fc73e982bd-kube-api-access-5mp2g\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:00:19.716691 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:19.716601 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895240-x7b9t"] Nov 03 14:00:19.722716 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:19.722682 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895240-x7b9t"] Nov 03 14:00:19.932052 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:19.932016 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895240-5qpqx"] Nov 03 14:00:19.935667 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:19.935639 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5faf5d-29895240-5qpqx"] Nov 03 14:00:20.116343 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:20.116261 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33a20f02-e3b2-4f59-a01f-68b3d837c090" path="/var/lib/kubelet/pods/33a20f02-e3b2-4f59-a01f-68b3d837c090/volumes" Nov 03 14:00:20.116789 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:20.116770 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61dbd591-8f07-493d-8a27-9475e5fc84a8" path="/var/lib/kubelet/pods/61dbd591-8f07-493d-8a27-9475e5fc84a8/volumes" Nov 03 14:00:20.117191 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:20.117177 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5fa16b1-200a-44b7-a5bf-4fa8305c7916" path="/var/lib/kubelet/pods/c5fa16b1-200a-44b7-a5bf-4fa8305c7916/volumes" Nov 03 14:00:20.289497 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:20.289440 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895240-2wbbp" Nov 03 14:00:20.289991 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:20.289441 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895240-2wbbp" event={"ID":"582acff3-15b0-4430-9ff2-25188540fcab","Type":"ContainerDied","Data":"f30b1c75963b6aab0afbaa664ed159ed02cce99d290c1d58695421a1ee4cbb31"} Nov 03 14:00:20.289991 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:20.289585 5979 scope.go:122] "RemoveContainer" containerID="f0571d54e4b2632f5628a401615a1dc28630e8995dde3838bde6b77b9a928f40" Nov 03 14:00:20.290837 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:20.290818 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895240-p7lk8" Nov 03 14:00:20.290914 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:20.290850 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895240-p7lk8" event={"ID":"7c1c368a-b074-4c74-afe6-77fc73e982bd","Type":"ContainerDied","Data":"7aebbc4e8d7f89d3570e381240e7814927a2a649f6a3154e77d2edc34dcdc5fb"} Nov 03 14:00:20.300438 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:20.300422 5979 scope.go:122] "RemoveContainer" containerID="3c826b7e6cc7ac1e440559a1c39264a4db4e4a0624b9a95169fb577db27bd24c" Nov 03 14:00:20.405972 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:20.405938 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895240-p9n95"] Nov 03 14:00:20.451236 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:20.451201 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895240-p7lk8"] Nov 03 14:00:20.454424 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:20.454398 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895240-p7lk8"] Nov 03 14:00:20.555649 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:20.555590 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895240-2wbbp"] Nov 03 14:00:20.570813 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:20.559499 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895240-2wbbp"] Nov 03 14:00:20.570813 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:20.566965 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895240-p9n95" Nov 03 14:00:20.601209 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:20.601170 5979 scope.go:122] "RemoveContainer" containerID="a39ccd5f2a1dd3eb6be9eeb32470e5dfc2ec0a59661f102acd7626dc686212be" Nov 03 14:00:20.613635 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:20.613609 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cd99276-aae5-4f57-bb7b-fde6e6821f20-kube-api-access-4z5s7\" (UniqueName: \"kubernetes.io/projected/6cd99276-aae5-4f57-bb7b-fde6e6821f20-kube-api-access-4z5s7\") pod \"6cd99276-aae5-4f57-bb7b-fde6e6821f20\" (UID: \"6cd99276-aae5-4f57-bb7b-fde6e6821f20\") " Nov 03 14:00:20.613762 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:20.613677 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cd99276-aae5-4f57-bb7b-fde6e6821f20-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cd99276-aae5-4f57-bb7b-fde6e6821f20-maas-api-service-ca\") pod \"6cd99276-aae5-4f57-bb7b-fde6e6821f20\" (UID: \"6cd99276-aae5-4f57-bb7b-fde6e6821f20\") " Nov 03 14:00:20.614119 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:20.614097 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cd99276-aae5-4f57-bb7b-fde6e6821f20-maas-api-service-ca" pod "6cd99276-aae5-4f57-bb7b-fde6e6821f20" (UID: "6cd99276-aae5-4f57-bb7b-fde6e6821f20"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:00:20.614201 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:20.614185 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cd99276-aae5-4f57-bb7b-fde6e6821f20-maas-api-service-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:00:20.615832 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:20.615803 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cd99276-aae5-4f57-bb7b-fde6e6821f20-kube-api-access-4z5s7" pod "6cd99276-aae5-4f57-bb7b-fde6e6821f20" (UID: "6cd99276-aae5-4f57-bb7b-fde6e6821f20"). InnerVolumeSpecName "kube-api-access-4z5s7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:00:20.715579 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:20.715462 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z5s7\" (UniqueName: \"kubernetes.io/projected/6cd99276-aae5-4f57-bb7b-fde6e6821f20-kube-api-access-4z5s7\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:00:21.302438 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:21.302401 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895240-p9n95" event={"ID":"6cd99276-aae5-4f57-bb7b-fde6e6821f20","Type":"ContainerDied","Data":"ea8658630120da80f58f25880f9155c7f4cb8383120c318891c8d0a9438e10c9"} Nov 03 14:00:21.302438 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:21.302441 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895240-p9n95" Nov 03 14:00:21.303061 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:21.302464 5979 scope.go:122] "RemoveContainer" containerID="8fc7b18459a542261055b1283f6d7f2c4590db0a7e8bd180626fe937018b5562" Nov 03 14:00:21.457677 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:21.457642 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895240-p9n95"] Nov 03 14:00:21.461711 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:21.461681 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895240-p9n95"] Nov 03 14:00:22.115580 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:22.115551 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="582acff3-15b0-4430-9ff2-25188540fcab" path="/var/lib/kubelet/pods/582acff3-15b0-4430-9ff2-25188540fcab/volumes" Nov 03 14:00:22.116019 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:22.116005 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cd99276-aae5-4f57-bb7b-fde6e6821f20" path="/var/lib/kubelet/pods/6cd99276-aae5-4f57-bb7b-fde6e6821f20/volumes" Nov 03 14:00:22.116381 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:22.116368 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c1c368a-b074-4c74-afe6-77fc73e982bd" path="/var/lib/kubelet/pods/7c1c368a-b074-4c74-afe6-77fc73e982bd/volumes" Nov 03 14:00:23.194928 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:23.194893 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-c4e398-5b4997669f-bxk4r" Nov 03 14:00:37.234624 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:00:37.234584 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:00:40.588233 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:40.588193 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-921466-openshift-default-5fc5946897-pc6bx"] Nov 03 14:00:40.588750 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:40.588485 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-921466-openshift-default-5fc5946897-pc6bx" podUID="313666bc-2334-42a4-8dfe-ab7054ef13dd" containerName="istio-proxy" containerID="cri-o://de623b561483fa5529aecfc623031856aee6898b8dfc28f5f01f8c5e04470973" gracePeriod=30 Nov 03 14:00:42.656304 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:42.656264 5979 patch_prober.go:29] interesting pod/e2e-noten-921466-openshift-default-5fc5946897-pc6bx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 14:00:42.656745 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:42.656320 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-921466-openshift-default-5fc5946897-pc6bx" podUID="313666bc-2334-42a4-8dfe-ab7054ef13dd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 14:00:44.545718 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:44.545669 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:00:45.922622 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:45.922595 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-921466-openshift-default-5fc5946897-pc6bx" Nov 03 14:00:45.937726 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:45.937691 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/313666bc-2334-42a4-8dfe-ab7054ef13dd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/313666bc-2334-42a4-8dfe-ab7054ef13dd-istio-envoy\") pod \"313666bc-2334-42a4-8dfe-ab7054ef13dd\" (UID: \"313666bc-2334-42a4-8dfe-ab7054ef13dd\") " Nov 03 14:00:45.937922 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:45.937738 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/313666bc-2334-42a4-8dfe-ab7054ef13dd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/313666bc-2334-42a4-8dfe-ab7054ef13dd-workload-socket\") pod \"313666bc-2334-42a4-8dfe-ab7054ef13dd\" (UID: \"313666bc-2334-42a4-8dfe-ab7054ef13dd\") " Nov 03 14:00:45.937922 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:45.937760 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/313666bc-2334-42a4-8dfe-ab7054ef13dd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/313666bc-2334-42a4-8dfe-ab7054ef13dd-istiod-ca-cert\") pod \"313666bc-2334-42a4-8dfe-ab7054ef13dd\" (UID: \"313666bc-2334-42a4-8dfe-ab7054ef13dd\") " Nov 03 14:00:45.937922 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:45.937801 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/313666bc-2334-42a4-8dfe-ab7054ef13dd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/313666bc-2334-42a4-8dfe-ab7054ef13dd-istio-podinfo\") pod \"313666bc-2334-42a4-8dfe-ab7054ef13dd\" (UID: \"313666bc-2334-42a4-8dfe-ab7054ef13dd\") " Nov 03 14:00:45.937922 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:45.937827 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/313666bc-2334-42a4-8dfe-ab7054ef13dd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/313666bc-2334-42a4-8dfe-ab7054ef13dd-credential-socket\") pod \"313666bc-2334-42a4-8dfe-ab7054ef13dd\" (UID: \"313666bc-2334-42a4-8dfe-ab7054ef13dd\") " Nov 03 14:00:45.937922 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:45.937910 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/313666bc-2334-42a4-8dfe-ab7054ef13dd-istio-token\" (UniqueName: \"kubernetes.io/projected/313666bc-2334-42a4-8dfe-ab7054ef13dd-istio-token\") pod \"313666bc-2334-42a4-8dfe-ab7054ef13dd\" (UID: \"313666bc-2334-42a4-8dfe-ab7054ef13dd\") " Nov 03 14:00:45.938233 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:45.937938 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/313666bc-2334-42a4-8dfe-ab7054ef13dd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/313666bc-2334-42a4-8dfe-ab7054ef13dd-istio-data\") pod \"313666bc-2334-42a4-8dfe-ab7054ef13dd\" (UID: \"313666bc-2334-42a4-8dfe-ab7054ef13dd\") " Nov 03 14:00:45.938233 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:45.937998 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/313666bc-2334-42a4-8dfe-ab7054ef13dd-kube-api-access-nr97b\" (UniqueName: \"kubernetes.io/projected/313666bc-2334-42a4-8dfe-ab7054ef13dd-kube-api-access-nr97b\") pod \"313666bc-2334-42a4-8dfe-ab7054ef13dd\" (UID: \"313666bc-2334-42a4-8dfe-ab7054ef13dd\") " Nov 03 14:00:45.938233 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:45.938054 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/313666bc-2334-42a4-8dfe-ab7054ef13dd-workload-socket" pod "313666bc-2334-42a4-8dfe-ab7054ef13dd" (UID: "313666bc-2334-42a4-8dfe-ab7054ef13dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:00:45.938233 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:45.938072 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/313666bc-2334-42a4-8dfe-ab7054ef13dd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/313666bc-2334-42a4-8dfe-ab7054ef13dd-workload-certs\") pod \"313666bc-2334-42a4-8dfe-ab7054ef13dd\" (UID: \"313666bc-2334-42a4-8dfe-ab7054ef13dd\") " Nov 03 14:00:45.938447 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:45.938275 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/313666bc-2334-42a4-8dfe-ab7054ef13dd-credential-socket" pod "313666bc-2334-42a4-8dfe-ab7054ef13dd" (UID: "313666bc-2334-42a4-8dfe-ab7054ef13dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:00:45.938447 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:45.938320 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/313666bc-2334-42a4-8dfe-ab7054ef13dd-workload-certs" pod "313666bc-2334-42a4-8dfe-ab7054ef13dd" (UID: "313666bc-2334-42a4-8dfe-ab7054ef13dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:00:45.938447 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:45.938332 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/313666bc-2334-42a4-8dfe-ab7054ef13dd-istiod-ca-cert" pod "313666bc-2334-42a4-8dfe-ab7054ef13dd" (UID: "313666bc-2334-42a4-8dfe-ab7054ef13dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:00:45.938652 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:45.938493 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/313666bc-2334-42a4-8dfe-ab7054ef13dd-istio-data" pod "313666bc-2334-42a4-8dfe-ab7054ef13dd" (UID: "313666bc-2334-42a4-8dfe-ab7054ef13dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:00:45.938722 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:45.938689 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/313666bc-2334-42a4-8dfe-ab7054ef13dd-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:00:45.938722 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:45.938710 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/313666bc-2334-42a4-8dfe-ab7054ef13dd-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:00:45.938834 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:45.938724 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/313666bc-2334-42a4-8dfe-ab7054ef13dd-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:00:45.938834 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:45.938737 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/313666bc-2334-42a4-8dfe-ab7054ef13dd-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:00:45.938834 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:45.938750 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/313666bc-2334-42a4-8dfe-ab7054ef13dd-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:00:45.940011 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:45.939987 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/313666bc-2334-42a4-8dfe-ab7054ef13dd-istio-envoy" pod "313666bc-2334-42a4-8dfe-ab7054ef13dd" (UID: "313666bc-2334-42a4-8dfe-ab7054ef13dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:00:45.940152 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:45.940020 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/313666bc-2334-42a4-8dfe-ab7054ef13dd-istio-podinfo" pod "313666bc-2334-42a4-8dfe-ab7054ef13dd" (UID: "313666bc-2334-42a4-8dfe-ab7054ef13dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:00:45.940364 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:45.940340 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/313666bc-2334-42a4-8dfe-ab7054ef13dd-istio-token" pod "313666bc-2334-42a4-8dfe-ab7054ef13dd" (UID: "313666bc-2334-42a4-8dfe-ab7054ef13dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:00:45.940459 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:45.940382 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/313666bc-2334-42a4-8dfe-ab7054ef13dd-kube-api-access-nr97b" pod "313666bc-2334-42a4-8dfe-ab7054ef13dd" (UID: "313666bc-2334-42a4-8dfe-ab7054ef13dd"). InnerVolumeSpecName "kube-api-access-nr97b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:00:46.039740 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:46.039706 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/313666bc-2334-42a4-8dfe-ab7054ef13dd-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:00:46.039740 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:46.039737 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr97b\" (UniqueName: \"kubernetes.io/projected/313666bc-2334-42a4-8dfe-ab7054ef13dd-kube-api-access-nr97b\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:00:46.039740 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:46.039747 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/313666bc-2334-42a4-8dfe-ab7054ef13dd-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:00:46.040029 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:46.039755 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/313666bc-2334-42a4-8dfe-ab7054ef13dd-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:00:46.480589 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:46.480543 5979 generic.go:356] "Generic (PLEG): container finished" podID="313666bc-2334-42a4-8dfe-ab7054ef13dd" containerID="de623b561483fa5529aecfc623031856aee6898b8dfc28f5f01f8c5e04470973" exitCode=0 Nov 03 14:00:46.480795 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:46.480593 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-921466-openshift-default-5fc5946897-pc6bx" Nov 03 14:00:46.480795 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:46.480634 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-921466-openshift-default-5fc5946897-pc6bx" event={"ID":"313666bc-2334-42a4-8dfe-ab7054ef13dd","Type":"ContainerDied","Data":"de623b561483fa5529aecfc623031856aee6898b8dfc28f5f01f8c5e04470973"} Nov 03 14:00:46.480795 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:46.480695 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-921466-openshift-default-5fc5946897-pc6bx" event={"ID":"313666bc-2334-42a4-8dfe-ab7054ef13dd","Type":"ContainerDied","Data":"dca67d2a23e0899d9e8642881b4ad04d4b875d5212585527e47aca2b74934c00"} Nov 03 14:00:46.480795 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:46.480721 5979 scope.go:122] "RemoveContainer" containerID="de623b561483fa5529aecfc623031856aee6898b8dfc28f5f01f8c5e04470973" Nov 03 14:00:46.491136 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:46.491110 5979 scope.go:122] "RemoveContainer" containerID="de623b561483fa5529aecfc623031856aee6898b8dfc28f5f01f8c5e04470973" Nov 03 14:00:46.491426 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:00:46.491406 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de623b561483fa5529aecfc623031856aee6898b8dfc28f5f01f8c5e04470973\": container with ID starting with de623b561483fa5529aecfc623031856aee6898b8dfc28f5f01f8c5e04470973 not found: ID does not exist" containerID="de623b561483fa5529aecfc623031856aee6898b8dfc28f5f01f8c5e04470973" Nov 03 14:00:46.491527 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:46.491435 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de623b561483fa5529aecfc623031856aee6898b8dfc28f5f01f8c5e04470973"} err="failed to get container status \"de623b561483fa5529aecfc623031856aee6898b8dfc28f5f01f8c5e04470973\": rpc error: code = NotFound desc = could not find container \"de623b561483fa5529aecfc623031856aee6898b8dfc28f5f01f8c5e04470973\": container with ID starting with de623b561483fa5529aecfc623031856aee6898b8dfc28f5f01f8c5e04470973 not found: ID does not exist" Nov 03 14:00:46.636609 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:46.636576 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-921466-openshift-default-5fc5946897-pc6bx"] Nov 03 14:00:46.640130 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:46.640102 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-921466-openshift-default-5fc5946897-pc6bx"] Nov 03 14:00:48.116134 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:48.116095 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="313666bc-2334-42a4-8dfe-ab7054ef13dd" path="/var/lib/kubelet/pods/313666bc-2334-42a4-8dfe-ab7054ef13dd/volumes" Nov 03 14:00:59.850146 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:59.850061 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae248d-789b45f9c9-8kbb6"] Nov 03 14:00:59.850745 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:59.850727 5979 cpu_manager.go:414] "containerMap: removing container" podUID="61dbd591-8f07-493d-8a27-9475e5fc84a8" containerName="cleanup" Nov 03 14:00:59.850797 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:59.850746 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61dbd591-8f07-493d-8a27-9475e5fc84a8" containerName="cleanup" Nov 03 14:00:59.850797 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:59.850757 5979 cpu_manager.go:414] "containerMap: removing container" podUID="33a20f02-e3b2-4f59-a01f-68b3d837c090" containerName="cleanup" Nov 03 14:00:59.850797 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:59.850761 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33a20f02-e3b2-4f59-a01f-68b3d837c090" containerName="cleanup" Nov 03 14:00:59.850797 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:59.850778 5979 cpu_manager.go:414] "containerMap: removing container" podUID="c5fa16b1-200a-44b7-a5bf-4fa8305c7916" containerName="cleanup" Nov 03 14:00:59.850797 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:59.850783 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5fa16b1-200a-44b7-a5bf-4fa8305c7916" containerName="cleanup" Nov 03 14:00:59.850797 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:59.850791 5979 cpu_manager.go:414] "containerMap: removing container" podUID="33a20f02-e3b2-4f59-a01f-68b3d837c090" containerName="cleanup" Nov 03 14:00:59.850797 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:59.850795 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33a20f02-e3b2-4f59-a01f-68b3d837c090" containerName="cleanup" Nov 03 14:00:59.851023 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:59.850803 5979 cpu_manager.go:414] "containerMap: removing container" podUID="582acff3-15b0-4430-9ff2-25188540fcab" containerName="cleanup" Nov 03 14:00:59.851023 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:59.850808 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="582acff3-15b0-4430-9ff2-25188540fcab" containerName="cleanup" Nov 03 14:00:59.851023 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:59.850815 5979 cpu_manager.go:414] "containerMap: removing container" podUID="61dbd591-8f07-493d-8a27-9475e5fc84a8" containerName="cleanup" Nov 03 14:00:59.851023 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:59.850820 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61dbd591-8f07-493d-8a27-9475e5fc84a8" containerName="cleanup" Nov 03 14:00:59.851023 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:59.850827 5979 cpu_manager.go:414] "containerMap: removing container" podUID="6cd99276-aae5-4f57-bb7b-fde6e6821f20" containerName="cleanup" Nov 03 14:00:59.851023 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:59.850831 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cd99276-aae5-4f57-bb7b-fde6e6821f20" containerName="cleanup" Nov 03 14:00:59.851023 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:59.850841 5979 cpu_manager.go:414] "containerMap: removing container" podUID="6cd99276-aae5-4f57-bb7b-fde6e6821f20" containerName="cleanup" Nov 03 14:00:59.851023 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:59.850844 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cd99276-aae5-4f57-bb7b-fde6e6821f20" containerName="cleanup" Nov 03 14:00:59.851023 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:59.850854 5979 cpu_manager.go:414] "containerMap: removing container" podUID="7c1c368a-b074-4c74-afe6-77fc73e982bd" containerName="cleanup" Nov 03 14:00:59.851023 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:59.850858 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c1c368a-b074-4c74-afe6-77fc73e982bd" containerName="cleanup" Nov 03 14:00:59.851023 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:59.850864 5979 cpu_manager.go:414] "containerMap: removing container" podUID="c5fa16b1-200a-44b7-a5bf-4fa8305c7916" containerName="cleanup" Nov 03 14:00:59.851023 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:59.850869 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5fa16b1-200a-44b7-a5bf-4fa8305c7916" containerName="cleanup" Nov 03 14:00:59.851023 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:59.850877 5979 cpu_manager.go:414] "containerMap: removing container" podUID="6cd99276-aae5-4f57-bb7b-fde6e6821f20" containerName="cleanup" Nov 03 14:00:59.851023 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:59.850881 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cd99276-aae5-4f57-bb7b-fde6e6821f20" containerName="cleanup" Nov 03 14:00:59.851023 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:59.850892 5979 cpu_manager.go:414] "containerMap: removing container" podUID="313666bc-2334-42a4-8dfe-ab7054ef13dd" containerName="istio-proxy" Nov 03 14:00:59.851023 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:59.850897 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="313666bc-2334-42a4-8dfe-ab7054ef13dd" containerName="istio-proxy" Nov 03 14:00:59.851023 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:59.850903 5979 cpu_manager.go:414] "containerMap: removing container" podUID="7c1c368a-b074-4c74-afe6-77fc73e982bd" containerName="cleanup" Nov 03 14:00:59.851023 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:59.850908 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c1c368a-b074-4c74-afe6-77fc73e982bd" containerName="cleanup" Nov 03 14:00:59.851023 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:59.850915 5979 cpu_manager.go:414] "containerMap: removing container" podUID="7c1c368a-b074-4c74-afe6-77fc73e982bd" containerName="cleanup" Nov 03 14:00:59.851023 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:59.850919 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c1c368a-b074-4c74-afe6-77fc73e982bd" containerName="cleanup" Nov 03 14:00:59.851023 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:59.850932 5979 cpu_manager.go:414] "containerMap: removing container" podUID="582acff3-15b0-4430-9ff2-25188540fcab" containerName="cleanup" Nov 03 14:00:59.851023 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:59.850937 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="582acff3-15b0-4430-9ff2-25188540fcab" containerName="cleanup" Nov 03 14:00:59.851023 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:59.851025 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c1c368a-b074-4c74-afe6-77fc73e982bd" containerName="cleanup" Nov 03 14:00:59.851023 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:59.851034 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="582acff3-15b0-4430-9ff2-25188540fcab" containerName="cleanup" Nov 03 14:00:59.851846 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:59.851041 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="582acff3-15b0-4430-9ff2-25188540fcab" containerName="cleanup" Nov 03 14:00:59.851846 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:59.851049 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="61dbd591-8f07-493d-8a27-9475e5fc84a8" containerName="cleanup" Nov 03 14:00:59.851846 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:59.851055 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="33a20f02-e3b2-4f59-a01f-68b3d837c090" containerName="cleanup" Nov 03 14:00:59.851846 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:59.851062 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="61dbd591-8f07-493d-8a27-9475e5fc84a8" containerName="cleanup" Nov 03 14:00:59.851846 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:59.851067 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="33a20f02-e3b2-4f59-a01f-68b3d837c090" containerName="cleanup" Nov 03 14:00:59.851846 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:59.851073 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5fa16b1-200a-44b7-a5bf-4fa8305c7916" containerName="cleanup" Nov 03 14:00:59.851846 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:59.851078 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="33a20f02-e3b2-4f59-a01f-68b3d837c090" containerName="cleanup" Nov 03 14:00:59.851846 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:59.851084 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="61dbd591-8f07-493d-8a27-9475e5fc84a8" containerName="cleanup" Nov 03 14:00:59.851846 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:59.851091 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cd99276-aae5-4f57-bb7b-fde6e6821f20" containerName="cleanup" Nov 03 14:00:59.851846 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:59.851097 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cd99276-aae5-4f57-bb7b-fde6e6821f20" containerName="cleanup" Nov 03 14:00:59.851846 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:59.851103 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5fa16b1-200a-44b7-a5bf-4fa8305c7916" containerName="cleanup" Nov 03 14:00:59.851846 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:59.851107 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cd99276-aae5-4f57-bb7b-fde6e6821f20" containerName="cleanup" Nov 03 14:00:59.851846 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:59.851114 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="313666bc-2334-42a4-8dfe-ab7054ef13dd" containerName="istio-proxy" Nov 03 14:00:59.851846 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:59.851118 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c1c368a-b074-4c74-afe6-77fc73e982bd" containerName="cleanup" Nov 03 14:00:59.854560 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:59.854538 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae248d-789b45f9c9-8kbb6" Nov 03 14:00:59.903595 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:59.903563 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-ae248d-serving-cert\"" Nov 03 14:00:59.925552 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:59.925515 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efd740a0-6578-40c0-8368-9d8bbeea64f1-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-ae248d-789b45f9c9-8kbb6\" (UID: \"efd740a0-6578-40c0-8368-9d8bbeea64f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae248d-789b45f9c9-8kbb6" Nov 03 14:00:59.925729 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:59.925574 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efd740a0-6578-40c0-8368-9d8bbeea64f1-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ae248d-789b45f9c9-8kbb6\" (UID: \"efd740a0-6578-40c0-8368-9d8bbeea64f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae248d-789b45f9c9-8kbb6" Nov 03 14:00:59.925781 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:59.925754 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efd740a0-6578-40c0-8368-9d8bbeea64f1-metrics-certs\") pod \"maas-api-e2e-extmod-a-ae248d-789b45f9c9-8kbb6\" (UID: \"efd740a0-6578-40c0-8368-9d8bbeea64f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae248d-789b45f9c9-8kbb6" Nov 03 14:00:59.925914 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:59.925891 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z22f\" (UniqueName: \"kubernetes.io/projected/efd740a0-6578-40c0-8368-9d8bbeea64f1-kube-api-access-8z22f\") pod \"maas-api-e2e-extmod-a-ae248d-789b45f9c9-8kbb6\" (UID: \"efd740a0-6578-40c0-8368-9d8bbeea64f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae248d-789b45f9c9-8kbb6" Nov 03 14:00:59.952621 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:00:59.952587 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae248d-789b45f9c9-8kbb6"] Nov 03 14:01:00.026963 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:00.026930 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efd740a0-6578-40c0-8368-9d8bbeea64f1-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-ae248d-789b45f9c9-8kbb6\" (UID: \"efd740a0-6578-40c0-8368-9d8bbeea64f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae248d-789b45f9c9-8kbb6" Nov 03 14:01:00.027146 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:00.026976 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efd740a0-6578-40c0-8368-9d8bbeea64f1-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ae248d-789b45f9c9-8kbb6\" (UID: \"efd740a0-6578-40c0-8368-9d8bbeea64f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae248d-789b45f9c9-8kbb6" Nov 03 14:01:00.027146 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:00.027037 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efd740a0-6578-40c0-8368-9d8bbeea64f1-metrics-certs\") pod \"maas-api-e2e-extmod-a-ae248d-789b45f9c9-8kbb6\" (UID: \"efd740a0-6578-40c0-8368-9d8bbeea64f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae248d-789b45f9c9-8kbb6" Nov 03 14:01:00.027146 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:00.027088 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z22f\" (UniqueName: \"kubernetes.io/projected/efd740a0-6578-40c0-8368-9d8bbeea64f1-kube-api-access-8z22f\") pod \"maas-api-e2e-extmod-a-ae248d-789b45f9c9-8kbb6\" (UID: \"efd740a0-6578-40c0-8368-9d8bbeea64f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae248d-789b45f9c9-8kbb6" Nov 03 14:01:00.027323 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:01:00.027163 5979 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae248d-serving-cert: secret "maas-api-e2e-extmod-a-ae248d-serving-cert" not found Nov 03 14:01:00.027323 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:01:00.027244 5979 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efd740a0-6578-40c0-8368-9d8bbeea64f1-maas-api-tls podName:efd740a0-6578-40c0-8368-9d8bbeea64f1 nodeName:}" failed. No retries permitted until 2026-11-03 14:01:00.527219292 +0000 UTC m=+3342.537907535 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/efd740a0-6578-40c0-8368-9d8bbeea64f1-maas-api-tls") pod "maas-api-e2e-extmod-a-ae248d-789b45f9c9-8kbb6" (UID: "efd740a0-6578-40c0-8368-9d8bbeea64f1") : secret "maas-api-e2e-extmod-a-ae248d-serving-cert" not found Nov 03 14:01:00.027729 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:00.027708 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efd740a0-6578-40c0-8368-9d8bbeea64f1-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-ae248d-789b45f9c9-8kbb6\" (UID: \"efd740a0-6578-40c0-8368-9d8bbeea64f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae248d-789b45f9c9-8kbb6" Nov 03 14:01:00.029461 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:00.029441 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efd740a0-6578-40c0-8368-9d8bbeea64f1-metrics-certs\") pod \"maas-api-e2e-extmod-a-ae248d-789b45f9c9-8kbb6\" (UID: \"efd740a0-6578-40c0-8368-9d8bbeea64f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae248d-789b45f9c9-8kbb6" Nov 03 14:01:00.081840 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:00.081803 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z22f\" (UniqueName: \"kubernetes.io/projected/efd740a0-6578-40c0-8368-9d8bbeea64f1-kube-api-access-8z22f\") pod \"maas-api-e2e-extmod-a-ae248d-789b45f9c9-8kbb6\" (UID: \"efd740a0-6578-40c0-8368-9d8bbeea64f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae248d-789b45f9c9-8kbb6" Nov 03 14:01:00.533358 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:00.533314 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efd740a0-6578-40c0-8368-9d8bbeea64f1-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ae248d-789b45f9c9-8kbb6\" (UID: \"efd740a0-6578-40c0-8368-9d8bbeea64f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae248d-789b45f9c9-8kbb6" Nov 03 14:01:00.535737 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:00.535712 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efd740a0-6578-40c0-8368-9d8bbeea64f1-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ae248d-789b45f9c9-8kbb6\" (UID: \"efd740a0-6578-40c0-8368-9d8bbeea64f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae248d-789b45f9c9-8kbb6" Nov 03 14:01:00.769041 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:00.769004 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae248d-789b45f9c9-8kbb6" Nov 03 14:01:01.142879 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:01.142847 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae248d-789b45f9c9-8kbb6"] Nov 03 14:01:01.150116 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:01:01.150087 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefd740a0_6578_40c0_8368_9d8bbeea64f1.slice/crio-086e884f6bc9c2a87e5547fca58aad8261bece1bc6c4786fad6dff6ae208db2c WatchSource:0}: Error finding container 086e884f6bc9c2a87e5547fca58aad8261bece1bc6c4786fad6dff6ae208db2c: Status 404 returned error can't find the container with id 086e884f6bc9c2a87e5547fca58aad8261bece1bc6c4786fad6dff6ae208db2c Nov 03 14:01:01.151529 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:01.151510 5979 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 14:01:01.585272 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:01.585180 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae248d-789b45f9c9-8kbb6" event={"ID":"efd740a0-6578-40c0-8368-9d8bbeea64f1","Type":"ContainerStarted","Data":"086e884f6bc9c2a87e5547fca58aad8261bece1bc6c4786fad6dff6ae208db2c"} Nov 03 14:01:02.593020 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:02.592931 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae248d-789b45f9c9-8kbb6" event={"ID":"efd740a0-6578-40c0-8368-9d8bbeea64f1","Type":"ContainerStarted","Data":"8ddfb0dc8dc5d40d1b857c9d5a0eac401db0ea981f135a57af0aaead2feecc84"} Nov 03 14:01:02.593396 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:02.593050 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae248d-789b45f9c9-8kbb6" Nov 03 14:01:02.704790 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:02.704731 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae248d-789b45f9c9-8kbb6" podStartSLOduration=2.667134275 podStartE2EDuration="3.704717847s" totalImagesPullingTime="1.037583572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:00:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:01:02.702088563 +0000 UTC m=+3344.712776805" watchObservedRunningTime="2026-11-03 14:01:02.704717847 +0000 UTC m=+3344.715406088" Nov 03 14:01:07.234807 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:01:07.234739 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:01:08.604249 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:08.604223 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae248d-789b45f9c9-8kbb6" Nov 03 14:01:34.896626 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:34.896585 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-58f896-67cbfc54bc-crds5"] Nov 03 14:01:34.897193 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:34.897179 5979 cpu_manager.go:414] "containerMap: removing container" podUID="582acff3-15b0-4430-9ff2-25188540fcab" containerName="cleanup" Nov 03 14:01:34.897256 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:34.897194 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="582acff3-15b0-4430-9ff2-25188540fcab" containerName="cleanup" Nov 03 14:01:34.897256 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:34.897231 5979 cpu_manager.go:414] "containerMap: removing container" podUID="33a20f02-e3b2-4f59-a01f-68b3d837c090" containerName="cleanup" Nov 03 14:01:34.897256 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:34.897236 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33a20f02-e3b2-4f59-a01f-68b3d837c090" containerName="cleanup" Nov 03 14:01:34.897256 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:34.897245 5979 cpu_manager.go:414] "containerMap: removing container" podUID="61dbd591-8f07-493d-8a27-9475e5fc84a8" containerName="cleanup" Nov 03 14:01:34.897256 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:34.897250 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61dbd591-8f07-493d-8a27-9475e5fc84a8" containerName="cleanup" Nov 03 14:01:34.897437 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:34.897263 5979 cpu_manager.go:414] "containerMap: removing container" podUID="c5fa16b1-200a-44b7-a5bf-4fa8305c7916" containerName="cleanup" Nov 03 14:01:34.897437 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:34.897268 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5fa16b1-200a-44b7-a5bf-4fa8305c7916" containerName="cleanup" Nov 03 14:01:34.897437 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:34.897353 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c1c368a-b074-4c74-afe6-77fc73e982bd" containerName="cleanup" Nov 03 14:01:34.897437 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:34.897364 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="582acff3-15b0-4430-9ff2-25188540fcab" containerName="cleanup" Nov 03 14:01:34.897437 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:34.897378 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5fa16b1-200a-44b7-a5bf-4fa8305c7916" containerName="cleanup" Nov 03 14:01:34.901037 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:34.901018 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-58f896-67cbfc54bc-crds5" Nov 03 14:01:34.960901 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:34.960849 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-58f896\"" Nov 03 14:01:34.998705 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:34.998672 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-58f896-dockercfg-h6hpz\"" Nov 03 14:01:35.006157 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:35.006105 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-58f896-67cbfc54bc-crds5"] Nov 03 14:01:35.077747 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:35.077716 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98afcadd-f2c0-44ca-bfbb-33752dc919d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-58f896-67cbfc54bc-crds5\" (UID: \"98afcadd-f2c0-44ca-bfbb-33752dc919d1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-58f896-67cbfc54bc-crds5" Nov 03 14:01:35.077914 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:35.077874 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnmrh\" (UniqueName: \"kubernetes.io/projected/98afcadd-f2c0-44ca-bfbb-33752dc919d1-kube-api-access-qnmrh\") pod \"payload-pre-processing-e2e-extmod-a-58f896-67cbfc54bc-crds5\" (UID: \"98afcadd-f2c0-44ca-bfbb-33752dc919d1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-58f896-67cbfc54bc-crds5" Nov 03 14:01:35.179220 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:35.179133 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnmrh\" (UniqueName: \"kubernetes.io/projected/98afcadd-f2c0-44ca-bfbb-33752dc919d1-kube-api-access-qnmrh\") pod \"payload-pre-processing-e2e-extmod-a-58f896-67cbfc54bc-crds5\" (UID: \"98afcadd-f2c0-44ca-bfbb-33752dc919d1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-58f896-67cbfc54bc-crds5" Nov 03 14:01:35.179600 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:35.179571 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98afcadd-f2c0-44ca-bfbb-33752dc919d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-58f896-67cbfc54bc-crds5\" (UID: \"98afcadd-f2c0-44ca-bfbb-33752dc919d1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-58f896-67cbfc54bc-crds5" Nov 03 14:01:35.180184 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:35.180165 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98afcadd-f2c0-44ca-bfbb-33752dc919d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-58f896-67cbfc54bc-crds5\" (UID: \"98afcadd-f2c0-44ca-bfbb-33752dc919d1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-58f896-67cbfc54bc-crds5" Nov 03 14:01:35.233087 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:35.233056 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnmrh\" (UniqueName: \"kubernetes.io/projected/98afcadd-f2c0-44ca-bfbb-33752dc919d1-kube-api-access-qnmrh\") pod \"payload-pre-processing-e2e-extmod-a-58f896-67cbfc54bc-crds5\" (UID: \"98afcadd-f2c0-44ca-bfbb-33752dc919d1\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-58f896-67cbfc54bc-crds5" Nov 03 14:01:35.514493 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:35.514390 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-58f896-67cbfc54bc-crds5" Nov 03 14:01:35.889310 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:35.889275 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-58f896-67cbfc54bc-crds5"] Nov 03 14:01:35.896297 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:01:35.896267 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98afcadd_f2c0_44ca_bfbb_33752dc919d1.slice/crio-637507142ef0ea69b0529e3305992a919607a2cee17be41abf0e3766b8a9c9fa WatchSource:0}: Error finding container 637507142ef0ea69b0529e3305992a919607a2cee17be41abf0e3766b8a9c9fa: Status 404 returned error can't find the container with id 637507142ef0ea69b0529e3305992a919607a2cee17be41abf0e3766b8a9c9fa Nov 03 14:01:36.847554 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:36.847512 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-58f896-67cbfc54bc-crds5" event={"ID":"98afcadd-f2c0-44ca-bfbb-33752dc919d1","Type":"ContainerStarted","Data":"60e13a353d92c4ae38a012704511d3ad89c24737e926eefac97c9e41d93503ed"} Nov 03 14:01:36.847978 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:36.847577 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-58f896-67cbfc54bc-crds5" event={"ID":"98afcadd-f2c0-44ca-bfbb-33752dc919d1","Type":"ContainerStarted","Data":"637507142ef0ea69b0529e3305992a919607a2cee17be41abf0e3766b8a9c9fa"} Nov 03 14:01:36.847978 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:36.847728 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-58f896-67cbfc54bc-crds5" Nov 03 14:01:36.980544 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:36.980453 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-58f896-67cbfc54bc-crds5" podStartSLOduration=2.980434849 podStartE2EDuration="2.980434849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:01:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:01:36.98012725 +0000 UTC m=+3378.990815500" watchObservedRunningTime="2026-11-03 14:01:36.980434849 +0000 UTC m=+3378.991123092" Nov 03 14:01:37.234865 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:01:37.234822 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:01:39.706817 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:39.706768 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:01:47.856682 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:47.856654 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-58f896-67cbfc54bc-crds5" Nov 03 14:01:49.935350 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:49.935316 5979 generic.go:356] "Generic (PLEG): container finished" podID="e0e4153d-37d8-4233-8744-7515f9878bdf" containerID="8313cbab19ffb9001c91a3a40203b463da8cb6e7b820af64d57a964a311ed62c" exitCode=0 Nov 03 14:01:49.935816 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:49.935392 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44" event={"ID":"e0e4153d-37d8-4233-8744-7515f9878bdf","Type":"ContainerDied","Data":"8313cbab19ffb9001c91a3a40203b463da8cb6e7b820af64d57a964a311ed62c"} Nov 03 14:01:49.935861 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:49.935827 5979 scope.go:122] "RemoveContainer" containerID="8313cbab19ffb9001c91a3a40203b463da8cb6e7b820af64d57a964a311ed62c" Nov 03 14:01:49.937552 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:49.937524 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:01:49.937624 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:49.937591 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:01:49.937664 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:49.937622 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:01:50.984667 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:50.984623 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44" event={"ID":"e0e4153d-37d8-4233-8744-7515f9878bdf","Type":"ContainerStarted","Data":"53b27204ee5be8299e66bef903229b786ab8cc099622126ce25c40863ab33643"} Nov 03 14:01:51.759524 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:51.758871 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44" Nov 03 14:01:51.759524 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:51.758999 5979 patch_prober.go:29] interesting pod/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.36:15021/healthz/ready\": dial tcp 10.134.1.36:15021: connect: connection refused" start-of-body= Nov 03 14:01:51.759524 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:51.759045 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44" podUID="e0e4153d-37d8-4233-8744-7515f9878bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.36:15021/healthz/ready\": dial tcp 10.134.1.36:15021: connect: connection refused" Nov 03 14:01:52.769016 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:52.768966 5979 patch_prober.go:29] interesting pod/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.36:15021/healthz/ready\": dial tcp 10.134.1.36:15021: connect: connection refused" start-of-body= Nov 03 14:01:52.769612 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:52.769038 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44" podUID="e0e4153d-37d8-4233-8744-7515f9878bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.36:15021/healthz/ready\": dial tcp 10.134.1.36:15021: connect: connection refused" Nov 03 14:01:53.763497 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:53.760857 5979 patch_prober.go:29] interesting pod/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.36:15021/healthz/ready\": dial tcp 10.134.1.36:15021: connect: connection refused" start-of-body= Nov 03 14:01:53.763497 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:53.760920 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44" podUID="e0e4153d-37d8-4233-8744-7515f9878bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.36:15021/healthz/ready\": dial tcp 10.134.1.36:15021: connect: connection refused" Nov 03 14:01:54.764496 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:54.758778 5979 patch_prober.go:29] interesting pod/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.36:15021/healthz/ready\": dial tcp 10.134.1.36:15021: connect: connection refused" start-of-body= Nov 03 14:01:54.764496 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:54.758840 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44" podUID="e0e4153d-37d8-4233-8744-7515f9878bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.36:15021/healthz/ready\": dial tcp 10.134.1.36:15021: connect: connection refused" Nov 03 14:01:55.761619 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:55.761582 5979 patch_prober.go:29] interesting pod/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.36:15021/healthz/ready\": dial tcp 10.134.1.36:15021: connect: connection refused" start-of-body= Nov 03 14:01:55.761885 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:55.761861 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44" podUID="e0e4153d-37d8-4233-8744-7515f9878bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.36:15021/healthz/ready\": dial tcp 10.134.1.36:15021: connect: connection refused" Nov 03 14:01:56.762065 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:56.762028 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44" Nov 03 14:01:56.762496 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:56.762289 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44" Nov 03 14:01:56.763063 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:01:56.763043 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44" Nov 03 14:02:07.235916 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:02:07.235881 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:02:20.162628 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:20.162585 5979 patch_prober.go:29] interesting pod/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:02:20.163060 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:20.162643 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6" podUID="5ab327a8-7012-4504-9557-6dd31ed7a365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:02:21.206398 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:21.206362 5979 generic.go:356] "Generic (PLEG): container finished" podID="5ab327a8-7012-4504-9557-6dd31ed7a365" containerID="b4f4f426888318cccd58a37b998208567c7222a935db9d014cfea1cb15ebf6bf" exitCode=0 Nov 03 14:02:21.206839 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:21.206446 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6" event={"ID":"5ab327a8-7012-4504-9557-6dd31ed7a365","Type":"ContainerDied","Data":"b4f4f426888318cccd58a37b998208567c7222a935db9d014cfea1cb15ebf6bf"} Nov 03 14:02:21.206914 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:21.206871 5979 scope.go:122] "RemoveContainer" containerID="b4f4f426888318cccd58a37b998208567c7222a935db9d014cfea1cb15ebf6bf" Nov 03 14:02:21.208556 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:21.208530 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:02:21.208637 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:21.208597 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:02:21.208637 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:21.208629 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:02:22.214601 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:22.214566 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6" event={"ID":"5ab327a8-7012-4504-9557-6dd31ed7a365","Type":"ContainerStarted","Data":"da2ca747ec56fffdd8cfab01245b312d773fc8b360eecd5c63a540e28dc57ade"} Nov 03 14:02:23.091067 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:23.091028 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6" Nov 03 14:02:23.091297 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:23.091274 5979 patch_prober.go:29] interesting pod/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused" start-of-body= Nov 03 14:02:23.091352 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:23.091329 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6" podUID="5ab327a8-7012-4504-9557-6dd31ed7a365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused" Nov 03 14:02:24.090358 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:24.090315 5979 patch_prober.go:29] interesting pod/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused" start-of-body= Nov 03 14:02:24.090813 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:24.090388 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6" podUID="5ab327a8-7012-4504-9557-6dd31ed7a365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused" Nov 03 14:02:25.090522 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:25.090464 5979 patch_prober.go:29] interesting pod/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused" start-of-body= Nov 03 14:02:25.090971 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:25.090553 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6" podUID="5ab327a8-7012-4504-9557-6dd31ed7a365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused" Nov 03 14:02:26.090614 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:26.090569 5979 patch_prober.go:29] interesting pod/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused" start-of-body= Nov 03 14:02:26.091104 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:26.091071 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6" podUID="5ab327a8-7012-4504-9557-6dd31ed7a365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused" Nov 03 14:02:27.090327 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:27.090294 5979 patch_prober.go:29] interesting pod/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused" start-of-body= Nov 03 14:02:27.090539 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:27.090354 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6" podUID="5ab327a8-7012-4504-9557-6dd31ed7a365" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused" Nov 03 14:02:28.095250 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:28.095212 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6" Nov 03 14:02:28.095775 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:28.095569 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6" Nov 03 14:02:28.096144 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:28.096127 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6" Nov 03 14:02:34.443407 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:34.443371 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb"] Nov 03 14:02:34.447052 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:34.447029 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb" Nov 03 14:02:34.496083 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:34.496048 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-6ebe57-openshift-default-dockercfg-hk6f8\"" Nov 03 14:02:34.546613 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:34.546577 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb"] Nov 03 14:02:34.613942 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:34.613904 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ded222c-3a0a-4c3a-bd70-493d97638c99-workload-socket\") pod \"e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb\" (UID: \"9ded222c-3a0a-4c3a-bd70-493d97638c99\") " pod="openshift-ingress/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb" Nov 03 14:02:34.614300 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:34.614271 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ded222c-3a0a-4c3a-bd70-493d97638c99-istio-podinfo\") pod \"e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb\" (UID: \"9ded222c-3a0a-4c3a-bd70-493d97638c99\") " pod="openshift-ingress/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb" Nov 03 14:02:34.614565 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:34.614545 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ded222c-3a0a-4c3a-bd70-493d97638c99-workload-certs\") pod \"e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb\" (UID: \"9ded222c-3a0a-4c3a-bd70-493d97638c99\") " pod="openshift-ingress/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb" Nov 03 14:02:34.614725 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:34.614701 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ded222c-3a0a-4c3a-bd70-493d97638c99-credential-socket\") pod \"e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb\" (UID: \"9ded222c-3a0a-4c3a-bd70-493d97638c99\") " pod="openshift-ingress/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb" Nov 03 14:02:34.614905 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:34.614887 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhsgf\" (UniqueName: \"kubernetes.io/projected/9ded222c-3a0a-4c3a-bd70-493d97638c99-kube-api-access-zhsgf\") pod \"e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb\" (UID: \"9ded222c-3a0a-4c3a-bd70-493d97638c99\") " pod="openshift-ingress/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb" Nov 03 14:02:34.615074 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:34.615056 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ded222c-3a0a-4c3a-bd70-493d97638c99-istio-envoy\") pod \"e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb\" (UID: \"9ded222c-3a0a-4c3a-bd70-493d97638c99\") " pod="openshift-ingress/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb" Nov 03 14:02:34.615237 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:34.615212 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ded222c-3a0a-4c3a-bd70-493d97638c99-istio-token\") pod \"e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb\" (UID: \"9ded222c-3a0a-4c3a-bd70-493d97638c99\") " pod="openshift-ingress/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb" Nov 03 14:02:34.615375 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:34.615305 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ded222c-3a0a-4c3a-bd70-493d97638c99-istiod-ca-cert\") pod \"e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb\" (UID: \"9ded222c-3a0a-4c3a-bd70-493d97638c99\") " pod="openshift-ingress/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb" Nov 03 14:02:34.615375 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:34.615348 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ded222c-3a0a-4c3a-bd70-493d97638c99-istio-data\") pod \"e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb\" (UID: \"9ded222c-3a0a-4c3a-bd70-493d97638c99\") " pod="openshift-ingress/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb" Nov 03 14:02:34.717312 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:34.717222 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ded222c-3a0a-4c3a-bd70-493d97638c99-istio-token\") pod \"e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb\" (UID: \"9ded222c-3a0a-4c3a-bd70-493d97638c99\") " pod="openshift-ingress/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb" Nov 03 14:02:34.717312 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:34.717286 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ded222c-3a0a-4c3a-bd70-493d97638c99-istiod-ca-cert\") pod \"e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb\" (UID: \"9ded222c-3a0a-4c3a-bd70-493d97638c99\") " pod="openshift-ingress/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb" Nov 03 14:02:34.717619 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:34.717321 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ded222c-3a0a-4c3a-bd70-493d97638c99-istio-data\") pod \"e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb\" (UID: \"9ded222c-3a0a-4c3a-bd70-493d97638c99\") " pod="openshift-ingress/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb" Nov 03 14:02:34.717619 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:34.717392 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ded222c-3a0a-4c3a-bd70-493d97638c99-workload-socket\") pod \"e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb\" (UID: \"9ded222c-3a0a-4c3a-bd70-493d97638c99\") " pod="openshift-ingress/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb" Nov 03 14:02:34.717619 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:34.717433 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ded222c-3a0a-4c3a-bd70-493d97638c99-istio-podinfo\") pod \"e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb\" (UID: \"9ded222c-3a0a-4c3a-bd70-493d97638c99\") " pod="openshift-ingress/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb" Nov 03 14:02:34.717619 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:34.717518 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ded222c-3a0a-4c3a-bd70-493d97638c99-workload-certs\") pod \"e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb\" (UID: \"9ded222c-3a0a-4c3a-bd70-493d97638c99\") " pod="openshift-ingress/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb" Nov 03 14:02:34.717880 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:34.717659 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ded222c-3a0a-4c3a-bd70-493d97638c99-credential-socket\") pod \"e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb\" (UID: \"9ded222c-3a0a-4c3a-bd70-493d97638c99\") " pod="openshift-ingress/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb" Nov 03 14:02:34.717880 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:34.717739 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhsgf\" (UniqueName: \"kubernetes.io/projected/9ded222c-3a0a-4c3a-bd70-493d97638c99-kube-api-access-zhsgf\") pod \"e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb\" (UID: \"9ded222c-3a0a-4c3a-bd70-493d97638c99\") " pod="openshift-ingress/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb" Nov 03 14:02:34.717880 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:34.717780 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ded222c-3a0a-4c3a-bd70-493d97638c99-istio-envoy\") pod \"e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb\" (UID: \"9ded222c-3a0a-4c3a-bd70-493d97638c99\") " pod="openshift-ingress/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb" Nov 03 14:02:34.718058 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:34.717917 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ded222c-3a0a-4c3a-bd70-493d97638c99-istio-data\") pod \"e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb\" (UID: \"9ded222c-3a0a-4c3a-bd70-493d97638c99\") " pod="openshift-ingress/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb" Nov 03 14:02:34.718222 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:34.717975 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ded222c-3a0a-4c3a-bd70-493d97638c99-workload-socket\") pod \"e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb\" (UID: \"9ded222c-3a0a-4c3a-bd70-493d97638c99\") " pod="openshift-ingress/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb" Nov 03 14:02:34.718600 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:34.718556 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ded222c-3a0a-4c3a-bd70-493d97638c99-credential-socket\") pod \"e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb\" (UID: \"9ded222c-3a0a-4c3a-bd70-493d97638c99\") " pod="openshift-ingress/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb" Nov 03 14:02:34.718766 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:34.718740 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ded222c-3a0a-4c3a-bd70-493d97638c99-workload-certs\") pod \"e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb\" (UID: \"9ded222c-3a0a-4c3a-bd70-493d97638c99\") " pod="openshift-ingress/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb" Nov 03 14:02:34.718849 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:34.718765 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ded222c-3a0a-4c3a-bd70-493d97638c99-istiod-ca-cert\") pod \"e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb\" (UID: \"9ded222c-3a0a-4c3a-bd70-493d97638c99\") " pod="openshift-ingress/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb" Nov 03 14:02:34.723295 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:34.721149 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ded222c-3a0a-4c3a-bd70-493d97638c99-istio-podinfo\") pod \"e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb\" (UID: \"9ded222c-3a0a-4c3a-bd70-493d97638c99\") " pod="openshift-ingress/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb" Nov 03 14:02:34.732982 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:34.732957 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ded222c-3a0a-4c3a-bd70-493d97638c99-istio-envoy\") pod \"e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb\" (UID: \"9ded222c-3a0a-4c3a-bd70-493d97638c99\") " pod="openshift-ingress/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb" Nov 03 14:02:34.772894 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:34.772857 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ded222c-3a0a-4c3a-bd70-493d97638c99-istio-token\") pod \"e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb\" (UID: \"9ded222c-3a0a-4c3a-bd70-493d97638c99\") " pod="openshift-ingress/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb" Nov 03 14:02:34.773085 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:34.772897 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhsgf\" (UniqueName: \"kubernetes.io/projected/9ded222c-3a0a-4c3a-bd70-493d97638c99-kube-api-access-zhsgf\") pod \"e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb\" (UID: \"9ded222c-3a0a-4c3a-bd70-493d97638c99\") " pod="openshift-ingress/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb" Nov 03 14:02:35.059401 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:35.059314 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb" Nov 03 14:02:35.489127 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:35.489084 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb"] Nov 03 14:02:35.495860 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:02:35.495832 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ded222c_3a0a_4c3a_bd70_493d97638c99.slice/crio-4f32e1c888abee0935c13f663cd247e58be61da799df73f50fd1cead9d413fe3 WatchSource:0}: Error finding container 4f32e1c888abee0935c13f663cd247e58be61da799df73f50fd1cead9d413fe3: Status 404 returned error can't find the container with id 4f32e1c888abee0935c13f663cd247e58be61da799df73f50fd1cead9d413fe3 Nov 03 14:02:35.498610 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:35.498578 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:02:35.498703 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:35.498649 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:02:35.498703 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:35.498683 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:02:36.333839 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:36.333801 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb" event={"ID":"9ded222c-3a0a-4c3a-bd70-493d97638c99","Type":"ContainerStarted","Data":"a0c8bf7f848148181fcfbf15c36f59265eab995cf50b73913c6179ec369dcd41"} Nov 03 14:02:36.334043 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:36.333859 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb" event={"ID":"9ded222c-3a0a-4c3a-bd70-493d97638c99","Type":"ContainerStarted","Data":"4f32e1c888abee0935c13f663cd247e58be61da799df73f50fd1cead9d413fe3"} Nov 03 14:02:37.059983 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:37.059941 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb" Nov 03 14:02:37.066035 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:37.066010 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb" Nov 03 14:02:37.178276 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:37.178223 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb" podStartSLOduration=3.178208512 podStartE2EDuration="3.178208512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:02:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:02:36.537221961 +0000 UTC m=+3438.547910214" watchObservedRunningTime="2026-11-03 14:02:37.178208512 +0000 UTC m=+3439.188896754" Nov 03 14:02:37.235572 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:02:37.235531 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:02:37.340001 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:37.339907 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb" Nov 03 14:02:37.340990 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:37.340972 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb" Nov 03 14:02:39.674993 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:39.674957 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:02:40.169783 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:40.169745 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d5d415-676fcbc795-x4w5c"] Nov 03 14:02:40.173384 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:40.173359 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d5d415-676fcbc795-x4w5c" Nov 03 14:02:40.247683 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:40.247648 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xg9z\" (UniqueName: \"kubernetes.io/projected/bd709159-02e1-4730-bac2-9a0fff9b2c0c-kube-api-access-9xg9z\") pod \"payload-pre-processing-e2e-extmod-b-d5d415-676fcbc795-x4w5c\" (UID: \"bd709159-02e1-4730-bac2-9a0fff9b2c0c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d5d415-676fcbc795-x4w5c" Nov 03 14:02:40.247929 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:40.247895 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd709159-02e1-4730-bac2-9a0fff9b2c0c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-d5d415-676fcbc795-x4w5c\" (UID: \"bd709159-02e1-4730-bac2-9a0fff9b2c0c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d5d415-676fcbc795-x4w5c" Nov 03 14:02:40.270270 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:40.270242 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-d5d415\"" Nov 03 14:02:40.270270 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:40.270243 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-d5d415-dockercfg-kwpq5\"" Nov 03 14:02:40.273763 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:40.273740 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d5d415-676fcbc795-x4w5c"] Nov 03 14:02:40.349203 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:40.349153 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd709159-02e1-4730-bac2-9a0fff9b2c0c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-d5d415-676fcbc795-x4w5c\" (UID: \"bd709159-02e1-4730-bac2-9a0fff9b2c0c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d5d415-676fcbc795-x4w5c" Nov 03 14:02:40.349436 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:40.349346 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xg9z\" (UniqueName: \"kubernetes.io/projected/bd709159-02e1-4730-bac2-9a0fff9b2c0c-kube-api-access-9xg9z\") pod \"payload-pre-processing-e2e-extmod-b-d5d415-676fcbc795-x4w5c\" (UID: \"bd709159-02e1-4730-bac2-9a0fff9b2c0c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d5d415-676fcbc795-x4w5c" Nov 03 14:02:40.350268 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:40.350249 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd709159-02e1-4730-bac2-9a0fff9b2c0c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-d5d415-676fcbc795-x4w5c\" (UID: \"bd709159-02e1-4730-bac2-9a0fff9b2c0c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d5d415-676fcbc795-x4w5c" Nov 03 14:02:40.404240 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:40.404206 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xg9z\" (UniqueName: \"kubernetes.io/projected/bd709159-02e1-4730-bac2-9a0fff9b2c0c-kube-api-access-9xg9z\") pod \"payload-pre-processing-e2e-extmod-b-d5d415-676fcbc795-x4w5c\" (UID: \"bd709159-02e1-4730-bac2-9a0fff9b2c0c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d5d415-676fcbc795-x4w5c" Nov 03 14:02:40.487923 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:40.487845 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d5d415-676fcbc795-x4w5c" Nov 03 14:02:40.944287 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:40.944231 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d5d415-676fcbc795-x4w5c"] Nov 03 14:02:40.967845 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:02:40.957771 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd709159_02e1_4730_bac2_9a0fff9b2c0c.slice/crio-b6dee20a71085d7604d575b6d365cab85d805523276ec780f2a68d1621e4577b WatchSource:0}: Error finding container b6dee20a71085d7604d575b6d365cab85d805523276ec780f2a68d1621e4577b: Status 404 returned error can't find the container with id b6dee20a71085d7604d575b6d365cab85d805523276ec780f2a68d1621e4577b Nov 03 14:02:41.368354 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:41.368249 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d5d415-676fcbc795-x4w5c" event={"ID":"bd709159-02e1-4730-bac2-9a0fff9b2c0c","Type":"ContainerStarted","Data":"466c1edcb2f35d007f360a99e267a01c41de315bd0d275424ce7effbe76e0022"} Nov 03 14:02:41.368354 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:41.368324 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d5d415-676fcbc795-x4w5c" event={"ID":"bd709159-02e1-4730-bac2-9a0fff9b2c0c","Type":"ContainerStarted","Data":"b6dee20a71085d7604d575b6d365cab85d805523276ec780f2a68d1621e4577b"} Nov 03 14:02:41.368678 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:41.368370 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d5d415-676fcbc795-x4w5c" Nov 03 14:02:41.477783 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:41.477712 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d5d415-676fcbc795-x4w5c" podStartSLOduration=1.477696496 podStartE2EDuration="1.477696496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:02:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:02:41.475324564 +0000 UTC m=+3443.486012806" watchObservedRunningTime="2026-11-03 14:02:41.477696496 +0000 UTC m=+3443.488384738" Nov 03 14:02:44.835073 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:44.835039 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6ebe57-98fcc7dd7-rf945"] Nov 03 14:02:44.838405 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:44.838385 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6ebe57-98fcc7dd7-rf945" Nov 03 14:02:44.891751 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:44.891719 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6ebe57-98fcc7dd7-rf945"] Nov 03 14:02:44.935348 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:44.935304 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-6ebe57\"" Nov 03 14:02:44.935584 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:44.935484 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-6ebe57-dockercfg-8t5hm\"" Nov 03 14:02:44.977499 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:44.971238 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pjnc\" (UniqueName: \"kubernetes.io/projected/680e1deb-10e9-4074-9b16-b741e3655e63-kube-api-access-2pjnc\") pod \"payload-pre-processing-e2e-extmod-a-6ebe57-98fcc7dd7-rf945\" (UID: \"680e1deb-10e9-4074-9b16-b741e3655e63\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6ebe57-98fcc7dd7-rf945" Nov 03 14:02:44.977499 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:44.971300 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/680e1deb-10e9-4074-9b16-b741e3655e63-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-6ebe57-98fcc7dd7-rf945\" (UID: \"680e1deb-10e9-4074-9b16-b741e3655e63\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6ebe57-98fcc7dd7-rf945" Nov 03 14:02:45.033020 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:45.029064 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6ebe57-58896bdfff-trblz"] Nov 03 14:02:45.037421 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:45.037361 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-6ebe57-58896bdfff-trblz" Nov 03 14:02:45.053340 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:45.053303 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6ebe57-58896bdfff-trblz"] Nov 03 14:02:45.073200 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:45.073169 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pjnc\" (UniqueName: \"kubernetes.io/projected/680e1deb-10e9-4074-9b16-b741e3655e63-kube-api-access-2pjnc\") pod \"payload-pre-processing-e2e-extmod-a-6ebe57-98fcc7dd7-rf945\" (UID: \"680e1deb-10e9-4074-9b16-b741e3655e63\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6ebe57-98fcc7dd7-rf945" Nov 03 14:02:45.073372 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:45.073206 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/680e1deb-10e9-4074-9b16-b741e3655e63-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-6ebe57-98fcc7dd7-rf945\" (UID: \"680e1deb-10e9-4074-9b16-b741e3655e63\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6ebe57-98fcc7dd7-rf945" Nov 03 14:02:45.073372 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:45.073332 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdvln\" (UniqueName: \"kubernetes.io/projected/d9ead347-e7c9-4fbc-b59f-6e137e8a979b-kube-api-access-wdvln\") pod \"payload-processing-e2e-extmod-a-6ebe57-58896bdfff-trblz\" (UID: \"d9ead347-e7c9-4fbc-b59f-6e137e8a979b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6ebe57-58896bdfff-trblz" Nov 03 14:02:45.073589 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:45.073399 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9ead347-e7c9-4fbc-b59f-6e137e8a979b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-6ebe57-58896bdfff-trblz\" (UID: \"d9ead347-e7c9-4fbc-b59f-6e137e8a979b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6ebe57-58896bdfff-trblz" Nov 03 14:02:45.074020 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:45.073995 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/680e1deb-10e9-4074-9b16-b741e3655e63-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-6ebe57-98fcc7dd7-rf945\" (UID: \"680e1deb-10e9-4074-9b16-b741e3655e63\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6ebe57-98fcc7dd7-rf945" Nov 03 14:02:45.126839 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:45.126767 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pjnc\" (UniqueName: \"kubernetes.io/projected/680e1deb-10e9-4074-9b16-b741e3655e63-kube-api-access-2pjnc\") pod \"payload-pre-processing-e2e-extmod-a-6ebe57-98fcc7dd7-rf945\" (UID: \"680e1deb-10e9-4074-9b16-b741e3655e63\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6ebe57-98fcc7dd7-rf945" Nov 03 14:02:45.151509 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:45.151479 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6ebe57-98fcc7dd7-rf945" Nov 03 14:02:45.174486 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:45.174446 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9ead347-e7c9-4fbc-b59f-6e137e8a979b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-6ebe57-58896bdfff-trblz\" (UID: \"d9ead347-e7c9-4fbc-b59f-6e137e8a979b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6ebe57-58896bdfff-trblz" Nov 03 14:02:45.174968 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:45.174935 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdvln\" (UniqueName: \"kubernetes.io/projected/d9ead347-e7c9-4fbc-b59f-6e137e8a979b-kube-api-access-wdvln\") pod \"payload-processing-e2e-extmod-a-6ebe57-58896bdfff-trblz\" (UID: \"d9ead347-e7c9-4fbc-b59f-6e137e8a979b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6ebe57-58896bdfff-trblz" Nov 03 14:02:45.175229 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:45.175208 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9ead347-e7c9-4fbc-b59f-6e137e8a979b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-6ebe57-58896bdfff-trblz\" (UID: \"d9ead347-e7c9-4fbc-b59f-6e137e8a979b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6ebe57-58896bdfff-trblz" Nov 03 14:02:45.229296 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:45.229265 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdvln\" (UniqueName: \"kubernetes.io/projected/d9ead347-e7c9-4fbc-b59f-6e137e8a979b-kube-api-access-wdvln\") pod \"payload-processing-e2e-extmod-a-6ebe57-58896bdfff-trblz\" (UID: \"d9ead347-e7c9-4fbc-b59f-6e137e8a979b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6ebe57-58896bdfff-trblz" Nov 03 14:02:45.356981 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:45.356946 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-6ebe57-58896bdfff-trblz" Nov 03 14:02:45.526791 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:45.526757 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6ebe57-98fcc7dd7-rf945"] Nov 03 14:02:45.533642 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:02:45.533612 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod680e1deb_10e9_4074_9b16_b741e3655e63.slice/crio-935b05bb6e1d4df37b34177072e448da8b677c4c122744e25f55fcca9cab1329 WatchSource:0}: Error finding container 935b05bb6e1d4df37b34177072e448da8b677c4c122744e25f55fcca9cab1329: Status 404 returned error can't find the container with id 935b05bb6e1d4df37b34177072e448da8b677c4c122744e25f55fcca9cab1329 Nov 03 14:02:45.754139 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:45.754047 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6ebe57-58896bdfff-trblz"] Nov 03 14:02:45.759983 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:02:45.759950 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9ead347_e7c9_4fbc_b59f_6e137e8a979b.slice/crio-4f160147160746cf89da963172856733274967e39d1bdfca2b9a562b44e03e53 WatchSource:0}: Error finding container 4f160147160746cf89da963172856733274967e39d1bdfca2b9a562b44e03e53: Status 404 returned error can't find the container with id 4f160147160746cf89da963172856733274967e39d1bdfca2b9a562b44e03e53 Nov 03 14:02:46.402156 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:46.402117 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6ebe57-98fcc7dd7-rf945" event={"ID":"680e1deb-10e9-4074-9b16-b741e3655e63","Type":"ContainerStarted","Data":"c2486f58f90898c5593ad78e6772de76ecd78cc7b0106ec49282d6de09f93465"} Nov 03 14:02:46.402677 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:46.402188 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6ebe57-98fcc7dd7-rf945" Nov 03 14:02:46.402677 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:46.402213 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6ebe57-98fcc7dd7-rf945" event={"ID":"680e1deb-10e9-4074-9b16-b741e3655e63","Type":"ContainerStarted","Data":"935b05bb6e1d4df37b34177072e448da8b677c4c122744e25f55fcca9cab1329"} Nov 03 14:02:46.403709 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:46.403683 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-6ebe57-58896bdfff-trblz" event={"ID":"d9ead347-e7c9-4fbc-b59f-6e137e8a979b","Type":"ContainerStarted","Data":"9d2a7efdeaf6314aa081e098256bd3e0470937fa9ba1e378f6f59fb6a04d063e"} Nov 03 14:02:46.403837 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:46.403728 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-6ebe57-58896bdfff-trblz" event={"ID":"d9ead347-e7c9-4fbc-b59f-6e137e8a979b","Type":"ContainerStarted","Data":"4f160147160746cf89da963172856733274967e39d1bdfca2b9a562b44e03e53"} Nov 03 14:02:46.403837 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:46.403769 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-6ebe57-58896bdfff-trblz" Nov 03 14:02:46.510670 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:46.510612 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6ebe57-98fcc7dd7-rf945" podStartSLOduration=2.510598845 podStartE2EDuration="2.510598845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:02:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:02:46.508100524 +0000 UTC m=+3448.518788765" watchObservedRunningTime="2026-11-03 14:02:46.510598845 +0000 UTC m=+3448.521287086" Nov 03 14:02:46.618131 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:46.618078 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-6ebe57-58896bdfff-trblz" podStartSLOduration=2.618062379 podStartE2EDuration="2.618062379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:02:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:02:46.614480753 +0000 UTC m=+3448.625168984" watchObservedRunningTime="2026-11-03 14:02:46.618062379 +0000 UTC m=+3448.628750621" Nov 03 14:02:52.377016 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:52.376986 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d5d415-676fcbc795-x4w5c" Nov 03 14:02:57.413287 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:57.413256 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6ebe57-98fcc7dd7-rf945" Nov 03 14:02:57.413732 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:02:57.413325 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-6ebe57-58896bdfff-trblz" Nov 03 14:03:07.234694 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:03:07.234658 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:03:12.672413 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:03:12.672378 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-966a03-7dd5866fdc-grhzp"] Nov 03 14:03:12.676050 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:03:12.676028 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-966a03-7dd5866fdc-grhzp" Nov 03 14:03:12.728229 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:03:12.728199 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-966a03\"" Nov 03 14:03:12.784533 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:03:12.772770 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3d6304e-6ef4-4c50-beb6-54ef6162d206-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-966a03-7dd5866fdc-grhzp\" (UID: \"c3d6304e-6ef4-4c50-beb6-54ef6162d206\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-966a03-7dd5866fdc-grhzp" Nov 03 14:03:12.784533 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:03:12.772966 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqt8j\" (UniqueName: \"kubernetes.io/projected/c3d6304e-6ef4-4c50-beb6-54ef6162d206-kube-api-access-bqt8j\") pod \"payload-pre-processing-e2e-extmod-b-966a03-7dd5866fdc-grhzp\" (UID: \"c3d6304e-6ef4-4c50-beb6-54ef6162d206\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-966a03-7dd5866fdc-grhzp" Nov 03 14:03:12.784533 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:03:12.775261 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-966a03-dockercfg-dg4js\"" Nov 03 14:03:12.784533 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:03:12.779794 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-966a03-7dd5866fdc-grhzp"] Nov 03 14:03:12.877819 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:03:12.877779 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3d6304e-6ef4-4c50-beb6-54ef6162d206-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-966a03-7dd5866fdc-grhzp\" (UID: \"c3d6304e-6ef4-4c50-beb6-54ef6162d206\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-966a03-7dd5866fdc-grhzp" Nov 03 14:03:12.878021 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:03:12.877897 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqt8j\" (UniqueName: \"kubernetes.io/projected/c3d6304e-6ef4-4c50-beb6-54ef6162d206-kube-api-access-bqt8j\") pod \"payload-pre-processing-e2e-extmod-b-966a03-7dd5866fdc-grhzp\" (UID: \"c3d6304e-6ef4-4c50-beb6-54ef6162d206\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-966a03-7dd5866fdc-grhzp" Nov 03 14:03:12.878682 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:03:12.878652 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3d6304e-6ef4-4c50-beb6-54ef6162d206-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-966a03-7dd5866fdc-grhzp\" (UID: \"c3d6304e-6ef4-4c50-beb6-54ef6162d206\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-966a03-7dd5866fdc-grhzp" Nov 03 14:03:12.931307 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:03:12.931238 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqt8j\" (UniqueName: \"kubernetes.io/projected/c3d6304e-6ef4-4c50-beb6-54ef6162d206-kube-api-access-bqt8j\") pod \"payload-pre-processing-e2e-extmod-b-966a03-7dd5866fdc-grhzp\" (UID: \"c3d6304e-6ef4-4c50-beb6-54ef6162d206\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-966a03-7dd5866fdc-grhzp" Nov 03 14:03:12.992647 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:03:12.992610 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-966a03-7dd5866fdc-grhzp" Nov 03 14:03:13.364286 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:03:13.364205 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-966a03-7dd5866fdc-grhzp"] Nov 03 14:03:13.370999 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:03:13.370972 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3d6304e_6ef4_4c50_beb6_54ef6162d206.slice/crio-79a2c4c11adabf23d4b95571ff846556c58c5523408d297fbf671d702f200903 WatchSource:0}: Error finding container 79a2c4c11adabf23d4b95571ff846556c58c5523408d297fbf671d702f200903: Status 404 returned error can't find the container with id 79a2c4c11adabf23d4b95571ff846556c58c5523408d297fbf671d702f200903 Nov 03 14:03:13.617147 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:03:13.617065 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-966a03-7dd5866fdc-grhzp" event={"ID":"c3d6304e-6ef4-4c50-beb6-54ef6162d206","Type":"ContainerStarted","Data":"7dbd17374f2e708c8e708b12106f128c755b2d1d8c38cfcf1874840d6b2b1e6c"} Nov 03 14:03:13.617147 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:03:13.617139 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-966a03-7dd5866fdc-grhzp" Nov 03 14:03:13.617376 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:03:13.617162 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-966a03-7dd5866fdc-grhzp" event={"ID":"c3d6304e-6ef4-4c50-beb6-54ef6162d206","Type":"ContainerStarted","Data":"79a2c4c11adabf23d4b95571ff846556c58c5523408d297fbf671d702f200903"} Nov 03 14:03:13.725172 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:03:13.725111 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-966a03-7dd5866fdc-grhzp" podStartSLOduration=1.725092071 podStartE2EDuration="1.725092071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:03:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:03:13.724599666 +0000 UTC m=+3475.735287935" watchObservedRunningTime="2026-11-03 14:03:13.725092071 +0000 UTC m=+3475.735780315" Nov 03 14:03:18.671857 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:03:18.671824 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:03:24.627715 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:03:24.627687 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-966a03-7dd5866fdc-grhzp" Nov 03 14:03:37.235960 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:03:37.235911 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:03:55.425113 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:03:55.425077 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c94937d-qcd7r"] Nov 03 14:03:55.429583 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:03:55.429563 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c94937d-qcd7r" Nov 03 14:03:55.479320 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:03:55.479291 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:03:55.483024 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:03:55.482990 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c94937d-qcd7r"] Nov 03 14:03:55.546039 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:03:55.546004 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2327b6e9-03af-454b-889d-a8c565de71f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c94937d-qcd7r\" (UID: \"2327b6e9-03af-454b-889d-a8c565de71f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c94937d-qcd7r" Nov 03 14:03:55.546349 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:03:55.546167 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2327b6e9-03af-454b-889d-a8c565de71f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c94937d-qcd7r\" (UID: \"2327b6e9-03af-454b-889d-a8c565de71f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c94937d-qcd7r" Nov 03 14:03:55.546349 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:03:55.546237 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4wqn\" (UniqueName: \"kubernetes.io/projected/2327b6e9-03af-454b-889d-a8c565de71f4-kube-api-access-d4wqn\") pod \"maas-api-revoke-sub-5c94937d-qcd7r\" (UID: \"2327b6e9-03af-454b-889d-a8c565de71f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c94937d-qcd7r" Nov 03 14:03:55.580504 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:03:55.580457 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 14:03:55.647702 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:03:55.647664 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2327b6e9-03af-454b-889d-a8c565de71f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c94937d-qcd7r\" (UID: \"2327b6e9-03af-454b-889d-a8c565de71f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c94937d-qcd7r" Nov 03 14:03:55.647898 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:03:55.647752 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4wqn\" (UniqueName: \"kubernetes.io/projected/2327b6e9-03af-454b-889d-a8c565de71f4-kube-api-access-d4wqn\") pod \"maas-api-revoke-sub-5c94937d-qcd7r\" (UID: \"2327b6e9-03af-454b-889d-a8c565de71f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c94937d-qcd7r" Nov 03 14:03:55.647957 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:03:55.647903 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2327b6e9-03af-454b-889d-a8c565de71f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c94937d-qcd7r\" (UID: \"2327b6e9-03af-454b-889d-a8c565de71f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c94937d-qcd7r" Nov 03 14:03:55.648583 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:03:55.648555 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2327b6e9-03af-454b-889d-a8c565de71f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c94937d-qcd7r\" (UID: \"2327b6e9-03af-454b-889d-a8c565de71f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c94937d-qcd7r" Nov 03 14:03:55.650146 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:03:55.650119 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2327b6e9-03af-454b-889d-a8c565de71f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c94937d-qcd7r\" (UID: \"2327b6e9-03af-454b-889d-a8c565de71f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c94937d-qcd7r" Nov 03 14:03:55.701142 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:03:55.701067 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4wqn\" (UniqueName: \"kubernetes.io/projected/2327b6e9-03af-454b-889d-a8c565de71f4-kube-api-access-d4wqn\") pod \"maas-api-revoke-sub-5c94937d-qcd7r\" (UID: \"2327b6e9-03af-454b-889d-a8c565de71f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c94937d-qcd7r" Nov 03 14:03:55.743108 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:03:55.743081 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c94937d-qcd7r" Nov 03 14:03:56.120279 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:03:56.120201 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c94937d-qcd7r"] Nov 03 14:03:56.127102 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:03:56.127076 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2327b6e9_03af_454b_889d_a8c565de71f4.slice/crio-66af42ff7a0bb062e467040552de0057ab4eb3dc09a17580d8e5a94369bf2ec5 WatchSource:0}: Error finding container 66af42ff7a0bb062e467040552de0057ab4eb3dc09a17580d8e5a94369bf2ec5: Status 404 returned error can't find the container with id 66af42ff7a0bb062e467040552de0057ab4eb3dc09a17580d8e5a94369bf2ec5 Nov 03 14:03:56.907146 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:03:56.907099 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c94937d-qcd7r" event={"ID":"2327b6e9-03af-454b-889d-a8c565de71f4","Type":"ContainerStarted","Data":"cb77d801977d6bdd72e65351590b7e64020823ac7807f2826ee25ac65eb1df95"} Nov 03 14:03:56.907589 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:03:56.907166 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c94937d-qcd7r" event={"ID":"2327b6e9-03af-454b-889d-a8c565de71f4","Type":"ContainerStarted","Data":"66af42ff7a0bb062e467040552de0057ab4eb3dc09a17580d8e5a94369bf2ec5"} Nov 03 14:03:57.016825 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:03:57.016766 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c94937d-qcd7r" podStartSLOduration=2.016751394 podStartE2EDuration="2.016751394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:03:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:03:57.014079405 +0000 UTC m=+3519.024767648" watchObservedRunningTime="2026-11-03 14:03:57.016751394 +0000 UTC m=+3519.027439679" Nov 03 14:03:57.919294 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:03:57.919257 5979 generic.go:356] "Generic (PLEG): container finished" podID="2327b6e9-03af-454b-889d-a8c565de71f4" containerID="cb77d801977d6bdd72e65351590b7e64020823ac7807f2826ee25ac65eb1df95" exitCode=0 Nov 03 14:03:57.919749 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:03:57.919343 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c94937d-qcd7r" event={"ID":"2327b6e9-03af-454b-889d-a8c565de71f4","Type":"ContainerDied","Data":"cb77d801977d6bdd72e65351590b7e64020823ac7807f2826ee25ac65eb1df95"} Nov 03 14:03:59.084143 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:03:59.084115 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c94937d-qcd7r" Nov 03 14:03:59.195330 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:03:59.195247 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2327b6e9-03af-454b-889d-a8c565de71f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2327b6e9-03af-454b-889d-a8c565de71f4-maas-api-openshift-ca\") pod \"2327b6e9-03af-454b-889d-a8c565de71f4\" (UID: \"2327b6e9-03af-454b-889d-a8c565de71f4\") " Nov 03 14:03:59.195551 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:03:59.195401 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2327b6e9-03af-454b-889d-a8c565de71f4-kube-api-access-d4wqn\" (UniqueName: \"kubernetes.io/projected/2327b6e9-03af-454b-889d-a8c565de71f4-kube-api-access-d4wqn\") pod \"2327b6e9-03af-454b-889d-a8c565de71f4\" (UID: \"2327b6e9-03af-454b-889d-a8c565de71f4\") " Nov 03 14:03:59.195551 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:03:59.195467 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2327b6e9-03af-454b-889d-a8c565de71f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2327b6e9-03af-454b-889d-a8c565de71f4-maas-api-xks-ca\") pod \"2327b6e9-03af-454b-889d-a8c565de71f4\" (UID: \"2327b6e9-03af-454b-889d-a8c565de71f4\") " Nov 03 14:03:59.195779 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:03:59.195684 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2327b6e9-03af-454b-889d-a8c565de71f4-maas-api-openshift-ca" pod "2327b6e9-03af-454b-889d-a8c565de71f4" (UID: "2327b6e9-03af-454b-889d-a8c565de71f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:03:59.196106 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:03:59.196079 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2327b6e9-03af-454b-889d-a8c565de71f4-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:03:59.198127 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:03:59.198102 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2327b6e9-03af-454b-889d-a8c565de71f4-maas-api-xks-ca" pod "2327b6e9-03af-454b-889d-a8c565de71f4" (UID: "2327b6e9-03af-454b-889d-a8c565de71f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:03:59.198225 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:03:59.198135 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2327b6e9-03af-454b-889d-a8c565de71f4-kube-api-access-d4wqn" pod "2327b6e9-03af-454b-889d-a8c565de71f4" (UID: "2327b6e9-03af-454b-889d-a8c565de71f4"). InnerVolumeSpecName "kube-api-access-d4wqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:03:59.297780 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:03:59.297738 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2327b6e9-03af-454b-889d-a8c565de71f4-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:03:59.297961 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:03:59.297802 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4wqn\" (UniqueName: \"kubernetes.io/projected/2327b6e9-03af-454b-889d-a8c565de71f4-kube-api-access-d4wqn\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:03:59.935539 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:03:59.935500 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c94937d-qcd7r" event={"ID":"2327b6e9-03af-454b-889d-a8c565de71f4","Type":"ContainerDied","Data":"66af42ff7a0bb062e467040552de0057ab4eb3dc09a17580d8e5a94369bf2ec5"} Nov 03 14:03:59.935727 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:03:59.935546 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c94937d-qcd7r" Nov 03 14:03:59.935727 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:03:59.935556 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66af42ff7a0bb062e467040552de0057ab4eb3dc09a17580d8e5a94369bf2ec5" Nov 03 14:04:06.176206 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:06.176166 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9347bb13-zktbd"] Nov 03 14:04:06.176942 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:06.176925 5979 cpu_manager.go:414] "containerMap: removing container" podUID="2327b6e9-03af-454b-889d-a8c565de71f4" containerName="revoke-keys" Nov 03 14:04:06.176992 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:06.176943 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2327b6e9-03af-454b-889d-a8c565de71f4" containerName="revoke-keys" Nov 03 14:04:06.177064 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:06.177055 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="2327b6e9-03af-454b-889d-a8c565de71f4" containerName="revoke-keys" Nov 03 14:04:06.180862 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:06.180839 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9347bb13-zktbd" Nov 03 14:04:06.234344 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:06.234312 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9347bb13-zktbd"] Nov 03 14:04:06.276602 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:06.276566 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:04:06.276774 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:06.276629 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 14:04:06.285911 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:06.285880 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9fft\" (UniqueName: \"kubernetes.io/projected/db0469df-0987-4c6a-bf09-3f66ca1c70c9-kube-api-access-x9fft\") pod \"maas-api-revoke-sub-9347bb13-zktbd\" (UID: \"db0469df-0987-4c6a-bf09-3f66ca1c70c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9347bb13-zktbd" Nov 03 14:04:06.286055 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:06.285931 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db0469df-0987-4c6a-bf09-3f66ca1c70c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9347bb13-zktbd\" (UID: \"db0469df-0987-4c6a-bf09-3f66ca1c70c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9347bb13-zktbd" Nov 03 14:04:06.286055 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:06.285951 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db0469df-0987-4c6a-bf09-3f66ca1c70c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9347bb13-zktbd\" (UID: \"db0469df-0987-4c6a-bf09-3f66ca1c70c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9347bb13-zktbd" Nov 03 14:04:06.387157 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:06.387126 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9fft\" (UniqueName: \"kubernetes.io/projected/db0469df-0987-4c6a-bf09-3f66ca1c70c9-kube-api-access-x9fft\") pod \"maas-api-revoke-sub-9347bb13-zktbd\" (UID: \"db0469df-0987-4c6a-bf09-3f66ca1c70c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9347bb13-zktbd" Nov 03 14:04:06.387354 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:06.387176 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db0469df-0987-4c6a-bf09-3f66ca1c70c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9347bb13-zktbd\" (UID: \"db0469df-0987-4c6a-bf09-3f66ca1c70c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9347bb13-zktbd" Nov 03 14:04:06.387354 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:06.387294 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db0469df-0987-4c6a-bf09-3f66ca1c70c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9347bb13-zktbd\" (UID: \"db0469df-0987-4c6a-bf09-3f66ca1c70c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9347bb13-zktbd" Nov 03 14:04:06.387980 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:06.387945 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db0469df-0987-4c6a-bf09-3f66ca1c70c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9347bb13-zktbd\" (UID: \"db0469df-0987-4c6a-bf09-3f66ca1c70c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9347bb13-zktbd" Nov 03 14:04:06.389460 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:06.389443 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db0469df-0987-4c6a-bf09-3f66ca1c70c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9347bb13-zktbd\" (UID: \"db0469df-0987-4c6a-bf09-3f66ca1c70c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9347bb13-zktbd" Nov 03 14:04:06.440962 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:06.440894 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9fft\" (UniqueName: \"kubernetes.io/projected/db0469df-0987-4c6a-bf09-3f66ca1c70c9-kube-api-access-x9fft\") pod \"maas-api-revoke-sub-9347bb13-zktbd\" (UID: \"db0469df-0987-4c6a-bf09-3f66ca1c70c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9347bb13-zktbd" Nov 03 14:04:06.495657 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:06.495618 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9347bb13-zktbd" Nov 03 14:04:06.873735 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:06.873646 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9347bb13-zktbd"] Nov 03 14:04:06.880825 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:04:06.880798 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb0469df_0987_4c6a_bf09_3f66ca1c70c9.slice/crio-20e33ece661ba8b0124af2d63e3ea0f1071b8d41164f90a864e27e9f47850a8a WatchSource:0}: Error finding container 20e33ece661ba8b0124af2d63e3ea0f1071b8d41164f90a864e27e9f47850a8a: Status 404 returned error can't find the container with id 20e33ece661ba8b0124af2d63e3ea0f1071b8d41164f90a864e27e9f47850a8a Nov 03 14:04:06.898968 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:06.898942 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:04:07.000434 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:07.000392 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9347bb13-zktbd" event={"ID":"db0469df-0987-4c6a-bf09-3f66ca1c70c9","Type":"ContainerStarted","Data":"20e33ece661ba8b0124af2d63e3ea0f1071b8d41164f90a864e27e9f47850a8a"} Nov 03 14:04:07.233934 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:04:07.233899 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:04:08.008745 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:08.008712 5979 generic.go:356] "Generic (PLEG): container finished" podID="db0469df-0987-4c6a-bf09-3f66ca1c70c9" containerID="6f77aa0d638fd47b33d7505aa060d4782a55a611aaf52c1557dfc58f09045035" exitCode=0 Nov 03 14:04:08.008946 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:08.008794 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9347bb13-zktbd" event={"ID":"db0469df-0987-4c6a-bf09-3f66ca1c70c9","Type":"ContainerDied","Data":"6f77aa0d638fd47b33d7505aa060d4782a55a611aaf52c1557dfc58f09045035"} Nov 03 14:04:09.169054 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:09.169020 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9347bb13-zktbd" Nov 03 14:04:09.232365 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:09.232332 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db0469df-0987-4c6a-bf09-3f66ca1c70c9-kube-api-access-x9fft\" (UniqueName: \"kubernetes.io/projected/db0469df-0987-4c6a-bf09-3f66ca1c70c9-kube-api-access-x9fft\") pod \"db0469df-0987-4c6a-bf09-3f66ca1c70c9\" (UID: \"db0469df-0987-4c6a-bf09-3f66ca1c70c9\") " Nov 03 14:04:09.232578 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:09.232384 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db0469df-0987-4c6a-bf09-3f66ca1c70c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db0469df-0987-4c6a-bf09-3f66ca1c70c9-maas-api-openshift-ca\") pod \"db0469df-0987-4c6a-bf09-3f66ca1c70c9\" (UID: \"db0469df-0987-4c6a-bf09-3f66ca1c70c9\") " Nov 03 14:04:09.232633 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:09.232583 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db0469df-0987-4c6a-bf09-3f66ca1c70c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db0469df-0987-4c6a-bf09-3f66ca1c70c9-maas-api-xks-ca\") pod \"db0469df-0987-4c6a-bf09-3f66ca1c70c9\" (UID: \"db0469df-0987-4c6a-bf09-3f66ca1c70c9\") " Nov 03 14:04:09.232765 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:09.232729 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db0469df-0987-4c6a-bf09-3f66ca1c70c9-maas-api-openshift-ca" pod "db0469df-0987-4c6a-bf09-3f66ca1c70c9" (UID: "db0469df-0987-4c6a-bf09-3f66ca1c70c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:04:09.233033 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:09.233015 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db0469df-0987-4c6a-bf09-3f66ca1c70c9-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:04:09.234480 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:09.234447 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db0469df-0987-4c6a-bf09-3f66ca1c70c9-kube-api-access-x9fft" pod "db0469df-0987-4c6a-bf09-3f66ca1c70c9" (UID: "db0469df-0987-4c6a-bf09-3f66ca1c70c9"). InnerVolumeSpecName "kube-api-access-x9fft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:04:09.234567 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:09.234491 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db0469df-0987-4c6a-bf09-3f66ca1c70c9-maas-api-xks-ca" pod "db0469df-0987-4c6a-bf09-3f66ca1c70c9" (UID: "db0469df-0987-4c6a-bf09-3f66ca1c70c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:04:09.334184 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:09.334089 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9fft\" (UniqueName: \"kubernetes.io/projected/db0469df-0987-4c6a-bf09-3f66ca1c70c9-kube-api-access-x9fft\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:04:09.334184 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:09.334121 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db0469df-0987-4c6a-bf09-3f66ca1c70c9-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:04:10.022444 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:10.022405 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9347bb13-zktbd" event={"ID":"db0469df-0987-4c6a-bf09-3f66ca1c70c9","Type":"ContainerDied","Data":"20e33ece661ba8b0124af2d63e3ea0f1071b8d41164f90a864e27e9f47850a8a"} Nov 03 14:04:10.022444 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:10.022446 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9347bb13-zktbd" Nov 03 14:04:10.022697 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:10.022466 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20e33ece661ba8b0124af2d63e3ea0f1071b8d41164f90a864e27e9f47850a8a" Nov 03 14:04:36.038998 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:36.038961 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-58f896-67cbfc54bc-crds5"] Nov 03 14:04:37.236262 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:04:37.236223 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:04:41.039448 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:41.039356 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-58f896-67cbfc54bc-crds5" podUID="98afcadd-f2c0-44ca-bfbb-33752dc919d1" containerName="payload-pre-processing" containerID="cri-o://60e13a353d92c4ae38a012704511d3ad89c24737e926eefac97c9e41d93503ed" gracePeriod=25 Nov 03 14:04:41.338887 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:41.338852 5979 generic.go:356] "Generic (PLEG): container finished" podID="98afcadd-f2c0-44ca-bfbb-33752dc919d1" containerID="60e13a353d92c4ae38a012704511d3ad89c24737e926eefac97c9e41d93503ed" exitCode=0 Nov 03 14:04:41.339153 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:41.338931 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-58f896-67cbfc54bc-crds5" event={"ID":"98afcadd-f2c0-44ca-bfbb-33752dc919d1","Type":"ContainerDied","Data":"60e13a353d92c4ae38a012704511d3ad89c24737e926eefac97c9e41d93503ed"} Nov 03 14:04:41.341463 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:41.341437 5979 generic.go:356] "Generic (PLEG): container finished" podID="9ded222c-3a0a-4c3a-bd70-493d97638c99" containerID="a0c8bf7f848148181fcfbf15c36f59265eab995cf50b73913c6179ec369dcd41" exitCode=0 Nov 03 14:04:41.341610 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:41.341550 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb" event={"ID":"9ded222c-3a0a-4c3a-bd70-493d97638c99","Type":"ContainerDied","Data":"a0c8bf7f848148181fcfbf15c36f59265eab995cf50b73913c6179ec369dcd41"} Nov 03 14:04:41.342006 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:41.341989 5979 scope.go:122] "RemoveContainer" containerID="a0c8bf7f848148181fcfbf15c36f59265eab995cf50b73913c6179ec369dcd41" Nov 03 14:04:41.343748 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:41.343722 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:04:41.343860 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:41.343788 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:04:41.343860 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:41.343820 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:04:41.380021 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:41.379990 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-58f896-67cbfc54bc-crds5" Nov 03 14:04:41.557085 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:41.556976 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98afcadd-f2c0-44ca-bfbb-33752dc919d1-kube-api-access-qnmrh\" (UniqueName: \"kubernetes.io/projected/98afcadd-f2c0-44ca-bfbb-33752dc919d1-kube-api-access-qnmrh\") pod \"98afcadd-f2c0-44ca-bfbb-33752dc919d1\" (UID: \"98afcadd-f2c0-44ca-bfbb-33752dc919d1\") " Nov 03 14:04:41.557278 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:41.557092 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98afcadd-f2c0-44ca-bfbb-33752dc919d1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98afcadd-f2c0-44ca-bfbb-33752dc919d1-plugins-config-volume\") pod \"98afcadd-f2c0-44ca-bfbb-33752dc919d1\" (UID: \"98afcadd-f2c0-44ca-bfbb-33752dc919d1\") " Nov 03 14:04:41.558375 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:41.558344 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98afcadd-f2c0-44ca-bfbb-33752dc919d1-plugins-config-volume" pod "98afcadd-f2c0-44ca-bfbb-33752dc919d1" (UID: "98afcadd-f2c0-44ca-bfbb-33752dc919d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:04:41.561689 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:41.561660 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98afcadd-f2c0-44ca-bfbb-33752dc919d1-kube-api-access-qnmrh" pod "98afcadd-f2c0-44ca-bfbb-33752dc919d1" (UID: "98afcadd-f2c0-44ca-bfbb-33752dc919d1"). InnerVolumeSpecName "kube-api-access-qnmrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:04:41.659010 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:41.658965 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnmrh\" (UniqueName: \"kubernetes.io/projected/98afcadd-f2c0-44ca-bfbb-33752dc919d1-kube-api-access-qnmrh\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:04:41.659010 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:41.658998 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98afcadd-f2c0-44ca-bfbb-33752dc919d1-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:04:42.371564 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:42.371528 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-58f896-67cbfc54bc-crds5" Nov 03 14:04:42.371564 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:42.371559 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-58f896-67cbfc54bc-crds5" event={"ID":"98afcadd-f2c0-44ca-bfbb-33752dc919d1","Type":"ContainerDied","Data":"637507142ef0ea69b0529e3305992a919607a2cee17be41abf0e3766b8a9c9fa"} Nov 03 14:04:42.372131 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:42.371629 5979 scope.go:122] "RemoveContainer" containerID="60e13a353d92c4ae38a012704511d3ad89c24737e926eefac97c9e41d93503ed" Nov 03 14:04:42.374396 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:42.374362 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb" event={"ID":"9ded222c-3a0a-4c3a-bd70-493d97638c99","Type":"ContainerStarted","Data":"f845790cd0005261edb6dce0ea61f0db17fa5835eb201c2caf78854eb37e2e68"} Nov 03 14:04:42.530502 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:42.530417 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-58f896-67cbfc54bc-crds5"] Nov 03 14:04:42.535983 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:42.535937 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-58f896-67cbfc54bc-crds5"] Nov 03 14:04:43.068498 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:43.066890 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb" Nov 03 14:04:43.068498 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:43.067057 5979 patch_prober.go:29] interesting pod/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" start-of-body= Nov 03 14:04:43.068498 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:43.067114 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb" podUID="9ded222c-3a0a-4c3a-bd70-493d97638c99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" Nov 03 14:04:44.061918 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:44.061878 5979 patch_prober.go:29] interesting pod/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" start-of-body= Nov 03 14:04:44.062559 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:44.062526 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb" podUID="9ded222c-3a0a-4c3a-bd70-493d97638c99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" Nov 03 14:04:44.116414 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:44.116383 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98afcadd-f2c0-44ca-bfbb-33752dc919d1" path="/var/lib/kubelet/pods/98afcadd-f2c0-44ca-bfbb-33752dc919d1/volumes" Nov 03 14:04:45.063493 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:45.062810 5979 patch_prober.go:29] interesting pod/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" start-of-body= Nov 03 14:04:45.063493 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:45.062873 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb" podUID="9ded222c-3a0a-4c3a-bd70-493d97638c99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" Nov 03 14:04:45.473544 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:45.473505 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-966a03-7cfcd9f66c-7dk75"] Nov 03 14:04:45.474590 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:45.474564 5979 cpu_manager.go:414] "containerMap: removing container" podUID="db0469df-0987-4c6a-bf09-3f66ca1c70c9" containerName="revoke-keys" Nov 03 14:04:45.474590 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:45.474586 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db0469df-0987-4c6a-bf09-3f66ca1c70c9" containerName="revoke-keys" Nov 03 14:04:45.474735 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:45.474645 5979 cpu_manager.go:414] "containerMap: removing container" podUID="98afcadd-f2c0-44ca-bfbb-33752dc919d1" containerName="payload-pre-processing" Nov 03 14:04:45.474735 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:45.474654 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98afcadd-f2c0-44ca-bfbb-33752dc919d1" containerName="payload-pre-processing" Nov 03 14:04:45.474809 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:45.474794 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="98afcadd-f2c0-44ca-bfbb-33752dc919d1" containerName="payload-pre-processing" Nov 03 14:04:45.474851 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:45.474820 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="db0469df-0987-4c6a-bf09-3f66ca1c70c9" containerName="revoke-keys" Nov 03 14:04:45.478942 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:45.478921 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-966a03-7cfcd9f66c-7dk75" Nov 03 14:04:45.514531 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:45.514502 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x89h5\" (UniqueName: \"kubernetes.io/projected/c8f215c3-1d1d-4b0c-8868-609bb451e8ce-kube-api-access-x89h5\") pod \"maas-api-e2e-extmod-b-966a03-7cfcd9f66c-7dk75\" (UID: \"c8f215c3-1d1d-4b0c-8868-609bb451e8ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-966a03-7cfcd9f66c-7dk75" Nov 03 14:04:45.514669 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:45.514587 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8f215c3-1d1d-4b0c-8868-609bb451e8ce-metrics-certs\") pod \"maas-api-e2e-extmod-b-966a03-7cfcd9f66c-7dk75\" (UID: \"c8f215c3-1d1d-4b0c-8868-609bb451e8ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-966a03-7cfcd9f66c-7dk75" Nov 03 14:04:45.514752 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:45.514723 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8f215c3-1d1d-4b0c-8868-609bb451e8ce-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-966a03-7cfcd9f66c-7dk75\" (UID: \"c8f215c3-1d1d-4b0c-8868-609bb451e8ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-966a03-7cfcd9f66c-7dk75" Nov 03 14:04:45.514838 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:45.514821 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8f215c3-1d1d-4b0c-8868-609bb451e8ce-maas-api-tls\") pod \"maas-api-e2e-extmod-b-966a03-7cfcd9f66c-7dk75\" (UID: \"c8f215c3-1d1d-4b0c-8868-609bb451e8ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-966a03-7cfcd9f66c-7dk75" Nov 03 14:04:45.517548 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:45.517525 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-966a03-7cfcd9f66c-7dk75"] Nov 03 14:04:45.531510 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:45.527771 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-966a03-serving-cert\"" Nov 03 14:04:45.616343 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:45.616310 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8f215c3-1d1d-4b0c-8868-609bb451e8ce-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-966a03-7cfcd9f66c-7dk75\" (UID: \"c8f215c3-1d1d-4b0c-8868-609bb451e8ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-966a03-7cfcd9f66c-7dk75" Nov 03 14:04:45.616670 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:45.616645 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8f215c3-1d1d-4b0c-8868-609bb451e8ce-maas-api-tls\") pod \"maas-api-e2e-extmod-b-966a03-7cfcd9f66c-7dk75\" (UID: \"c8f215c3-1d1d-4b0c-8868-609bb451e8ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-966a03-7cfcd9f66c-7dk75" Nov 03 14:04:45.616793 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:45.616766 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x89h5\" (UniqueName: \"kubernetes.io/projected/c8f215c3-1d1d-4b0c-8868-609bb451e8ce-kube-api-access-x89h5\") pod \"maas-api-e2e-extmod-b-966a03-7cfcd9f66c-7dk75\" (UID: \"c8f215c3-1d1d-4b0c-8868-609bb451e8ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-966a03-7cfcd9f66c-7dk75" Nov 03 14:04:45.616862 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:04:45.616803 5979 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-966a03-serving-cert: secret "maas-api-e2e-extmod-b-966a03-serving-cert" not found Nov 03 14:04:45.616862 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:45.616853 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8f215c3-1d1d-4b0c-8868-609bb451e8ce-metrics-certs\") pod \"maas-api-e2e-extmod-b-966a03-7cfcd9f66c-7dk75\" (UID: \"c8f215c3-1d1d-4b0c-8868-609bb451e8ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-966a03-7cfcd9f66c-7dk75" Nov 03 14:04:45.616981 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:04:45.616888 5979 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8f215c3-1d1d-4b0c-8868-609bb451e8ce-maas-api-tls podName:c8f215c3-1d1d-4b0c-8868-609bb451e8ce nodeName:}" failed. No retries permitted until 2026-11-03 14:04:46.116865747 +0000 UTC m=+3568.127553969 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c8f215c3-1d1d-4b0c-8868-609bb451e8ce-maas-api-tls") pod "maas-api-e2e-extmod-b-966a03-7cfcd9f66c-7dk75" (UID: "c8f215c3-1d1d-4b0c-8868-609bb451e8ce") : secret "maas-api-e2e-extmod-b-966a03-serving-cert" not found Nov 03 14:04:45.617412 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:45.617379 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8f215c3-1d1d-4b0c-8868-609bb451e8ce-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-966a03-7cfcd9f66c-7dk75\" (UID: \"c8f215c3-1d1d-4b0c-8868-609bb451e8ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-966a03-7cfcd9f66c-7dk75" Nov 03 14:04:45.619572 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:45.619546 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8f215c3-1d1d-4b0c-8868-609bb451e8ce-metrics-certs\") pod \"maas-api-e2e-extmod-b-966a03-7cfcd9f66c-7dk75\" (UID: \"c8f215c3-1d1d-4b0c-8868-609bb451e8ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-966a03-7cfcd9f66c-7dk75" Nov 03 14:04:45.671404 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:45.671374 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x89h5\" (UniqueName: \"kubernetes.io/projected/c8f215c3-1d1d-4b0c-8868-609bb451e8ce-kube-api-access-x89h5\") pod \"maas-api-e2e-extmod-b-966a03-7cfcd9f66c-7dk75\" (UID: \"c8f215c3-1d1d-4b0c-8868-609bb451e8ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-966a03-7cfcd9f66c-7dk75" Nov 03 14:04:46.066495 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:46.065905 5979 patch_prober.go:29] interesting pod/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" start-of-body= Nov 03 14:04:46.066495 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:46.065968 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb" podUID="9ded222c-3a0a-4c3a-bd70-493d97638c99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" Nov 03 14:04:46.130365 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:46.130329 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8f215c3-1d1d-4b0c-8868-609bb451e8ce-maas-api-tls\") pod \"maas-api-e2e-extmod-b-966a03-7cfcd9f66c-7dk75\" (UID: \"c8f215c3-1d1d-4b0c-8868-609bb451e8ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-966a03-7cfcd9f66c-7dk75" Nov 03 14:04:46.132940 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:46.132921 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8f215c3-1d1d-4b0c-8868-609bb451e8ce-maas-api-tls\") pod \"maas-api-e2e-extmod-b-966a03-7cfcd9f66c-7dk75\" (UID: \"c8f215c3-1d1d-4b0c-8868-609bb451e8ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-966a03-7cfcd9f66c-7dk75" Nov 03 14:04:46.392904 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:46.392872 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-966a03-7cfcd9f66c-7dk75" Nov 03 14:04:46.800124 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:46.800031 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-966a03-7cfcd9f66c-7dk75"] Nov 03 14:04:46.807161 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:04:46.807134 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8f215c3_1d1d_4b0c_8868_609bb451e8ce.slice/crio-796a9bbed3bbb2a067fc6c0c00b6bc0f5f5da600a5b61021fb538bde62a8c30b WatchSource:0}: Error finding container 796a9bbed3bbb2a067fc6c0c00b6bc0f5f5da600a5b61021fb538bde62a8c30b: Status 404 returned error can't find the container with id 796a9bbed3bbb2a067fc6c0c00b6bc0f5f5da600a5b61021fb538bde62a8c30b Nov 03 14:04:47.068725 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:47.068614 5979 patch_prober.go:29] interesting pod/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" start-of-body= Nov 03 14:04:47.069383 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:47.069343 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb" podUID="9ded222c-3a0a-4c3a-bd70-493d97638c99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.49:15021/healthz/ready\": dial tcp 10.134.1.49:15021: connect: connection refused" Nov 03 14:04:47.412966 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:47.412932 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-966a03-7cfcd9f66c-7dk75" event={"ID":"c8f215c3-1d1d-4b0c-8868-609bb451e8ce","Type":"ContainerStarted","Data":"796a9bbed3bbb2a067fc6c0c00b6bc0f5f5da600a5b61021fb538bde62a8c30b"} Nov 03 14:04:48.065245 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:48.065163 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb" Nov 03 14:04:48.065668 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:48.065646 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb" Nov 03 14:04:48.066420 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:48.066402 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb" Nov 03 14:04:48.422729 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:48.422691 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-966a03-7cfcd9f66c-7dk75" event={"ID":"c8f215c3-1d1d-4b0c-8868-609bb451e8ce","Type":"ContainerStarted","Data":"7f79a28182b5964b901b9d452212cf8af1f664fc3e19e95c73e531d1739a5c77"} Nov 03 14:04:48.423143 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:48.422928 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-966a03-7cfcd9f66c-7dk75" Nov 03 14:04:48.533820 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:48.533771 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-966a03-7cfcd9f66c-7dk75" podStartSLOduration=2.7232687799999997 podStartE2EDuration="3.53375765s" totalImagesPullingTime="810.48887ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:04:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:04:48.532112568 +0000 UTC m=+3570.542800804" watchObservedRunningTime="2026-11-03 14:04:48.53375765 +0000 UTC m=+3570.544445892" Nov 03 14:04:54.435041 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:04:54.435005 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-966a03-7cfcd9f66c-7dk75" Nov 03 14:05:06.862383 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:06.862343 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:05:07.233426 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:05:07.233387 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:05:37.235406 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:05:37.235366 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:05:40.150460 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:40.150415 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ae248d-6d79d84655-b88z9"] Nov 03 14:05:40.258335 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:40.258296 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ae248d-7456846d9f-wntx2"] Nov 03 14:05:45.151158 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:45.151099 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-ae248d-6d79d84655-b88z9" podUID="2c5348d9-47e4-428f-b789-558b167b9ae5" containerName="payload-processing" containerID="cri-o://362365845d0e40cbd118f4e94e761c771496f90d8c7a17b57c4ee7336314ce9e" gracePeriod=25 Nov 03 14:05:45.259597 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:45.259540 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ae248d-7456846d9f-wntx2" podUID="acda8eb6-a116-4a80-8572-4fc538ddb5b6" containerName="payload-pre-processing" containerID="cri-o://efff3607abe96de52392d0f3f00188c14a528e774c973de8306db829e2a122d1" gracePeriod=25 Nov 03 14:05:45.348158 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:45.348120 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6ebe57-58896bdfff-trblz"] Nov 03 14:05:45.430533 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:45.430505 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ae248d-6d79d84655-b88z9" Nov 03 14:05:45.459553 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:45.459432 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6ebe57-98fcc7dd7-rf945"] Nov 03 14:05:45.482142 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:45.481272 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c5348d9-47e4-428f-b789-558b167b9ae5-kube-api-access-j4cx9\" (UniqueName: \"kubernetes.io/projected/2c5348d9-47e4-428f-b789-558b167b9ae5-kube-api-access-j4cx9\") pod \"2c5348d9-47e4-428f-b789-558b167b9ae5\" (UID: \"2c5348d9-47e4-428f-b789-558b167b9ae5\") " Nov 03 14:05:45.482142 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:45.481385 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c5348d9-47e4-428f-b789-558b167b9ae5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c5348d9-47e4-428f-b789-558b167b9ae5-plugins-config-volume\") pod \"2c5348d9-47e4-428f-b789-558b167b9ae5\" (UID: \"2c5348d9-47e4-428f-b789-558b167b9ae5\") " Nov 03 14:05:45.483008 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:45.482962 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c5348d9-47e4-428f-b789-558b167b9ae5-plugins-config-volume" pod "2c5348d9-47e4-428f-b789-558b167b9ae5" (UID: "2c5348d9-47e4-428f-b789-558b167b9ae5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:05:45.484778 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:45.484738 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c5348d9-47e4-428f-b789-558b167b9ae5-kube-api-access-j4cx9" pod "2c5348d9-47e4-428f-b789-558b167b9ae5" (UID: "2c5348d9-47e4-428f-b789-558b167b9ae5"). InnerVolumeSpecName "kube-api-access-j4cx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:05:45.531195 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:45.531174 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ae248d-7456846d9f-wntx2" Nov 03 14:05:45.582705 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:45.582674 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acda8eb6-a116-4a80-8572-4fc538ddb5b6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acda8eb6-a116-4a80-8572-4fc538ddb5b6-plugins-config-volume\") pod \"acda8eb6-a116-4a80-8572-4fc538ddb5b6\" (UID: \"acda8eb6-a116-4a80-8572-4fc538ddb5b6\") " Nov 03 14:05:45.582705 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:45.582712 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acda8eb6-a116-4a80-8572-4fc538ddb5b6-kube-api-access-6hg2d\" (UniqueName: \"kubernetes.io/projected/acda8eb6-a116-4a80-8572-4fc538ddb5b6-kube-api-access-6hg2d\") pod \"acda8eb6-a116-4a80-8572-4fc538ddb5b6\" (UID: \"acda8eb6-a116-4a80-8572-4fc538ddb5b6\") " Nov 03 14:05:45.583145 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:45.583121 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acda8eb6-a116-4a80-8572-4fc538ddb5b6-plugins-config-volume" pod "acda8eb6-a116-4a80-8572-4fc538ddb5b6" (UID: "acda8eb6-a116-4a80-8572-4fc538ddb5b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:05:45.583191 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:45.583181 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4cx9\" (UniqueName: \"kubernetes.io/projected/2c5348d9-47e4-428f-b789-558b167b9ae5-kube-api-access-j4cx9\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:05:45.583231 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:45.583196 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c5348d9-47e4-428f-b789-558b167b9ae5-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:05:45.584876 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:45.584855 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acda8eb6-a116-4a80-8572-4fc538ddb5b6-kube-api-access-6hg2d" pod "acda8eb6-a116-4a80-8572-4fc538ddb5b6" (UID: "acda8eb6-a116-4a80-8572-4fc538ddb5b6"). InnerVolumeSpecName "kube-api-access-6hg2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:05:45.684249 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:45.684170 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acda8eb6-a116-4a80-8572-4fc538ddb5b6-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:05:45.684460 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:45.684442 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hg2d\" (UniqueName: \"kubernetes.io/projected/acda8eb6-a116-4a80-8572-4fc538ddb5b6-kube-api-access-6hg2d\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:05:45.837394 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:45.837352 5979 generic.go:356] "Generic (PLEG): container finished" podID="acda8eb6-a116-4a80-8572-4fc538ddb5b6" containerID="efff3607abe96de52392d0f3f00188c14a528e774c973de8306db829e2a122d1" exitCode=0 Nov 03 14:05:45.837639 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:45.837411 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ae248d-7456846d9f-wntx2" Nov 03 14:05:45.837639 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:45.837548 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ae248d-7456846d9f-wntx2" event={"ID":"acda8eb6-a116-4a80-8572-4fc538ddb5b6","Type":"ContainerDied","Data":"efff3607abe96de52392d0f3f00188c14a528e774c973de8306db829e2a122d1"} Nov 03 14:05:45.837639 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:45.837601 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ae248d-7456846d9f-wntx2" event={"ID":"acda8eb6-a116-4a80-8572-4fc538ddb5b6","Type":"ContainerDied","Data":"7fea12a3d43cc84fccfa80523ad17a2e8738967d70b49cdc17cc50a790d75f59"} Nov 03 14:05:45.837639 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:45.837631 5979 scope.go:122] "RemoveContainer" containerID="efff3607abe96de52392d0f3f00188c14a528e774c973de8306db829e2a122d1" Nov 03 14:05:45.840144 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:45.840118 5979 generic.go:356] "Generic (PLEG): container finished" podID="2c5348d9-47e4-428f-b789-558b167b9ae5" containerID="362365845d0e40cbd118f4e94e761c771496f90d8c7a17b57c4ee7336314ce9e" exitCode=0 Nov 03 14:05:45.840313 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:45.840256 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ae248d-6d79d84655-b88z9" event={"ID":"2c5348d9-47e4-428f-b789-558b167b9ae5","Type":"ContainerDied","Data":"362365845d0e40cbd118f4e94e761c771496f90d8c7a17b57c4ee7336314ce9e"} Nov 03 14:05:45.840387 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:45.840344 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ae248d-6d79d84655-b88z9" event={"ID":"2c5348d9-47e4-428f-b789-558b167b9ae5","Type":"ContainerDied","Data":"8b58c1a5083f289f249bad68b2ca62474757fc4716fc0b1375463cbcf3114b8e"} Nov 03 14:05:45.840455 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:45.840417 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ae248d-6d79d84655-b88z9" Nov 03 14:05:45.852387 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:45.852366 5979 scope.go:122] "RemoveContainer" containerID="efff3607abe96de52392d0f3f00188c14a528e774c973de8306db829e2a122d1" Nov 03 14:05:45.852704 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:05:45.852676 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efff3607abe96de52392d0f3f00188c14a528e774c973de8306db829e2a122d1\": container with ID starting with efff3607abe96de52392d0f3f00188c14a528e774c973de8306db829e2a122d1 not found: ID does not exist" containerID="efff3607abe96de52392d0f3f00188c14a528e774c973de8306db829e2a122d1" Nov 03 14:05:45.852811 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:45.852713 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efff3607abe96de52392d0f3f00188c14a528e774c973de8306db829e2a122d1"} err="failed to get container status \"efff3607abe96de52392d0f3f00188c14a528e774c973de8306db829e2a122d1\": rpc error: code = NotFound desc = could not find container \"efff3607abe96de52392d0f3f00188c14a528e774c973de8306db829e2a122d1\": container with ID starting with efff3607abe96de52392d0f3f00188c14a528e774c973de8306db829e2a122d1 not found: ID does not exist" Nov 03 14:05:45.852811 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:45.852739 5979 scope.go:122] "RemoveContainer" containerID="362365845d0e40cbd118f4e94e761c771496f90d8c7a17b57c4ee7336314ce9e" Nov 03 14:05:45.862253 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:45.862235 5979 scope.go:122] "RemoveContainer" containerID="362365845d0e40cbd118f4e94e761c771496f90d8c7a17b57c4ee7336314ce9e" Nov 03 14:05:45.862545 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:05:45.862524 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"362365845d0e40cbd118f4e94e761c771496f90d8c7a17b57c4ee7336314ce9e\": container with ID starting with 362365845d0e40cbd118f4e94e761c771496f90d8c7a17b57c4ee7336314ce9e not found: ID does not exist" containerID="362365845d0e40cbd118f4e94e761c771496f90d8c7a17b57c4ee7336314ce9e" Nov 03 14:05:45.862626 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:45.862555 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"362365845d0e40cbd118f4e94e761c771496f90d8c7a17b57c4ee7336314ce9e"} err="failed to get container status \"362365845d0e40cbd118f4e94e761c771496f90d8c7a17b57c4ee7336314ce9e\": rpc error: code = NotFound desc = could not find container \"362365845d0e40cbd118f4e94e761c771496f90d8c7a17b57c4ee7336314ce9e\": container with ID starting with 362365845d0e40cbd118f4e94e761c771496f90d8c7a17b57c4ee7336314ce9e not found: ID does not exist" Nov 03 14:05:46.012733 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:46.012646 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ae248d-6d79d84655-b88z9"] Nov 03 14:05:46.016569 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:46.016543 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ae248d-6d79d84655-b88z9"] Nov 03 14:05:46.116055 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:46.116024 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c5348d9-47e4-428f-b789-558b167b9ae5" path="/var/lib/kubelet/pods/2c5348d9-47e4-428f-b789-558b167b9ae5/volumes" Nov 03 14:05:46.120330 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:46.120291 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ae248d-7456846d9f-wntx2"] Nov 03 14:05:46.124650 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:46.124625 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ae248d-7456846d9f-wntx2"] Nov 03 14:05:48.115782 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:48.115750 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acda8eb6-a116-4a80-8572-4fc538ddb5b6" path="/var/lib/kubelet/pods/acda8eb6-a116-4a80-8572-4fc538ddb5b6/volumes" Nov 03 14:05:50.349143 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:50.349091 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-6ebe57-58896bdfff-trblz" podUID="d9ead347-e7c9-4fbc-b59f-6e137e8a979b" containerName="payload-processing" containerID="cri-o://9d2a7efdeaf6314aa081e098256bd3e0470937fa9ba1e378f6f59fb6a04d063e" gracePeriod=25 Nov 03 14:05:50.461127 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:50.461090 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6ebe57-98fcc7dd7-rf945" podUID="680e1deb-10e9-4074-9b16-b741e3655e63" containerName="payload-pre-processing" containerID="cri-o://c2486f58f90898c5593ad78e6772de76ecd78cc7b0106ec49282d6de09f93465" gracePeriod=25 Nov 03 14:05:50.624360 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:50.624285 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-6ebe57-58896bdfff-trblz" Nov 03 14:05:50.675688 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:50.675656 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9ead347-e7c9-4fbc-b59f-6e137e8a979b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9ead347-e7c9-4fbc-b59f-6e137e8a979b-plugins-config-volume\") pod \"d9ead347-e7c9-4fbc-b59f-6e137e8a979b\" (UID: \"d9ead347-e7c9-4fbc-b59f-6e137e8a979b\") " Nov 03 14:05:50.675885 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:50.675794 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9ead347-e7c9-4fbc-b59f-6e137e8a979b-kube-api-access-wdvln\" (UniqueName: \"kubernetes.io/projected/d9ead347-e7c9-4fbc-b59f-6e137e8a979b-kube-api-access-wdvln\") pod \"d9ead347-e7c9-4fbc-b59f-6e137e8a979b\" (UID: \"d9ead347-e7c9-4fbc-b59f-6e137e8a979b\") " Nov 03 14:05:50.676119 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:50.676089 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9ead347-e7c9-4fbc-b59f-6e137e8a979b-plugins-config-volume" pod "d9ead347-e7c9-4fbc-b59f-6e137e8a979b" (UID: "d9ead347-e7c9-4fbc-b59f-6e137e8a979b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:05:50.676382 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:50.676357 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9ead347-e7c9-4fbc-b59f-6e137e8a979b-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:05:50.677990 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:50.677961 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9ead347-e7c9-4fbc-b59f-6e137e8a979b-kube-api-access-wdvln" pod "d9ead347-e7c9-4fbc-b59f-6e137e8a979b" (UID: "d9ead347-e7c9-4fbc-b59f-6e137e8a979b"). InnerVolumeSpecName "kube-api-access-wdvln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:05:50.721939 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:50.721913 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6ebe57-98fcc7dd7-rf945" Nov 03 14:05:50.776943 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:50.776912 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/680e1deb-10e9-4074-9b16-b741e3655e63-kube-api-access-2pjnc\" (UniqueName: \"kubernetes.io/projected/680e1deb-10e9-4074-9b16-b741e3655e63-kube-api-access-2pjnc\") pod \"680e1deb-10e9-4074-9b16-b741e3655e63\" (UID: \"680e1deb-10e9-4074-9b16-b741e3655e63\") " Nov 03 14:05:50.777149 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:50.776974 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/680e1deb-10e9-4074-9b16-b741e3655e63-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/680e1deb-10e9-4074-9b16-b741e3655e63-plugins-config-volume\") pod \"680e1deb-10e9-4074-9b16-b741e3655e63\" (UID: \"680e1deb-10e9-4074-9b16-b741e3655e63\") " Nov 03 14:05:50.777397 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:50.777379 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdvln\" (UniqueName: \"kubernetes.io/projected/d9ead347-e7c9-4fbc-b59f-6e137e8a979b-kube-api-access-wdvln\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:05:50.777480 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:50.777415 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/680e1deb-10e9-4074-9b16-b741e3655e63-plugins-config-volume" pod "680e1deb-10e9-4074-9b16-b741e3655e63" (UID: "680e1deb-10e9-4074-9b16-b741e3655e63"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:05:50.779078 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:50.779046 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/680e1deb-10e9-4074-9b16-b741e3655e63-kube-api-access-2pjnc" pod "680e1deb-10e9-4074-9b16-b741e3655e63" (UID: "680e1deb-10e9-4074-9b16-b741e3655e63"). InnerVolumeSpecName "kube-api-access-2pjnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:05:50.878642 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:50.878559 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pjnc\" (UniqueName: \"kubernetes.io/projected/680e1deb-10e9-4074-9b16-b741e3655e63-kube-api-access-2pjnc\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:05:50.878642 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:50.878594 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/680e1deb-10e9-4074-9b16-b741e3655e63-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:05:50.912881 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:50.912849 5979 generic.go:356] "Generic (PLEG): container finished" podID="d9ead347-e7c9-4fbc-b59f-6e137e8a979b" containerID="9d2a7efdeaf6314aa081e098256bd3e0470937fa9ba1e378f6f59fb6a04d063e" exitCode=0 Nov 03 14:05:50.913054 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:50.912889 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-6ebe57-58896bdfff-trblz" Nov 03 14:05:50.913054 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:50.912937 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-6ebe57-58896bdfff-trblz" event={"ID":"d9ead347-e7c9-4fbc-b59f-6e137e8a979b","Type":"ContainerDied","Data":"9d2a7efdeaf6314aa081e098256bd3e0470937fa9ba1e378f6f59fb6a04d063e"} Nov 03 14:05:50.913054 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:50.912988 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-6ebe57-58896bdfff-trblz" event={"ID":"d9ead347-e7c9-4fbc-b59f-6e137e8a979b","Type":"ContainerDied","Data":"4f160147160746cf89da963172856733274967e39d1bdfca2b9a562b44e03e53"} Nov 03 14:05:50.913054 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:50.913018 5979 scope.go:122] "RemoveContainer" containerID="9d2a7efdeaf6314aa081e098256bd3e0470937fa9ba1e378f6f59fb6a04d063e" Nov 03 14:05:50.914677 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:50.914649 5979 generic.go:356] "Generic (PLEG): container finished" podID="680e1deb-10e9-4074-9b16-b741e3655e63" containerID="c2486f58f90898c5593ad78e6772de76ecd78cc7b0106ec49282d6de09f93465" exitCode=0 Nov 03 14:05:50.914801 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:50.914723 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6ebe57-98fcc7dd7-rf945" Nov 03 14:05:50.914866 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:50.914727 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6ebe57-98fcc7dd7-rf945" event={"ID":"680e1deb-10e9-4074-9b16-b741e3655e63","Type":"ContainerDied","Data":"c2486f58f90898c5593ad78e6772de76ecd78cc7b0106ec49282d6de09f93465"} Nov 03 14:05:50.914925 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:50.914869 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6ebe57-98fcc7dd7-rf945" event={"ID":"680e1deb-10e9-4074-9b16-b741e3655e63","Type":"ContainerDied","Data":"935b05bb6e1d4df37b34177072e448da8b677c4c122744e25f55fcca9cab1329"} Nov 03 14:05:50.923648 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:50.923519 5979 scope.go:122] "RemoveContainer" containerID="9d2a7efdeaf6314aa081e098256bd3e0470937fa9ba1e378f6f59fb6a04d063e" Nov 03 14:05:50.923841 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:05:50.923814 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d2a7efdeaf6314aa081e098256bd3e0470937fa9ba1e378f6f59fb6a04d063e\": container with ID starting with 9d2a7efdeaf6314aa081e098256bd3e0470937fa9ba1e378f6f59fb6a04d063e not found: ID does not exist" containerID="9d2a7efdeaf6314aa081e098256bd3e0470937fa9ba1e378f6f59fb6a04d063e" Nov 03 14:05:50.923964 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:50.923850 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d2a7efdeaf6314aa081e098256bd3e0470937fa9ba1e378f6f59fb6a04d063e"} err="failed to get container status \"9d2a7efdeaf6314aa081e098256bd3e0470937fa9ba1e378f6f59fb6a04d063e\": rpc error: code = NotFound desc = could not find container \"9d2a7efdeaf6314aa081e098256bd3e0470937fa9ba1e378f6f59fb6a04d063e\": container with ID starting with 9d2a7efdeaf6314aa081e098256bd3e0470937fa9ba1e378f6f59fb6a04d063e not found: ID does not exist" Nov 03 14:05:50.923964 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:50.923874 5979 scope.go:122] "RemoveContainer" containerID="c2486f58f90898c5593ad78e6772de76ecd78cc7b0106ec49282d6de09f93465" Nov 03 14:05:50.934463 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:50.934444 5979 scope.go:122] "RemoveContainer" containerID="c2486f58f90898c5593ad78e6772de76ecd78cc7b0106ec49282d6de09f93465" Nov 03 14:05:50.934734 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:05:50.934712 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2486f58f90898c5593ad78e6772de76ecd78cc7b0106ec49282d6de09f93465\": container with ID starting with c2486f58f90898c5593ad78e6772de76ecd78cc7b0106ec49282d6de09f93465 not found: ID does not exist" containerID="c2486f58f90898c5593ad78e6772de76ecd78cc7b0106ec49282d6de09f93465" Nov 03 14:05:50.934803 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:50.934742 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2486f58f90898c5593ad78e6772de76ecd78cc7b0106ec49282d6de09f93465"} err="failed to get container status \"c2486f58f90898c5593ad78e6772de76ecd78cc7b0106ec49282d6de09f93465\": rpc error: code = NotFound desc = could not find container \"c2486f58f90898c5593ad78e6772de76ecd78cc7b0106ec49282d6de09f93465\": container with ID starting with c2486f58f90898c5593ad78e6772de76ecd78cc7b0106ec49282d6de09f93465 not found: ID does not exist" Nov 03 14:05:51.067579 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:51.067540 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6ebe57-98fcc7dd7-rf945"] Nov 03 14:05:51.070368 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:51.070335 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6ebe57-98fcc7dd7-rf945"] Nov 03 14:05:51.171266 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:51.171228 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6ebe57-58896bdfff-trblz"] Nov 03 14:05:51.176115 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:51.176087 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6ebe57-58896bdfff-trblz"] Nov 03 14:05:52.116173 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:52.116142 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="680e1deb-10e9-4074-9b16-b741e3655e63" path="/var/lib/kubelet/pods/680e1deb-10e9-4074-9b16-b741e3655e63/volumes" Nov 03 14:05:52.116622 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:05:52.116585 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9ead347-e7c9-4fbc-b59f-6e137e8a979b" path="/var/lib/kubelet/pods/d9ead347-e7c9-4fbc-b59f-6e137e8a979b/volumes" Nov 03 14:06:00.170979 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:00.170937 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-58f896-ttlqv"] Nov 03 14:06:00.171691 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:00.171675 5979 cpu_manager.go:414] "containerMap: removing container" podUID="acda8eb6-a116-4a80-8572-4fc538ddb5b6" containerName="payload-pre-processing" Nov 03 14:06:00.171748 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:00.171692 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acda8eb6-a116-4a80-8572-4fc538ddb5b6" containerName="payload-pre-processing" Nov 03 14:06:00.171807 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:00.171796 5979 cpu_manager.go:414] "containerMap: removing container" podUID="2c5348d9-47e4-428f-b789-558b167b9ae5" containerName="payload-processing" Nov 03 14:06:00.171850 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:00.171806 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c5348d9-47e4-428f-b789-558b167b9ae5" containerName="payload-processing" Nov 03 14:06:00.171850 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:00.171820 5979 cpu_manager.go:414] "containerMap: removing container" podUID="d9ead347-e7c9-4fbc-b59f-6e137e8a979b" containerName="payload-processing" Nov 03 14:06:00.171850 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:00.171825 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9ead347-e7c9-4fbc-b59f-6e137e8a979b" containerName="payload-processing" Nov 03 14:06:00.171850 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:00.171845 5979 cpu_manager.go:414] "containerMap: removing container" podUID="680e1deb-10e9-4074-9b16-b741e3655e63" containerName="payload-pre-processing" Nov 03 14:06:00.172006 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:00.171852 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="680e1deb-10e9-4074-9b16-b741e3655e63" containerName="payload-pre-processing" Nov 03 14:06:00.172060 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:00.172045 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="acda8eb6-a116-4a80-8572-4fc538ddb5b6" containerName="payload-pre-processing" Nov 03 14:06:00.172099 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:00.172074 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c5348d9-47e4-428f-b789-558b167b9ae5" containerName="payload-processing" Nov 03 14:06:00.172099 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:00.172082 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9ead347-e7c9-4fbc-b59f-6e137e8a979b" containerName="payload-processing" Nov 03 14:06:00.172099 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:00.172089 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="680e1deb-10e9-4074-9b16-b741e3655e63" containerName="payload-pre-processing" Nov 03 14:06:00.174859 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:00.174840 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-58f896-ttlqv" Nov 03 14:06:00.271586 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:00.271554 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:06:00.271775 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:00.271689 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 14:06:00.274579 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:00.274554 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-58f896-ttlqv"] Nov 03 14:06:00.330980 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:00.330861 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fa8b828-eff2-45b5-9027-84cbce81a3f4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-58f896-ttlqv\" (UID: \"3fa8b828-eff2-45b5-9027-84cbce81a3f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-58f896-ttlqv" Nov 03 14:06:00.331383 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:00.331362 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5ccp\" (UniqueName: \"kubernetes.io/projected/3fa8b828-eff2-45b5-9027-84cbce81a3f4-kube-api-access-n5ccp\") pod \"maas-api-revoke-keys-e2e-extmod-a-58f896-ttlqv\" (UID: \"3fa8b828-eff2-45b5-9027-84cbce81a3f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-58f896-ttlqv" Nov 03 14:06:00.331792 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:00.331770 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fa8b828-eff2-45b5-9027-84cbce81a3f4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-58f896-ttlqv\" (UID: \"3fa8b828-eff2-45b5-9027-84cbce81a3f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-58f896-ttlqv" Nov 03 14:06:00.433658 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:00.433579 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fa8b828-eff2-45b5-9027-84cbce81a3f4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-58f896-ttlqv\" (UID: \"3fa8b828-eff2-45b5-9027-84cbce81a3f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-58f896-ttlqv" Nov 03 14:06:00.433829 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:00.433664 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fa8b828-eff2-45b5-9027-84cbce81a3f4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-58f896-ttlqv\" (UID: \"3fa8b828-eff2-45b5-9027-84cbce81a3f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-58f896-ttlqv" Nov 03 14:06:00.433829 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:00.433765 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5ccp\" (UniqueName: \"kubernetes.io/projected/3fa8b828-eff2-45b5-9027-84cbce81a3f4-kube-api-access-n5ccp\") pod \"maas-api-revoke-keys-e2e-extmod-a-58f896-ttlqv\" (UID: \"3fa8b828-eff2-45b5-9027-84cbce81a3f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-58f896-ttlqv" Nov 03 14:06:00.434318 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:00.434294 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fa8b828-eff2-45b5-9027-84cbce81a3f4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-58f896-ttlqv\" (UID: \"3fa8b828-eff2-45b5-9027-84cbce81a3f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-58f896-ttlqv" Nov 03 14:06:00.436031 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:00.436009 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fa8b828-eff2-45b5-9027-84cbce81a3f4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-58f896-ttlqv\" (UID: \"3fa8b828-eff2-45b5-9027-84cbce81a3f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-58f896-ttlqv" Nov 03 14:06:00.495193 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:00.493280 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5ccp\" (UniqueName: \"kubernetes.io/projected/3fa8b828-eff2-45b5-9027-84cbce81a3f4-kube-api-access-n5ccp\") pod \"maas-api-revoke-keys-e2e-extmod-a-58f896-ttlqv\" (UID: \"3fa8b828-eff2-45b5-9027-84cbce81a3f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-58f896-ttlqv" Nov 03 14:06:00.788666 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:00.788576 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-58f896-ttlqv" Nov 03 14:06:01.168393 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:01.168350 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-58f896-ttlqv"] Nov 03 14:06:01.175396 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:06:01.175369 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fa8b828_eff2_45b5_9027_84cbce81a3f4.slice/crio-1623cf770ebd79d2a3ed4d5d8c449818ebfe5e9ca9c439268b1a6e48cf863131 WatchSource:0}: Error finding container 1623cf770ebd79d2a3ed4d5d8c449818ebfe5e9ca9c439268b1a6e48cf863131: Status 404 returned error can't find the container with id 1623cf770ebd79d2a3ed4d5d8c449818ebfe5e9ca9c439268b1a6e48cf863131 Nov 03 14:06:01.991408 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:01.991369 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-58f896-ttlqv" event={"ID":"3fa8b828-eff2-45b5-9027-84cbce81a3f4","Type":"ContainerStarted","Data":"a6bf3aea33b38fe5f3627446cfbd115b66e64be1dac21b1d6683e9d1511f0db2"} Nov 03 14:06:01.991661 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:01.991435 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-58f896-ttlqv" event={"ID":"3fa8b828-eff2-45b5-9027-84cbce81a3f4","Type":"ContainerStarted","Data":"1623cf770ebd79d2a3ed4d5d8c449818ebfe5e9ca9c439268b1a6e48cf863131"} Nov 03 14:06:02.136531 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:02.136478 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-58f896-ttlqv" podStartSLOduration=2.136453538 podStartE2EDuration="2.136453538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:06:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:06:02.099378009 +0000 UTC m=+3644.110066254" watchObservedRunningTime="2026-11-03 14:06:02.136453538 +0000 UTC m=+3644.147141780" Nov 03 14:06:02.999615 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:02.999581 5979 generic.go:356] "Generic (PLEG): container finished" podID="3fa8b828-eff2-45b5-9027-84cbce81a3f4" containerID="a6bf3aea33b38fe5f3627446cfbd115b66e64be1dac21b1d6683e9d1511f0db2" exitCode=0 Nov 03 14:06:03.000047 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:02.999672 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-58f896-ttlqv" event={"ID":"3fa8b828-eff2-45b5-9027-84cbce81a3f4","Type":"ContainerDied","Data":"a6bf3aea33b38fe5f3627446cfbd115b66e64be1dac21b1d6683e9d1511f0db2"} Nov 03 14:06:04.178863 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:04.178831 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-58f896-ttlqv" Nov 03 14:06:04.287154 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:04.287058 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fa8b828-eff2-45b5-9027-84cbce81a3f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fa8b828-eff2-45b5-9027-84cbce81a3f4-maas-api-xks-ca\") pod \"3fa8b828-eff2-45b5-9027-84cbce81a3f4\" (UID: \"3fa8b828-eff2-45b5-9027-84cbce81a3f4\") " Nov 03 14:06:04.287447 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:04.287420 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fa8b828-eff2-45b5-9027-84cbce81a3f4-kube-api-access-n5ccp\" (UniqueName: \"kubernetes.io/projected/3fa8b828-eff2-45b5-9027-84cbce81a3f4-kube-api-access-n5ccp\") pod \"3fa8b828-eff2-45b5-9027-84cbce81a3f4\" (UID: \"3fa8b828-eff2-45b5-9027-84cbce81a3f4\") " Nov 03 14:06:04.287775 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:04.287754 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fa8b828-eff2-45b5-9027-84cbce81a3f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fa8b828-eff2-45b5-9027-84cbce81a3f4-maas-api-openshift-ca\") pod \"3fa8b828-eff2-45b5-9027-84cbce81a3f4\" (UID: \"3fa8b828-eff2-45b5-9027-84cbce81a3f4\") " Nov 03 14:06:04.288351 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:04.288315 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fa8b828-eff2-45b5-9027-84cbce81a3f4-maas-api-openshift-ca" pod "3fa8b828-eff2-45b5-9027-84cbce81a3f4" (UID: "3fa8b828-eff2-45b5-9027-84cbce81a3f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:06:04.289217 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:04.289187 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fa8b828-eff2-45b5-9027-84cbce81a3f4-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:06:04.290601 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:04.290575 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fa8b828-eff2-45b5-9027-84cbce81a3f4-maas-api-xks-ca" pod "3fa8b828-eff2-45b5-9027-84cbce81a3f4" (UID: "3fa8b828-eff2-45b5-9027-84cbce81a3f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:06:04.291390 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:04.291356 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fa8b828-eff2-45b5-9027-84cbce81a3f4-kube-api-access-n5ccp" pod "3fa8b828-eff2-45b5-9027-84cbce81a3f4" (UID: "3fa8b828-eff2-45b5-9027-84cbce81a3f4"). InnerVolumeSpecName "kube-api-access-n5ccp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:06:04.391000 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:04.390960 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fa8b828-eff2-45b5-9027-84cbce81a3f4-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:06:04.391000 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:04.390995 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5ccp\" (UniqueName: \"kubernetes.io/projected/3fa8b828-eff2-45b5-9027-84cbce81a3f4-kube-api-access-n5ccp\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:06:05.014373 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:05.014340 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-58f896-ttlqv" Nov 03 14:06:05.014595 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:05.014374 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-58f896-ttlqv" event={"ID":"3fa8b828-eff2-45b5-9027-84cbce81a3f4","Type":"ContainerDied","Data":"1623cf770ebd79d2a3ed4d5d8c449818ebfe5e9ca9c439268b1a6e48cf863131"} Nov 03 14:06:05.014595 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:05.014457 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1623cf770ebd79d2a3ed4d5d8c449818ebfe5e9ca9c439268b1a6e48cf863131" Nov 03 14:06:06.721688 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:06.721651 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb"] Nov 03 14:06:06.722128 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:06.721854 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb" podUID="9ded222c-3a0a-4c3a-bd70-493d97638c99" containerName="istio-proxy" containerID="cri-o://f845790cd0005261edb6dce0ea61f0db17fa5835eb201c2caf78854eb37e2e68" gracePeriod=30 Nov 03 14:06:07.234923 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:06:07.234886 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:06:12.001322 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:12.001259 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb" Nov 03 14:06:12.062157 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:12.062119 5979 generic.go:356] "Generic (PLEG): container finished" podID="9ded222c-3a0a-4c3a-bd70-493d97638c99" containerID="f845790cd0005261edb6dce0ea61f0db17fa5835eb201c2caf78854eb37e2e68" exitCode=0 Nov 03 14:06:12.062361 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:12.062183 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb" event={"ID":"9ded222c-3a0a-4c3a-bd70-493d97638c99","Type":"ContainerDied","Data":"f845790cd0005261edb6dce0ea61f0db17fa5835eb201c2caf78854eb37e2e68"} Nov 03 14:06:12.062361 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:12.062198 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb" Nov 03 14:06:12.062553 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:12.062381 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb" event={"ID":"9ded222c-3a0a-4c3a-bd70-493d97638c99","Type":"ContainerDied","Data":"4f32e1c888abee0935c13f663cd247e58be61da799df73f50fd1cead9d413fe3"} Nov 03 14:06:12.062553 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:12.062411 5979 scope.go:122] "RemoveContainer" containerID="f845790cd0005261edb6dce0ea61f0db17fa5835eb201c2caf78854eb37e2e68" Nov 03 14:06:12.073228 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:12.073211 5979 scope.go:122] "RemoveContainer" containerID="a0c8bf7f848148181fcfbf15c36f59265eab995cf50b73913c6179ec369dcd41" Nov 03 14:06:12.082297 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:12.082280 5979 scope.go:122] "RemoveContainer" containerID="f845790cd0005261edb6dce0ea61f0db17fa5835eb201c2caf78854eb37e2e68" Nov 03 14:06:12.082566 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:06:12.082547 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f845790cd0005261edb6dce0ea61f0db17fa5835eb201c2caf78854eb37e2e68\": container with ID starting with f845790cd0005261edb6dce0ea61f0db17fa5835eb201c2caf78854eb37e2e68 not found: ID does not exist" containerID="f845790cd0005261edb6dce0ea61f0db17fa5835eb201c2caf78854eb37e2e68" Nov 03 14:06:12.082627 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:12.082574 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f845790cd0005261edb6dce0ea61f0db17fa5835eb201c2caf78854eb37e2e68"} err="failed to get container status \"f845790cd0005261edb6dce0ea61f0db17fa5835eb201c2caf78854eb37e2e68\": rpc error: code = NotFound desc = could not find container \"f845790cd0005261edb6dce0ea61f0db17fa5835eb201c2caf78854eb37e2e68\": container with ID starting with f845790cd0005261edb6dce0ea61f0db17fa5835eb201c2caf78854eb37e2e68 not found: ID does not exist" Nov 03 14:06:12.082627 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:12.082591 5979 scope.go:122] "RemoveContainer" containerID="a0c8bf7f848148181fcfbf15c36f59265eab995cf50b73913c6179ec369dcd41" Nov 03 14:06:12.082824 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:06:12.082809 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0c8bf7f848148181fcfbf15c36f59265eab995cf50b73913c6179ec369dcd41\": container with ID starting with a0c8bf7f848148181fcfbf15c36f59265eab995cf50b73913c6179ec369dcd41 not found: ID does not exist" containerID="a0c8bf7f848148181fcfbf15c36f59265eab995cf50b73913c6179ec369dcd41" Nov 03 14:06:12.082868 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:12.082829 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0c8bf7f848148181fcfbf15c36f59265eab995cf50b73913c6179ec369dcd41"} err="failed to get container status \"a0c8bf7f848148181fcfbf15c36f59265eab995cf50b73913c6179ec369dcd41\": rpc error: code = NotFound desc = could not find container \"a0c8bf7f848148181fcfbf15c36f59265eab995cf50b73913c6179ec369dcd41\": container with ID starting with a0c8bf7f848148181fcfbf15c36f59265eab995cf50b73913c6179ec369dcd41 not found: ID does not exist" Nov 03 14:06:12.092318 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:12.092296 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ded222c-3a0a-4c3a-bd70-493d97638c99-istio-token\" (UniqueName: \"kubernetes.io/projected/9ded222c-3a0a-4c3a-bd70-493d97638c99-istio-token\") pod \"9ded222c-3a0a-4c3a-bd70-493d97638c99\" (UID: \"9ded222c-3a0a-4c3a-bd70-493d97638c99\") " Nov 03 14:06:12.092402 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:12.092343 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ded222c-3a0a-4c3a-bd70-493d97638c99-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ded222c-3a0a-4c3a-bd70-493d97638c99-workload-certs\") pod \"9ded222c-3a0a-4c3a-bd70-493d97638c99\" (UID: \"9ded222c-3a0a-4c3a-bd70-493d97638c99\") " Nov 03 14:06:12.092402 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:12.092389 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ded222c-3a0a-4c3a-bd70-493d97638c99-kube-api-access-zhsgf\" (UniqueName: \"kubernetes.io/projected/9ded222c-3a0a-4c3a-bd70-493d97638c99-kube-api-access-zhsgf\") pod \"9ded222c-3a0a-4c3a-bd70-493d97638c99\" (UID: \"9ded222c-3a0a-4c3a-bd70-493d97638c99\") " Nov 03 14:06:12.092506 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:12.092441 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ded222c-3a0a-4c3a-bd70-493d97638c99-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ded222c-3a0a-4c3a-bd70-493d97638c99-credential-socket\") pod \"9ded222c-3a0a-4c3a-bd70-493d97638c99\" (UID: \"9ded222c-3a0a-4c3a-bd70-493d97638c99\") " Nov 03 14:06:12.092636 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:12.092619 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9ded222c-3a0a-4c3a-bd70-493d97638c99-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ded222c-3a0a-4c3a-bd70-493d97638c99-istio-podinfo\") pod \"9ded222c-3a0a-4c3a-bd70-493d97638c99\" (UID: \"9ded222c-3a0a-4c3a-bd70-493d97638c99\") " Nov 03 14:06:12.092694 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:12.092678 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ded222c-3a0a-4c3a-bd70-493d97638c99-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ded222c-3a0a-4c3a-bd70-493d97638c99-workload-socket\") pod \"9ded222c-3a0a-4c3a-bd70-493d97638c99\" (UID: \"9ded222c-3a0a-4c3a-bd70-493d97638c99\") " Nov 03 14:06:12.092760 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:12.092714 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ded222c-3a0a-4c3a-bd70-493d97638c99-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ded222c-3a0a-4c3a-bd70-493d97638c99-istio-data\") pod \"9ded222c-3a0a-4c3a-bd70-493d97638c99\" (UID: \"9ded222c-3a0a-4c3a-bd70-493d97638c99\") " Nov 03 14:06:12.092760 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:12.092718 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ded222c-3a0a-4c3a-bd70-493d97638c99-credential-socket" pod "9ded222c-3a0a-4c3a-bd70-493d97638c99" (UID: "9ded222c-3a0a-4c3a-bd70-493d97638c99"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:06:12.092760 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:12.092725 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ded222c-3a0a-4c3a-bd70-493d97638c99-workload-certs" pod "9ded222c-3a0a-4c3a-bd70-493d97638c99" (UID: "9ded222c-3a0a-4c3a-bd70-493d97638c99"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:06:12.092760 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:12.092745 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ded222c-3a0a-4c3a-bd70-493d97638c99-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ded222c-3a0a-4c3a-bd70-493d97638c99-istio-envoy\") pod \"9ded222c-3a0a-4c3a-bd70-493d97638c99\" (UID: \"9ded222c-3a0a-4c3a-bd70-493d97638c99\") " Nov 03 14:06:12.093023 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:12.092863 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ded222c-3a0a-4c3a-bd70-493d97638c99-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ded222c-3a0a-4c3a-bd70-493d97638c99-istiod-ca-cert\") pod \"9ded222c-3a0a-4c3a-bd70-493d97638c99\" (UID: \"9ded222c-3a0a-4c3a-bd70-493d97638c99\") " Nov 03 14:06:12.093023 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:12.092915 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ded222c-3a0a-4c3a-bd70-493d97638c99-istio-data" pod "9ded222c-3a0a-4c3a-bd70-493d97638c99" (UID: "9ded222c-3a0a-4c3a-bd70-493d97638c99"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:06:12.093023 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:12.092924 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ded222c-3a0a-4c3a-bd70-493d97638c99-workload-socket" pod "9ded222c-3a0a-4c3a-bd70-493d97638c99" (UID: "9ded222c-3a0a-4c3a-bd70-493d97638c99"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:06:12.093308 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:12.093282 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ded222c-3a0a-4c3a-bd70-493d97638c99-istiod-ca-cert" pod "9ded222c-3a0a-4c3a-bd70-493d97638c99" (UID: "9ded222c-3a0a-4c3a-bd70-493d97638c99"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:06:12.093395 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:12.093362 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ded222c-3a0a-4c3a-bd70-493d97638c99-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:06:12.093395 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:12.093385 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ded222c-3a0a-4c3a-bd70-493d97638c99-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:06:12.093708 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:12.093399 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ded222c-3a0a-4c3a-bd70-493d97638c99-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:06:12.093708 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:12.093414 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ded222c-3a0a-4c3a-bd70-493d97638c99-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:06:12.094727 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:12.094701 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ded222c-3a0a-4c3a-bd70-493d97638c99-kube-api-access-zhsgf" pod "9ded222c-3a0a-4c3a-bd70-493d97638c99" (UID: "9ded222c-3a0a-4c3a-bd70-493d97638c99"). InnerVolumeSpecName "kube-api-access-zhsgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:06:12.094816 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:12.094745 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ded222c-3a0a-4c3a-bd70-493d97638c99-istio-token" pod "9ded222c-3a0a-4c3a-bd70-493d97638c99" (UID: "9ded222c-3a0a-4c3a-bd70-493d97638c99"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:06:12.094997 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:12.094975 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ded222c-3a0a-4c3a-bd70-493d97638c99-istio-envoy" pod "9ded222c-3a0a-4c3a-bd70-493d97638c99" (UID: "9ded222c-3a0a-4c3a-bd70-493d97638c99"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:06:12.095062 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:12.094993 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ded222c-3a0a-4c3a-bd70-493d97638c99-istio-podinfo" pod "9ded222c-3a0a-4c3a-bd70-493d97638c99" (UID: "9ded222c-3a0a-4c3a-bd70-493d97638c99"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:06:12.194160 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:12.194125 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ded222c-3a0a-4c3a-bd70-493d97638c99-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:06:12.194160 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:12.194158 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ded222c-3a0a-4c3a-bd70-493d97638c99-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:06:12.194369 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:12.194169 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ded222c-3a0a-4c3a-bd70-493d97638c99-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:06:12.194369 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:12.194180 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ded222c-3a0a-4c3a-bd70-493d97638c99-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:06:12.194369 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:12.194189 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhsgf\" (UniqueName: \"kubernetes.io/projected/9ded222c-3a0a-4c3a-bd70-493d97638c99-kube-api-access-zhsgf\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:06:12.470940 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:12.470906 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb"] Nov 03 14:06:12.475274 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:12.475244 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-6ebe57-openshift-default-f8cd59df7-tn7xb"] Nov 03 14:06:13.287119 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:13.287083 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c4e398-5b4997669f-bxk4r"] Nov 03 14:06:14.116521 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:14.116484 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ded222c-3a0a-4c3a-bd70-493d97638c99" path="/var/lib/kubelet/pods/9ded222c-3a0a-4c3a-bd70-493d97638c99/volumes" Nov 03 14:06:18.177164 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:18.177118 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7469666f9f-vpmg2"] Nov 03 14:06:18.177866 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:18.177848 5979 cpu_manager.go:414] "containerMap: removing container" podUID="9ded222c-3a0a-4c3a-bd70-493d97638c99" containerName="istio-proxy" Nov 03 14:06:18.177913 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:18.177867 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ded222c-3a0a-4c3a-bd70-493d97638c99" containerName="istio-proxy" Nov 03 14:06:18.177955 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:18.177915 5979 cpu_manager.go:414] "containerMap: removing container" podUID="3fa8b828-eff2-45b5-9027-84cbce81a3f4" containerName="revoke-keys" Nov 03 14:06:18.177955 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:18.177920 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fa8b828-eff2-45b5-9027-84cbce81a3f4" containerName="revoke-keys" Nov 03 14:06:18.178045 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:18.178032 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fa8b828-eff2-45b5-9027-84cbce81a3f4" containerName="revoke-keys" Nov 03 14:06:18.178082 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:18.178054 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ded222c-3a0a-4c3a-bd70-493d97638c99" containerName="istio-proxy" Nov 03 14:06:18.178082 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:18.178064 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ded222c-3a0a-4c3a-bd70-493d97638c99" containerName="istio-proxy" Nov 03 14:06:18.183187 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:18.183167 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7469666f9f-vpmg2" Nov 03 14:06:18.234451 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:18.234417 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7469666f9f-vpmg2"] Nov 03 14:06:18.277216 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:18.277180 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 14:06:18.277216 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:18.277196 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-e86f9ab11580\"" Nov 03 14:06:18.277216 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:18.277206 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 14:06:18.277457 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:18.277275 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 14:06:18.278682 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:18.278662 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-n87hv\"" Nov 03 14:06:18.278757 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:18.278744 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 14:06:18.287360 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:18.287335 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-c4e398-5b4997669f-bxk4r" podUID="84b8df5b-8fd5-4de1-819f-8b543a53ebf2" containerName="payload-processing" containerID="cri-o://f61cd932758e41bbeac4c5ea3ac5fba52bca7eeb08f82504cf20314cfb9d3856" gracePeriod=25 Nov 03 14:06:18.329815 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:18.329785 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/192eaf94-c0d0-4ce0-959e-5e297a072ab1-routing-overlay-volume\") pod \"payload-processing-external-model-7469666f9f-vpmg2\" (UID: \"192eaf94-c0d0-4ce0-959e-5e297a072ab1\") " pod="models-as-a-service/payload-processing-external-model-7469666f9f-vpmg2" Nov 03 14:06:18.329940 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:18.329835 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/192eaf94-c0d0-4ce0-959e-5e297a072ab1-plugins-config-volume\") pod \"payload-processing-external-model-7469666f9f-vpmg2\" (UID: \"192eaf94-c0d0-4ce0-959e-5e297a072ab1\") " pod="models-as-a-service/payload-processing-external-model-7469666f9f-vpmg2" Nov 03 14:06:18.330041 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:18.330020 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/192eaf94-c0d0-4ce0-959e-5e297a072ab1-provider-credentials\") pod \"payload-processing-external-model-7469666f9f-vpmg2\" (UID: \"192eaf94-c0d0-4ce0-959e-5e297a072ab1\") " pod="models-as-a-service/payload-processing-external-model-7469666f9f-vpmg2" Nov 03 14:06:18.330094 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:18.330062 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4mn4\" (UniqueName: \"kubernetes.io/projected/192eaf94-c0d0-4ce0-959e-5e297a072ab1-kube-api-access-b4mn4\") pod \"payload-processing-external-model-7469666f9f-vpmg2\" (UID: \"192eaf94-c0d0-4ce0-959e-5e297a072ab1\") " pod="models-as-a-service/payload-processing-external-model-7469666f9f-vpmg2" Nov 03 14:06:18.431063 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:18.430985 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/192eaf94-c0d0-4ce0-959e-5e297a072ab1-routing-overlay-volume\") pod \"payload-processing-external-model-7469666f9f-vpmg2\" (UID: \"192eaf94-c0d0-4ce0-959e-5e297a072ab1\") " pod="models-as-a-service/payload-processing-external-model-7469666f9f-vpmg2" Nov 03 14:06:18.431063 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:18.431041 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/192eaf94-c0d0-4ce0-959e-5e297a072ab1-plugins-config-volume\") pod \"payload-processing-external-model-7469666f9f-vpmg2\" (UID: \"192eaf94-c0d0-4ce0-959e-5e297a072ab1\") " pod="models-as-a-service/payload-processing-external-model-7469666f9f-vpmg2" Nov 03 14:06:18.431241 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:18.431165 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/192eaf94-c0d0-4ce0-959e-5e297a072ab1-provider-credentials\") pod \"payload-processing-external-model-7469666f9f-vpmg2\" (UID: \"192eaf94-c0d0-4ce0-959e-5e297a072ab1\") " pod="models-as-a-service/payload-processing-external-model-7469666f9f-vpmg2" Nov 03 14:06:18.431241 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:18.431197 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4mn4\" (UniqueName: \"kubernetes.io/projected/192eaf94-c0d0-4ce0-959e-5e297a072ab1-kube-api-access-b4mn4\") pod \"payload-processing-external-model-7469666f9f-vpmg2\" (UID: \"192eaf94-c0d0-4ce0-959e-5e297a072ab1\") " pod="models-as-a-service/payload-processing-external-model-7469666f9f-vpmg2" Nov 03 14:06:18.431836 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:18.431808 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/192eaf94-c0d0-4ce0-959e-5e297a072ab1-routing-overlay-volume\") pod \"payload-processing-external-model-7469666f9f-vpmg2\" (UID: \"192eaf94-c0d0-4ce0-959e-5e297a072ab1\") " pod="models-as-a-service/payload-processing-external-model-7469666f9f-vpmg2" Nov 03 14:06:18.431950 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:18.431887 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/192eaf94-c0d0-4ce0-959e-5e297a072ab1-plugins-config-volume\") pod \"payload-processing-external-model-7469666f9f-vpmg2\" (UID: \"192eaf94-c0d0-4ce0-959e-5e297a072ab1\") " pod="models-as-a-service/payload-processing-external-model-7469666f9f-vpmg2" Nov 03 14:06:18.446988 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:18.446952 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/192eaf94-c0d0-4ce0-959e-5e297a072ab1-provider-credentials\") pod \"payload-processing-external-model-7469666f9f-vpmg2\" (UID: \"192eaf94-c0d0-4ce0-959e-5e297a072ab1\") " pod="models-as-a-service/payload-processing-external-model-7469666f9f-vpmg2" Nov 03 14:06:18.484787 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:18.484755 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4mn4\" (UniqueName: \"kubernetes.io/projected/192eaf94-c0d0-4ce0-959e-5e297a072ab1-kube-api-access-b4mn4\") pod \"payload-processing-external-model-7469666f9f-vpmg2\" (UID: \"192eaf94-c0d0-4ce0-959e-5e297a072ab1\") " pod="models-as-a-service/payload-processing-external-model-7469666f9f-vpmg2" Nov 03 14:06:18.497392 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:18.497361 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7469666f9f-vpmg2" Nov 03 14:06:18.565672 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:18.565513 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c4e398-5b4997669f-bxk4r" Nov 03 14:06:18.634179 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:18.634149 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84b8df5b-8fd5-4de1-819f-8b543a53ebf2-kube-api-access-bfrfq\" (UniqueName: \"kubernetes.io/projected/84b8df5b-8fd5-4de1-819f-8b543a53ebf2-kube-api-access-bfrfq\") pod \"84b8df5b-8fd5-4de1-819f-8b543a53ebf2\" (UID: \"84b8df5b-8fd5-4de1-819f-8b543a53ebf2\") " Nov 03 14:06:18.634462 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:18.634348 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84b8df5b-8fd5-4de1-819f-8b543a53ebf2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84b8df5b-8fd5-4de1-819f-8b543a53ebf2-plugins-config-volume\") pod \"84b8df5b-8fd5-4de1-819f-8b543a53ebf2\" (UID: \"84b8df5b-8fd5-4de1-819f-8b543a53ebf2\") " Nov 03 14:06:18.635101 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:18.635071 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84b8df5b-8fd5-4de1-819f-8b543a53ebf2-plugins-config-volume" pod "84b8df5b-8fd5-4de1-819f-8b543a53ebf2" (UID: "84b8df5b-8fd5-4de1-819f-8b543a53ebf2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:06:18.636617 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:18.636580 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84b8df5b-8fd5-4de1-819f-8b543a53ebf2-kube-api-access-bfrfq" pod "84b8df5b-8fd5-4de1-819f-8b543a53ebf2" (UID: "84b8df5b-8fd5-4de1-819f-8b543a53ebf2"). InnerVolumeSpecName "kube-api-access-bfrfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:06:18.735503 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:18.735407 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84b8df5b-8fd5-4de1-819f-8b543a53ebf2-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:06:18.735503 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:18.735437 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfrfq\" (UniqueName: \"kubernetes.io/projected/84b8df5b-8fd5-4de1-819f-8b543a53ebf2-kube-api-access-bfrfq\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:06:18.888602 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:18.888567 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7469666f9f-vpmg2"] Nov 03 14:06:18.895330 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:06:18.895300 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod192eaf94_c0d0_4ce0_959e_5e297a072ab1.slice/crio-8eca31a0ea17b050b1332fc8e06d81b084c91306cfd139408309b0175ba7237c WatchSource:0}: Error finding container 8eca31a0ea17b050b1332fc8e06d81b084c91306cfd139408309b0175ba7237c: Status 404 returned error can't find the container with id 8eca31a0ea17b050b1332fc8e06d81b084c91306cfd139408309b0175ba7237c Nov 03 14:06:19.122848 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:19.122751 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7469666f9f-vpmg2" event={"ID":"192eaf94-c0d0-4ce0-959e-5e297a072ab1","Type":"ContainerStarted","Data":"3bd6092d13f0834afef2f365c244e2e320fee2045033bff865be01178aecc980"} Nov 03 14:06:19.122848 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:19.122806 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7469666f9f-vpmg2" event={"ID":"192eaf94-c0d0-4ce0-959e-5e297a072ab1","Type":"ContainerStarted","Data":"8eca31a0ea17b050b1332fc8e06d81b084c91306cfd139408309b0175ba7237c"} Nov 03 14:06:19.123085 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:19.122872 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7469666f9f-vpmg2" Nov 03 14:06:19.124302 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:19.124280 5979 generic.go:356] "Generic (PLEG): container finished" podID="84b8df5b-8fd5-4de1-819f-8b543a53ebf2" containerID="f61cd932758e41bbeac4c5ea3ac5fba52bca7eeb08f82504cf20314cfb9d3856" exitCode=0 Nov 03 14:06:19.124400 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:19.124317 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c4e398-5b4997669f-bxk4r" event={"ID":"84b8df5b-8fd5-4de1-819f-8b543a53ebf2","Type":"ContainerDied","Data":"f61cd932758e41bbeac4c5ea3ac5fba52bca7eeb08f82504cf20314cfb9d3856"} Nov 03 14:06:19.124400 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:19.124345 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c4e398-5b4997669f-bxk4r" event={"ID":"84b8df5b-8fd5-4de1-819f-8b543a53ebf2","Type":"ContainerDied","Data":"13bb1a495b7b5e70dbf5d22c01fd44ad31d625dfefcc9ad4cc0db38c560ceaad"} Nov 03 14:06:19.124400 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:19.124366 5979 scope.go:122] "RemoveContainer" containerID="f61cd932758e41bbeac4c5ea3ac5fba52bca7eeb08f82504cf20314cfb9d3856" Nov 03 14:06:19.124532 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:19.124318 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c4e398-5b4997669f-bxk4r" Nov 03 14:06:19.135464 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:19.135442 5979 scope.go:122] "RemoveContainer" containerID="f61cd932758e41bbeac4c5ea3ac5fba52bca7eeb08f82504cf20314cfb9d3856" Nov 03 14:06:19.135853 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:06:19.135832 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f61cd932758e41bbeac4c5ea3ac5fba52bca7eeb08f82504cf20314cfb9d3856\": container with ID starting with f61cd932758e41bbeac4c5ea3ac5fba52bca7eeb08f82504cf20314cfb9d3856 not found: ID does not exist" containerID="f61cd932758e41bbeac4c5ea3ac5fba52bca7eeb08f82504cf20314cfb9d3856" Nov 03 14:06:19.135965 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:19.135860 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f61cd932758e41bbeac4c5ea3ac5fba52bca7eeb08f82504cf20314cfb9d3856"} err="failed to get container status \"f61cd932758e41bbeac4c5ea3ac5fba52bca7eeb08f82504cf20314cfb9d3856\": rpc error: code = NotFound desc = could not find container \"f61cd932758e41bbeac4c5ea3ac5fba52bca7eeb08f82504cf20314cfb9d3856\": container with ID starting with f61cd932758e41bbeac4c5ea3ac5fba52bca7eeb08f82504cf20314cfb9d3856 not found: ID does not exist" Nov 03 14:06:19.231511 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:19.231431 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7469666f9f-vpmg2" podStartSLOduration=1.231415783 podStartE2EDuration="1.231415783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:06:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:06:19.230303223 +0000 UTC m=+3661.240991464" watchObservedRunningTime="2026-11-03 14:06:19.231415783 +0000 UTC m=+3661.242104025" Nov 03 14:06:19.277943 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:19.277552 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:06:19.343996 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:19.343964 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c4e398-5b4997669f-bxk4r"] Nov 03 14:06:19.353249 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:19.353220 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c4e398-5b4997669f-bxk4r"] Nov 03 14:06:19.449579 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:19.449545 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-58f896-ttlqv"] Nov 03 14:06:19.452328 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:19.452303 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-58f896-ttlqv"] Nov 03 14:06:20.116613 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:20.116579 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fa8b828-eff2-45b5-9027-84cbce81a3f4" path="/var/lib/kubelet/pods/3fa8b828-eff2-45b5-9027-84cbce81a3f4/volumes" Nov 03 14:06:20.117051 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:20.117036 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84b8df5b-8fd5-4de1-819f-8b543a53ebf2" path="/var/lib/kubelet/pods/84b8df5b-8fd5-4de1-819f-8b543a53ebf2/volumes" Nov 03 14:06:29.427404 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:29.427367 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-84dcd755bf-c9rnp"] Nov 03 14:06:29.428041 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:29.428002 5979 cpu_manager.go:414] "containerMap: removing container" podUID="84b8df5b-8fd5-4de1-819f-8b543a53ebf2" containerName="payload-processing" Nov 03 14:06:29.428041 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:29.428017 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84b8df5b-8fd5-4de1-819f-8b543a53ebf2" containerName="payload-processing" Nov 03 14:06:29.428041 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:29.428030 5979 cpu_manager.go:414] "containerMap: removing container" podUID="9ded222c-3a0a-4c3a-bd70-493d97638c99" containerName="istio-proxy" Nov 03 14:06:29.428041 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:29.428035 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ded222c-3a0a-4c3a-bd70-493d97638c99" containerName="istio-proxy" Nov 03 14:06:29.428268 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:29.428177 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="84b8df5b-8fd5-4de1-819f-8b543a53ebf2" containerName="payload-processing" Nov 03 14:06:29.431673 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:29.431648 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-84dcd755bf-c9rnp" Nov 03 14:06:29.481894 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:29.481860 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-3eb0ea556450\"" Nov 03 14:06:29.489685 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:29.489647 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/797bcf35-fa9e-47b8-887c-bb8942af277b-plugins-config-volume\") pod \"payload-processing-external-model-84dcd755bf-c9rnp\" (UID: \"797bcf35-fa9e-47b8-887c-bb8942af277b\") " pod="models-as-a-service/payload-processing-external-model-84dcd755bf-c9rnp" Nov 03 14:06:29.489846 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:29.489768 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/797bcf35-fa9e-47b8-887c-bb8942af277b-provider-credentials\") pod \"payload-processing-external-model-84dcd755bf-c9rnp\" (UID: \"797bcf35-fa9e-47b8-887c-bb8942af277b\") " pod="models-as-a-service/payload-processing-external-model-84dcd755bf-c9rnp" Nov 03 14:06:29.489967 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:29.489946 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b859h\" (UniqueName: \"kubernetes.io/projected/797bcf35-fa9e-47b8-887c-bb8942af277b-kube-api-access-b859h\") pod \"payload-processing-external-model-84dcd755bf-c9rnp\" (UID: \"797bcf35-fa9e-47b8-887c-bb8942af277b\") " pod="models-as-a-service/payload-processing-external-model-84dcd755bf-c9rnp" Nov 03 14:06:29.490105 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:29.490060 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/797bcf35-fa9e-47b8-887c-bb8942af277b-routing-overlay-volume\") pod \"payload-processing-external-model-84dcd755bf-c9rnp\" (UID: \"797bcf35-fa9e-47b8-887c-bb8942af277b\") " pod="models-as-a-service/payload-processing-external-model-84dcd755bf-c9rnp" Nov 03 14:06:29.533464 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:29.533427 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-84dcd755bf-c9rnp"] Nov 03 14:06:29.591382 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:29.591335 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/797bcf35-fa9e-47b8-887c-bb8942af277b-plugins-config-volume\") pod \"payload-processing-external-model-84dcd755bf-c9rnp\" (UID: \"797bcf35-fa9e-47b8-887c-bb8942af277b\") " pod="models-as-a-service/payload-processing-external-model-84dcd755bf-c9rnp" Nov 03 14:06:29.591563 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:29.591436 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/797bcf35-fa9e-47b8-887c-bb8942af277b-provider-credentials\") pod \"payload-processing-external-model-84dcd755bf-c9rnp\" (UID: \"797bcf35-fa9e-47b8-887c-bb8942af277b\") " pod="models-as-a-service/payload-processing-external-model-84dcd755bf-c9rnp" Nov 03 14:06:29.591636 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:29.591617 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b859h\" (UniqueName: \"kubernetes.io/projected/797bcf35-fa9e-47b8-887c-bb8942af277b-kube-api-access-b859h\") pod \"payload-processing-external-model-84dcd755bf-c9rnp\" (UID: \"797bcf35-fa9e-47b8-887c-bb8942af277b\") " pod="models-as-a-service/payload-processing-external-model-84dcd755bf-c9rnp" Nov 03 14:06:29.591746 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:29.591722 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/797bcf35-fa9e-47b8-887c-bb8942af277b-routing-overlay-volume\") pod \"payload-processing-external-model-84dcd755bf-c9rnp\" (UID: \"797bcf35-fa9e-47b8-887c-bb8942af277b\") " pod="models-as-a-service/payload-processing-external-model-84dcd755bf-c9rnp" Nov 03 14:06:29.592225 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:29.592195 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/797bcf35-fa9e-47b8-887c-bb8942af277b-plugins-config-volume\") pod \"payload-processing-external-model-84dcd755bf-c9rnp\" (UID: \"797bcf35-fa9e-47b8-887c-bb8942af277b\") " pod="models-as-a-service/payload-processing-external-model-84dcd755bf-c9rnp" Nov 03 14:06:29.592355 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:29.592255 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/797bcf35-fa9e-47b8-887c-bb8942af277b-routing-overlay-volume\") pod \"payload-processing-external-model-84dcd755bf-c9rnp\" (UID: \"797bcf35-fa9e-47b8-887c-bb8942af277b\") " pod="models-as-a-service/payload-processing-external-model-84dcd755bf-c9rnp" Nov 03 14:06:29.593982 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:29.593956 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/797bcf35-fa9e-47b8-887c-bb8942af277b-provider-credentials\") pod \"payload-processing-external-model-84dcd755bf-c9rnp\" (UID: \"797bcf35-fa9e-47b8-887c-bb8942af277b\") " pod="models-as-a-service/payload-processing-external-model-84dcd755bf-c9rnp" Nov 03 14:06:29.645497 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:29.645438 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b859h\" (UniqueName: \"kubernetes.io/projected/797bcf35-fa9e-47b8-887c-bb8942af277b-kube-api-access-b859h\") pod \"payload-processing-external-model-84dcd755bf-c9rnp\" (UID: \"797bcf35-fa9e-47b8-887c-bb8942af277b\") " pod="models-as-a-service/payload-processing-external-model-84dcd755bf-c9rnp" Nov 03 14:06:29.746064 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:29.745977 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-84dcd755bf-c9rnp" Nov 03 14:06:30.121600 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:30.121511 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-84dcd755bf-c9rnp"] Nov 03 14:06:30.127539 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:06:30.127513 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod797bcf35_fa9e_47b8_887c_bb8942af277b.slice/crio-cb7e262a4fb23ae615389a67eb479700bffec883797e67fe5e7297351237178d WatchSource:0}: Error finding container cb7e262a4fb23ae615389a67eb479700bffec883797e67fe5e7297351237178d: Status 404 returned error can't find the container with id cb7e262a4fb23ae615389a67eb479700bffec883797e67fe5e7297351237178d Nov 03 14:06:30.135405 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:30.135383 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-7469666f9f-vpmg2" Nov 03 14:06:30.217604 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:30.217567 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-84dcd755bf-c9rnp" event={"ID":"797bcf35-fa9e-47b8-887c-bb8942af277b","Type":"ContainerStarted","Data":"56b7486b76019cadddbbc0091d32603ae9dd4e92673b0d4e72b583b7ac271548"} Nov 03 14:06:30.217767 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:30.217628 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-84dcd755bf-c9rnp" event={"ID":"797bcf35-fa9e-47b8-887c-bb8942af277b","Type":"ContainerStarted","Data":"cb7e262a4fb23ae615389a67eb479700bffec883797e67fe5e7297351237178d"} Nov 03 14:06:30.217767 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:30.217650 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-84dcd755bf-c9rnp" Nov 03 14:06:30.351803 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:30.351749 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-84dcd755bf-c9rnp" podStartSLOduration=1.351734183 podStartE2EDuration="1.351734183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:06:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:06:30.350852598 +0000 UTC m=+3672.361540840" watchObservedRunningTime="2026-11-03 14:06:30.351734183 +0000 UTC m=+3672.362422454" Nov 03 14:06:37.235425 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:06:37.235387 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:06:41.228253 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:41.228224 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-84dcd755bf-c9rnp" Nov 03 14:06:41.387175 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:41.387134 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7469666f9f-vpmg2"] Nov 03 14:06:46.388137 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:46.388092 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7469666f9f-vpmg2" podUID="192eaf94-c0d0-4ce0-959e-5e297a072ab1" containerName="payload-processing" containerID="cri-o://3bd6092d13f0834afef2f365c244e2e320fee2045033bff865be01178aecc980" gracePeriod=25 Nov 03 14:06:46.663296 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:46.663230 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7469666f9f-vpmg2" Nov 03 14:06:46.733799 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:46.733765 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/192eaf94-c0d0-4ce0-959e-5e297a072ab1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/192eaf94-c0d0-4ce0-959e-5e297a072ab1-provider-credentials\") pod \"192eaf94-c0d0-4ce0-959e-5e297a072ab1\" (UID: \"192eaf94-c0d0-4ce0-959e-5e297a072ab1\") " Nov 03 14:06:46.733993 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:46.733846 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/192eaf94-c0d0-4ce0-959e-5e297a072ab1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/192eaf94-c0d0-4ce0-959e-5e297a072ab1-routing-overlay-volume\") pod \"192eaf94-c0d0-4ce0-959e-5e297a072ab1\" (UID: \"192eaf94-c0d0-4ce0-959e-5e297a072ab1\") " Nov 03 14:06:46.733993 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:46.733871 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/192eaf94-c0d0-4ce0-959e-5e297a072ab1-kube-api-access-b4mn4\" (UniqueName: \"kubernetes.io/projected/192eaf94-c0d0-4ce0-959e-5e297a072ab1-kube-api-access-b4mn4\") pod \"192eaf94-c0d0-4ce0-959e-5e297a072ab1\" (UID: \"192eaf94-c0d0-4ce0-959e-5e297a072ab1\") " Nov 03 14:06:46.733993 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:46.733942 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/192eaf94-c0d0-4ce0-959e-5e297a072ab1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/192eaf94-c0d0-4ce0-959e-5e297a072ab1-plugins-config-volume\") pod \"192eaf94-c0d0-4ce0-959e-5e297a072ab1\" (UID: \"192eaf94-c0d0-4ce0-959e-5e297a072ab1\") " Nov 03 14:06:46.734285 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:46.734258 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/192eaf94-c0d0-4ce0-959e-5e297a072ab1-routing-overlay-volume" pod "192eaf94-c0d0-4ce0-959e-5e297a072ab1" (UID: "192eaf94-c0d0-4ce0-959e-5e297a072ab1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:06:46.734408 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:46.734393 5979 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/192eaf94-c0d0-4ce0-959e-5e297a072ab1-routing-overlay-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:06:46.734502 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:46.734400 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/192eaf94-c0d0-4ce0-959e-5e297a072ab1-plugins-config-volume" pod "192eaf94-c0d0-4ce0-959e-5e297a072ab1" (UID: "192eaf94-c0d0-4ce0-959e-5e297a072ab1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:06:46.736071 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:46.736045 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/192eaf94-c0d0-4ce0-959e-5e297a072ab1-provider-credentials" pod "192eaf94-c0d0-4ce0-959e-5e297a072ab1" (UID: "192eaf94-c0d0-4ce0-959e-5e297a072ab1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:06:46.736071 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:46.736052 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/192eaf94-c0d0-4ce0-959e-5e297a072ab1-kube-api-access-b4mn4" pod "192eaf94-c0d0-4ce0-959e-5e297a072ab1" (UID: "192eaf94-c0d0-4ce0-959e-5e297a072ab1"). InnerVolumeSpecName "kube-api-access-b4mn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:06:46.835928 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:46.835896 5979 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/192eaf94-c0d0-4ce0-959e-5e297a072ab1-provider-credentials\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:06:46.835928 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:46.835930 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4mn4\" (UniqueName: \"kubernetes.io/projected/192eaf94-c0d0-4ce0-959e-5e297a072ab1-kube-api-access-b4mn4\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:06:46.836153 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:46.835941 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/192eaf94-c0d0-4ce0-959e-5e297a072ab1-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:06:47.340970 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:47.340937 5979 generic.go:356] "Generic (PLEG): container finished" podID="192eaf94-c0d0-4ce0-959e-5e297a072ab1" containerID="3bd6092d13f0834afef2f365c244e2e320fee2045033bff865be01178aecc980" exitCode=0 Nov 03 14:06:47.341172 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:47.340977 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7469666f9f-vpmg2" event={"ID":"192eaf94-c0d0-4ce0-959e-5e297a072ab1","Type":"ContainerDied","Data":"3bd6092d13f0834afef2f365c244e2e320fee2045033bff865be01178aecc980"} Nov 03 14:06:47.341172 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:47.340983 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7469666f9f-vpmg2" Nov 03 14:06:47.341172 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:47.341016 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7469666f9f-vpmg2" event={"ID":"192eaf94-c0d0-4ce0-959e-5e297a072ab1","Type":"ContainerDied","Data":"8eca31a0ea17b050b1332fc8e06d81b084c91306cfd139408309b0175ba7237c"} Nov 03 14:06:47.341172 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:47.341037 5979 scope.go:122] "RemoveContainer" containerID="3bd6092d13f0834afef2f365c244e2e320fee2045033bff865be01178aecc980" Nov 03 14:06:47.350911 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:47.350894 5979 scope.go:122] "RemoveContainer" containerID="3bd6092d13f0834afef2f365c244e2e320fee2045033bff865be01178aecc980" Nov 03 14:06:47.351202 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:06:47.351180 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bd6092d13f0834afef2f365c244e2e320fee2045033bff865be01178aecc980\": container with ID starting with 3bd6092d13f0834afef2f365c244e2e320fee2045033bff865be01178aecc980 not found: ID does not exist" containerID="3bd6092d13f0834afef2f365c244e2e320fee2045033bff865be01178aecc980" Nov 03 14:06:47.351268 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:47.351209 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bd6092d13f0834afef2f365c244e2e320fee2045033bff865be01178aecc980"} err="failed to get container status \"3bd6092d13f0834afef2f365c244e2e320fee2045033bff865be01178aecc980\": rpc error: code = NotFound desc = could not find container \"3bd6092d13f0834afef2f365c244e2e320fee2045033bff865be01178aecc980\": container with ID starting with 3bd6092d13f0834afef2f365c244e2e320fee2045033bff865be01178aecc980 not found: ID does not exist" Nov 03 14:06:47.496130 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:47.496084 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7469666f9f-vpmg2"] Nov 03 14:06:47.544018 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:47.543982 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7469666f9f-vpmg2"] Nov 03 14:06:48.116292 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:06:48.116257 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="192eaf94-c0d0-4ce0-959e-5e297a072ab1" path="/var/lib/kubelet/pods/192eaf94-c0d0-4ce0-959e-5e297a072ab1/volumes" Nov 03 14:07:03.269492 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:03.269386 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f945362d-vvx7f"] Nov 03 14:07:03.270107 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:03.270089 5979 cpu_manager.go:414] "containerMap: removing container" podUID="192eaf94-c0d0-4ce0-959e-5e297a072ab1" containerName="payload-processing" Nov 03 14:07:03.270176 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:03.270108 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="192eaf94-c0d0-4ce0-959e-5e297a072ab1" containerName="payload-processing" Nov 03 14:07:03.270215 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:03.270197 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="192eaf94-c0d0-4ce0-959e-5e297a072ab1" containerName="payload-processing" Nov 03 14:07:03.273817 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:03.273795 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f945362d-vvx7f" Nov 03 14:07:03.323978 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:03.323942 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:07:03.327088 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:03.327058 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f945362d-vvx7f"] Nov 03 14:07:03.352868 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:03.352835 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtb7b\" (UniqueName: \"kubernetes.io/projected/798ad31e-2eea-448a-b2e8-bc02a98191c6-kube-api-access-dtb7b\") pod \"maas-api-revoke-sub-f945362d-vvx7f\" (UID: \"798ad31e-2eea-448a-b2e8-bc02a98191c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f945362d-vvx7f" Nov 03 14:07:03.353075 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:03.352881 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/798ad31e-2eea-448a-b2e8-bc02a98191c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f945362d-vvx7f\" (UID: \"798ad31e-2eea-448a-b2e8-bc02a98191c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f945362d-vvx7f" Nov 03 14:07:03.353075 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:03.352943 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/798ad31e-2eea-448a-b2e8-bc02a98191c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f945362d-vvx7f\" (UID: \"798ad31e-2eea-448a-b2e8-bc02a98191c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f945362d-vvx7f" Nov 03 14:07:03.370196 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:03.370170 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 14:07:03.454746 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:03.454707 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtb7b\" (UniqueName: \"kubernetes.io/projected/798ad31e-2eea-448a-b2e8-bc02a98191c6-kube-api-access-dtb7b\") pod \"maas-api-revoke-sub-f945362d-vvx7f\" (UID: \"798ad31e-2eea-448a-b2e8-bc02a98191c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f945362d-vvx7f" Nov 03 14:07:03.454746 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:03.454750 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/798ad31e-2eea-448a-b2e8-bc02a98191c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f945362d-vvx7f\" (UID: \"798ad31e-2eea-448a-b2e8-bc02a98191c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f945362d-vvx7f" Nov 03 14:07:03.454990 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:03.454791 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/798ad31e-2eea-448a-b2e8-bc02a98191c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f945362d-vvx7f\" (UID: \"798ad31e-2eea-448a-b2e8-bc02a98191c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f945362d-vvx7f" Nov 03 14:07:03.455442 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:03.455420 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/798ad31e-2eea-448a-b2e8-bc02a98191c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f945362d-vvx7f\" (UID: \"798ad31e-2eea-448a-b2e8-bc02a98191c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f945362d-vvx7f" Nov 03 14:07:03.457377 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:03.457350 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/798ad31e-2eea-448a-b2e8-bc02a98191c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f945362d-vvx7f\" (UID: \"798ad31e-2eea-448a-b2e8-bc02a98191c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f945362d-vvx7f" Nov 03 14:07:03.509401 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:03.509370 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtb7b\" (UniqueName: \"kubernetes.io/projected/798ad31e-2eea-448a-b2e8-bc02a98191c6-kube-api-access-dtb7b\") pod \"maas-api-revoke-sub-f945362d-vvx7f\" (UID: \"798ad31e-2eea-448a-b2e8-bc02a98191c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f945362d-vvx7f" Nov 03 14:07:03.588186 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:03.588101 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f945362d-vvx7f" Nov 03 14:07:03.964014 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:03.963973 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f945362d-vvx7f"] Nov 03 14:07:03.971275 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:07:03.971243 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod798ad31e_2eea_448a_b2e8_bc02a98191c6.slice/crio-d774f2730199b4b091056491d25b6b52a6b607e9bc7fa317f1543480cf275979 WatchSource:0}: Error finding container d774f2730199b4b091056491d25b6b52a6b607e9bc7fa317f1543480cf275979: Status 404 returned error can't find the container with id d774f2730199b4b091056491d25b6b52a6b607e9bc7fa317f1543480cf275979 Nov 03 14:07:04.456022 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:04.455986 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f945362d-vvx7f" event={"ID":"798ad31e-2eea-448a-b2e8-bc02a98191c6","Type":"ContainerStarted","Data":"626a215e5a921b1394818f5737f54ae2be66f4e7e7938bb44e558f1cc23bc873"} Nov 03 14:07:04.456418 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:04.456036 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f945362d-vvx7f" event={"ID":"798ad31e-2eea-448a-b2e8-bc02a98191c6","Type":"ContainerStarted","Data":"d774f2730199b4b091056491d25b6b52a6b607e9bc7fa317f1543480cf275979"} Nov 03 14:07:04.567739 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:04.567625 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f945362d-vvx7f" podStartSLOduration=1.5676123579999999 podStartE2EDuration="1.567612358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:07:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:07:04.563603693 +0000 UTC m=+3706.574291938" watchObservedRunningTime="2026-11-03 14:07:04.567612358 +0000 UTC m=+3706.578300638" Nov 03 14:07:05.464451 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:05.464415 5979 generic.go:356] "Generic (PLEG): container finished" podID="798ad31e-2eea-448a-b2e8-bc02a98191c6" containerID="626a215e5a921b1394818f5737f54ae2be66f4e7e7938bb44e558f1cc23bc873" exitCode=0 Nov 03 14:07:05.464850 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:05.464513 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f945362d-vvx7f" event={"ID":"798ad31e-2eea-448a-b2e8-bc02a98191c6","Type":"ContainerDied","Data":"626a215e5a921b1394818f5737f54ae2be66f4e7e7938bb44e558f1cc23bc873"} Nov 03 14:07:05.698643 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:05.698609 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae248d-vvfwm"] Nov 03 14:07:05.702763 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:05.702741 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae248d-vvfwm" Nov 03 14:07:05.757412 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:05.757335 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae248d-vvfwm"] Nov 03 14:07:05.820277 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:05.820239 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9896368-75e8-48ca-a563-a95befb454ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-ae248d-vvfwm\" (UID: \"b9896368-75e8-48ca-a563-a95befb454ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae248d-vvfwm" Nov 03 14:07:05.820542 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:05.820338 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9896368-75e8-48ca-a563-a95befb454ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-ae248d-vvfwm\" (UID: \"b9896368-75e8-48ca-a563-a95befb454ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae248d-vvfwm" Nov 03 14:07:05.820542 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:05.820448 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brg2f\" (UniqueName: \"kubernetes.io/projected/b9896368-75e8-48ca-a563-a95befb454ef-kube-api-access-brg2f\") pod \"maas-api-revoke-keys-e2e-extmod-a-ae248d-vvfwm\" (UID: \"b9896368-75e8-48ca-a563-a95befb454ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae248d-vvfwm" Nov 03 14:07:05.922393 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:05.922359 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brg2f\" (UniqueName: \"kubernetes.io/projected/b9896368-75e8-48ca-a563-a95befb454ef-kube-api-access-brg2f\") pod \"maas-api-revoke-keys-e2e-extmod-a-ae248d-vvfwm\" (UID: \"b9896368-75e8-48ca-a563-a95befb454ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae248d-vvfwm" Nov 03 14:07:05.922615 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:05.922516 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9896368-75e8-48ca-a563-a95befb454ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-ae248d-vvfwm\" (UID: \"b9896368-75e8-48ca-a563-a95befb454ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae248d-vvfwm" Nov 03 14:07:05.922615 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:05.922567 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9896368-75e8-48ca-a563-a95befb454ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-ae248d-vvfwm\" (UID: \"b9896368-75e8-48ca-a563-a95befb454ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae248d-vvfwm" Nov 03 14:07:05.923300 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:05.923280 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9896368-75e8-48ca-a563-a95befb454ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-ae248d-vvfwm\" (UID: \"b9896368-75e8-48ca-a563-a95befb454ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae248d-vvfwm" Nov 03 14:07:05.924934 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:05.924914 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9896368-75e8-48ca-a563-a95befb454ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-ae248d-vvfwm\" (UID: \"b9896368-75e8-48ca-a563-a95befb454ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae248d-vvfwm" Nov 03 14:07:05.976395 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:05.976360 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brg2f\" (UniqueName: \"kubernetes.io/projected/b9896368-75e8-48ca-a563-a95befb454ef-kube-api-access-brg2f\") pod \"maas-api-revoke-keys-e2e-extmod-a-ae248d-vvfwm\" (UID: \"b9896368-75e8-48ca-a563-a95befb454ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae248d-vvfwm" Nov 03 14:07:06.016508 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:06.016415 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae248d-vvfwm" Nov 03 14:07:06.397132 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:06.397097 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae248d-vvfwm"] Nov 03 14:07:06.403739 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:07:06.403709 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9896368_75e8_48ca_a563_a95befb454ef.slice/crio-1289183c85308c45ae2934079aaef2507a25edb4bfdeabb4906d1bc048fb2b9a WatchSource:0}: Error finding container 1289183c85308c45ae2934079aaef2507a25edb4bfdeabb4906d1bc048fb2b9a: Status 404 returned error can't find the container with id 1289183c85308c45ae2934079aaef2507a25edb4bfdeabb4906d1bc048fb2b9a Nov 03 14:07:06.471698 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:06.471665 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae248d-vvfwm" event={"ID":"b9896368-75e8-48ca-a563-a95befb454ef","Type":"ContainerStarted","Data":"1289183c85308c45ae2934079aaef2507a25edb4bfdeabb4906d1bc048fb2b9a"} Nov 03 14:07:06.599497 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:06.599441 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f945362d-vvx7f" Nov 03 14:07:06.632546 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:06.632515 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/798ad31e-2eea-448a-b2e8-bc02a98191c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/798ad31e-2eea-448a-b2e8-bc02a98191c6-maas-api-openshift-ca\") pod \"798ad31e-2eea-448a-b2e8-bc02a98191c6\" (UID: \"798ad31e-2eea-448a-b2e8-bc02a98191c6\") " Nov 03 14:07:06.632711 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:06.632601 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/798ad31e-2eea-448a-b2e8-bc02a98191c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/798ad31e-2eea-448a-b2e8-bc02a98191c6-maas-api-xks-ca\") pod \"798ad31e-2eea-448a-b2e8-bc02a98191c6\" (UID: \"798ad31e-2eea-448a-b2e8-bc02a98191c6\") " Nov 03 14:07:06.632781 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:06.632764 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/798ad31e-2eea-448a-b2e8-bc02a98191c6-kube-api-access-dtb7b\" (UniqueName: \"kubernetes.io/projected/798ad31e-2eea-448a-b2e8-bc02a98191c6-kube-api-access-dtb7b\") pod \"798ad31e-2eea-448a-b2e8-bc02a98191c6\" (UID: \"798ad31e-2eea-448a-b2e8-bc02a98191c6\") " Nov 03 14:07:06.632917 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:06.632897 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/798ad31e-2eea-448a-b2e8-bc02a98191c6-maas-api-openshift-ca" pod "798ad31e-2eea-448a-b2e8-bc02a98191c6" (UID: "798ad31e-2eea-448a-b2e8-bc02a98191c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:07:06.633195 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:06.633178 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/798ad31e-2eea-448a-b2e8-bc02a98191c6-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:06.634643 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:06.634616 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/798ad31e-2eea-448a-b2e8-bc02a98191c6-maas-api-xks-ca" pod "798ad31e-2eea-448a-b2e8-bc02a98191c6" (UID: "798ad31e-2eea-448a-b2e8-bc02a98191c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:07:06.634738 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:06.634683 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/798ad31e-2eea-448a-b2e8-bc02a98191c6-kube-api-access-dtb7b" pod "798ad31e-2eea-448a-b2e8-bc02a98191c6" (UID: "798ad31e-2eea-448a-b2e8-bc02a98191c6"). InnerVolumeSpecName "kube-api-access-dtb7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:07:06.734952 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:06.734905 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/798ad31e-2eea-448a-b2e8-bc02a98191c6-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:06.735134 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:06.734956 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtb7b\" (UniqueName: \"kubernetes.io/projected/798ad31e-2eea-448a-b2e8-bc02a98191c6-kube-api-access-dtb7b\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:07.234266 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:07:07.234185 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:07:07.479657 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:07.479623 5979 generic.go:356] "Generic (PLEG): container finished" podID="b9896368-75e8-48ca-a563-a95befb454ef" containerID="c161ec0e32fb414b336a446ef932ddc1257d83a72ebb9c0ec5e358d3cc9b0818" exitCode=0 Nov 03 14:07:07.480094 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:07.479709 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae248d-vvfwm" event={"ID":"b9896368-75e8-48ca-a563-a95befb454ef","Type":"ContainerDied","Data":"c161ec0e32fb414b336a446ef932ddc1257d83a72ebb9c0ec5e358d3cc9b0818"} Nov 03 14:07:07.481093 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:07.481071 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f945362d-vvx7f" event={"ID":"798ad31e-2eea-448a-b2e8-bc02a98191c6","Type":"ContainerDied","Data":"d774f2730199b4b091056491d25b6b52a6b607e9bc7fa317f1543480cf275979"} Nov 03 14:07:07.481205 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:07.481105 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f945362d-vvx7f" Nov 03 14:07:07.481205 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:07.481112 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d774f2730199b4b091056491d25b6b52a6b607e9bc7fa317f1543480cf275979" Nov 03 14:07:08.637554 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:08.637526 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae248d-vvfwm" Nov 03 14:07:08.658777 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:08.658745 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9896368-75e8-48ca-a563-a95befb454ef-kube-api-access-brg2f\" (UniqueName: \"kubernetes.io/projected/b9896368-75e8-48ca-a563-a95befb454ef-kube-api-access-brg2f\") pod \"b9896368-75e8-48ca-a563-a95befb454ef\" (UID: \"b9896368-75e8-48ca-a563-a95befb454ef\") " Nov 03 14:07:08.658945 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:08.658809 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9896368-75e8-48ca-a563-a95befb454ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9896368-75e8-48ca-a563-a95befb454ef-maas-api-openshift-ca\") pod \"b9896368-75e8-48ca-a563-a95befb454ef\" (UID: \"b9896368-75e8-48ca-a563-a95befb454ef\") " Nov 03 14:07:08.658993 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:08.658945 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9896368-75e8-48ca-a563-a95befb454ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9896368-75e8-48ca-a563-a95befb454ef-maas-api-xks-ca\") pod \"b9896368-75e8-48ca-a563-a95befb454ef\" (UID: \"b9896368-75e8-48ca-a563-a95befb454ef\") " Nov 03 14:07:08.659185 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:08.659161 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9896368-75e8-48ca-a563-a95befb454ef-maas-api-openshift-ca" pod "b9896368-75e8-48ca-a563-a95befb454ef" (UID: "b9896368-75e8-48ca-a563-a95befb454ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:07:08.659511 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:08.659460 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9896368-75e8-48ca-a563-a95befb454ef-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:08.660967 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:08.660940 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9896368-75e8-48ca-a563-a95befb454ef-maas-api-xks-ca" pod "b9896368-75e8-48ca-a563-a95befb454ef" (UID: "b9896368-75e8-48ca-a563-a95befb454ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:07:08.661064 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:08.660996 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9896368-75e8-48ca-a563-a95befb454ef-kube-api-access-brg2f" pod "b9896368-75e8-48ca-a563-a95befb454ef" (UID: "b9896368-75e8-48ca-a563-a95befb454ef"). InnerVolumeSpecName "kube-api-access-brg2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:07:08.760442 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:08.760354 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brg2f\" (UniqueName: \"kubernetes.io/projected/b9896368-75e8-48ca-a563-a95befb454ef-kube-api-access-brg2f\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:08.760442 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:08.760387 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9896368-75e8-48ca-a563-a95befb454ef-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:09.496763 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:09.496731 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae248d-vvfwm" Nov 03 14:07:09.496946 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:09.496762 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae248d-vvfwm" event={"ID":"b9896368-75e8-48ca-a563-a95befb454ef","Type":"ContainerDied","Data":"1289183c85308c45ae2934079aaef2507a25edb4bfdeabb4906d1bc048fb2b9a"} Nov 03 14:07:09.496946 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:09.496820 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1289183c85308c45ae2934079aaef2507a25edb4bfdeabb4906d1bc048fb2b9a" Nov 03 14:07:16.603502 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:16.603439 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-78f749b598-jjswn"] Nov 03 14:07:16.604185 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:16.604168 5979 cpu_manager.go:414] "containerMap: removing container" podUID="798ad31e-2eea-448a-b2e8-bc02a98191c6" containerName="revoke-keys" Nov 03 14:07:16.604242 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:16.604187 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="798ad31e-2eea-448a-b2e8-bc02a98191c6" containerName="revoke-keys" Nov 03 14:07:16.604242 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:16.604205 5979 cpu_manager.go:414] "containerMap: removing container" podUID="b9896368-75e8-48ca-a563-a95befb454ef" containerName="revoke-keys" Nov 03 14:07:16.604242 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:16.604211 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9896368-75e8-48ca-a563-a95befb454ef" containerName="revoke-keys" Nov 03 14:07:16.604348 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:16.604308 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9896368-75e8-48ca-a563-a95befb454ef" containerName="revoke-keys" Nov 03 14:07:16.604348 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:16.604326 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="798ad31e-2eea-448a-b2e8-bc02a98191c6" containerName="revoke-keys" Nov 03 14:07:16.608172 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:16.608152 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-78f749b598-jjswn" Nov 03 14:07:16.661184 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:16.661149 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-78f749b598-jjswn"] Nov 03 14:07:16.684078 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:16.684041 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6a3788db-a41a-47f8-b73c-c695cfdc2801-provider-credentials\") pod \"payload-processing-external-model-78f749b598-jjswn\" (UID: \"6a3788db-a41a-47f8-b73c-c695cfdc2801\") " pod="models-as-a-service/payload-processing-external-model-78f749b598-jjswn" Nov 03 14:07:16.684241 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:16.684158 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6a3788db-a41a-47f8-b73c-c695cfdc2801-routing-overlay-volume\") pod \"payload-processing-external-model-78f749b598-jjswn\" (UID: \"6a3788db-a41a-47f8-b73c-c695cfdc2801\") " pod="models-as-a-service/payload-processing-external-model-78f749b598-jjswn" Nov 03 14:07:16.684241 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:16.684235 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk84t\" (UniqueName: \"kubernetes.io/projected/6a3788db-a41a-47f8-b73c-c695cfdc2801-kube-api-access-pk84t\") pod \"payload-processing-external-model-78f749b598-jjswn\" (UID: \"6a3788db-a41a-47f8-b73c-c695cfdc2801\") " pod="models-as-a-service/payload-processing-external-model-78f749b598-jjswn" Nov 03 14:07:16.684340 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:16.684285 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a3788db-a41a-47f8-b73c-c695cfdc2801-plugins-config-volume\") pod \"payload-processing-external-model-78f749b598-jjswn\" (UID: \"6a3788db-a41a-47f8-b73c-c695cfdc2801\") " pod="models-as-a-service/payload-processing-external-model-78f749b598-jjswn" Nov 03 14:07:16.785286 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:16.785249 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6a3788db-a41a-47f8-b73c-c695cfdc2801-provider-credentials\") pod \"payload-processing-external-model-78f749b598-jjswn\" (UID: \"6a3788db-a41a-47f8-b73c-c695cfdc2801\") " pod="models-as-a-service/payload-processing-external-model-78f749b598-jjswn" Nov 03 14:07:16.785528 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:16.785340 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6a3788db-a41a-47f8-b73c-c695cfdc2801-routing-overlay-volume\") pod \"payload-processing-external-model-78f749b598-jjswn\" (UID: \"6a3788db-a41a-47f8-b73c-c695cfdc2801\") " pod="models-as-a-service/payload-processing-external-model-78f749b598-jjswn" Nov 03 14:07:16.785528 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:16.785407 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk84t\" (UniqueName: \"kubernetes.io/projected/6a3788db-a41a-47f8-b73c-c695cfdc2801-kube-api-access-pk84t\") pod \"payload-processing-external-model-78f749b598-jjswn\" (UID: \"6a3788db-a41a-47f8-b73c-c695cfdc2801\") " pod="models-as-a-service/payload-processing-external-model-78f749b598-jjswn" Nov 03 14:07:16.785528 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:16.785444 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a3788db-a41a-47f8-b73c-c695cfdc2801-plugins-config-volume\") pod \"payload-processing-external-model-78f749b598-jjswn\" (UID: \"6a3788db-a41a-47f8-b73c-c695cfdc2801\") " pod="models-as-a-service/payload-processing-external-model-78f749b598-jjswn" Nov 03 14:07:16.786027 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:16.786006 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6a3788db-a41a-47f8-b73c-c695cfdc2801-routing-overlay-volume\") pod \"payload-processing-external-model-78f749b598-jjswn\" (UID: \"6a3788db-a41a-47f8-b73c-c695cfdc2801\") " pod="models-as-a-service/payload-processing-external-model-78f749b598-jjswn" Nov 03 14:07:16.786175 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:16.786155 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a3788db-a41a-47f8-b73c-c695cfdc2801-plugins-config-volume\") pod \"payload-processing-external-model-78f749b598-jjswn\" (UID: \"6a3788db-a41a-47f8-b73c-c695cfdc2801\") " pod="models-as-a-service/payload-processing-external-model-78f749b598-jjswn" Nov 03 14:07:16.787845 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:16.787823 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6a3788db-a41a-47f8-b73c-c695cfdc2801-provider-credentials\") pod \"payload-processing-external-model-78f749b598-jjswn\" (UID: \"6a3788db-a41a-47f8-b73c-c695cfdc2801\") " pod="models-as-a-service/payload-processing-external-model-78f749b598-jjswn" Nov 03 14:07:16.838808 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:16.838779 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk84t\" (UniqueName: \"kubernetes.io/projected/6a3788db-a41a-47f8-b73c-c695cfdc2801-kube-api-access-pk84t\") pod \"payload-processing-external-model-78f749b598-jjswn\" (UID: \"6a3788db-a41a-47f8-b73c-c695cfdc2801\") " pod="models-as-a-service/payload-processing-external-model-78f749b598-jjswn" Nov 03 14:07:16.922565 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:16.922532 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-78f749b598-jjswn" Nov 03 14:07:17.377782 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:17.377697 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-78f749b598-jjswn"] Nov 03 14:07:17.384642 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:07:17.384613 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a3788db_a41a_47f8_b73c_c695cfdc2801.slice/crio-bee14bf11708dd496f94e3d2cc184bc06d0b7cb9964479a315f32cc33f9986ff WatchSource:0}: Error finding container bee14bf11708dd496f94e3d2cc184bc06d0b7cb9964479a315f32cc33f9986ff: Status 404 returned error can't find the container with id bee14bf11708dd496f94e3d2cc184bc06d0b7cb9964479a315f32cc33f9986ff Nov 03 14:07:17.565850 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:17.565815 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:07:17.567213 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:17.567173 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-78f749b598-jjswn" event={"ID":"6a3788db-a41a-47f8-b73c-c695cfdc2801","Type":"ContainerStarted","Data":"234529210fbf5507b524fb3f5b70a244c1e8be15c098457998c01e2d15b89330"} Nov 03 14:07:17.567386 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:17.567233 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-78f749b598-jjswn" event={"ID":"6a3788db-a41a-47f8-b73c-c695cfdc2801","Type":"ContainerStarted","Data":"bee14bf11708dd496f94e3d2cc184bc06d0b7cb9964479a315f32cc33f9986ff"} Nov 03 14:07:17.567386 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:17.567270 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-78f749b598-jjswn" Nov 03 14:07:17.675631 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:17.675581 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-78f749b598-jjswn" podStartSLOduration=1.67556583 podStartE2EDuration="1.67556583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:07:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:07:17.674209867 +0000 UTC m=+3719.684898112" watchObservedRunningTime="2026-11-03 14:07:17.67556583 +0000 UTC m=+3719.686254087" Nov 03 14:07:20.391706 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:20.391675 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6"] Nov 03 14:07:20.392146 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:20.391873 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6" podUID="5ab327a8-7012-4504-9557-6dd31ed7a365" containerName="istio-proxy" containerID="cri-o://da2ca747ec56fffdd8cfab01245b312d773fc8b360eecd5c63a540e28dc57ade" gracePeriod=30 Nov 03 14:07:21.524223 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:21.524181 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-78f749b598-jjswn"] Nov 03 14:07:21.552730 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:21.552696 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7458df57f5-zrrlx"] Nov 03 14:07:21.558320 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:21.558280 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7458df57f5-zrrlx" Nov 03 14:07:21.602392 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:21.602360 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae248d-789b45f9c9-8kbb6"] Nov 03 14:07:21.602588 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:21.602568 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae248d-789b45f9c9-8kbb6" podUID="efd740a0-6578-40c0-8368-9d8bbeea64f1" containerName="maas-api" containerID="cri-o://8ddfb0dc8dc5d40d1b857c9d5a0eac401db0ea981f135a57af0aaead2feecc84" gracePeriod=30 Nov 03 14:07:21.609241 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:21.609217 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-d4ee3f6c80de\"" Nov 03 14:07:21.631226 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:21.631192 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7458df57f5-zrrlx"] Nov 03 14:07:21.677313 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:21.677275 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b34d5b3f-3449-4b91-8919-d278d0dc5112-routing-overlay-volume\") pod \"payload-processing-external-model-7458df57f5-zrrlx\" (UID: \"b34d5b3f-3449-4b91-8919-d278d0dc5112\") " pod="models-as-a-service/payload-processing-external-model-7458df57f5-zrrlx" Nov 03 14:07:21.677925 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:21.677527 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b34d5b3f-3449-4b91-8919-d278d0dc5112-provider-credentials\") pod \"payload-processing-external-model-7458df57f5-zrrlx\" (UID: \"b34d5b3f-3449-4b91-8919-d278d0dc5112\") " pod="models-as-a-service/payload-processing-external-model-7458df57f5-zrrlx" Nov 03 14:07:21.677925 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:21.677607 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b34d5b3f-3449-4b91-8919-d278d0dc5112-plugins-config-volume\") pod \"payload-processing-external-model-7458df57f5-zrrlx\" (UID: \"b34d5b3f-3449-4b91-8919-d278d0dc5112\") " pod="models-as-a-service/payload-processing-external-model-7458df57f5-zrrlx" Nov 03 14:07:21.677925 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:21.677651 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rd8s\" (UniqueName: \"kubernetes.io/projected/b34d5b3f-3449-4b91-8919-d278d0dc5112-kube-api-access-5rd8s\") pod \"payload-processing-external-model-7458df57f5-zrrlx\" (UID: \"b34d5b3f-3449-4b91-8919-d278d0dc5112\") " pod="models-as-a-service/payload-processing-external-model-7458df57f5-zrrlx" Nov 03 14:07:21.780651 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:21.778751 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b34d5b3f-3449-4b91-8919-d278d0dc5112-routing-overlay-volume\") pod \"payload-processing-external-model-7458df57f5-zrrlx\" (UID: \"b34d5b3f-3449-4b91-8919-d278d0dc5112\") " pod="models-as-a-service/payload-processing-external-model-7458df57f5-zrrlx" Nov 03 14:07:21.780651 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:21.778912 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b34d5b3f-3449-4b91-8919-d278d0dc5112-provider-credentials\") pod \"payload-processing-external-model-7458df57f5-zrrlx\" (UID: \"b34d5b3f-3449-4b91-8919-d278d0dc5112\") " pod="models-as-a-service/payload-processing-external-model-7458df57f5-zrrlx" Nov 03 14:07:21.780651 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:21.778985 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b34d5b3f-3449-4b91-8919-d278d0dc5112-plugins-config-volume\") pod \"payload-processing-external-model-7458df57f5-zrrlx\" (UID: \"b34d5b3f-3449-4b91-8919-d278d0dc5112\") " pod="models-as-a-service/payload-processing-external-model-7458df57f5-zrrlx" Nov 03 14:07:21.780651 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:21.779024 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rd8s\" (UniqueName: \"kubernetes.io/projected/b34d5b3f-3449-4b91-8919-d278d0dc5112-kube-api-access-5rd8s\") pod \"payload-processing-external-model-7458df57f5-zrrlx\" (UID: \"b34d5b3f-3449-4b91-8919-d278d0dc5112\") " pod="models-as-a-service/payload-processing-external-model-7458df57f5-zrrlx" Nov 03 14:07:21.780651 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:21.780210 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b34d5b3f-3449-4b91-8919-d278d0dc5112-routing-overlay-volume\") pod \"payload-processing-external-model-7458df57f5-zrrlx\" (UID: \"b34d5b3f-3449-4b91-8919-d278d0dc5112\") " pod="models-as-a-service/payload-processing-external-model-7458df57f5-zrrlx" Nov 03 14:07:21.781940 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:21.781922 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b34d5b3f-3449-4b91-8919-d278d0dc5112-plugins-config-volume\") pod \"payload-processing-external-model-7458df57f5-zrrlx\" (UID: \"b34d5b3f-3449-4b91-8919-d278d0dc5112\") " pod="models-as-a-service/payload-processing-external-model-7458df57f5-zrrlx" Nov 03 14:07:21.788140 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:21.788115 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b34d5b3f-3449-4b91-8919-d278d0dc5112-provider-credentials\") pod \"payload-processing-external-model-7458df57f5-zrrlx\" (UID: \"b34d5b3f-3449-4b91-8919-d278d0dc5112\") " pod="models-as-a-service/payload-processing-external-model-7458df57f5-zrrlx" Nov 03 14:07:21.863945 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:21.863908 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rd8s\" (UniqueName: \"kubernetes.io/projected/b34d5b3f-3449-4b91-8919-d278d0dc5112-kube-api-access-5rd8s\") pod \"payload-processing-external-model-7458df57f5-zrrlx\" (UID: \"b34d5b3f-3449-4b91-8919-d278d0dc5112\") " pod="models-as-a-service/payload-processing-external-model-7458df57f5-zrrlx" Nov 03 14:07:21.876051 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:21.876020 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7458df57f5-zrrlx" Nov 03 14:07:21.909417 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:21.909383 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae248d-789b45f9c9-8kbb6" Nov 03 14:07:22.082785 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:22.082695 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efd740a0-6578-40c0-8368-9d8bbeea64f1-kube-api-access-8z22f\" (UniqueName: \"kubernetes.io/projected/efd740a0-6578-40c0-8368-9d8bbeea64f1-kube-api-access-8z22f\") pod \"efd740a0-6578-40c0-8368-9d8bbeea64f1\" (UID: \"efd740a0-6578-40c0-8368-9d8bbeea64f1\") " Nov 03 14:07:22.082966 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:22.082810 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efd740a0-6578-40c0-8368-9d8bbeea64f1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efd740a0-6578-40c0-8368-9d8bbeea64f1-maas-api-tls\") pod \"efd740a0-6578-40c0-8368-9d8bbeea64f1\" (UID: \"efd740a0-6578-40c0-8368-9d8bbeea64f1\") " Nov 03 14:07:22.082966 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:22.082872 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efd740a0-6578-40c0-8368-9d8bbeea64f1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efd740a0-6578-40c0-8368-9d8bbeea64f1-openshift-service-ca\") pod \"efd740a0-6578-40c0-8368-9d8bbeea64f1\" (UID: \"efd740a0-6578-40c0-8368-9d8bbeea64f1\") " Nov 03 14:07:22.082966 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:22.082915 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efd740a0-6578-40c0-8368-9d8bbeea64f1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/efd740a0-6578-40c0-8368-9d8bbeea64f1-metrics-certs\") pod \"efd740a0-6578-40c0-8368-9d8bbeea64f1\" (UID: \"efd740a0-6578-40c0-8368-9d8bbeea64f1\") " Nov 03 14:07:22.083319 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:22.083287 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efd740a0-6578-40c0-8368-9d8bbeea64f1-openshift-service-ca" pod "efd740a0-6578-40c0-8368-9d8bbeea64f1" (UID: "efd740a0-6578-40c0-8368-9d8bbeea64f1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:07:22.085098 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:22.085064 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efd740a0-6578-40c0-8368-9d8bbeea64f1-maas-api-tls" pod "efd740a0-6578-40c0-8368-9d8bbeea64f1" (UID: "efd740a0-6578-40c0-8368-9d8bbeea64f1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:07:22.085225 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:22.085180 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efd740a0-6578-40c0-8368-9d8bbeea64f1-metrics-certs" pod "efd740a0-6578-40c0-8368-9d8bbeea64f1" (UID: "efd740a0-6578-40c0-8368-9d8bbeea64f1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:07:22.085225 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:22.085203 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efd740a0-6578-40c0-8368-9d8bbeea64f1-kube-api-access-8z22f" pod "efd740a0-6578-40c0-8368-9d8bbeea64f1" (UID: "efd740a0-6578-40c0-8368-9d8bbeea64f1"). InnerVolumeSpecName "kube-api-access-8z22f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:07:22.184512 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:22.184434 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8z22f\" (UniqueName: \"kubernetes.io/projected/efd740a0-6578-40c0-8368-9d8bbeea64f1-kube-api-access-8z22f\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:22.184512 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:22.184495 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efd740a0-6578-40c0-8368-9d8bbeea64f1-maas-api-tls\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:22.184512 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:22.184513 5979 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efd740a0-6578-40c0-8368-9d8bbeea64f1-openshift-service-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:22.184856 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:22.184530 5979 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/efd740a0-6578-40c0-8368-9d8bbeea64f1-metrics-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:22.253848 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:22.253811 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7458df57f5-zrrlx"] Nov 03 14:07:22.260360 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:07:22.260333 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb34d5b3f_3449_4b91_8919_d278d0dc5112.slice/crio-b2cbf0c71c99603e06597b4cc3b6efba8259a81c876f62c297ac7ef9079c7ed7 WatchSource:0}: Error finding container b2cbf0c71c99603e06597b4cc3b6efba8259a81c876f62c297ac7ef9079c7ed7: Status 404 returned error can't find the container with id b2cbf0c71c99603e06597b4cc3b6efba8259a81c876f62c297ac7ef9079c7ed7 Nov 03 14:07:22.642590 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:22.641047 5979 generic.go:356] "Generic (PLEG): container finished" podID="efd740a0-6578-40c0-8368-9d8bbeea64f1" containerID="8ddfb0dc8dc5d40d1b857c9d5a0eac401db0ea981f135a57af0aaead2feecc84" exitCode=0 Nov 03 14:07:22.642590 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:22.641137 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae248d-789b45f9c9-8kbb6" event={"ID":"efd740a0-6578-40c0-8368-9d8bbeea64f1","Type":"ContainerDied","Data":"8ddfb0dc8dc5d40d1b857c9d5a0eac401db0ea981f135a57af0aaead2feecc84"} Nov 03 14:07:22.642590 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:22.641193 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae248d-789b45f9c9-8kbb6" event={"ID":"efd740a0-6578-40c0-8368-9d8bbeea64f1","Type":"ContainerDied","Data":"086e884f6bc9c2a87e5547fca58aad8261bece1bc6c4786fad6dff6ae208db2c"} Nov 03 14:07:22.642590 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:22.641219 5979 scope.go:122] "RemoveContainer" containerID="8ddfb0dc8dc5d40d1b857c9d5a0eac401db0ea981f135a57af0aaead2feecc84" Nov 03 14:07:22.642590 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:22.641968 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae248d-789b45f9c9-8kbb6" Nov 03 14:07:22.645610 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:22.644465 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7458df57f5-zrrlx" event={"ID":"b34d5b3f-3449-4b91-8919-d278d0dc5112","Type":"ContainerStarted","Data":"69253c9896a05dcf8e0a39833cab05d01cdfbf9f0ef2d45f39e8307e0a59d70f"} Nov 03 14:07:22.645610 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:22.644539 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7458df57f5-zrrlx" event={"ID":"b34d5b3f-3449-4b91-8919-d278d0dc5112","Type":"ContainerStarted","Data":"b2cbf0c71c99603e06597b4cc3b6efba8259a81c876f62c297ac7ef9079c7ed7"} Nov 03 14:07:22.645610 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:22.645285 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7458df57f5-zrrlx" Nov 03 14:07:22.654958 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:22.654931 5979 scope.go:122] "RemoveContainer" containerID="8ddfb0dc8dc5d40d1b857c9d5a0eac401db0ea981f135a57af0aaead2feecc84" Nov 03 14:07:22.655273 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:07:22.655252 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ddfb0dc8dc5d40d1b857c9d5a0eac401db0ea981f135a57af0aaead2feecc84\": container with ID starting with 8ddfb0dc8dc5d40d1b857c9d5a0eac401db0ea981f135a57af0aaead2feecc84 not found: ID does not exist" containerID="8ddfb0dc8dc5d40d1b857c9d5a0eac401db0ea981f135a57af0aaead2feecc84" Nov 03 14:07:22.655339 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:22.655283 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ddfb0dc8dc5d40d1b857c9d5a0eac401db0ea981f135a57af0aaead2feecc84"} err="failed to get container status \"8ddfb0dc8dc5d40d1b857c9d5a0eac401db0ea981f135a57af0aaead2feecc84\": rpc error: code = NotFound desc = could not find container \"8ddfb0dc8dc5d40d1b857c9d5a0eac401db0ea981f135a57af0aaead2feecc84\": container with ID starting with 8ddfb0dc8dc5d40d1b857c9d5a0eac401db0ea981f135a57af0aaead2feecc84 not found: ID does not exist" Nov 03 14:07:22.800692 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:22.800639 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7458df57f5-zrrlx" podStartSLOduration=1.8006238460000001 podStartE2EDuration="1.800623846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:07:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:07:22.799303941 +0000 UTC m=+3724.809992187" watchObservedRunningTime="2026-11-03 14:07:22.800623846 +0000 UTC m=+3724.811312088" Nov 03 14:07:22.905183 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:22.905103 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae248d-789b45f9c9-8kbb6"] Nov 03 14:07:22.909229 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:22.909204 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ae248d-789b45f9c9-8kbb6"] Nov 03 14:07:23.674512 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:23.674460 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae248d-vvfwm"] Nov 03 14:07:23.677628 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:23.677600 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-ae248d-vvfwm"] Nov 03 14:07:23.889912 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:23.889876 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44"] Nov 03 14:07:23.890417 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:23.890389 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44" podUID="e0e4153d-37d8-4233-8744-7515f9878bdf" containerName="istio-proxy" containerID="cri-o://53b27204ee5be8299e66bef903229b786ab8cc099622126ce25c40863ab33643" gracePeriod=30 Nov 03 14:07:24.117458 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:24.117378 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9896368-75e8-48ca-a563-a95befb454ef" path="/var/lib/kubelet/pods/b9896368-75e8-48ca-a563-a95befb454ef/volumes" Nov 03 14:07:24.117880 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:24.117865 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efd740a0-6578-40c0-8368-9d8bbeea64f1" path="/var/lib/kubelet/pods/efd740a0-6578-40c0-8368-9d8bbeea64f1/volumes" Nov 03 14:07:24.742562 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:07:24.742530 5979 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-d4ee3f6c80de: secret "agc-cred-d4ee3f6c80de" not found Nov 03 14:07:24.742562 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:07:24.742556 5979 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7458df57f5-zrrlx: secret "agc-cred-d4ee3f6c80de" not found Nov 03 14:07:24.743290 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:07:24.742614 5979 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b34d5b3f-3449-4b91-8919-d278d0dc5112-provider-credentials podName:b34d5b3f-3449-4b91-8919-d278d0dc5112 nodeName:}" failed. No retries permitted until 2026-11-03 14:07:25.242593641 +0000 UTC m=+3727.253281862 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b34d5b3f-3449-4b91-8919-d278d0dc5112-provider-credentials") pod "payload-processing-external-model-7458df57f5-zrrlx" (UID: "b34d5b3f-3449-4b91-8919-d278d0dc5112") : secret "agc-cred-d4ee3f6c80de" not found Nov 03 14:07:25.247288 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:07:25.247254 5979 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-d4ee3f6c80de: secret "agc-cred-d4ee3f6c80de" not found Nov 03 14:07:25.247288 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:07:25.247284 5979 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7458df57f5-zrrlx: secret "agc-cred-d4ee3f6c80de" not found Nov 03 14:07:25.247572 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:07:25.247334 5979 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b34d5b3f-3449-4b91-8919-d278d0dc5112-provider-credentials podName:b34d5b3f-3449-4b91-8919-d278d0dc5112 nodeName:}" failed. No retries permitted until 2026-11-03 14:07:26.247321424 +0000 UTC m=+3728.258009645 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b34d5b3f-3449-4b91-8919-d278d0dc5112-provider-credentials") pod "payload-processing-external-model-7458df57f5-zrrlx" (UID: "b34d5b3f-3449-4b91-8919-d278d0dc5112") : secret "agc-cred-d4ee3f6c80de" not found Nov 03 14:07:25.702520 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:25.702451 5979 generic.go:356] "Generic (PLEG): container finished" podID="5ab327a8-7012-4504-9557-6dd31ed7a365" containerID="da2ca747ec56fffdd8cfab01245b312d773fc8b360eecd5c63a540e28dc57ade" exitCode=0 Nov 03 14:07:25.702520 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:25.702500 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6" event={"ID":"5ab327a8-7012-4504-9557-6dd31ed7a365","Type":"ContainerDied","Data":"da2ca747ec56fffdd8cfab01245b312d773fc8b360eecd5c63a540e28dc57ade"} Nov 03 14:07:25.702777 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:25.702580 5979 scope.go:122] "RemoveContainer" containerID="b4f4f426888318cccd58a37b998208567c7222a935db9d014cfea1cb15ebf6bf" Nov 03 14:07:25.754836 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:25.754813 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6" Nov 03 14:07:25.758626 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:25.758598 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ab327a8-7012-4504-9557-6dd31ed7a365-istio-token\" (UniqueName: \"kubernetes.io/projected/5ab327a8-7012-4504-9557-6dd31ed7a365-istio-token\") pod \"5ab327a8-7012-4504-9557-6dd31ed7a365\" (UID: \"5ab327a8-7012-4504-9557-6dd31ed7a365\") " Nov 03 14:07:25.758771 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:25.758663 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ab327a8-7012-4504-9557-6dd31ed7a365-kube-api-access-5pvz2\" (UniqueName: \"kubernetes.io/projected/5ab327a8-7012-4504-9557-6dd31ed7a365-kube-api-access-5pvz2\") pod \"5ab327a8-7012-4504-9557-6dd31ed7a365\" (UID: \"5ab327a8-7012-4504-9557-6dd31ed7a365\") " Nov 03 14:07:25.758771 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:25.758750 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ab327a8-7012-4504-9557-6dd31ed7a365-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ab327a8-7012-4504-9557-6dd31ed7a365-istiod-ca-cert\") pod \"5ab327a8-7012-4504-9557-6dd31ed7a365\" (UID: \"5ab327a8-7012-4504-9557-6dd31ed7a365\") " Nov 03 14:07:25.758895 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:25.758828 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ab327a8-7012-4504-9557-6dd31ed7a365-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ab327a8-7012-4504-9557-6dd31ed7a365-istio-podinfo\") pod \"5ab327a8-7012-4504-9557-6dd31ed7a365\" (UID: \"5ab327a8-7012-4504-9557-6dd31ed7a365\") " Nov 03 14:07:25.758958 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:25.758926 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ab327a8-7012-4504-9557-6dd31ed7a365-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ab327a8-7012-4504-9557-6dd31ed7a365-istio-data\") pod \"5ab327a8-7012-4504-9557-6dd31ed7a365\" (UID: \"5ab327a8-7012-4504-9557-6dd31ed7a365\") " Nov 03 14:07:25.759020 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:25.758959 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ab327a8-7012-4504-9557-6dd31ed7a365-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ab327a8-7012-4504-9557-6dd31ed7a365-workload-socket\") pod \"5ab327a8-7012-4504-9557-6dd31ed7a365\" (UID: \"5ab327a8-7012-4504-9557-6dd31ed7a365\") " Nov 03 14:07:25.759077 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:25.759056 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ab327a8-7012-4504-9557-6dd31ed7a365-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ab327a8-7012-4504-9557-6dd31ed7a365-credential-socket\") pod \"5ab327a8-7012-4504-9557-6dd31ed7a365\" (UID: \"5ab327a8-7012-4504-9557-6dd31ed7a365\") " Nov 03 14:07:25.759135 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:25.759111 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ab327a8-7012-4504-9557-6dd31ed7a365-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ab327a8-7012-4504-9557-6dd31ed7a365-istio-envoy\") pod \"5ab327a8-7012-4504-9557-6dd31ed7a365\" (UID: \"5ab327a8-7012-4504-9557-6dd31ed7a365\") " Nov 03 14:07:25.759197 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:25.759174 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ab327a8-7012-4504-9557-6dd31ed7a365-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ab327a8-7012-4504-9557-6dd31ed7a365-workload-certs\") pod \"5ab327a8-7012-4504-9557-6dd31ed7a365\" (UID: \"5ab327a8-7012-4504-9557-6dd31ed7a365\") " Nov 03 14:07:25.759255 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:25.759190 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ab327a8-7012-4504-9557-6dd31ed7a365-istio-data" pod "5ab327a8-7012-4504-9557-6dd31ed7a365" (UID: "5ab327a8-7012-4504-9557-6dd31ed7a365"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:07:25.759255 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:25.759206 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ab327a8-7012-4504-9557-6dd31ed7a365-istiod-ca-cert" pod "5ab327a8-7012-4504-9557-6dd31ed7a365" (UID: "5ab327a8-7012-4504-9557-6dd31ed7a365"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:07:25.759444 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:25.759419 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ab327a8-7012-4504-9557-6dd31ed7a365-credential-socket" pod "5ab327a8-7012-4504-9557-6dd31ed7a365" (UID: "5ab327a8-7012-4504-9557-6dd31ed7a365"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:07:25.759699 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:25.759674 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ab327a8-7012-4504-9557-6dd31ed7a365-workload-socket" pod "5ab327a8-7012-4504-9557-6dd31ed7a365" (UID: "5ab327a8-7012-4504-9557-6dd31ed7a365"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:07:25.759893 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:25.759865 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ab327a8-7012-4504-9557-6dd31ed7a365-workload-certs" pod "5ab327a8-7012-4504-9557-6dd31ed7a365" (UID: "5ab327a8-7012-4504-9557-6dd31ed7a365"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:07:25.759893 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:25.759869 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ab327a8-7012-4504-9557-6dd31ed7a365-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:25.760094 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:25.759918 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ab327a8-7012-4504-9557-6dd31ed7a365-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:25.760094 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:25.759937 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ab327a8-7012-4504-9557-6dd31ed7a365-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:25.761110 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:25.761041 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ab327a8-7012-4504-9557-6dd31ed7a365-istio-podinfo" pod "5ab327a8-7012-4504-9557-6dd31ed7a365" (UID: "5ab327a8-7012-4504-9557-6dd31ed7a365"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:07:25.761350 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:25.761331 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ab327a8-7012-4504-9557-6dd31ed7a365-istio-token" pod "5ab327a8-7012-4504-9557-6dd31ed7a365" (UID: "5ab327a8-7012-4504-9557-6dd31ed7a365"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:07:25.761464 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:25.761438 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ab327a8-7012-4504-9557-6dd31ed7a365-kube-api-access-5pvz2" pod "5ab327a8-7012-4504-9557-6dd31ed7a365" (UID: "5ab327a8-7012-4504-9557-6dd31ed7a365"). InnerVolumeSpecName "kube-api-access-5pvz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:07:25.761764 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:25.761739 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ab327a8-7012-4504-9557-6dd31ed7a365-istio-envoy" pod "5ab327a8-7012-4504-9557-6dd31ed7a365" (UID: "5ab327a8-7012-4504-9557-6dd31ed7a365"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:07:25.861180 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:25.861144 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ab327a8-7012-4504-9557-6dd31ed7a365-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:25.861180 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:25.861176 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ab327a8-7012-4504-9557-6dd31ed7a365-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:25.861180 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:25.861186 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ab327a8-7012-4504-9557-6dd31ed7a365-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:25.861434 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:25.861195 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pvz2\" (UniqueName: \"kubernetes.io/projected/5ab327a8-7012-4504-9557-6dd31ed7a365-kube-api-access-5pvz2\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:25.861434 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:25.861207 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ab327a8-7012-4504-9557-6dd31ed7a365-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:25.861434 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:25.861216 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ab327a8-7012-4504-9557-6dd31ed7a365-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:26.266507 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:07:26.266431 5979 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-d4ee3f6c80de: secret "agc-cred-d4ee3f6c80de" not found Nov 03 14:07:26.266507 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:07:26.266501 5979 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7458df57f5-zrrlx: secret "agc-cred-d4ee3f6c80de" not found Nov 03 14:07:26.266720 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:07:26.266579 5979 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b34d5b3f-3449-4b91-8919-d278d0dc5112-provider-credentials podName:b34d5b3f-3449-4b91-8919-d278d0dc5112 nodeName:}" failed. No retries permitted until 2026-11-03 14:07:28.266558393 +0000 UTC m=+3730.277246630 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b34d5b3f-3449-4b91-8919-d278d0dc5112-provider-credentials") pod "payload-processing-external-model-7458df57f5-zrrlx" (UID: "b34d5b3f-3449-4b91-8919-d278d0dc5112") : secret "agc-cred-d4ee3f6c80de" not found Nov 03 14:07:26.525443 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:26.525369 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-78f749b598-jjswn" podUID="6a3788db-a41a-47f8-b73c-c695cfdc2801" containerName="payload-processing" containerID="cri-o://234529210fbf5507b524fb3f5b70a244c1e8be15c098457998c01e2d15b89330" gracePeriod=25 Nov 03 14:07:26.693318 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:26.693282 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7458df57f5-zrrlx"] Nov 03 14:07:26.718057 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:26.718021 5979 generic.go:356] "Generic (PLEG): container finished" podID="6a3788db-a41a-47f8-b73c-c695cfdc2801" containerID="234529210fbf5507b524fb3f5b70a244c1e8be15c098457998c01e2d15b89330" exitCode=0 Nov 03 14:07:26.718222 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:26.718076 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-78f749b598-jjswn" event={"ID":"6a3788db-a41a-47f8-b73c-c695cfdc2801","Type":"ContainerDied","Data":"234529210fbf5507b524fb3f5b70a244c1e8be15c098457998c01e2d15b89330"} Nov 03 14:07:26.719999 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:26.719951 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6" event={"ID":"5ab327a8-7012-4504-9557-6dd31ed7a365","Type":"ContainerDied","Data":"9d4abc6a4bdc7c7267a375e4d83d4e7d86ef70ae32a433c643dbf3e15e59c639"} Nov 03 14:07:26.719999 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:26.719982 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6" Nov 03 14:07:26.720213 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:26.720018 5979 scope.go:122] "RemoveContainer" containerID="da2ca747ec56fffdd8cfab01245b312d773fc8b360eecd5c63a540e28dc57ade" Nov 03 14:07:26.727020 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:26.726916 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-78f749b598-z7shp"] Nov 03 14:07:26.727830 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:26.727787 5979 cpu_manager.go:414] "containerMap: removing container" podUID="5ab327a8-7012-4504-9557-6dd31ed7a365" containerName="istio-proxy" Nov 03 14:07:26.727830 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:26.727810 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ab327a8-7012-4504-9557-6dd31ed7a365" containerName="istio-proxy" Nov 03 14:07:26.728043 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:26.727849 5979 cpu_manager.go:414] "containerMap: removing container" podUID="efd740a0-6578-40c0-8368-9d8bbeea64f1" containerName="maas-api" Nov 03 14:07:26.728043 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:26.727858 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efd740a0-6578-40c0-8368-9d8bbeea64f1" containerName="maas-api" Nov 03 14:07:26.728043 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:26.727885 5979 cpu_manager.go:414] "containerMap: removing container" podUID="5ab327a8-7012-4504-9557-6dd31ed7a365" containerName="istio-proxy" Nov 03 14:07:26.728043 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:26.727893 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ab327a8-7012-4504-9557-6dd31ed7a365" containerName="istio-proxy" Nov 03 14:07:26.728257 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:26.728061 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="efd740a0-6578-40c0-8368-9d8bbeea64f1" containerName="maas-api" Nov 03 14:07:26.728257 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:26.728083 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ab327a8-7012-4504-9557-6dd31ed7a365" containerName="istio-proxy" Nov 03 14:07:26.732001 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:26.731981 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-78f749b598-z7shp" Nov 03 14:07:26.764979 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:26.764946 5979 patch_prober.go:29] interesting pod/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 14:07:26.765404 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:26.765015 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44" podUID="e0e4153d-37d8-4233-8744-7515f9878bdf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 14:07:26.772683 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:26.772645 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9174be51-25e6-4996-b121-c15133d0a923-provider-credentials\") pod \"payload-processing-external-model-78f749b598-z7shp\" (UID: \"9174be51-25e6-4996-b121-c15133d0a923\") " pod="models-as-a-service/payload-processing-external-model-78f749b598-z7shp" Nov 03 14:07:26.772845 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:26.772815 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9174be51-25e6-4996-b121-c15133d0a923-plugins-config-volume\") pod \"payload-processing-external-model-78f749b598-z7shp\" (UID: \"9174be51-25e6-4996-b121-c15133d0a923\") " pod="models-as-a-service/payload-processing-external-model-78f749b598-z7shp" Nov 03 14:07:26.773009 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:26.772910 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn8cn\" (UniqueName: \"kubernetes.io/projected/9174be51-25e6-4996-b121-c15133d0a923-kube-api-access-xn8cn\") pod \"payload-processing-external-model-78f749b598-z7shp\" (UID: \"9174be51-25e6-4996-b121-c15133d0a923\") " pod="models-as-a-service/payload-processing-external-model-78f749b598-z7shp" Nov 03 14:07:26.773009 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:26.772948 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9174be51-25e6-4996-b121-c15133d0a923-routing-overlay-volume\") pod \"payload-processing-external-model-78f749b598-z7shp\" (UID: \"9174be51-25e6-4996-b121-c15133d0a923\") " pod="models-as-a-service/payload-processing-external-model-78f749b598-z7shp" Nov 03 14:07:26.799360 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:26.799335 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-78f749b598-z7shp"] Nov 03 14:07:26.807795 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:26.804396 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-78f749b598-jjswn" Nov 03 14:07:26.874336 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:26.874302 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a3788db-a41a-47f8-b73c-c695cfdc2801-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6a3788db-a41a-47f8-b73c-c695cfdc2801-routing-overlay-volume\") pod \"6a3788db-a41a-47f8-b73c-c695cfdc2801\" (UID: \"6a3788db-a41a-47f8-b73c-c695cfdc2801\") " Nov 03 14:07:26.874567 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:26.874399 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a3788db-a41a-47f8-b73c-c695cfdc2801-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6a3788db-a41a-47f8-b73c-c695cfdc2801-provider-credentials\") pod \"6a3788db-a41a-47f8-b73c-c695cfdc2801\" (UID: \"6a3788db-a41a-47f8-b73c-c695cfdc2801\") " Nov 03 14:07:26.874567 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:26.874442 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a3788db-a41a-47f8-b73c-c695cfdc2801-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a3788db-a41a-47f8-b73c-c695cfdc2801-plugins-config-volume\") pod \"6a3788db-a41a-47f8-b73c-c695cfdc2801\" (UID: \"6a3788db-a41a-47f8-b73c-c695cfdc2801\") " Nov 03 14:07:26.874731 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:26.874711 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a3788db-a41a-47f8-b73c-c695cfdc2801-kube-api-access-pk84t\" (UniqueName: \"kubernetes.io/projected/6a3788db-a41a-47f8-b73c-c695cfdc2801-kube-api-access-pk84t\") pod \"6a3788db-a41a-47f8-b73c-c695cfdc2801\" (UID: \"6a3788db-a41a-47f8-b73c-c695cfdc2801\") " Nov 03 14:07:26.874817 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:26.874798 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a3788db-a41a-47f8-b73c-c695cfdc2801-routing-overlay-volume" pod "6a3788db-a41a-47f8-b73c-c695cfdc2801" (UID: "6a3788db-a41a-47f8-b73c-c695cfdc2801"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:07:26.874919 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:26.874893 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a3788db-a41a-47f8-b73c-c695cfdc2801-plugins-config-volume" pod "6a3788db-a41a-47f8-b73c-c695cfdc2801" (UID: "6a3788db-a41a-47f8-b73c-c695cfdc2801"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:07:26.874919 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:26.874907 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn8cn\" (UniqueName: \"kubernetes.io/projected/9174be51-25e6-4996-b121-c15133d0a923-kube-api-access-xn8cn\") pod \"payload-processing-external-model-78f749b598-z7shp\" (UID: \"9174be51-25e6-4996-b121-c15133d0a923\") " pod="models-as-a-service/payload-processing-external-model-78f749b598-z7shp" Nov 03 14:07:26.875095 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:26.874980 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9174be51-25e6-4996-b121-c15133d0a923-routing-overlay-volume\") pod \"payload-processing-external-model-78f749b598-z7shp\" (UID: \"9174be51-25e6-4996-b121-c15133d0a923\") " pod="models-as-a-service/payload-processing-external-model-78f749b598-z7shp" Nov 03 14:07:26.875095 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:26.875028 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9174be51-25e6-4996-b121-c15133d0a923-provider-credentials\") pod \"payload-processing-external-model-78f749b598-z7shp\" (UID: \"9174be51-25e6-4996-b121-c15133d0a923\") " pod="models-as-a-service/payload-processing-external-model-78f749b598-z7shp" Nov 03 14:07:26.875213 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:26.875175 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9174be51-25e6-4996-b121-c15133d0a923-plugins-config-volume\") pod \"payload-processing-external-model-78f749b598-z7shp\" (UID: \"9174be51-25e6-4996-b121-c15133d0a923\") " pod="models-as-a-service/payload-processing-external-model-78f749b598-z7shp" Nov 03 14:07:26.875303 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:26.875288 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a3788db-a41a-47f8-b73c-c695cfdc2801-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:26.875367 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:26.875306 5979 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6a3788db-a41a-47f8-b73c-c695cfdc2801-routing-overlay-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:26.875882 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:26.875852 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9174be51-25e6-4996-b121-c15133d0a923-routing-overlay-volume\") pod \"payload-processing-external-model-78f749b598-z7shp\" (UID: \"9174be51-25e6-4996-b121-c15133d0a923\") " pod="models-as-a-service/payload-processing-external-model-78f749b598-z7shp" Nov 03 14:07:26.876030 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:26.876009 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9174be51-25e6-4996-b121-c15133d0a923-plugins-config-volume\") pod \"payload-processing-external-model-78f749b598-z7shp\" (UID: \"9174be51-25e6-4996-b121-c15133d0a923\") " pod="models-as-a-service/payload-processing-external-model-78f749b598-z7shp" Nov 03 14:07:26.876944 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:26.876919 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a3788db-a41a-47f8-b73c-c695cfdc2801-provider-credentials" pod "6a3788db-a41a-47f8-b73c-c695cfdc2801" (UID: "6a3788db-a41a-47f8-b73c-c695cfdc2801"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:07:26.877049 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:26.876953 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a3788db-a41a-47f8-b73c-c695cfdc2801-kube-api-access-pk84t" pod "6a3788db-a41a-47f8-b73c-c695cfdc2801" (UID: "6a3788db-a41a-47f8-b73c-c695cfdc2801"). InnerVolumeSpecName "kube-api-access-pk84t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:07:26.877538 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:26.877521 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9174be51-25e6-4996-b121-c15133d0a923-provider-credentials\") pod \"payload-processing-external-model-78f749b598-z7shp\" (UID: \"9174be51-25e6-4996-b121-c15133d0a923\") " pod="models-as-a-service/payload-processing-external-model-78f749b598-z7shp" Nov 03 14:07:26.928829 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:26.928804 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn8cn\" (UniqueName: \"kubernetes.io/projected/9174be51-25e6-4996-b121-c15133d0a923-kube-api-access-xn8cn\") pod \"payload-processing-external-model-78f749b598-z7shp\" (UID: \"9174be51-25e6-4996-b121-c15133d0a923\") " pod="models-as-a-service/payload-processing-external-model-78f749b598-z7shp" Nov 03 14:07:26.976338 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:26.976304 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pk84t\" (UniqueName: \"kubernetes.io/projected/6a3788db-a41a-47f8-b73c-c695cfdc2801-kube-api-access-pk84t\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:26.976338 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:26.976338 5979 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6a3788db-a41a-47f8-b73c-c695cfdc2801-provider-credentials\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:27.018137 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:27.018103 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6"] Nov 03 14:07:27.025531 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:27.025498 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-c4e398-openshift-default-7df68c6c4f-k5sq6"] Nov 03 14:07:27.049222 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:27.049159 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-78f749b598-z7shp" Nov 03 14:07:27.424744 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:27.424711 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-78f749b598-z7shp"] Nov 03 14:07:27.431375 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:07:27.431344 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9174be51_25e6_4996_b121_c15133d0a923.slice/crio-fa9db13d96817026a1336696a093171b355945c86072b3ad7edb3d3fc0f372f7 WatchSource:0}: Error finding container fa9db13d96817026a1336696a093171b355945c86072b3ad7edb3d3fc0f372f7: Status 404 returned error can't find the container with id fa9db13d96817026a1336696a093171b355945c86072b3ad7edb3d3fc0f372f7 Nov 03 14:07:27.728546 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:27.728425 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-78f749b598-z7shp" event={"ID":"9174be51-25e6-4996-b121-c15133d0a923","Type":"ContainerStarted","Data":"206dfc764264207d0a7044fd908b159992ebf6e2ca8b014f40e8d3e910633290"} Nov 03 14:07:27.728546 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:27.728501 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-78f749b598-z7shp" event={"ID":"9174be51-25e6-4996-b121-c15133d0a923","Type":"ContainerStarted","Data":"fa9db13d96817026a1336696a093171b355945c86072b3ad7edb3d3fc0f372f7"} Nov 03 14:07:27.728768 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:27.728646 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-78f749b598-z7shp" Nov 03 14:07:27.730970 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:27.730944 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-78f749b598-jjswn" event={"ID":"6a3788db-a41a-47f8-b73c-c695cfdc2801","Type":"ContainerDied","Data":"bee14bf11708dd496f94e3d2cc184bc06d0b7cb9964479a315f32cc33f9986ff"} Nov 03 14:07:27.731131 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:27.730973 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-78f749b598-jjswn" Nov 03 14:07:27.731131 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:27.730993 5979 scope.go:122] "RemoveContainer" containerID="234529210fbf5507b524fb3f5b70a244c1e8be15c098457998c01e2d15b89330" Nov 03 14:07:27.836766 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:27.836707 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-78f749b598-z7shp" podStartSLOduration=1.8366878469999999 podStartE2EDuration="1.836687847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:07:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:07:27.83605812 +0000 UTC m=+3729.846746374" watchObservedRunningTime="2026-11-03 14:07:27.836687847 +0000 UTC m=+3729.847376088" Nov 03 14:07:27.941620 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:27.941582 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-78f749b598-jjswn"] Nov 03 14:07:27.945607 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:27.945582 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-78f749b598-jjswn"] Nov 03 14:07:28.116327 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:28.116250 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ab327a8-7012-4504-9557-6dd31ed7a365" path="/var/lib/kubelet/pods/5ab327a8-7012-4504-9557-6dd31ed7a365/volumes" Nov 03 14:07:28.116727 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:28.116713 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a3788db-a41a-47f8-b73c-c695cfdc2801" path="/var/lib/kubelet/pods/6a3788db-a41a-47f8-b73c-c695cfdc2801/volumes" Nov 03 14:07:28.296872 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:07:28.296837 5979 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-d4ee3f6c80de: secret "agc-cred-d4ee3f6c80de" not found Nov 03 14:07:28.296872 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:07:28.296866 5979 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7458df57f5-zrrlx: secret "agc-cred-d4ee3f6c80de" not found Nov 03 14:07:28.297102 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:07:28.296918 5979 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b34d5b3f-3449-4b91-8919-d278d0dc5112-provider-credentials podName:b34d5b3f-3449-4b91-8919-d278d0dc5112 nodeName:}" failed. No retries permitted until 2026-11-03 14:07:32.296903578 +0000 UTC m=+3734.307591798 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b34d5b3f-3449-4b91-8919-d278d0dc5112-provider-credentials") pod "payload-processing-external-model-7458df57f5-zrrlx" (UID: "b34d5b3f-3449-4b91-8919-d278d0dc5112") : secret "agc-cred-d4ee3f6c80de" not found Nov 03 14:07:29.154143 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:29.154117 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44" Nov 03 14:07:29.208252 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:29.208166 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e0e4153d-37d8-4233-8744-7515f9878bdf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0e4153d-37d8-4233-8744-7515f9878bdf-istio-podinfo\") pod \"e0e4153d-37d8-4233-8744-7515f9878bdf\" (UID: \"e0e4153d-37d8-4233-8744-7515f9878bdf\") " Nov 03 14:07:29.208252 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:29.208204 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0e4153d-37d8-4233-8744-7515f9878bdf-istio-token\" (UniqueName: \"kubernetes.io/projected/e0e4153d-37d8-4233-8744-7515f9878bdf-istio-token\") pod \"e0e4153d-37d8-4233-8744-7515f9878bdf\" (UID: \"e0e4153d-37d8-4233-8744-7515f9878bdf\") " Nov 03 14:07:29.208531 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:29.208276 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0e4153d-37d8-4233-8744-7515f9878bdf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0e4153d-37d8-4233-8744-7515f9878bdf-istio-data\") pod \"e0e4153d-37d8-4233-8744-7515f9878bdf\" (UID: \"e0e4153d-37d8-4233-8744-7515f9878bdf\") " Nov 03 14:07:29.208531 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:29.208399 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0e4153d-37d8-4233-8744-7515f9878bdf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0e4153d-37d8-4233-8744-7515f9878bdf-workload-certs\") pod \"e0e4153d-37d8-4233-8744-7515f9878bdf\" (UID: \"e0e4153d-37d8-4233-8744-7515f9878bdf\") " Nov 03 14:07:29.208531 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:29.208501 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0e4153d-37d8-4233-8744-7515f9878bdf-kube-api-access-9tpl4\" (UniqueName: \"kubernetes.io/projected/e0e4153d-37d8-4233-8744-7515f9878bdf-kube-api-access-9tpl4\") pod \"e0e4153d-37d8-4233-8744-7515f9878bdf\" (UID: \"e0e4153d-37d8-4233-8744-7515f9878bdf\") " Nov 03 14:07:29.208726 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:29.208558 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0e4153d-37d8-4233-8744-7515f9878bdf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0e4153d-37d8-4233-8744-7515f9878bdf-istiod-ca-cert\") pod \"e0e4153d-37d8-4233-8744-7515f9878bdf\" (UID: \"e0e4153d-37d8-4233-8744-7515f9878bdf\") " Nov 03 14:07:29.208726 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:29.208598 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0e4153d-37d8-4233-8744-7515f9878bdf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0e4153d-37d8-4233-8744-7515f9878bdf-credential-socket\") pod \"e0e4153d-37d8-4233-8744-7515f9878bdf\" (UID: \"e0e4153d-37d8-4233-8744-7515f9878bdf\") " Nov 03 14:07:29.208726 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:29.208645 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0e4153d-37d8-4233-8744-7515f9878bdf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0e4153d-37d8-4233-8744-7515f9878bdf-istio-envoy\") pod \"e0e4153d-37d8-4233-8744-7515f9878bdf\" (UID: \"e0e4153d-37d8-4233-8744-7515f9878bdf\") " Nov 03 14:07:29.208726 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:29.208662 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0e4153d-37d8-4233-8744-7515f9878bdf-istio-data" pod "e0e4153d-37d8-4233-8744-7515f9878bdf" (UID: "e0e4153d-37d8-4233-8744-7515f9878bdf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:07:29.208726 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:29.208706 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0e4153d-37d8-4233-8744-7515f9878bdf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0e4153d-37d8-4233-8744-7515f9878bdf-workload-socket\") pod \"e0e4153d-37d8-4233-8744-7515f9878bdf\" (UID: \"e0e4153d-37d8-4233-8744-7515f9878bdf\") " Nov 03 14:07:29.208726 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:29.208707 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0e4153d-37d8-4233-8744-7515f9878bdf-workload-certs" pod "e0e4153d-37d8-4233-8744-7515f9878bdf" (UID: "e0e4153d-37d8-4233-8744-7515f9878bdf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:07:29.209070 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:29.208922 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0e4153d-37d8-4233-8744-7515f9878bdf-credential-socket" pod "e0e4153d-37d8-4233-8744-7515f9878bdf" (UID: "e0e4153d-37d8-4233-8744-7515f9878bdf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:07:29.209070 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:29.209003 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0e4153d-37d8-4233-8744-7515f9878bdf-istiod-ca-cert" pod "e0e4153d-37d8-4233-8744-7515f9878bdf" (UID: "e0e4153d-37d8-4233-8744-7515f9878bdf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:07:29.209186 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:29.209096 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0e4153d-37d8-4233-8744-7515f9878bdf-workload-socket" pod "e0e4153d-37d8-4233-8744-7515f9878bdf" (UID: "e0e4153d-37d8-4233-8744-7515f9878bdf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:07:29.209373 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:29.209349 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0e4153d-37d8-4233-8744-7515f9878bdf-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:29.209446 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:29.209378 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0e4153d-37d8-4233-8744-7515f9878bdf-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:29.209446 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:29.209393 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0e4153d-37d8-4233-8744-7515f9878bdf-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:29.209446 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:29.209410 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0e4153d-37d8-4233-8744-7515f9878bdf-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:29.209446 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:29.209424 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0e4153d-37d8-4233-8744-7515f9878bdf-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:29.210709 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:29.210684 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0e4153d-37d8-4233-8744-7515f9878bdf-istio-token" pod "e0e4153d-37d8-4233-8744-7515f9878bdf" (UID: "e0e4153d-37d8-4233-8744-7515f9878bdf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:07:29.210709 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:29.210695 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e0e4153d-37d8-4233-8744-7515f9878bdf-istio-podinfo" pod "e0e4153d-37d8-4233-8744-7515f9878bdf" (UID: "e0e4153d-37d8-4233-8744-7515f9878bdf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:07:29.211177 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:29.211155 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0e4153d-37d8-4233-8744-7515f9878bdf-istio-envoy" pod "e0e4153d-37d8-4233-8744-7515f9878bdf" (UID: "e0e4153d-37d8-4233-8744-7515f9878bdf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:07:29.211177 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:29.211177 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0e4153d-37d8-4233-8744-7515f9878bdf-kube-api-access-9tpl4" pod "e0e4153d-37d8-4233-8744-7515f9878bdf" (UID: "e0e4153d-37d8-4233-8744-7515f9878bdf"). InnerVolumeSpecName "kube-api-access-9tpl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:07:29.311012 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:29.310978 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tpl4\" (UniqueName: \"kubernetes.io/projected/e0e4153d-37d8-4233-8744-7515f9878bdf-kube-api-access-9tpl4\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:29.311012 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:29.311008 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0e4153d-37d8-4233-8744-7515f9878bdf-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:29.311012 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:29.311018 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0e4153d-37d8-4233-8744-7515f9878bdf-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:29.311284 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:29.311027 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0e4153d-37d8-4233-8744-7515f9878bdf-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:29.748047 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:29.748007 5979 generic.go:356] "Generic (PLEG): container finished" podID="e0e4153d-37d8-4233-8744-7515f9878bdf" containerID="53b27204ee5be8299e66bef903229b786ab8cc099622126ce25c40863ab33643" exitCode=0 Nov 03 14:07:29.748254 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:29.748092 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44" Nov 03 14:07:29.748254 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:29.748099 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44" event={"ID":"e0e4153d-37d8-4233-8744-7515f9878bdf","Type":"ContainerDied","Data":"53b27204ee5be8299e66bef903229b786ab8cc099622126ce25c40863ab33643"} Nov 03 14:07:29.748254 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:29.748215 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44" event={"ID":"e0e4153d-37d8-4233-8744-7515f9878bdf","Type":"ContainerDied","Data":"6b965580cf5acf314b6709e30d9fda871ce94e52eab9bb375f7fda0b5fbb3daf"} Nov 03 14:07:29.748422 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:29.748258 5979 scope.go:122] "RemoveContainer" containerID="53b27204ee5be8299e66bef903229b786ab8cc099622126ce25c40863ab33643" Nov 03 14:07:29.760244 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:29.760226 5979 scope.go:122] "RemoveContainer" containerID="8313cbab19ffb9001c91a3a40203b463da8cb6e7b820af64d57a964a311ed62c" Nov 03 14:07:29.769527 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:29.769504 5979 scope.go:122] "RemoveContainer" containerID="53b27204ee5be8299e66bef903229b786ab8cc099622126ce25c40863ab33643" Nov 03 14:07:29.769793 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:07:29.769774 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53b27204ee5be8299e66bef903229b786ab8cc099622126ce25c40863ab33643\": container with ID starting with 53b27204ee5be8299e66bef903229b786ab8cc099622126ce25c40863ab33643 not found: ID does not exist" containerID="53b27204ee5be8299e66bef903229b786ab8cc099622126ce25c40863ab33643" Nov 03 14:07:29.769861 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:29.769801 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53b27204ee5be8299e66bef903229b786ab8cc099622126ce25c40863ab33643"} err="failed to get container status \"53b27204ee5be8299e66bef903229b786ab8cc099622126ce25c40863ab33643\": rpc error: code = NotFound desc = could not find container \"53b27204ee5be8299e66bef903229b786ab8cc099622126ce25c40863ab33643\": container with ID starting with 53b27204ee5be8299e66bef903229b786ab8cc099622126ce25c40863ab33643 not found: ID does not exist" Nov 03 14:07:29.769861 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:29.769818 5979 scope.go:122] "RemoveContainer" containerID="8313cbab19ffb9001c91a3a40203b463da8cb6e7b820af64d57a964a311ed62c" Nov 03 14:07:29.770028 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:07:29.770013 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8313cbab19ffb9001c91a3a40203b463da8cb6e7b820af64d57a964a311ed62c\": container with ID starting with 8313cbab19ffb9001c91a3a40203b463da8cb6e7b820af64d57a964a311ed62c not found: ID does not exist" containerID="8313cbab19ffb9001c91a3a40203b463da8cb6e7b820af64d57a964a311ed62c" Nov 03 14:07:29.770073 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:29.770033 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8313cbab19ffb9001c91a3a40203b463da8cb6e7b820af64d57a964a311ed62c"} err="failed to get container status \"8313cbab19ffb9001c91a3a40203b463da8cb6e7b820af64d57a964a311ed62c\": rpc error: code = NotFound desc = could not find container \"8313cbab19ffb9001c91a3a40203b463da8cb6e7b820af64d57a964a311ed62c\": container with ID starting with 8313cbab19ffb9001c91a3a40203b463da8cb6e7b820af64d57a964a311ed62c not found: ID does not exist" Nov 03 14:07:29.902795 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:29.902762 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44"] Nov 03 14:07:29.949519 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:29.949487 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-ae248d-openshift-default-5768d77597-rbw44"] Nov 03 14:07:30.115359 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:30.115278 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0e4153d-37d8-4233-8744-7515f9878bdf" path="/var/lib/kubelet/pods/e0e4153d-37d8-4233-8744-7515f9878bdf/volumes" Nov 03 14:07:30.860960 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:30.860925 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d5d415-676fcbc795-x4w5c"] Nov 03 14:07:31.694430 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:31.694390 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7458df57f5-zrrlx" podUID="b34d5b3f-3449-4b91-8919-d278d0dc5112" containerName="payload-processing" containerID="cri-o://69253c9896a05dcf8e0a39833cab05d01cdfbf9f0ef2d45f39e8307e0a59d70f" gracePeriod=25 Nov 03 14:07:31.966682 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:31.966657 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7458df57f5-zrrlx" Nov 03 14:07:32.049511 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:32.049465 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b34d5b3f-3449-4b91-8919-d278d0dc5112-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b34d5b3f-3449-4b91-8919-d278d0dc5112-routing-overlay-volume\") pod \"b34d5b3f-3449-4b91-8919-d278d0dc5112\" (UID: \"b34d5b3f-3449-4b91-8919-d278d0dc5112\") " Nov 03 14:07:32.049689 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:32.049581 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b34d5b3f-3449-4b91-8919-d278d0dc5112-kube-api-access-5rd8s\" (UniqueName: \"kubernetes.io/projected/b34d5b3f-3449-4b91-8919-d278d0dc5112-kube-api-access-5rd8s\") pod \"b34d5b3f-3449-4b91-8919-d278d0dc5112\" (UID: \"b34d5b3f-3449-4b91-8919-d278d0dc5112\") " Nov 03 14:07:32.049751 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:32.049736 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b34d5b3f-3449-4b91-8919-d278d0dc5112-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b34d5b3f-3449-4b91-8919-d278d0dc5112-provider-credentials\") pod \"b34d5b3f-3449-4b91-8919-d278d0dc5112\" (UID: \"b34d5b3f-3449-4b91-8919-d278d0dc5112\") " Nov 03 14:07:32.049791 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:32.049781 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b34d5b3f-3449-4b91-8919-d278d0dc5112-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b34d5b3f-3449-4b91-8919-d278d0dc5112-plugins-config-volume\") pod \"b34d5b3f-3449-4b91-8919-d278d0dc5112\" (UID: \"b34d5b3f-3449-4b91-8919-d278d0dc5112\") " Nov 03 14:07:32.049862 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:32.049792 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b34d5b3f-3449-4b91-8919-d278d0dc5112-routing-overlay-volume" pod "b34d5b3f-3449-4b91-8919-d278d0dc5112" (UID: "b34d5b3f-3449-4b91-8919-d278d0dc5112"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:07:32.050093 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:32.050071 5979 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b34d5b3f-3449-4b91-8919-d278d0dc5112-routing-overlay-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:32.050245 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:32.050179 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b34d5b3f-3449-4b91-8919-d278d0dc5112-plugins-config-volume" pod "b34d5b3f-3449-4b91-8919-d278d0dc5112" (UID: "b34d5b3f-3449-4b91-8919-d278d0dc5112"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:07:32.051818 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:32.051795 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b34d5b3f-3449-4b91-8919-d278d0dc5112-provider-credentials" pod "b34d5b3f-3449-4b91-8919-d278d0dc5112" (UID: "b34d5b3f-3449-4b91-8919-d278d0dc5112"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:07:32.051920 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:32.051853 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b34d5b3f-3449-4b91-8919-d278d0dc5112-kube-api-access-5rd8s" pod "b34d5b3f-3449-4b91-8919-d278d0dc5112" (UID: "b34d5b3f-3449-4b91-8919-d278d0dc5112"). InnerVolumeSpecName "kube-api-access-5rd8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:07:32.151807 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:32.151751 5979 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b34d5b3f-3449-4b91-8919-d278d0dc5112-provider-credentials\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:32.151807 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:32.151787 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b34d5b3f-3449-4b91-8919-d278d0dc5112-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:32.151807 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:32.151805 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rd8s\" (UniqueName: \"kubernetes.io/projected/b34d5b3f-3449-4b91-8919-d278d0dc5112-kube-api-access-5rd8s\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:32.773989 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:32.773956 5979 generic.go:356] "Generic (PLEG): container finished" podID="b34d5b3f-3449-4b91-8919-d278d0dc5112" containerID="69253c9896a05dcf8e0a39833cab05d01cdfbf9f0ef2d45f39e8307e0a59d70f" exitCode=0 Nov 03 14:07:32.774183 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:32.774010 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7458df57f5-zrrlx" Nov 03 14:07:32.774183 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:32.774025 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7458df57f5-zrrlx" event={"ID":"b34d5b3f-3449-4b91-8919-d278d0dc5112","Type":"ContainerDied","Data":"69253c9896a05dcf8e0a39833cab05d01cdfbf9f0ef2d45f39e8307e0a59d70f"} Nov 03 14:07:32.774183 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:32.774108 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7458df57f5-zrrlx" event={"ID":"b34d5b3f-3449-4b91-8919-d278d0dc5112","Type":"ContainerDied","Data":"b2cbf0c71c99603e06597b4cc3b6efba8259a81c876f62c297ac7ef9079c7ed7"} Nov 03 14:07:32.774183 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:32.774132 5979 scope.go:122] "RemoveContainer" containerID="69253c9896a05dcf8e0a39833cab05d01cdfbf9f0ef2d45f39e8307e0a59d70f" Nov 03 14:07:32.784485 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:32.784455 5979 scope.go:122] "RemoveContainer" containerID="69253c9896a05dcf8e0a39833cab05d01cdfbf9f0ef2d45f39e8307e0a59d70f" Nov 03 14:07:32.784743 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:07:32.784721 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69253c9896a05dcf8e0a39833cab05d01cdfbf9f0ef2d45f39e8307e0a59d70f\": container with ID starting with 69253c9896a05dcf8e0a39833cab05d01cdfbf9f0ef2d45f39e8307e0a59d70f not found: ID does not exist" containerID="69253c9896a05dcf8e0a39833cab05d01cdfbf9f0ef2d45f39e8307e0a59d70f" Nov 03 14:07:32.784825 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:32.784755 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69253c9896a05dcf8e0a39833cab05d01cdfbf9f0ef2d45f39e8307e0a59d70f"} err="failed to get container status \"69253c9896a05dcf8e0a39833cab05d01cdfbf9f0ef2d45f39e8307e0a59d70f\": rpc error: code = NotFound desc = could not find container \"69253c9896a05dcf8e0a39833cab05d01cdfbf9f0ef2d45f39e8307e0a59d70f\": container with ID starting with 69253c9896a05dcf8e0a39833cab05d01cdfbf9f0ef2d45f39e8307e0a59d70f not found: ID does not exist" Nov 03 14:07:32.928181 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:32.928148 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7458df57f5-zrrlx"] Nov 03 14:07:32.975145 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:32.975114 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7458df57f5-zrrlx"] Nov 03 14:07:34.116891 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:34.116858 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b34d5b3f-3449-4b91-8919-d278d0dc5112" path="/var/lib/kubelet/pods/b34d5b3f-3449-4b91-8919-d278d0dc5112/volumes" Nov 03 14:07:34.358975 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:34.358938 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-78f749b598-z7shp"] Nov 03 14:07:34.362315 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:34.362285 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-78f749b598-z7shp" podUID="9174be51-25e6-4996-b121-c15133d0a923" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:07:34.390673 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:34.390640 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7458df57f5-vs9gv"] Nov 03 14:07:34.391275 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:34.391260 5979 cpu_manager.go:414] "containerMap: removing container" podUID="e0e4153d-37d8-4233-8744-7515f9878bdf" containerName="istio-proxy" Nov 03 14:07:34.391326 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:34.391277 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0e4153d-37d8-4233-8744-7515f9878bdf" containerName="istio-proxy" Nov 03 14:07:34.391326 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:34.391293 5979 cpu_manager.go:414] "containerMap: removing container" podUID="e0e4153d-37d8-4233-8744-7515f9878bdf" containerName="istio-proxy" Nov 03 14:07:34.391326 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:34.391299 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0e4153d-37d8-4233-8744-7515f9878bdf" containerName="istio-proxy" Nov 03 14:07:34.391326 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:34.391317 5979 cpu_manager.go:414] "containerMap: removing container" podUID="b34d5b3f-3449-4b91-8919-d278d0dc5112" containerName="payload-processing" Nov 03 14:07:34.391326 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:34.391323 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b34d5b3f-3449-4b91-8919-d278d0dc5112" containerName="payload-processing" Nov 03 14:07:34.391523 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:34.391333 5979 cpu_manager.go:414] "containerMap: removing container" podUID="6a3788db-a41a-47f8-b73c-c695cfdc2801" containerName="payload-processing" Nov 03 14:07:34.391523 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:34.391338 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a3788db-a41a-47f8-b73c-c695cfdc2801" containerName="payload-processing" Nov 03 14:07:34.391523 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:34.391412 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0e4153d-37d8-4233-8744-7515f9878bdf" containerName="istio-proxy" Nov 03 14:07:34.391523 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:34.391424 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0e4153d-37d8-4233-8744-7515f9878bdf" containerName="istio-proxy" Nov 03 14:07:34.391523 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:34.391435 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="b34d5b3f-3449-4b91-8919-d278d0dc5112" containerName="payload-processing" Nov 03 14:07:34.391523 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:34.391440 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a3788db-a41a-47f8-b73c-c695cfdc2801" containerName="payload-processing" Nov 03 14:07:34.391523 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:34.391447 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ab327a8-7012-4504-9557-6dd31ed7a365" containerName="istio-proxy" Nov 03 14:07:34.396310 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:34.396291 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7458df57f5-vs9gv" Nov 03 14:07:34.445530 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:34.445466 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-d4ee3f6c80de\"" Nov 03 14:07:34.464675 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:34.464639 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7458df57f5-vs9gv"] Nov 03 14:07:34.480311 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:34.480264 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f61493e-2883-4333-afbb-ee29a52f7a1e-plugins-config-volume\") pod \"payload-processing-external-model-7458df57f5-vs9gv\" (UID: \"3f61493e-2883-4333-afbb-ee29a52f7a1e\") " pod="models-as-a-service/payload-processing-external-model-7458df57f5-vs9gv" Nov 03 14:07:34.480512 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:34.480360 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3f61493e-2883-4333-afbb-ee29a52f7a1e-provider-credentials\") pod \"payload-processing-external-model-7458df57f5-vs9gv\" (UID: \"3f61493e-2883-4333-afbb-ee29a52f7a1e\") " pod="models-as-a-service/payload-processing-external-model-7458df57f5-vs9gv" Nov 03 14:07:34.480634 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:34.480610 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvx6k\" (UniqueName: \"kubernetes.io/projected/3f61493e-2883-4333-afbb-ee29a52f7a1e-kube-api-access-jvx6k\") pod \"payload-processing-external-model-7458df57f5-vs9gv\" (UID: \"3f61493e-2883-4333-afbb-ee29a52f7a1e\") " pod="models-as-a-service/payload-processing-external-model-7458df57f5-vs9gv" Nov 03 14:07:34.480735 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:34.480716 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3f61493e-2883-4333-afbb-ee29a52f7a1e-routing-overlay-volume\") pod \"payload-processing-external-model-7458df57f5-vs9gv\" (UID: \"3f61493e-2883-4333-afbb-ee29a52f7a1e\") " pod="models-as-a-service/payload-processing-external-model-7458df57f5-vs9gv" Nov 03 14:07:34.582138 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:34.582103 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3f61493e-2883-4333-afbb-ee29a52f7a1e-provider-credentials\") pod \"payload-processing-external-model-7458df57f5-vs9gv\" (UID: \"3f61493e-2883-4333-afbb-ee29a52f7a1e\") " pod="models-as-a-service/payload-processing-external-model-7458df57f5-vs9gv" Nov 03 14:07:34.582420 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:34.582311 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvx6k\" (UniqueName: \"kubernetes.io/projected/3f61493e-2883-4333-afbb-ee29a52f7a1e-kube-api-access-jvx6k\") pod \"payload-processing-external-model-7458df57f5-vs9gv\" (UID: \"3f61493e-2883-4333-afbb-ee29a52f7a1e\") " pod="models-as-a-service/payload-processing-external-model-7458df57f5-vs9gv" Nov 03 14:07:34.582977 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:34.582808 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3f61493e-2883-4333-afbb-ee29a52f7a1e-routing-overlay-volume\") pod \"payload-processing-external-model-7458df57f5-vs9gv\" (UID: \"3f61493e-2883-4333-afbb-ee29a52f7a1e\") " pod="models-as-a-service/payload-processing-external-model-7458df57f5-vs9gv" Nov 03 14:07:34.582977 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:34.582917 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f61493e-2883-4333-afbb-ee29a52f7a1e-plugins-config-volume\") pod \"payload-processing-external-model-7458df57f5-vs9gv\" (UID: \"3f61493e-2883-4333-afbb-ee29a52f7a1e\") " pod="models-as-a-service/payload-processing-external-model-7458df57f5-vs9gv" Nov 03 14:07:34.583970 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:34.583936 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f61493e-2883-4333-afbb-ee29a52f7a1e-plugins-config-volume\") pod \"payload-processing-external-model-7458df57f5-vs9gv\" (UID: \"3f61493e-2883-4333-afbb-ee29a52f7a1e\") " pod="models-as-a-service/payload-processing-external-model-7458df57f5-vs9gv" Nov 03 14:07:34.584255 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:34.584235 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3f61493e-2883-4333-afbb-ee29a52f7a1e-routing-overlay-volume\") pod \"payload-processing-external-model-7458df57f5-vs9gv\" (UID: \"3f61493e-2883-4333-afbb-ee29a52f7a1e\") " pod="models-as-a-service/payload-processing-external-model-7458df57f5-vs9gv" Nov 03 14:07:34.586413 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:34.586391 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3f61493e-2883-4333-afbb-ee29a52f7a1e-provider-credentials\") pod \"payload-processing-external-model-7458df57f5-vs9gv\" (UID: \"3f61493e-2883-4333-afbb-ee29a52f7a1e\") " pod="models-as-a-service/payload-processing-external-model-7458df57f5-vs9gv" Nov 03 14:07:34.636124 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:34.636092 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvx6k\" (UniqueName: \"kubernetes.io/projected/3f61493e-2883-4333-afbb-ee29a52f7a1e-kube-api-access-jvx6k\") pod \"payload-processing-external-model-7458df57f5-vs9gv\" (UID: \"3f61493e-2883-4333-afbb-ee29a52f7a1e\") " pod="models-as-a-service/payload-processing-external-model-7458df57f5-vs9gv" Nov 03 14:07:34.710464 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:34.710388 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7458df57f5-vs9gv" Nov 03 14:07:35.086726 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:35.086633 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7458df57f5-vs9gv"] Nov 03 14:07:35.093566 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:07:35.093537 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f61493e_2883_4333_afbb_ee29a52f7a1e.slice/crio-325c99864a48be31c8efc75dff2634f7137d46bcd635ea34e9c6f742927fd6b8 WatchSource:0}: Error finding container 325c99864a48be31c8efc75dff2634f7137d46bcd635ea34e9c6f742927fd6b8: Status 404 returned error can't find the container with id 325c99864a48be31c8efc75dff2634f7137d46bcd635ea34e9c6f742927fd6b8 Nov 03 14:07:35.798779 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:35.798744 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7458df57f5-vs9gv" event={"ID":"3f61493e-2883-4333-afbb-ee29a52f7a1e","Type":"ContainerStarted","Data":"6995723110abac2817b6bb48663c6d373e94c8c833b6e869fec4cb83c7ee4f23"} Nov 03 14:07:35.799203 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:35.798794 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7458df57f5-vs9gv" event={"ID":"3f61493e-2883-4333-afbb-ee29a52f7a1e","Type":"ContainerStarted","Data":"325c99864a48be31c8efc75dff2634f7137d46bcd635ea34e9c6f742927fd6b8"} Nov 03 14:07:35.799203 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:35.798826 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7458df57f5-vs9gv" Nov 03 14:07:35.862181 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:35.862145 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d5d415-676fcbc795-x4w5c" podUID="bd709159-02e1-4730-bac2-9a0fff9b2c0c" containerName="payload-pre-processing" containerID="cri-o://466c1edcb2f35d007f360a99e267a01c41de315bd0d275424ce7effbe76e0022" gracePeriod=25 Nov 03 14:07:35.909520 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:35.909442 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7458df57f5-vs9gv" podStartSLOduration=1.909427808 podStartE2EDuration="1.909427808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:07:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:07:35.907669058 +0000 UTC m=+3737.918357301" watchObservedRunningTime="2026-11-03 14:07:35.909427808 +0000 UTC m=+3737.920116050" Nov 03 14:07:36.138910 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:36.138884 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d5d415-676fcbc795-x4w5c" Nov 03 14:07:36.204044 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:36.204012 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd709159-02e1-4730-bac2-9a0fff9b2c0c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd709159-02e1-4730-bac2-9a0fff9b2c0c-plugins-config-volume\") pod \"bd709159-02e1-4730-bac2-9a0fff9b2c0c\" (UID: \"bd709159-02e1-4730-bac2-9a0fff9b2c0c\") " Nov 03 14:07:36.204242 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:36.204211 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd709159-02e1-4730-bac2-9a0fff9b2c0c-kube-api-access-9xg9z\" (UniqueName: \"kubernetes.io/projected/bd709159-02e1-4730-bac2-9a0fff9b2c0c-kube-api-access-9xg9z\") pod \"bd709159-02e1-4730-bac2-9a0fff9b2c0c\" (UID: \"bd709159-02e1-4730-bac2-9a0fff9b2c0c\") " Nov 03 14:07:36.204448 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:36.204422 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd709159-02e1-4730-bac2-9a0fff9b2c0c-plugins-config-volume" pod "bd709159-02e1-4730-bac2-9a0fff9b2c0c" (UID: "bd709159-02e1-4730-bac2-9a0fff9b2c0c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:07:36.204705 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:36.204684 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd709159-02e1-4730-bac2-9a0fff9b2c0c-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:36.206235 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:36.206215 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd709159-02e1-4730-bac2-9a0fff9b2c0c-kube-api-access-9xg9z" pod "bd709159-02e1-4730-bac2-9a0fff9b2c0c" (UID: "bd709159-02e1-4730-bac2-9a0fff9b2c0c"). InnerVolumeSpecName "kube-api-access-9xg9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:07:36.306254 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:36.306216 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xg9z\" (UniqueName: \"kubernetes.io/projected/bd709159-02e1-4730-bac2-9a0fff9b2c0c-kube-api-access-9xg9z\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:36.808339 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:36.808305 5979 generic.go:356] "Generic (PLEG): container finished" podID="bd709159-02e1-4730-bac2-9a0fff9b2c0c" containerID="466c1edcb2f35d007f360a99e267a01c41de315bd0d275424ce7effbe76e0022" exitCode=0 Nov 03 14:07:36.808886 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:36.808359 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d5d415-676fcbc795-x4w5c" Nov 03 14:07:36.808886 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:36.808393 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d5d415-676fcbc795-x4w5c" event={"ID":"bd709159-02e1-4730-bac2-9a0fff9b2c0c","Type":"ContainerDied","Data":"466c1edcb2f35d007f360a99e267a01c41de315bd0d275424ce7effbe76e0022"} Nov 03 14:07:36.808886 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:36.808450 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d5d415-676fcbc795-x4w5c" event={"ID":"bd709159-02e1-4730-bac2-9a0fff9b2c0c","Type":"ContainerDied","Data":"b6dee20a71085d7604d575b6d365cab85d805523276ec780f2a68d1621e4577b"} Nov 03 14:07:36.808886 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:36.808488 5979 scope.go:122] "RemoveContainer" containerID="466c1edcb2f35d007f360a99e267a01c41de315bd0d275424ce7effbe76e0022" Nov 03 14:07:36.819312 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:36.819296 5979 scope.go:122] "RemoveContainer" containerID="466c1edcb2f35d007f360a99e267a01c41de315bd0d275424ce7effbe76e0022" Nov 03 14:07:36.819586 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:07:36.819564 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"466c1edcb2f35d007f360a99e267a01c41de315bd0d275424ce7effbe76e0022\": container with ID starting with 466c1edcb2f35d007f360a99e267a01c41de315bd0d275424ce7effbe76e0022 not found: ID does not exist" containerID="466c1edcb2f35d007f360a99e267a01c41de315bd0d275424ce7effbe76e0022" Nov 03 14:07:36.819668 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:36.819593 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"466c1edcb2f35d007f360a99e267a01c41de315bd0d275424ce7effbe76e0022"} err="failed to get container status \"466c1edcb2f35d007f360a99e267a01c41de315bd0d275424ce7effbe76e0022\": rpc error: code = NotFound desc = could not find container \"466c1edcb2f35d007f360a99e267a01c41de315bd0d275424ce7effbe76e0022\": container with ID starting with 466c1edcb2f35d007f360a99e267a01c41de315bd0d275424ce7effbe76e0022 not found: ID does not exist" Nov 03 14:07:36.962927 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:36.962886 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d5d415-676fcbc795-x4w5c"] Nov 03 14:07:37.009014 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:37.008978 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d5d415-676fcbc795-x4w5c"] Nov 03 14:07:37.237681 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:07:37.237639 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:07:37.570416 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:37.570338 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c4e398-2jzm2"] Nov 03 14:07:37.570994 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:37.570979 5979 cpu_manager.go:414] "containerMap: removing container" podUID="bd709159-02e1-4730-bac2-9a0fff9b2c0c" containerName="payload-pre-processing" Nov 03 14:07:37.571041 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:37.570995 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd709159-02e1-4730-bac2-9a0fff9b2c0c" containerName="payload-pre-processing" Nov 03 14:07:37.571113 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:37.571104 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd709159-02e1-4730-bac2-9a0fff9b2c0c" containerName="payload-pre-processing" Nov 03 14:07:37.575698 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:37.575674 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c4e398-2jzm2" Nov 03 14:07:37.624773 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:37.624713 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:07:37.625083 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:37.625033 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k8s6\" (UniqueName: \"kubernetes.io/projected/d6a0821f-4fce-409e-aa34-1c96dca44942-kube-api-access-4k8s6\") pod \"maas-api-revoke-keys-e2e-extmod-a-c4e398-2jzm2\" (UID: \"d6a0821f-4fce-409e-aa34-1c96dca44942\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c4e398-2jzm2" Nov 03 14:07:37.625209 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:37.625176 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6a0821f-4fce-409e-aa34-1c96dca44942-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-c4e398-2jzm2\" (UID: \"d6a0821f-4fce-409e-aa34-1c96dca44942\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c4e398-2jzm2" Nov 03 14:07:37.625336 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:37.625235 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6a0821f-4fce-409e-aa34-1c96dca44942-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-c4e398-2jzm2\" (UID: \"d6a0821f-4fce-409e-aa34-1c96dca44942\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c4e398-2jzm2" Nov 03 14:07:37.625531 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:37.625507 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 14:07:37.629274 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:37.629246 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c4e398-2jzm2"] Nov 03 14:07:37.726011 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:37.725975 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6a0821f-4fce-409e-aa34-1c96dca44942-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-c4e398-2jzm2\" (UID: \"d6a0821f-4fce-409e-aa34-1c96dca44942\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c4e398-2jzm2" Nov 03 14:07:37.726208 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:37.726033 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6a0821f-4fce-409e-aa34-1c96dca44942-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-c4e398-2jzm2\" (UID: \"d6a0821f-4fce-409e-aa34-1c96dca44942\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c4e398-2jzm2" Nov 03 14:07:37.726208 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:37.726122 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k8s6\" (UniqueName: \"kubernetes.io/projected/d6a0821f-4fce-409e-aa34-1c96dca44942-kube-api-access-4k8s6\") pod \"maas-api-revoke-keys-e2e-extmod-a-c4e398-2jzm2\" (UID: \"d6a0821f-4fce-409e-aa34-1c96dca44942\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c4e398-2jzm2" Nov 03 14:07:37.726639 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:37.726616 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6a0821f-4fce-409e-aa34-1c96dca44942-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-c4e398-2jzm2\" (UID: \"d6a0821f-4fce-409e-aa34-1c96dca44942\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c4e398-2jzm2" Nov 03 14:07:37.728373 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:37.728353 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6a0821f-4fce-409e-aa34-1c96dca44942-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-c4e398-2jzm2\" (UID: \"d6a0821f-4fce-409e-aa34-1c96dca44942\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c4e398-2jzm2" Nov 03 14:07:37.779336 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:37.779298 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k8s6\" (UniqueName: \"kubernetes.io/projected/d6a0821f-4fce-409e-aa34-1c96dca44942-kube-api-access-4k8s6\") pod \"maas-api-revoke-keys-e2e-extmod-a-c4e398-2jzm2\" (UID: \"d6a0821f-4fce-409e-aa34-1c96dca44942\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c4e398-2jzm2" Nov 03 14:07:37.890094 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:37.890050 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c4e398-2jzm2" Nov 03 14:07:38.117253 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:38.117219 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd709159-02e1-4730-bac2-9a0fff9b2c0c" path="/var/lib/kubelet/pods/bd709159-02e1-4730-bac2-9a0fff9b2c0c/volumes" Nov 03 14:07:38.265597 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:38.265517 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c4e398-2jzm2"] Nov 03 14:07:38.271936 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:07:38.271912 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6a0821f_4fce_409e_aa34_1c96dca44942.slice/crio-b410a099111332095a783d45a589e169d4e355c528103c2022d885cea25c5e94 WatchSource:0}: Error finding container b410a099111332095a783d45a589e169d4e355c528103c2022d885cea25c5e94: Status 404 returned error can't find the container with id b410a099111332095a783d45a589e169d4e355c528103c2022d885cea25c5e94 Nov 03 14:07:38.824456 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:38.824404 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c4e398-2jzm2" event={"ID":"d6a0821f-4fce-409e-aa34-1c96dca44942","Type":"ContainerStarted","Data":"9c39bdd958b2057337ec83a0df00aa13e6e3b1960360aefa588a518cf8a741ec"} Nov 03 14:07:38.824666 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:38.824512 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c4e398-2jzm2" event={"ID":"d6a0821f-4fce-409e-aa34-1c96dca44942","Type":"ContainerStarted","Data":"b410a099111332095a783d45a589e169d4e355c528103c2022d885cea25c5e94"} Nov 03 14:07:38.933845 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:38.933733 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c4e398-2jzm2" podStartSLOduration=1.9337128460000002 podStartE2EDuration="1.933712846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:07:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:07:38.930697742 +0000 UTC m=+3740.941385987" watchObservedRunningTime="2026-11-03 14:07:38.933712846 +0000 UTC m=+3740.944401088" Nov 03 14:07:39.360154 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:39.360108 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-78f749b598-z7shp" podUID="9174be51-25e6-4996-b121-c15133d0a923" containerName="payload-processing" containerID="cri-o://206dfc764264207d0a7044fd908b159992ebf6e2ca8b014f40e8d3e910633290" gracePeriod=25 Nov 03 14:07:39.718636 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:39.718610 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-78f749b598-z7shp" Nov 03 14:07:39.796624 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:39.793268 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9174be51-25e6-4996-b121-c15133d0a923-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9174be51-25e6-4996-b121-c15133d0a923-routing-overlay-volume\") pod \"9174be51-25e6-4996-b121-c15133d0a923\" (UID: \"9174be51-25e6-4996-b121-c15133d0a923\") " Nov 03 14:07:39.796624 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:39.793936 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9174be51-25e6-4996-b121-c15133d0a923-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9174be51-25e6-4996-b121-c15133d0a923-provider-credentials\") pod \"9174be51-25e6-4996-b121-c15133d0a923\" (UID: \"9174be51-25e6-4996-b121-c15133d0a923\") " Nov 03 14:07:39.796624 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:39.794072 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9174be51-25e6-4996-b121-c15133d0a923-kube-api-access-xn8cn\" (UniqueName: \"kubernetes.io/projected/9174be51-25e6-4996-b121-c15133d0a923-kube-api-access-xn8cn\") pod \"9174be51-25e6-4996-b121-c15133d0a923\" (UID: \"9174be51-25e6-4996-b121-c15133d0a923\") " Nov 03 14:07:39.796624 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:39.794158 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9174be51-25e6-4996-b121-c15133d0a923-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9174be51-25e6-4996-b121-c15133d0a923-plugins-config-volume\") pod \"9174be51-25e6-4996-b121-c15133d0a923\" (UID: \"9174be51-25e6-4996-b121-c15133d0a923\") " Nov 03 14:07:39.796624 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:39.793769 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9174be51-25e6-4996-b121-c15133d0a923-routing-overlay-volume" pod "9174be51-25e6-4996-b121-c15133d0a923" (UID: "9174be51-25e6-4996-b121-c15133d0a923"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:07:39.796624 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:39.795359 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9174be51-25e6-4996-b121-c15133d0a923-plugins-config-volume" pod "9174be51-25e6-4996-b121-c15133d0a923" (UID: "9174be51-25e6-4996-b121-c15133d0a923"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:07:39.800634 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:39.800600 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9174be51-25e6-4996-b121-c15133d0a923-kube-api-access-xn8cn" pod "9174be51-25e6-4996-b121-c15133d0a923" (UID: "9174be51-25e6-4996-b121-c15133d0a923"). InnerVolumeSpecName "kube-api-access-xn8cn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:07:39.804617 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:39.804588 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9174be51-25e6-4996-b121-c15133d0a923-provider-credentials" pod "9174be51-25e6-4996-b121-c15133d0a923" (UID: "9174be51-25e6-4996-b121-c15133d0a923"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:07:39.834088 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:39.834052 5979 generic.go:356] "Generic (PLEG): container finished" podID="9174be51-25e6-4996-b121-c15133d0a923" containerID="206dfc764264207d0a7044fd908b159992ebf6e2ca8b014f40e8d3e910633290" exitCode=0 Nov 03 14:07:39.834257 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:39.834094 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-78f749b598-z7shp" Nov 03 14:07:39.834257 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:39.834138 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-78f749b598-z7shp" event={"ID":"9174be51-25e6-4996-b121-c15133d0a923","Type":"ContainerDied","Data":"206dfc764264207d0a7044fd908b159992ebf6e2ca8b014f40e8d3e910633290"} Nov 03 14:07:39.834257 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:39.834192 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-78f749b598-z7shp" event={"ID":"9174be51-25e6-4996-b121-c15133d0a923","Type":"ContainerDied","Data":"fa9db13d96817026a1336696a093171b355945c86072b3ad7edb3d3fc0f372f7"} Nov 03 14:07:39.834257 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:39.834219 5979 scope.go:122] "RemoveContainer" containerID="206dfc764264207d0a7044fd908b159992ebf6e2ca8b014f40e8d3e910633290" Nov 03 14:07:39.836611 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:39.836162 5979 generic.go:356] "Generic (PLEG): container finished" podID="d6a0821f-4fce-409e-aa34-1c96dca44942" containerID="9c39bdd958b2057337ec83a0df00aa13e6e3b1960360aefa588a518cf8a741ec" exitCode=0 Nov 03 14:07:39.836611 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:39.836288 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c4e398-2jzm2" event={"ID":"d6a0821f-4fce-409e-aa34-1c96dca44942","Type":"ContainerDied","Data":"9c39bdd958b2057337ec83a0df00aa13e6e3b1960360aefa588a518cf8a741ec"} Nov 03 14:07:39.856482 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:39.856424 5979 scope.go:122] "RemoveContainer" containerID="206dfc764264207d0a7044fd908b159992ebf6e2ca8b014f40e8d3e910633290" Nov 03 14:07:39.856909 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:07:39.856875 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"206dfc764264207d0a7044fd908b159992ebf6e2ca8b014f40e8d3e910633290\": container with ID starting with 206dfc764264207d0a7044fd908b159992ebf6e2ca8b014f40e8d3e910633290 not found: ID does not exist" containerID="206dfc764264207d0a7044fd908b159992ebf6e2ca8b014f40e8d3e910633290" Nov 03 14:07:39.857039 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:39.856922 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"206dfc764264207d0a7044fd908b159992ebf6e2ca8b014f40e8d3e910633290"} err="failed to get container status \"206dfc764264207d0a7044fd908b159992ebf6e2ca8b014f40e8d3e910633290\": rpc error: code = NotFound desc = could not find container \"206dfc764264207d0a7044fd908b159992ebf6e2ca8b014f40e8d3e910633290\": container with ID starting with 206dfc764264207d0a7044fd908b159992ebf6e2ca8b014f40e8d3e910633290 not found: ID does not exist" Nov 03 14:07:39.896038 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:39.896000 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn8cn\" (UniqueName: \"kubernetes.io/projected/9174be51-25e6-4996-b121-c15133d0a923-kube-api-access-xn8cn\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:39.896038 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:39.896031 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9174be51-25e6-4996-b121-c15133d0a923-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:39.896038 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:39.896041 5979 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9174be51-25e6-4996-b121-c15133d0a923-routing-overlay-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:39.896359 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:39.896052 5979 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9174be51-25e6-4996-b121-c15133d0a923-provider-credentials\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:40.103910 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:40.103815 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-78f749b598-z7shp"] Nov 03 14:07:40.118687 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:40.118653 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-78f749b598-z7shp"] Nov 03 14:07:41.006074 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:41.006047 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c4e398-2jzm2" Nov 03 14:07:41.117395 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:41.117305 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6a0821f-4fce-409e-aa34-1c96dca44942-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6a0821f-4fce-409e-aa34-1c96dca44942-maas-api-xks-ca\") pod \"d6a0821f-4fce-409e-aa34-1c96dca44942\" (UID: \"d6a0821f-4fce-409e-aa34-1c96dca44942\") " Nov 03 14:07:41.117395 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:41.117370 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6a0821f-4fce-409e-aa34-1c96dca44942-kube-api-access-4k8s6\" (UniqueName: \"kubernetes.io/projected/d6a0821f-4fce-409e-aa34-1c96dca44942-kube-api-access-4k8s6\") pod \"d6a0821f-4fce-409e-aa34-1c96dca44942\" (UID: \"d6a0821f-4fce-409e-aa34-1c96dca44942\") " Nov 03 14:07:41.117932 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:41.117699 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6a0821f-4fce-409e-aa34-1c96dca44942-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6a0821f-4fce-409e-aa34-1c96dca44942-maas-api-openshift-ca\") pod \"d6a0821f-4fce-409e-aa34-1c96dca44942\" (UID: \"d6a0821f-4fce-409e-aa34-1c96dca44942\") " Nov 03 14:07:41.118113 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:41.118085 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6a0821f-4fce-409e-aa34-1c96dca44942-maas-api-openshift-ca" pod "d6a0821f-4fce-409e-aa34-1c96dca44942" (UID: "d6a0821f-4fce-409e-aa34-1c96dca44942"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:07:41.119683 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:41.119654 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6a0821f-4fce-409e-aa34-1c96dca44942-maas-api-xks-ca" pod "d6a0821f-4fce-409e-aa34-1c96dca44942" (UID: "d6a0821f-4fce-409e-aa34-1c96dca44942"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:07:41.119819 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:41.119797 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6a0821f-4fce-409e-aa34-1c96dca44942-kube-api-access-4k8s6" pod "d6a0821f-4fce-409e-aa34-1c96dca44942" (UID: "d6a0821f-4fce-409e-aa34-1c96dca44942"). InnerVolumeSpecName "kube-api-access-4k8s6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:07:41.219186 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:41.219143 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6a0821f-4fce-409e-aa34-1c96dca44942-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:41.219186 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:41.219180 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6a0821f-4fce-409e-aa34-1c96dca44942-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:41.219186 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:41.219194 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4k8s6\" (UniqueName: \"kubernetes.io/projected/d6a0821f-4fce-409e-aa34-1c96dca44942-kube-api-access-4k8s6\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:41.852060 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:41.852026 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c4e398-2jzm2" Nov 03 14:07:41.852254 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:41.852048 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c4e398-2jzm2" event={"ID":"d6a0821f-4fce-409e-aa34-1c96dca44942","Type":"ContainerDied","Data":"b410a099111332095a783d45a589e169d4e355c528103c2022d885cea25c5e94"} Nov 03 14:07:41.852254 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:41.852129 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b410a099111332095a783d45a589e169d4e355c528103c2022d885cea25c5e94" Nov 03 14:07:41.961247 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:41.961212 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7458df57f5-vs9gv"] Nov 03 14:07:41.964350 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:41.964325 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-7458df57f5-vs9gv" Nov 03 14:07:41.971655 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:41.971632 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7898dd9db6-tfrv8"] Nov 03 14:07:41.972235 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:41.972221 5979 cpu_manager.go:414] "containerMap: removing container" podUID="9174be51-25e6-4996-b121-c15133d0a923" containerName="payload-processing" Nov 03 14:07:41.972278 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:41.972237 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9174be51-25e6-4996-b121-c15133d0a923" containerName="payload-processing" Nov 03 14:07:41.972278 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:41.972245 5979 cpu_manager.go:414] "containerMap: removing container" podUID="d6a0821f-4fce-409e-aa34-1c96dca44942" containerName="revoke-keys" Nov 03 14:07:41.972278 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:41.972250 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6a0821f-4fce-409e-aa34-1c96dca44942" containerName="revoke-keys" Nov 03 14:07:41.972386 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:41.972368 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="9174be51-25e6-4996-b121-c15133d0a923" containerName="payload-processing" Nov 03 14:07:41.972386 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:41.972375 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6a0821f-4fce-409e-aa34-1c96dca44942" containerName="revoke-keys" Nov 03 14:07:41.976236 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:41.976215 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7898dd9db6-tfrv8" Nov 03 14:07:42.026311 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:42.026282 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-59e97a2ad282\"" Nov 03 14:07:42.046707 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:42.046676 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7898dd9db6-tfrv8"] Nov 03 14:07:42.117330 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:42.117255 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9174be51-25e6-4996-b121-c15133d0a923" path="/var/lib/kubelet/pods/9174be51-25e6-4996-b121-c15133d0a923/volumes" Nov 03 14:07:42.131547 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:42.131506 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk6wf\" (UniqueName: \"kubernetes.io/projected/0c2807d6-84df-4d2c-8872-1c7635c02850-kube-api-access-rk6wf\") pod \"payload-processing-external-model-7898dd9db6-tfrv8\" (UID: \"0c2807d6-84df-4d2c-8872-1c7635c02850\") " pod="models-as-a-service/payload-processing-external-model-7898dd9db6-tfrv8" Nov 03 14:07:42.131921 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:42.131548 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c2807d6-84df-4d2c-8872-1c7635c02850-plugins-config-volume\") pod \"payload-processing-external-model-7898dd9db6-tfrv8\" (UID: \"0c2807d6-84df-4d2c-8872-1c7635c02850\") " pod="models-as-a-service/payload-processing-external-model-7898dd9db6-tfrv8" Nov 03 14:07:42.131921 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:42.131692 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0c2807d6-84df-4d2c-8872-1c7635c02850-provider-credentials\") pod \"payload-processing-external-model-7898dd9db6-tfrv8\" (UID: \"0c2807d6-84df-4d2c-8872-1c7635c02850\") " pod="models-as-a-service/payload-processing-external-model-7898dd9db6-tfrv8" Nov 03 14:07:42.131921 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:42.131712 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0c2807d6-84df-4d2c-8872-1c7635c02850-routing-overlay-volume\") pod \"payload-processing-external-model-7898dd9db6-tfrv8\" (UID: \"0c2807d6-84df-4d2c-8872-1c7635c02850\") " pod="models-as-a-service/payload-processing-external-model-7898dd9db6-tfrv8" Nov 03 14:07:42.232691 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:42.232653 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk6wf\" (UniqueName: \"kubernetes.io/projected/0c2807d6-84df-4d2c-8872-1c7635c02850-kube-api-access-rk6wf\") pod \"payload-processing-external-model-7898dd9db6-tfrv8\" (UID: \"0c2807d6-84df-4d2c-8872-1c7635c02850\") " pod="models-as-a-service/payload-processing-external-model-7898dd9db6-tfrv8" Nov 03 14:07:42.232908 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:42.232700 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c2807d6-84df-4d2c-8872-1c7635c02850-plugins-config-volume\") pod \"payload-processing-external-model-7898dd9db6-tfrv8\" (UID: \"0c2807d6-84df-4d2c-8872-1c7635c02850\") " pod="models-as-a-service/payload-processing-external-model-7898dd9db6-tfrv8" Nov 03 14:07:42.232908 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:42.232830 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0c2807d6-84df-4d2c-8872-1c7635c02850-provider-credentials\") pod \"payload-processing-external-model-7898dd9db6-tfrv8\" (UID: \"0c2807d6-84df-4d2c-8872-1c7635c02850\") " pod="models-as-a-service/payload-processing-external-model-7898dd9db6-tfrv8" Nov 03 14:07:42.232908 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:42.232864 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0c2807d6-84df-4d2c-8872-1c7635c02850-routing-overlay-volume\") pod \"payload-processing-external-model-7898dd9db6-tfrv8\" (UID: \"0c2807d6-84df-4d2c-8872-1c7635c02850\") " pod="models-as-a-service/payload-processing-external-model-7898dd9db6-tfrv8" Nov 03 14:07:42.233095 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:07:42.232949 5979 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-e86f9ab11580: secret "agc-cred-e86f9ab11580" not found Nov 03 14:07:42.233095 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:07:42.232972 5979 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-84dcd755bf-c9rnp: secret "agc-cred-e86f9ab11580" not found Nov 03 14:07:42.233095 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:07:42.233046 5979 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/797bcf35-fa9e-47b8-887c-bb8942af277b-provider-credentials podName:797bcf35-fa9e-47b8-887c-bb8942af277b nodeName:}" failed. No retries permitted until 2026-11-03 14:07:42.733025975 +0000 UTC m=+3744.743714196 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/797bcf35-fa9e-47b8-887c-bb8942af277b-provider-credentials") pod "payload-processing-external-model-84dcd755bf-c9rnp" (UID: "797bcf35-fa9e-47b8-887c-bb8942af277b") : secret "agc-cred-e86f9ab11580" not found Nov 03 14:07:42.233770 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:42.233746 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c2807d6-84df-4d2c-8872-1c7635c02850-plugins-config-volume\") pod \"payload-processing-external-model-7898dd9db6-tfrv8\" (UID: \"0c2807d6-84df-4d2c-8872-1c7635c02850\") " pod="models-as-a-service/payload-processing-external-model-7898dd9db6-tfrv8" Nov 03 14:07:42.233949 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:42.233926 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0c2807d6-84df-4d2c-8872-1c7635c02850-routing-overlay-volume\") pod \"payload-processing-external-model-7898dd9db6-tfrv8\" (UID: \"0c2807d6-84df-4d2c-8872-1c7635c02850\") " pod="models-as-a-service/payload-processing-external-model-7898dd9db6-tfrv8" Nov 03 14:07:42.235720 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:42.235698 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0c2807d6-84df-4d2c-8872-1c7635c02850-provider-credentials\") pod \"payload-processing-external-model-7898dd9db6-tfrv8\" (UID: \"0c2807d6-84df-4d2c-8872-1c7635c02850\") " pod="models-as-a-service/payload-processing-external-model-7898dd9db6-tfrv8" Nov 03 14:07:42.287636 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:42.287608 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk6wf\" (UniqueName: \"kubernetes.io/projected/0c2807d6-84df-4d2c-8872-1c7635c02850-kube-api-access-rk6wf\") pod \"payload-processing-external-model-7898dd9db6-tfrv8\" (UID: \"0c2807d6-84df-4d2c-8872-1c7635c02850\") " pod="models-as-a-service/payload-processing-external-model-7898dd9db6-tfrv8" Nov 03 14:07:42.291369 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:42.291347 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7898dd9db6-tfrv8" Nov 03 14:07:42.673243 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:42.673210 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7898dd9db6-tfrv8"] Nov 03 14:07:42.680868 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:07:42.680839 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c2807d6_84df_4d2c_8872_1c7635c02850.slice/crio-3e82746961431a74f419ad1d11be42ad80337dc5a099a1e0bed2354a61bc01f8 WatchSource:0}: Error finding container 3e82746961431a74f419ad1d11be42ad80337dc5a099a1e0bed2354a61bc01f8: Status 404 returned error can't find the container with id 3e82746961431a74f419ad1d11be42ad80337dc5a099a1e0bed2354a61bc01f8 Nov 03 14:07:42.740521 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:07:42.740460 5979 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-e86f9ab11580: secret "agc-cred-e86f9ab11580" not found Nov 03 14:07:42.740521 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:07:42.740495 5979 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-84dcd755bf-c9rnp: secret "agc-cred-e86f9ab11580" not found Nov 03 14:07:42.740666 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:07:42.740553 5979 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/797bcf35-fa9e-47b8-887c-bb8942af277b-provider-credentials podName:797bcf35-fa9e-47b8-887c-bb8942af277b nodeName:}" failed. No retries permitted until 2026-11-03 14:07:43.740535885 +0000 UTC m=+3745.751224124 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/797bcf35-fa9e-47b8-887c-bb8942af277b-provider-credentials") pod "payload-processing-external-model-84dcd755bf-c9rnp" (UID: "797bcf35-fa9e-47b8-887c-bb8942af277b") : secret "agc-cred-e86f9ab11580" not found Nov 03 14:07:42.860948 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:42.860909 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7898dd9db6-tfrv8" event={"ID":"0c2807d6-84df-4d2c-8872-1c7635c02850","Type":"ContainerStarted","Data":"d60ef85027afd45576ee91bddbfae26eaa99d34d87c6e8c4a5660cbae95e2cde"} Nov 03 14:07:42.861130 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:42.860976 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7898dd9db6-tfrv8" event={"ID":"0c2807d6-84df-4d2c-8872-1c7635c02850","Type":"ContainerStarted","Data":"3e82746961431a74f419ad1d11be42ad80337dc5a099a1e0bed2354a61bc01f8"} Nov 03 14:07:42.861130 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:42.861006 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7898dd9db6-tfrv8" Nov 03 14:07:42.970595 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:42.970492 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7898dd9db6-tfrv8" podStartSLOduration=1.9704587519999999 podStartE2EDuration="1.970458752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:07:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:07:42.968966322 +0000 UTC m=+3744.979654565" watchObservedRunningTime="2026-11-03 14:07:42.970458752 +0000 UTC m=+3744.981147044" Nov 03 14:07:43.753276 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:07:43.753244 5979 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-e86f9ab11580: secret "agc-cred-e86f9ab11580" not found Nov 03 14:07:43.753276 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:07:43.753271 5979 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-84dcd755bf-c9rnp: secret "agc-cred-e86f9ab11580" not found Nov 03 14:07:43.753853 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:07:43.753334 5979 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/797bcf35-fa9e-47b8-887c-bb8942af277b-provider-credentials podName:797bcf35-fa9e-47b8-887c-bb8942af277b nodeName:}" failed. No retries permitted until 2026-11-03 14:07:45.753314796 +0000 UTC m=+3747.764003021 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/797bcf35-fa9e-47b8-887c-bb8942af277b-provider-credentials") pod "payload-processing-external-model-84dcd755bf-c9rnp" (UID: "797bcf35-fa9e-47b8-887c-bb8942af277b") : secret "agc-cred-e86f9ab11580" not found Nov 03 14:07:45.777396 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:07:45.777365 5979 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-e86f9ab11580: secret "agc-cred-e86f9ab11580" not found Nov 03 14:07:45.777396 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:07:45.777390 5979 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-84dcd755bf-c9rnp: secret "agc-cred-e86f9ab11580" not found Nov 03 14:07:45.777864 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:07:45.777443 5979 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/797bcf35-fa9e-47b8-887c-bb8942af277b-provider-credentials podName:797bcf35-fa9e-47b8-887c-bb8942af277b nodeName:}" failed. No retries permitted until 2026-11-03 14:07:49.777429063 +0000 UTC m=+3751.788117286 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/797bcf35-fa9e-47b8-887c-bb8942af277b-provider-credentials") pod "payload-processing-external-model-84dcd755bf-c9rnp" (UID: "797bcf35-fa9e-47b8-887c-bb8942af277b") : secret "agc-cred-e86f9ab11580" not found Nov 03 14:07:46.961984 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:46.961904 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7458df57f5-vs9gv" podUID="3f61493e-2883-4333-afbb-ee29a52f7a1e" containerName="payload-processing" containerID="cri-o://6995723110abac2817b6bb48663c6d373e94c8c833b6e869fec4cb83c7ee4f23" gracePeriod=25 Nov 03 14:07:47.218606 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:47.218541 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7898dd9db6-tfrv8"] Nov 03 14:07:47.220409 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:47.220382 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7458df57f5-vs9gv" Nov 03 14:07:47.221225 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:47.221201 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7898dd9db6-tfrv8" podUID="0c2807d6-84df-4d2c-8872-1c7635c02850" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:07:47.255955 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:47.255922 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-78f749b598-6hsx4"] Nov 03 14:07:47.256584 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:47.256567 5979 cpu_manager.go:414] "containerMap: removing container" podUID="3f61493e-2883-4333-afbb-ee29a52f7a1e" containerName="payload-processing" Nov 03 14:07:47.256584 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:47.256583 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f61493e-2883-4333-afbb-ee29a52f7a1e" containerName="payload-processing" Nov 03 14:07:47.256748 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:47.256737 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f61493e-2883-4333-afbb-ee29a52f7a1e" containerName="payload-processing" Nov 03 14:07:47.260176 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:47.260159 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" Nov 03 14:07:47.296996 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:47.296962 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f61493e-2883-4333-afbb-ee29a52f7a1e-kube-api-access-jvx6k\" (UniqueName: \"kubernetes.io/projected/3f61493e-2883-4333-afbb-ee29a52f7a1e-kube-api-access-jvx6k\") pod \"3f61493e-2883-4333-afbb-ee29a52f7a1e\" (UID: \"3f61493e-2883-4333-afbb-ee29a52f7a1e\") " Nov 03 14:07:47.297149 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:47.297009 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f61493e-2883-4333-afbb-ee29a52f7a1e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f61493e-2883-4333-afbb-ee29a52f7a1e-plugins-config-volume\") pod \"3f61493e-2883-4333-afbb-ee29a52f7a1e\" (UID: \"3f61493e-2883-4333-afbb-ee29a52f7a1e\") " Nov 03 14:07:47.297217 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:47.297202 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f61493e-2883-4333-afbb-ee29a52f7a1e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3f61493e-2883-4333-afbb-ee29a52f7a1e-provider-credentials\") pod \"3f61493e-2883-4333-afbb-ee29a52f7a1e\" (UID: \"3f61493e-2883-4333-afbb-ee29a52f7a1e\") " Nov 03 14:07:47.297281 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:47.297246 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f61493e-2883-4333-afbb-ee29a52f7a1e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3f61493e-2883-4333-afbb-ee29a52f7a1e-routing-overlay-volume\") pod \"3f61493e-2883-4333-afbb-ee29a52f7a1e\" (UID: \"3f61493e-2883-4333-afbb-ee29a52f7a1e\") " Nov 03 14:07:47.297424 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:47.297399 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f61493e-2883-4333-afbb-ee29a52f7a1e-plugins-config-volume" pod "3f61493e-2883-4333-afbb-ee29a52f7a1e" (UID: "3f61493e-2883-4333-afbb-ee29a52f7a1e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:07:47.297424 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:47.297407 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d09478af-d618-425d-a4e6-e0889e0f416d-plugins-config-volume\") pod \"payload-processing-external-model-78f749b598-6hsx4\" (UID: \"d09478af-d618-425d-a4e6-e0889e0f416d\") " pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" Nov 03 14:07:47.297603 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:47.297551 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d09478af-d618-425d-a4e6-e0889e0f416d-routing-overlay-volume\") pod \"payload-processing-external-model-78f749b598-6hsx4\" (UID: \"d09478af-d618-425d-a4e6-e0889e0f416d\") " pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" Nov 03 14:07:47.297728 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:47.297707 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f61493e-2883-4333-afbb-ee29a52f7a1e-routing-overlay-volume" pod "3f61493e-2883-4333-afbb-ee29a52f7a1e" (UID: "3f61493e-2883-4333-afbb-ee29a52f7a1e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:07:47.297728 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:47.297708 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t82t\" (UniqueName: \"kubernetes.io/projected/d09478af-d618-425d-a4e6-e0889e0f416d-kube-api-access-5t82t\") pod \"payload-processing-external-model-78f749b598-6hsx4\" (UID: \"d09478af-d618-425d-a4e6-e0889e0f416d\") " pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" Nov 03 14:07:47.297868 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:47.297849 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d09478af-d618-425d-a4e6-e0889e0f416d-provider-credentials\") pod \"payload-processing-external-model-78f749b598-6hsx4\" (UID: \"d09478af-d618-425d-a4e6-e0889e0f416d\") " pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" Nov 03 14:07:47.298039 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:47.298022 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f61493e-2883-4333-afbb-ee29a52f7a1e-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:47.298080 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:47.298047 5979 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3f61493e-2883-4333-afbb-ee29a52f7a1e-routing-overlay-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:47.299322 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:47.299295 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f61493e-2883-4333-afbb-ee29a52f7a1e-provider-credentials" pod "3f61493e-2883-4333-afbb-ee29a52f7a1e" (UID: "3f61493e-2883-4333-afbb-ee29a52f7a1e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:07:47.299415 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:47.299331 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f61493e-2883-4333-afbb-ee29a52f7a1e-kube-api-access-jvx6k" pod "3f61493e-2883-4333-afbb-ee29a52f7a1e" (UID: "3f61493e-2883-4333-afbb-ee29a52f7a1e"). InnerVolumeSpecName "kube-api-access-jvx6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:07:47.322444 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:47.322371 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-78f749b598-6hsx4"] Nov 03 14:07:47.400492 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:47.399693 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d09478af-d618-425d-a4e6-e0889e0f416d-routing-overlay-volume\") pod \"payload-processing-external-model-78f749b598-6hsx4\" (UID: \"d09478af-d618-425d-a4e6-e0889e0f416d\") " pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" Nov 03 14:07:47.400492 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:47.399826 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t82t\" (UniqueName: \"kubernetes.io/projected/d09478af-d618-425d-a4e6-e0889e0f416d-kube-api-access-5t82t\") pod \"payload-processing-external-model-78f749b598-6hsx4\" (UID: \"d09478af-d618-425d-a4e6-e0889e0f416d\") " pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" Nov 03 14:07:47.400492 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:47.399942 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d09478af-d618-425d-a4e6-e0889e0f416d-provider-credentials\") pod \"payload-processing-external-model-78f749b598-6hsx4\" (UID: \"d09478af-d618-425d-a4e6-e0889e0f416d\") " pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" Nov 03 14:07:47.400492 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:47.400024 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d09478af-d618-425d-a4e6-e0889e0f416d-plugins-config-volume\") pod \"payload-processing-external-model-78f749b598-6hsx4\" (UID: \"d09478af-d618-425d-a4e6-e0889e0f416d\") " pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" Nov 03 14:07:47.400492 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:47.400124 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvx6k\" (UniqueName: \"kubernetes.io/projected/3f61493e-2883-4333-afbb-ee29a52f7a1e-kube-api-access-jvx6k\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:47.400492 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:47.400140 5979 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3f61493e-2883-4333-afbb-ee29a52f7a1e-provider-credentials\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:47.402504 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:47.401060 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d09478af-d618-425d-a4e6-e0889e0f416d-plugins-config-volume\") pod \"payload-processing-external-model-78f749b598-6hsx4\" (UID: \"d09478af-d618-425d-a4e6-e0889e0f416d\") " pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" Nov 03 14:07:47.402504 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:47.401430 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d09478af-d618-425d-a4e6-e0889e0f416d-routing-overlay-volume\") pod \"payload-processing-external-model-78f749b598-6hsx4\" (UID: \"d09478af-d618-425d-a4e6-e0889e0f416d\") " pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" Nov 03 14:07:47.407347 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:47.407319 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d09478af-d618-425d-a4e6-e0889e0f416d-provider-credentials\") pod \"payload-processing-external-model-78f749b598-6hsx4\" (UID: \"d09478af-d618-425d-a4e6-e0889e0f416d\") " pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" Nov 03 14:07:47.455751 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:47.455715 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t82t\" (UniqueName: \"kubernetes.io/projected/d09478af-d618-425d-a4e6-e0889e0f416d-kube-api-access-5t82t\") pod \"payload-processing-external-model-78f749b598-6hsx4\" (UID: \"d09478af-d618-425d-a4e6-e0889e0f416d\") " pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" Nov 03 14:07:47.574551 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:47.574428 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" Nov 03 14:07:47.897229 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:47.897197 5979 generic.go:356] "Generic (PLEG): container finished" podID="3f61493e-2883-4333-afbb-ee29a52f7a1e" containerID="6995723110abac2817b6bb48663c6d373e94c8c833b6e869fec4cb83c7ee4f23" exitCode=0 Nov 03 14:07:47.897447 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:47.897253 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7458df57f5-vs9gv" Nov 03 14:07:47.897447 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:47.897291 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7458df57f5-vs9gv" event={"ID":"3f61493e-2883-4333-afbb-ee29a52f7a1e","Type":"ContainerDied","Data":"6995723110abac2817b6bb48663c6d373e94c8c833b6e869fec4cb83c7ee4f23"} Nov 03 14:07:47.897447 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:47.897346 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7458df57f5-vs9gv" event={"ID":"3f61493e-2883-4333-afbb-ee29a52f7a1e","Type":"ContainerDied","Data":"325c99864a48be31c8efc75dff2634f7137d46bcd635ea34e9c6f742927fd6b8"} Nov 03 14:07:47.897447 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:47.897373 5979 scope.go:122] "RemoveContainer" containerID="6995723110abac2817b6bb48663c6d373e94c8c833b6e869fec4cb83c7ee4f23" Nov 03 14:07:47.907664 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:47.907646 5979 scope.go:122] "RemoveContainer" containerID="6995723110abac2817b6bb48663c6d373e94c8c833b6e869fec4cb83c7ee4f23" Nov 03 14:07:47.907930 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:07:47.907913 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6995723110abac2817b6bb48663c6d373e94c8c833b6e869fec4cb83c7ee4f23\": container with ID starting with 6995723110abac2817b6bb48663c6d373e94c8c833b6e869fec4cb83c7ee4f23 not found: ID does not exist" containerID="6995723110abac2817b6bb48663c6d373e94c8c833b6e869fec4cb83c7ee4f23" Nov 03 14:07:47.907978 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:47.907939 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6995723110abac2817b6bb48663c6d373e94c8c833b6e869fec4cb83c7ee4f23"} err="failed to get container status \"6995723110abac2817b6bb48663c6d373e94c8c833b6e869fec4cb83c7ee4f23\": rpc error: code = NotFound desc = could not find container \"6995723110abac2817b6bb48663c6d373e94c8c833b6e869fec4cb83c7ee4f23\": container with ID starting with 6995723110abac2817b6bb48663c6d373e94c8c833b6e869fec4cb83c7ee4f23 not found: ID does not exist" Nov 03 14:07:47.952329 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:47.952303 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-78f749b598-6hsx4"] Nov 03 14:07:47.959245 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:07:47.959219 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd09478af_d618_425d_a4e6_e0889e0f416d.slice/crio-e6c8ba1640c087f8af1422528ef613b3c8e4de7e315601a3a51d48429053408d WatchSource:0}: Error finding container e6c8ba1640c087f8af1422528ef613b3c8e4de7e315601a3a51d48429053408d: Status 404 returned error can't find the container with id e6c8ba1640c087f8af1422528ef613b3c8e4de7e315601a3a51d48429053408d Nov 03 14:07:48.056422 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:48.056394 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7458df57f5-vs9gv"] Nov 03 14:07:48.060496 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:48.060446 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7458df57f5-vs9gv"] Nov 03 14:07:48.116316 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:48.116283 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f61493e-2883-4333-afbb-ee29a52f7a1e" path="/var/lib/kubelet/pods/3f61493e-2883-4333-afbb-ee29a52f7a1e/volumes" Nov 03 14:07:48.906493 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:48.906444 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" event={"ID":"d09478af-d618-425d-a4e6-e0889e0f416d","Type":"ContainerStarted","Data":"471394e3f03a245f6eee2c31b1e7c1fa191108205c4be0d8a4d2fb1fe5214fd8"} Nov 03 14:07:48.906698 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:48.906612 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" event={"ID":"d09478af-d618-425d-a4e6-e0889e0f416d","Type":"ContainerStarted","Data":"e6c8ba1640c087f8af1422528ef613b3c8e4de7e315601a3a51d48429053408d"} Nov 03 14:07:48.906698 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:48.906657 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" Nov 03 14:07:49.014706 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:49.014650 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" podStartSLOduration=2.014635239 podStartE2EDuration="2.014635239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:07:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:07:49.013586216 +0000 UTC m=+3751.024274470" watchObservedRunningTime="2026-11-03 14:07:49.014635239 +0000 UTC m=+3751.025323492" Nov 03 14:07:49.828726 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:07:49.828689 5979 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-e86f9ab11580: secret "agc-cred-e86f9ab11580" not found Nov 03 14:07:49.828726 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:07:49.828718 5979 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-84dcd755bf-c9rnp: secret "agc-cred-e86f9ab11580" not found Nov 03 14:07:49.829170 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:07:49.828767 5979 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/797bcf35-fa9e-47b8-887c-bb8942af277b-provider-credentials podName:797bcf35-fa9e-47b8-887c-bb8942af277b nodeName:}" failed. No retries permitted until 2026-11-03 14:07:57.828751199 +0000 UTC m=+3759.839439436 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/797bcf35-fa9e-47b8-887c-bb8942af277b-provider-credentials") pod "payload-processing-external-model-84dcd755bf-c9rnp" (UID: "797bcf35-fa9e-47b8-887c-bb8942af277b") : secret "agc-cred-e86f9ab11580" not found Nov 03 14:07:50.974834 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:50.974798 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b"] Nov 03 14:07:50.980349 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:50.980326 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b" Nov 03 14:07:51.046161 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:51.046117 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5178f75c-2f52-4a5b-bf00-c315bf7e110b-istio-envoy\") pod \"e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b\" (UID: \"5178f75c-2f52-4a5b-bf00-c315bf7e110b\") " pod="openshift-ingress/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b" Nov 03 14:07:51.046367 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:51.046174 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5178f75c-2f52-4a5b-bf00-c315bf7e110b-istio-token\") pod \"e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b\" (UID: \"5178f75c-2f52-4a5b-bf00-c315bf7e110b\") " pod="openshift-ingress/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b" Nov 03 14:07:51.046367 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:51.046249 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5178f75c-2f52-4a5b-bf00-c315bf7e110b-istio-data\") pod \"e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b\" (UID: \"5178f75c-2f52-4a5b-bf00-c315bf7e110b\") " pod="openshift-ingress/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b" Nov 03 14:07:51.046552 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:51.046431 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5178f75c-2f52-4a5b-bf00-c315bf7e110b-workload-certs\") pod \"e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b\" (UID: \"5178f75c-2f52-4a5b-bf00-c315bf7e110b\") " pod="openshift-ingress/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b" Nov 03 14:07:51.046626 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:51.046591 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5178f75c-2f52-4a5b-bf00-c315bf7e110b-credential-socket\") pod \"e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b\" (UID: \"5178f75c-2f52-4a5b-bf00-c315bf7e110b\") " pod="openshift-ingress/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b" Nov 03 14:07:51.046691 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:51.046632 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr26v\" (UniqueName: \"kubernetes.io/projected/5178f75c-2f52-4a5b-bf00-c315bf7e110b-kube-api-access-cr26v\") pod \"e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b\" (UID: \"5178f75c-2f52-4a5b-bf00-c315bf7e110b\") " pod="openshift-ingress/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b" Nov 03 14:07:51.046750 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:51.046693 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5178f75c-2f52-4a5b-bf00-c315bf7e110b-workload-socket\") pod \"e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b\" (UID: \"5178f75c-2f52-4a5b-bf00-c315bf7e110b\") " pod="openshift-ingress/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b" Nov 03 14:07:51.046750 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:51.046726 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5178f75c-2f52-4a5b-bf00-c315bf7e110b-istio-podinfo\") pod \"e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b\" (UID: \"5178f75c-2f52-4a5b-bf00-c315bf7e110b\") " pod="openshift-ingress/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b" Nov 03 14:07:51.046867 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:51.046780 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5178f75c-2f52-4a5b-bf00-c315bf7e110b-istiod-ca-cert\") pod \"e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b\" (UID: \"5178f75c-2f52-4a5b-bf00-c315bf7e110b\") " pod="openshift-ingress/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b" Nov 03 14:07:51.051100 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:51.051072 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b"] Nov 03 14:07:51.077684 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:51.077658 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-d6963e-openshift-default-dockercfg-m8wqb\"" Nov 03 14:07:51.148534 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:51.148080 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5178f75c-2f52-4a5b-bf00-c315bf7e110b-credential-socket\") pod \"e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b\" (UID: \"5178f75c-2f52-4a5b-bf00-c315bf7e110b\") " pod="openshift-ingress/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b" Nov 03 14:07:51.148534 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:51.148123 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr26v\" (UniqueName: \"kubernetes.io/projected/5178f75c-2f52-4a5b-bf00-c315bf7e110b-kube-api-access-cr26v\") pod \"e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b\" (UID: \"5178f75c-2f52-4a5b-bf00-c315bf7e110b\") " pod="openshift-ingress/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b" Nov 03 14:07:51.148534 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:51.148186 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5178f75c-2f52-4a5b-bf00-c315bf7e110b-workload-socket\") pod \"e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b\" (UID: \"5178f75c-2f52-4a5b-bf00-c315bf7e110b\") " pod="openshift-ingress/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b" Nov 03 14:07:51.148534 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:51.148218 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5178f75c-2f52-4a5b-bf00-c315bf7e110b-istio-podinfo\") pod \"e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b\" (UID: \"5178f75c-2f52-4a5b-bf00-c315bf7e110b\") " pod="openshift-ingress/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b" Nov 03 14:07:51.148534 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:51.148272 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5178f75c-2f52-4a5b-bf00-c315bf7e110b-istiod-ca-cert\") pod \"e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b\" (UID: \"5178f75c-2f52-4a5b-bf00-c315bf7e110b\") " pod="openshift-ingress/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b" Nov 03 14:07:51.148534 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:51.148344 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5178f75c-2f52-4a5b-bf00-c315bf7e110b-istio-envoy\") pod \"e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b\" (UID: \"5178f75c-2f52-4a5b-bf00-c315bf7e110b\") " pod="openshift-ingress/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b" Nov 03 14:07:51.148534 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:51.148374 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5178f75c-2f52-4a5b-bf00-c315bf7e110b-istio-token\") pod \"e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b\" (UID: \"5178f75c-2f52-4a5b-bf00-c315bf7e110b\") " pod="openshift-ingress/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b" Nov 03 14:07:51.149078 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:51.148673 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5178f75c-2f52-4a5b-bf00-c315bf7e110b-credential-socket\") pod \"e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b\" (UID: \"5178f75c-2f52-4a5b-bf00-c315bf7e110b\") " pod="openshift-ingress/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b" Nov 03 14:07:51.149078 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:51.148757 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5178f75c-2f52-4a5b-bf00-c315bf7e110b-istio-data\") pod \"e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b\" (UID: \"5178f75c-2f52-4a5b-bf00-c315bf7e110b\") " pod="openshift-ingress/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b" Nov 03 14:07:51.149078 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:51.148878 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5178f75c-2f52-4a5b-bf00-c315bf7e110b-workload-certs\") pod \"e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b\" (UID: \"5178f75c-2f52-4a5b-bf00-c315bf7e110b\") " pod="openshift-ingress/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b" Nov 03 14:07:51.149262 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:51.149192 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5178f75c-2f52-4a5b-bf00-c315bf7e110b-workload-socket\") pod \"e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b\" (UID: \"5178f75c-2f52-4a5b-bf00-c315bf7e110b\") " pod="openshift-ingress/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b" Nov 03 14:07:51.149262 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:51.149213 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5178f75c-2f52-4a5b-bf00-c315bf7e110b-workload-certs\") pod \"e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b\" (UID: \"5178f75c-2f52-4a5b-bf00-c315bf7e110b\") " pod="openshift-ingress/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b" Nov 03 14:07:51.149606 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:51.149580 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5178f75c-2f52-4a5b-bf00-c315bf7e110b-istio-data\") pod \"e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b\" (UID: \"5178f75c-2f52-4a5b-bf00-c315bf7e110b\") " pod="openshift-ingress/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b" Nov 03 14:07:51.149894 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:51.149872 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5178f75c-2f52-4a5b-bf00-c315bf7e110b-istiod-ca-cert\") pod \"e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b\" (UID: \"5178f75c-2f52-4a5b-bf00-c315bf7e110b\") " pod="openshift-ingress/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b" Nov 03 14:07:51.151951 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:51.151928 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5178f75c-2f52-4a5b-bf00-c315bf7e110b-istio-envoy\") pod \"e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b\" (UID: \"5178f75c-2f52-4a5b-bf00-c315bf7e110b\") " pod="openshift-ingress/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b" Nov 03 14:07:51.152698 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:51.152677 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5178f75c-2f52-4a5b-bf00-c315bf7e110b-istio-podinfo\") pod \"e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b\" (UID: \"5178f75c-2f52-4a5b-bf00-c315bf7e110b\") " pod="openshift-ingress/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b" Nov 03 14:07:51.205757 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:51.205728 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5178f75c-2f52-4a5b-bf00-c315bf7e110b-istio-token\") pod \"e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b\" (UID: \"5178f75c-2f52-4a5b-bf00-c315bf7e110b\") " pod="openshift-ingress/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b" Nov 03 14:07:51.207327 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:51.207293 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr26v\" (UniqueName: \"kubernetes.io/projected/5178f75c-2f52-4a5b-bf00-c315bf7e110b-kube-api-access-cr26v\") pod \"e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b\" (UID: \"5178f75c-2f52-4a5b-bf00-c315bf7e110b\") " pod="openshift-ingress/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b" Nov 03 14:07:51.295498 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:51.295395 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b" Nov 03 14:07:51.741771 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:51.741731 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b"] Nov 03 14:07:51.747642 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:07:51.747605 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5178f75c_2f52_4a5b_bf00_c315bf7e110b.slice/crio-d9c9c3a474e9943f6725ad0f08dc4677ace6187e5559acec7a1540e6dade174d WatchSource:0}: Error finding container d9c9c3a474e9943f6725ad0f08dc4677ace6187e5559acec7a1540e6dade174d: Status 404 returned error can't find the container with id d9c9c3a474e9943f6725ad0f08dc4677ace6187e5559acec7a1540e6dade174d Nov 03 14:07:51.750399 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:51.750371 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:07:51.750522 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:51.750440 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:07:51.750522 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:51.750503 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:07:51.932350 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:51.932312 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b" event={"ID":"5178f75c-2f52-4a5b-bf00-c315bf7e110b","Type":"ContainerStarted","Data":"2b48c25bf1e790d5210106dc9685a1370fa259f24c4ab7508fb17bf685d3be43"} Nov 03 14:07:51.932545 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:51.932371 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b" event={"ID":"5178f75c-2f52-4a5b-bf00-c315bf7e110b","Type":"ContainerStarted","Data":"d9c9c3a474e9943f6725ad0f08dc4677ace6187e5559acec7a1540e6dade174d"} Nov 03 14:07:52.043269 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:52.043159 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b" podStartSLOduration=2.04314075 podStartE2EDuration="2.04314075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:07:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:07:52.041010738 +0000 UTC m=+3754.051698982" watchObservedRunningTime="2026-11-03 14:07:52.04314075 +0000 UTC m=+3754.053828994" Nov 03 14:07:52.219393 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:52.219347 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7898dd9db6-tfrv8" podUID="0c2807d6-84df-4d2c-8872-1c7635c02850" containerName="payload-processing" containerID="cri-o://d60ef85027afd45576ee91bddbfae26eaa99d34d87c6e8c4a5660cbae95e2cde" gracePeriod=25 Nov 03 14:07:52.295713 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:52.295616 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b" Nov 03 14:07:52.301865 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:52.301841 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b" Nov 03 14:07:52.470673 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:52.470642 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c4e398-2jzm2"] Nov 03 14:07:52.473424 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:52.473401 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c4e398-2jzm2"] Nov 03 14:07:52.500318 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:52.500289 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7898dd9db6-tfrv8" Nov 03 14:07:52.598504 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:52.598411 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c2807d6-84df-4d2c-8872-1c7635c02850-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0c2807d6-84df-4d2c-8872-1c7635c02850-routing-overlay-volume\") pod \"0c2807d6-84df-4d2c-8872-1c7635c02850\" (UID: \"0c2807d6-84df-4d2c-8872-1c7635c02850\") " Nov 03 14:07:52.598669 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:52.598559 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c2807d6-84df-4d2c-8872-1c7635c02850-kube-api-access-rk6wf\" (UniqueName: \"kubernetes.io/projected/0c2807d6-84df-4d2c-8872-1c7635c02850-kube-api-access-rk6wf\") pod \"0c2807d6-84df-4d2c-8872-1c7635c02850\" (UID: \"0c2807d6-84df-4d2c-8872-1c7635c02850\") " Nov 03 14:07:52.598669 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:52.598592 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c2807d6-84df-4d2c-8872-1c7635c02850-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0c2807d6-84df-4d2c-8872-1c7635c02850-provider-credentials\") pod \"0c2807d6-84df-4d2c-8872-1c7635c02850\" (UID: \"0c2807d6-84df-4d2c-8872-1c7635c02850\") " Nov 03 14:07:52.598746 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:52.598722 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c2807d6-84df-4d2c-8872-1c7635c02850-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c2807d6-84df-4d2c-8872-1c7635c02850-plugins-config-volume\") pod \"0c2807d6-84df-4d2c-8872-1c7635c02850\" (UID: \"0c2807d6-84df-4d2c-8872-1c7635c02850\") " Nov 03 14:07:52.598884 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:52.598862 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c2807d6-84df-4d2c-8872-1c7635c02850-routing-overlay-volume" pod "0c2807d6-84df-4d2c-8872-1c7635c02850" (UID: "0c2807d6-84df-4d2c-8872-1c7635c02850"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:07:52.599231 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:52.599206 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c2807d6-84df-4d2c-8872-1c7635c02850-plugins-config-volume" pod "0c2807d6-84df-4d2c-8872-1c7635c02850" (UID: "0c2807d6-84df-4d2c-8872-1c7635c02850"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:07:52.599331 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:52.599311 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c2807d6-84df-4d2c-8872-1c7635c02850-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:52.599371 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:52.599335 5979 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0c2807d6-84df-4d2c-8872-1c7635c02850-routing-overlay-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:52.600708 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:52.600691 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c2807d6-84df-4d2c-8872-1c7635c02850-kube-api-access-rk6wf" pod "0c2807d6-84df-4d2c-8872-1c7635c02850" (UID: "0c2807d6-84df-4d2c-8872-1c7635c02850"). InnerVolumeSpecName "kube-api-access-rk6wf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:07:52.600859 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:52.600844 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c2807d6-84df-4d2c-8872-1c7635c02850-provider-credentials" pod "0c2807d6-84df-4d2c-8872-1c7635c02850" (UID: "0c2807d6-84df-4d2c-8872-1c7635c02850"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:07:52.701052 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:52.701015 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk6wf\" (UniqueName: \"kubernetes.io/projected/0c2807d6-84df-4d2c-8872-1c7635c02850-kube-api-access-rk6wf\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:52.701052 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:52.701055 5979 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0c2807d6-84df-4d2c-8872-1c7635c02850-provider-credentials\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:07:52.940543 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:52.940511 5979 generic.go:356] "Generic (PLEG): container finished" podID="0c2807d6-84df-4d2c-8872-1c7635c02850" containerID="d60ef85027afd45576ee91bddbfae26eaa99d34d87c6e8c4a5660cbae95e2cde" exitCode=0 Nov 03 14:07:52.940747 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:52.940564 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7898dd9db6-tfrv8" Nov 03 14:07:52.940747 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:52.940594 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7898dd9db6-tfrv8" event={"ID":"0c2807d6-84df-4d2c-8872-1c7635c02850","Type":"ContainerDied","Data":"d60ef85027afd45576ee91bddbfae26eaa99d34d87c6e8c4a5660cbae95e2cde"} Nov 03 14:07:52.940747 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:52.940646 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7898dd9db6-tfrv8" event={"ID":"0c2807d6-84df-4d2c-8872-1c7635c02850","Type":"ContainerDied","Data":"3e82746961431a74f419ad1d11be42ad80337dc5a099a1e0bed2354a61bc01f8"} Nov 03 14:07:52.940747 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:52.940674 5979 scope.go:122] "RemoveContainer" containerID="d60ef85027afd45576ee91bddbfae26eaa99d34d87c6e8c4a5660cbae95e2cde" Nov 03 14:07:52.941114 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:52.941056 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b" Nov 03 14:07:52.942143 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:52.942125 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b" Nov 03 14:07:52.950989 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:52.950971 5979 scope.go:122] "RemoveContainer" containerID="d60ef85027afd45576ee91bddbfae26eaa99d34d87c6e8c4a5660cbae95e2cde" Nov 03 14:07:52.951267 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:07:52.951248 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d60ef85027afd45576ee91bddbfae26eaa99d34d87c6e8c4a5660cbae95e2cde\": container with ID starting with d60ef85027afd45576ee91bddbfae26eaa99d34d87c6e8c4a5660cbae95e2cde not found: ID does not exist" containerID="d60ef85027afd45576ee91bddbfae26eaa99d34d87c6e8c4a5660cbae95e2cde" Nov 03 14:07:52.951340 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:52.951284 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d60ef85027afd45576ee91bddbfae26eaa99d34d87c6e8c4a5660cbae95e2cde"} err="failed to get container status \"d60ef85027afd45576ee91bddbfae26eaa99d34d87c6e8c4a5660cbae95e2cde\": rpc error: code = NotFound desc = could not find container \"d60ef85027afd45576ee91bddbfae26eaa99d34d87c6e8c4a5660cbae95e2cde\": container with ID starting with d60ef85027afd45576ee91bddbfae26eaa99d34d87c6e8c4a5660cbae95e2cde not found: ID does not exist" Nov 03 14:07:53.095373 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:53.095340 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7898dd9db6-tfrv8"] Nov 03 14:07:53.099952 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:53.099924 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7898dd9db6-tfrv8"] Nov 03 14:07:54.116341 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:54.116307 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c2807d6-84df-4d2c-8872-1c7635c02850" path="/var/lib/kubelet/pods/0c2807d6-84df-4d2c-8872-1c7635c02850/volumes" Nov 03 14:07:54.116814 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:54.116798 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6a0821f-4fce-409e-aa34-1c96dca44942" path="/var/lib/kubelet/pods/d6a0821f-4fce-409e-aa34-1c96dca44942/volumes" Nov 03 14:07:57.878573 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:07:57.878530 5979 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-e86f9ab11580: secret "agc-cred-e86f9ab11580" not found Nov 03 14:07:57.878573 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:07:57.878566 5979 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-84dcd755bf-c9rnp: secret "agc-cred-e86f9ab11580" not found Nov 03 14:07:57.879030 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:07:57.878628 5979 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/797bcf35-fa9e-47b8-887c-bb8942af277b-provider-credentials podName:797bcf35-fa9e-47b8-887c-bb8942af277b nodeName:}" failed. No retries permitted until 2026-11-03 14:08:13.8786087 +0000 UTC m=+3775.889296920 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/797bcf35-fa9e-47b8-887c-bb8942af277b-provider-credentials") pod "payload-processing-external-model-84dcd755bf-c9rnp" (UID: "797bcf35-fa9e-47b8-887c-bb8942af277b") : secret "agc-cred-e86f9ab11580" not found Nov 03 14:07:59.448100 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:59.448063 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-966a03-7dd5866fdc-grhzp"] Nov 03 14:07:59.922031 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:07:59.921988 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" podUID="d09478af-d618-425d-a4e6-e0889e0f416d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:08:03.609834 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:03.609794 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d6963e-cc4d8848c-bf2p7"] Nov 03 14:08:03.610525 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:03.610489 5979 cpu_manager.go:414] "containerMap: removing container" podUID="0c2807d6-84df-4d2c-8872-1c7635c02850" containerName="payload-processing" Nov 03 14:08:03.610525 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:03.610508 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c2807d6-84df-4d2c-8872-1c7635c02850" containerName="payload-processing" Nov 03 14:08:03.610712 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:03.610654 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c2807d6-84df-4d2c-8872-1c7635c02850" containerName="payload-processing" Nov 03 14:08:03.614373 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:03.614352 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d6963e-cc4d8848c-bf2p7" Nov 03 14:08:03.665035 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:03.664996 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-d6963e\"" Nov 03 14:08:03.712859 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:03.707602 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv4bz\" (UniqueName: \"kubernetes.io/projected/8c10c218-9344-4504-a060-2d533a772408-kube-api-access-qv4bz\") pod \"payload-pre-processing-e2e-inf-a-d6963e-cc4d8848c-bf2p7\" (UID: \"8c10c218-9344-4504-a060-2d533a772408\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d6963e-cc4d8848c-bf2p7" Nov 03 14:08:03.712859 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:03.707704 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c10c218-9344-4504-a060-2d533a772408-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d6963e-cc4d8848c-bf2p7\" (UID: \"8c10c218-9344-4504-a060-2d533a772408\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d6963e-cc4d8848c-bf2p7" Nov 03 14:08:03.720992 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:03.720952 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-d6963e-dockercfg-jhjk8\"" Nov 03 14:08:03.732123 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:03.731982 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d6963e-cc4d8848c-bf2p7"] Nov 03 14:08:03.795744 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:03.795707 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d6963e-8679d489c9-7n5xs"] Nov 03 14:08:03.800903 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:03.800882 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d6963e-8679d489c9-7n5xs" Nov 03 14:08:03.808669 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:03.808640 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv4bz\" (UniqueName: \"kubernetes.io/projected/8c10c218-9344-4504-a060-2d533a772408-kube-api-access-qv4bz\") pod \"payload-pre-processing-e2e-inf-a-d6963e-cc4d8848c-bf2p7\" (UID: \"8c10c218-9344-4504-a060-2d533a772408\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d6963e-cc4d8848c-bf2p7" Nov 03 14:08:03.808783 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:03.808709 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c10c218-9344-4504-a060-2d533a772408-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d6963e-cc4d8848c-bf2p7\" (UID: \"8c10c218-9344-4504-a060-2d533a772408\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d6963e-cc4d8848c-bf2p7" Nov 03 14:08:03.809407 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:03.809383 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c10c218-9344-4504-a060-2d533a772408-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d6963e-cc4d8848c-bf2p7\" (UID: \"8c10c218-9344-4504-a060-2d533a772408\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d6963e-cc4d8848c-bf2p7" Nov 03 14:08:03.850041 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:03.850009 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d6963e-8679d489c9-7n5xs"] Nov 03 14:08:03.863904 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:03.863818 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv4bz\" (UniqueName: \"kubernetes.io/projected/8c10c218-9344-4504-a060-2d533a772408-kube-api-access-qv4bz\") pod \"payload-pre-processing-e2e-inf-a-d6963e-cc4d8848c-bf2p7\" (UID: \"8c10c218-9344-4504-a060-2d533a772408\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d6963e-cc4d8848c-bf2p7" Nov 03 14:08:03.910338 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:03.910296 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlbdt\" (UniqueName: \"kubernetes.io/projected/f95dd442-db70-44a1-b6ac-533be225068c-kube-api-access-dlbdt\") pod \"payload-processing-e2e-inf-a-d6963e-8679d489c9-7n5xs\" (UID: \"f95dd442-db70-44a1-b6ac-533be225068c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d6963e-8679d489c9-7n5xs" Nov 03 14:08:03.910535 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:03.910368 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f95dd442-db70-44a1-b6ac-533be225068c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d6963e-8679d489c9-7n5xs\" (UID: \"f95dd442-db70-44a1-b6ac-533be225068c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d6963e-8679d489c9-7n5xs" Nov 03 14:08:03.929118 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:03.929086 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d6963e-cc4d8848c-bf2p7" Nov 03 14:08:04.011405 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:04.011370 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlbdt\" (UniqueName: \"kubernetes.io/projected/f95dd442-db70-44a1-b6ac-533be225068c-kube-api-access-dlbdt\") pod \"payload-processing-e2e-inf-a-d6963e-8679d489c9-7n5xs\" (UID: \"f95dd442-db70-44a1-b6ac-533be225068c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d6963e-8679d489c9-7n5xs" Nov 03 14:08:04.011575 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:04.011433 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f95dd442-db70-44a1-b6ac-533be225068c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d6963e-8679d489c9-7n5xs\" (UID: \"f95dd442-db70-44a1-b6ac-533be225068c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d6963e-8679d489c9-7n5xs" Nov 03 14:08:04.012166 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:04.012147 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f95dd442-db70-44a1-b6ac-533be225068c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-d6963e-8679d489c9-7n5xs\" (UID: \"f95dd442-db70-44a1-b6ac-533be225068c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d6963e-8679d489c9-7n5xs" Nov 03 14:08:04.065562 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:04.065514 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlbdt\" (UniqueName: \"kubernetes.io/projected/f95dd442-db70-44a1-b6ac-533be225068c-kube-api-access-dlbdt\") pod \"payload-processing-e2e-inf-a-d6963e-8679d489c9-7n5xs\" (UID: \"f95dd442-db70-44a1-b6ac-533be225068c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-d6963e-8679d489c9-7n5xs" Nov 03 14:08:04.115647 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:04.115556 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d6963e-8679d489c9-7n5xs" Nov 03 14:08:04.334324 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:04.334285 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d6963e-cc4d8848c-bf2p7"] Nov 03 14:08:04.340682 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:08:04.340653 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c10c218_9344_4504_a060_2d533a772408.slice/crio-a6791350d7034a59d25352b5f4ec399008f938862b63b21aa07663abc807033b WatchSource:0}: Error finding container a6791350d7034a59d25352b5f4ec399008f938862b63b21aa07663abc807033b: Status 404 returned error can't find the container with id a6791350d7034a59d25352b5f4ec399008f938862b63b21aa07663abc807033b Nov 03 14:08:04.449266 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:04.449216 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-966a03-7dd5866fdc-grhzp" podUID="c3d6304e-6ef4-4c50-beb6-54ef6162d206" containerName="payload-pre-processing" containerID="cri-o://7dbd17374f2e708c8e708b12106f128c755b2d1d8c38cfcf1874840d6b2b1e6c" gracePeriod=25 Nov 03 14:08:04.494534 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:04.494501 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d6963e-8679d489c9-7n5xs"] Nov 03 14:08:04.512452 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:08:04.512423 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf95dd442_db70_44a1_b6ac_533be225068c.slice/crio-c17de2af77aa503e3336b30390502b137ee3e93cc479a27b4b03c2d32ee93526 WatchSource:0}: Error finding container c17de2af77aa503e3336b30390502b137ee3e93cc479a27b4b03c2d32ee93526: Status 404 returned error can't find the container with id c17de2af77aa503e3336b30390502b137ee3e93cc479a27b4b03c2d32ee93526 Nov 03 14:08:04.626143 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:08:04.625536 5979 logging.go:55] [core] [Channel #9464 SubChannel #9465]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.1.53:50052", ServerName: "10.134.1.53:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.1.53:50052: connect: connection refused" Nov 03 14:08:04.751417 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:04.751390 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-966a03-7dd5866fdc-grhzp" Nov 03 14:08:04.922192 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:04.922107 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3d6304e-6ef4-4c50-beb6-54ef6162d206-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3d6304e-6ef4-4c50-beb6-54ef6162d206-plugins-config-volume\") pod \"c3d6304e-6ef4-4c50-beb6-54ef6162d206\" (UID: \"c3d6304e-6ef4-4c50-beb6-54ef6162d206\") " Nov 03 14:08:04.922376 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:04.922254 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3d6304e-6ef4-4c50-beb6-54ef6162d206-kube-api-access-bqt8j\" (UniqueName: \"kubernetes.io/projected/c3d6304e-6ef4-4c50-beb6-54ef6162d206-kube-api-access-bqt8j\") pod \"c3d6304e-6ef4-4c50-beb6-54ef6162d206\" (UID: \"c3d6304e-6ef4-4c50-beb6-54ef6162d206\") " Nov 03 14:08:04.922610 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:04.922585 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3d6304e-6ef4-4c50-beb6-54ef6162d206-plugins-config-volume" pod "c3d6304e-6ef4-4c50-beb6-54ef6162d206" (UID: "c3d6304e-6ef4-4c50-beb6-54ef6162d206"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:08:04.922888 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:04.922868 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3d6304e-6ef4-4c50-beb6-54ef6162d206-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:08:04.924383 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:04.924360 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3d6304e-6ef4-4c50-beb6-54ef6162d206-kube-api-access-bqt8j" pod "c3d6304e-6ef4-4c50-beb6-54ef6162d206" (UID: "c3d6304e-6ef4-4c50-beb6-54ef6162d206"). InnerVolumeSpecName "kube-api-access-bqt8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:08:05.024632 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:05.024592 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqt8j\" (UniqueName: \"kubernetes.io/projected/c3d6304e-6ef4-4c50-beb6-54ef6162d206-kube-api-access-bqt8j\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:08:05.058668 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:05.058633 5979 generic.go:356] "Generic (PLEG): container finished" podID="c3d6304e-6ef4-4c50-beb6-54ef6162d206" containerID="7dbd17374f2e708c8e708b12106f128c755b2d1d8c38cfcf1874840d6b2b1e6c" exitCode=0 Nov 03 14:08:05.058859 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:05.058698 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-966a03-7dd5866fdc-grhzp" Nov 03 14:08:05.058859 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:05.058713 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-966a03-7dd5866fdc-grhzp" event={"ID":"c3d6304e-6ef4-4c50-beb6-54ef6162d206","Type":"ContainerDied","Data":"7dbd17374f2e708c8e708b12106f128c755b2d1d8c38cfcf1874840d6b2b1e6c"} Nov 03 14:08:05.058859 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:05.058780 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-966a03-7dd5866fdc-grhzp" event={"ID":"c3d6304e-6ef4-4c50-beb6-54ef6162d206","Type":"ContainerDied","Data":"79a2c4c11adabf23d4b95571ff846556c58c5523408d297fbf671d702f200903"} Nov 03 14:08:05.058859 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:05.058810 5979 scope.go:122] "RemoveContainer" containerID="7dbd17374f2e708c8e708b12106f128c755b2d1d8c38cfcf1874840d6b2b1e6c" Nov 03 14:08:05.060503 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:05.060456 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d6963e-cc4d8848c-bf2p7" event={"ID":"8c10c218-9344-4504-a060-2d533a772408","Type":"ContainerStarted","Data":"4a5733889df53b6a1dd0a57aa050ce805132fdc2664195b5829345e288663e24"} Nov 03 14:08:05.060640 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:05.060521 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d6963e-cc4d8848c-bf2p7" event={"ID":"8c10c218-9344-4504-a060-2d533a772408","Type":"ContainerStarted","Data":"a6791350d7034a59d25352b5f4ec399008f938862b63b21aa07663abc807033b"} Nov 03 14:08:05.060640 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:05.060567 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d6963e-cc4d8848c-bf2p7" Nov 03 14:08:05.062168 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:05.062147 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d6963e-8679d489c9-7n5xs" event={"ID":"f95dd442-db70-44a1-b6ac-533be225068c","Type":"ContainerStarted","Data":"9d641f843c83d02b0d6fdf82c4032024d66ba841f7b9281d576d16a40a4ac7eb"} Nov 03 14:08:05.062295 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:05.062213 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d6963e-8679d489c9-7n5xs" event={"ID":"f95dd442-db70-44a1-b6ac-533be225068c","Type":"ContainerStarted","Data":"c17de2af77aa503e3336b30390502b137ee3e93cc479a27b4b03c2d32ee93526"} Nov 03 14:08:05.062295 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:05.062233 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d6963e-8679d489c9-7n5xs" Nov 03 14:08:05.069598 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:05.069578 5979 scope.go:122] "RemoveContainer" containerID="7dbd17374f2e708c8e708b12106f128c755b2d1d8c38cfcf1874840d6b2b1e6c" Nov 03 14:08:05.069873 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:08:05.069852 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dbd17374f2e708c8e708b12106f128c755b2d1d8c38cfcf1874840d6b2b1e6c\": container with ID starting with 7dbd17374f2e708c8e708b12106f128c755b2d1d8c38cfcf1874840d6b2b1e6c not found: ID does not exist" containerID="7dbd17374f2e708c8e708b12106f128c755b2d1d8c38cfcf1874840d6b2b1e6c" Nov 03 14:08:05.069980 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:05.069877 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dbd17374f2e708c8e708b12106f128c755b2d1d8c38cfcf1874840d6b2b1e6c"} err="failed to get container status \"7dbd17374f2e708c8e708b12106f128c755b2d1d8c38cfcf1874840d6b2b1e6c\": rpc error: code = NotFound desc = could not find container \"7dbd17374f2e708c8e708b12106f128c755b2d1d8c38cfcf1874840d6b2b1e6c\": container with ID starting with 7dbd17374f2e708c8e708b12106f128c755b2d1d8c38cfcf1874840d6b2b1e6c not found: ID does not exist" Nov 03 14:08:05.216314 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:05.216105 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-d6963e-8679d489c9-7n5xs" podStartSLOduration=2.216088479 podStartE2EDuration="2.216088479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:08:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:08:05.215035472 +0000 UTC m=+3767.225723714" watchObservedRunningTime="2026-11-03 14:08:05.216088479 +0000 UTC m=+3767.226776722" Nov 03 14:08:05.325713 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:05.325643 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d6963e-cc4d8848c-bf2p7" podStartSLOduration=2.325624368 podStartE2EDuration="2.325624368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:08:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:08:05.32164682 +0000 UTC m=+3767.332335062" watchObservedRunningTime="2026-11-03 14:08:05.325624368 +0000 UTC m=+3767.336312610" Nov 03 14:08:05.427080 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:05.427045 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-966a03-7dd5866fdc-grhzp"] Nov 03 14:08:05.431379 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:05.431353 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-966a03-7dd5866fdc-grhzp"] Nov 03 14:08:05.625457 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:05.625360 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-966a03-7dd5866fdc-grhzp" podUID="c3d6304e-6ef4-4c50-beb6-54ef6162d206" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.1.53:50052\" within 1s: context deadline exceeded" Nov 03 14:08:06.116666 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:06.116634 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3d6304e-6ef4-4c50-beb6-54ef6162d206" path="/var/lib/kubelet/pods/c3d6304e-6ef4-4c50-beb6-54ef6162d206/volumes" Nov 03 14:08:07.233598 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:08:07.233561 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:08:09.919577 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:09.919538 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" podUID="d09478af-d618-425d-a4e6-e0889e0f416d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:08:12.267857 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:12.267822 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l"] Nov 03 14:08:12.268573 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:12.268555 5979 cpu_manager.go:414] "containerMap: removing container" podUID="c3d6304e-6ef4-4c50-beb6-54ef6162d206" containerName="payload-pre-processing" Nov 03 14:08:12.268642 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:12.268574 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3d6304e-6ef4-4c50-beb6-54ef6162d206" containerName="payload-pre-processing" Nov 03 14:08:12.268703 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:12.268694 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3d6304e-6ef4-4c50-beb6-54ef6162d206" containerName="payload-pre-processing" Nov 03 14:08:12.274059 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:12.274041 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l" Nov 03 14:08:12.348361 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:12.348315 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-istiod-ca-cert\") pod \"e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l\" (UID: \"8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130\") " pod="openshift-ingress/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l" Nov 03 14:08:12.348788 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:12.348759 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-workload-certs\") pod \"e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l\" (UID: \"8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130\") " pod="openshift-ingress/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l" Nov 03 14:08:12.348935 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:12.348920 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-workload-socket\") pod \"e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l\" (UID: \"8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130\") " pod="openshift-ingress/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l" Nov 03 14:08:12.349060 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:12.349045 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-istio-data\") pod \"e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l\" (UID: \"8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130\") " pod="openshift-ingress/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l" Nov 03 14:08:12.353840 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:12.353808 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-istio-token\") pod \"e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l\" (UID: \"8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130\") " pod="openshift-ingress/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l" Nov 03 14:08:12.367611 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:12.367571 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-credential-socket\") pod \"e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l\" (UID: \"8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130\") " pod="openshift-ingress/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l" Nov 03 14:08:12.367859 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:12.367829 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-istio-envoy\") pod \"e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l\" (UID: \"8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130\") " pod="openshift-ingress/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l" Nov 03 14:08:12.368002 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:12.367869 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-istio-podinfo\") pod \"e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l\" (UID: \"8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130\") " pod="openshift-ingress/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l" Nov 03 14:08:12.368002 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:12.367917 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tpml\" (UniqueName: \"kubernetes.io/projected/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-kube-api-access-6tpml\") pod \"e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l\" (UID: \"8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130\") " pod="openshift-ingress/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l" Nov 03 14:08:12.393064 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:12.393024 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l"] Nov 03 14:08:12.401410 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:12.401377 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-f11801-openshift-default-dockercfg-qpkp6\"" Nov 03 14:08:12.471137 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:12.471096 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-istio-token\") pod \"e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l\" (UID: \"8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130\") " pod="openshift-ingress/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l" Nov 03 14:08:12.471332 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:12.471144 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-credential-socket\") pod \"e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l\" (UID: \"8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130\") " pod="openshift-ingress/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l" Nov 03 14:08:12.471332 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:12.471182 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-istio-envoy\") pod \"e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l\" (UID: \"8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130\") " pod="openshift-ingress/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l" Nov 03 14:08:12.471332 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:12.471206 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-istio-podinfo\") pod \"e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l\" (UID: \"8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130\") " pod="openshift-ingress/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l" Nov 03 14:08:12.471332 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:12.471262 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tpml\" (UniqueName: \"kubernetes.io/projected/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-kube-api-access-6tpml\") pod \"e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l\" (UID: \"8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130\") " pod="openshift-ingress/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l" Nov 03 14:08:12.471596 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:12.471354 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-istiod-ca-cert\") pod \"e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l\" (UID: \"8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130\") " pod="openshift-ingress/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l" Nov 03 14:08:12.471596 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:12.471510 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-workload-certs\") pod \"e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l\" (UID: \"8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130\") " pod="openshift-ingress/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l" Nov 03 14:08:12.471596 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:12.471543 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-workload-socket\") pod \"e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l\" (UID: \"8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130\") " pod="openshift-ingress/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l" Nov 03 14:08:12.471766 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:12.471595 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-istio-data\") pod \"e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l\" (UID: \"8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130\") " pod="openshift-ingress/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l" Nov 03 14:08:12.472211 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:12.472171 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-istio-data\") pod \"e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l\" (UID: \"8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130\") " pod="openshift-ingress/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l" Nov 03 14:08:12.472528 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:12.472502 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-workload-certs\") pod \"e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l\" (UID: \"8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130\") " pod="openshift-ingress/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l" Nov 03 14:08:12.472932 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:12.472912 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-workload-socket\") pod \"e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l\" (UID: \"8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130\") " pod="openshift-ingress/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l" Nov 03 14:08:12.473149 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:12.473120 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-credential-socket\") pod \"e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l\" (UID: \"8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130\") " pod="openshift-ingress/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l" Nov 03 14:08:12.473649 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:12.473620 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-istiod-ca-cert\") pod \"e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l\" (UID: \"8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130\") " pod="openshift-ingress/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l" Nov 03 14:08:12.474978 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:12.474956 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-istio-envoy\") pod \"e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l\" (UID: \"8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130\") " pod="openshift-ingress/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l" Nov 03 14:08:12.477001 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:12.476973 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-istio-podinfo\") pod \"e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l\" (UID: \"8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130\") " pod="openshift-ingress/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l" Nov 03 14:08:12.527017 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:12.526949 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-istio-token\") pod \"e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l\" (UID: \"8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130\") " pod="openshift-ingress/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l" Nov 03 14:08:12.527154 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:12.527110 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tpml\" (UniqueName: \"kubernetes.io/projected/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-kube-api-access-6tpml\") pod \"e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l\" (UID: \"8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130\") " pod="openshift-ingress/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l" Nov 03 14:08:12.588567 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:12.588532 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l" Nov 03 14:08:13.031584 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:13.031548 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l"] Nov 03 14:08:13.038714 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:08:13.038686 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f547ad2_b677_4e3f_bbc0_f6e1dc3ee130.slice/crio-ae2698aff290b6d452c9940b1839a8e9b0d6e200bf5fe16a6670d2d27b8ca81f WatchSource:0}: Error finding container ae2698aff290b6d452c9940b1839a8e9b0d6e200bf5fe16a6670d2d27b8ca81f: Status 404 returned error can't find the container with id ae2698aff290b6d452c9940b1839a8e9b0d6e200bf5fe16a6670d2d27b8ca81f Nov 03 14:08:13.041622 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:13.041589 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:08:13.041723 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:13.041658 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:08:13.041723 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:13.041687 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:08:13.124917 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:13.124886 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l" event={"ID":"8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130","Type":"ContainerStarted","Data":"8f530b2c5e7d54c1b2dc8044d2f741f34ed5993a3e584960e01e0546d9f28312"} Nov 03 14:08:13.125044 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:13.124954 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l" event={"ID":"8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130","Type":"ContainerStarted","Data":"ae2698aff290b6d452c9940b1839a8e9b0d6e200bf5fe16a6670d2d27b8ca81f"} Nov 03 14:08:13.239313 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:13.239255 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l" podStartSLOduration=1.239242662 podStartE2EDuration="1.239242662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:08:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:08:13.234228499 +0000 UTC m=+3775.244916752" watchObservedRunningTime="2026-11-03 14:08:13.239242662 +0000 UTC m=+3775.249930904" Nov 03 14:08:13.589619 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:13.589586 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l" Nov 03 14:08:13.892350 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:08:13.892308 5979 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-e86f9ab11580: secret "agc-cred-e86f9ab11580" not found Nov 03 14:08:13.892350 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:08:13.892342 5979 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-84dcd755bf-c9rnp: secret "agc-cred-e86f9ab11580" not found Nov 03 14:08:13.892702 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:08:13.892409 5979 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/797bcf35-fa9e-47b8-887c-bb8942af277b-provider-credentials podName:797bcf35-fa9e-47b8-887c-bb8942af277b nodeName:}" failed. No retries permitted until 2026-11-03 14:08:45.892390699 +0000 UTC m=+3807.903078938 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/797bcf35-fa9e-47b8-887c-bb8942af277b-provider-credentials") pod "payload-processing-external-model-84dcd755bf-c9rnp" (UID: "797bcf35-fa9e-47b8-887c-bb8942af277b") : secret "agc-cred-e86f9ab11580" not found Nov 03 14:08:14.594524 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:14.594488 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l" Nov 03 14:08:15.138565 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:15.138526 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l" Nov 03 14:08:15.139516 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:15.139494 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l" Nov 03 14:08:16.074136 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:16.074107 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-d6963e-8679d489c9-7n5xs" Nov 03 14:08:16.074540 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:16.074276 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d6963e-cc4d8848c-bf2p7" Nov 03 14:08:19.674648 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:19.674607 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp"] Nov 03 14:08:19.681720 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:19.681689 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp" Nov 03 14:08:19.803855 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:19.803797 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nx2cp\" (UID: \"215d3ba5-9175-474d-9ad2-ff3cdb5ed411\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp" Nov 03 14:08:19.817866 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:19.817830 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nx2cp\" (UID: \"215d3ba5-9175-474d-9ad2-ff3cdb5ed411\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp" Nov 03 14:08:19.818169 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:19.818150 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nx2cp\" (UID: \"215d3ba5-9175-474d-9ad2-ff3cdb5ed411\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp" Nov 03 14:08:19.818454 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:19.818427 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nx2cp\" (UID: \"215d3ba5-9175-474d-9ad2-ff3cdb5ed411\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp" Nov 03 14:08:19.818618 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:19.818600 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nx2cp\" (UID: \"215d3ba5-9175-474d-9ad2-ff3cdb5ed411\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp" Nov 03 14:08:19.818728 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:19.818715 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nx2cp\" (UID: \"215d3ba5-9175-474d-9ad2-ff3cdb5ed411\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp" Nov 03 14:08:19.818899 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:19.818886 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nx2cp\" (UID: \"215d3ba5-9175-474d-9ad2-ff3cdb5ed411\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp" Nov 03 14:08:19.819039 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:19.819019 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nx2cp\" (UID: \"215d3ba5-9175-474d-9ad2-ff3cdb5ed411\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp" Nov 03 14:08:19.819218 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:19.819202 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xttd7\" (UniqueName: \"kubernetes.io/projected/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-kube-api-access-xttd7\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nx2cp\" (UID: \"215d3ba5-9175-474d-9ad2-ff3cdb5ed411\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp" Nov 03 14:08:19.819410 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:19.817301 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp"] Nov 03 14:08:19.920234 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:19.920193 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nx2cp\" (UID: \"215d3ba5-9175-474d-9ad2-ff3cdb5ed411\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp" Nov 03 14:08:19.920234 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:19.920234 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nx2cp\" (UID: \"215d3ba5-9175-474d-9ad2-ff3cdb5ed411\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp" Nov 03 14:08:19.920535 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:19.920363 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nx2cp\" (UID: \"215d3ba5-9175-474d-9ad2-ff3cdb5ed411\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp" Nov 03 14:08:19.920535 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:19.920409 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" podUID="d09478af-d618-425d-a4e6-e0889e0f416d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:08:19.920535 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:19.920497 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nx2cp\" (UID: \"215d3ba5-9175-474d-9ad2-ff3cdb5ed411\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp" Nov 03 14:08:19.920721 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:19.920557 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nx2cp\" (UID: \"215d3ba5-9175-474d-9ad2-ff3cdb5ed411\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp" Nov 03 14:08:19.920721 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:19.920626 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xttd7\" (UniqueName: \"kubernetes.io/projected/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-kube-api-access-xttd7\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nx2cp\" (UID: \"215d3ba5-9175-474d-9ad2-ff3cdb5ed411\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp" Nov 03 14:08:19.920721 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:19.920642 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nx2cp\" (UID: \"215d3ba5-9175-474d-9ad2-ff3cdb5ed411\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp" Nov 03 14:08:19.920721 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:19.920670 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nx2cp\" (UID: \"215d3ba5-9175-474d-9ad2-ff3cdb5ed411\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp" Nov 03 14:08:19.920721 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:19.920695 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nx2cp\" (UID: \"215d3ba5-9175-474d-9ad2-ff3cdb5ed411\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp" Nov 03 14:08:19.920721 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:19.920710 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nx2cp\" (UID: \"215d3ba5-9175-474d-9ad2-ff3cdb5ed411\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp" Nov 03 14:08:19.921061 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:19.920803 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nx2cp\" (UID: \"215d3ba5-9175-474d-9ad2-ff3cdb5ed411\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp" Nov 03 14:08:19.921124 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:19.921058 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nx2cp\" (UID: \"215d3ba5-9175-474d-9ad2-ff3cdb5ed411\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp" Nov 03 14:08:19.921186 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:19.921160 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nx2cp\" (UID: \"215d3ba5-9175-474d-9ad2-ff3cdb5ed411\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp" Nov 03 14:08:19.921247 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:19.921230 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nx2cp\" (UID: \"215d3ba5-9175-474d-9ad2-ff3cdb5ed411\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp" Nov 03 14:08:19.922746 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:19.922724 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nx2cp\" (UID: \"215d3ba5-9175-474d-9ad2-ff3cdb5ed411\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp" Nov 03 14:08:19.922855 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:19.922787 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nx2cp\" (UID: \"215d3ba5-9175-474d-9ad2-ff3cdb5ed411\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp" Nov 03 14:08:19.986538 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:19.985760 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xttd7\" (UniqueName: \"kubernetes.io/projected/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-kube-api-access-xttd7\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nx2cp\" (UID: \"215d3ba5-9175-474d-9ad2-ff3cdb5ed411\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp" Nov 03 14:08:19.986538 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:19.985807 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-nx2cp\" (UID: \"215d3ba5-9175-474d-9ad2-ff3cdb5ed411\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp" Nov 03 14:08:20.004523 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:20.001756 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp" Nov 03 14:08:20.223065 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:20.222134 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-f11801-74b6fc964f-mf2fc"] Nov 03 14:08:20.236700 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:20.236619 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-f11801-74b6fc964f-mf2fc" Nov 03 14:08:20.284744 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:20.284709 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-f11801-74b6fc964f-mf2fc"] Nov 03 14:08:20.290199 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:20.290169 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-f11801\"" Nov 03 14:08:20.295154 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:20.295128 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-f11801-dockercfg-t945f\"" Nov 03 14:08:20.325913 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:20.325883 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rd77\" (UniqueName: \"kubernetes.io/projected/db161f59-c4de-4fd3-bc2b-0a6262dffd38-kube-api-access-6rd77\") pod \"payload-processing-e2e-extmod-a-f11801-74b6fc964f-mf2fc\" (UID: \"db161f59-c4de-4fd3-bc2b-0a6262dffd38\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-f11801-74b6fc964f-mf2fc" Nov 03 14:08:20.326153 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:20.326129 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db161f59-c4de-4fd3-bc2b-0a6262dffd38-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-f11801-74b6fc964f-mf2fc\" (UID: \"db161f59-c4de-4fd3-bc2b-0a6262dffd38\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-f11801-74b6fc964f-mf2fc" Nov 03 14:08:20.427197 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:20.427162 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db161f59-c4de-4fd3-bc2b-0a6262dffd38-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-f11801-74b6fc964f-mf2fc\" (UID: \"db161f59-c4de-4fd3-bc2b-0a6262dffd38\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-f11801-74b6fc964f-mf2fc" Nov 03 14:08:20.427396 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:20.427285 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rd77\" (UniqueName: \"kubernetes.io/projected/db161f59-c4de-4fd3-bc2b-0a6262dffd38-kube-api-access-6rd77\") pod \"payload-processing-e2e-extmod-a-f11801-74b6fc964f-mf2fc\" (UID: \"db161f59-c4de-4fd3-bc2b-0a6262dffd38\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-f11801-74b6fc964f-mf2fc" Nov 03 14:08:20.427923 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:20.427902 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db161f59-c4de-4fd3-bc2b-0a6262dffd38-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-f11801-74b6fc964f-mf2fc\" (UID: \"db161f59-c4de-4fd3-bc2b-0a6262dffd38\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-f11801-74b6fc964f-mf2fc" Nov 03 14:08:20.439126 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:20.439099 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp"] Nov 03 14:08:20.445370 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:08:20.445339 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod215d3ba5_9175_474d_9ad2_ff3cdb5ed411.slice/crio-3a693f6a44a742dc1fcc7f6c0b1629196bfab1872285e1a66e9dbf8e431431f8 WatchSource:0}: Error finding container 3a693f6a44a742dc1fcc7f6c0b1629196bfab1872285e1a66e9dbf8e431431f8: Status 404 returned error can't find the container with id 3a693f6a44a742dc1fcc7f6c0b1629196bfab1872285e1a66e9dbf8e431431f8 Nov 03 14:08:20.448010 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:20.447979 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:08:20.448098 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:20.448045 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:08:20.448098 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:20.448074 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:08:20.483202 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:20.483181 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rd77\" (UniqueName: \"kubernetes.io/projected/db161f59-c4de-4fd3-bc2b-0a6262dffd38-kube-api-access-6rd77\") pod \"payload-processing-e2e-extmod-a-f11801-74b6fc964f-mf2fc\" (UID: \"db161f59-c4de-4fd3-bc2b-0a6262dffd38\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-f11801-74b6fc964f-mf2fc" Nov 03 14:08:20.555412 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:20.555380 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-f11801-74b6fc964f-mf2fc" Nov 03 14:08:20.938250 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:20.938215 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-f11801-74b6fc964f-mf2fc"] Nov 03 14:08:20.946057 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:08:20.944954 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb161f59_c4de_4fd3_bc2b_0a6262dffd38.slice/crio-666195638c5cc3b0458930c19ab10575a5cddc27bf267f654fb5a814a88ebdfb WatchSource:0}: Error finding container 666195638c5cc3b0458930c19ab10575a5cddc27bf267f654fb5a814a88ebdfb: Status 404 returned error can't find the container with id 666195638c5cc3b0458930c19ab10575a5cddc27bf267f654fb5a814a88ebdfb Nov 03 14:08:21.246134 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:21.246101 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-f11801-74b6fc964f-mf2fc" event={"ID":"db161f59-c4de-4fd3-bc2b-0a6262dffd38","Type":"ContainerStarted","Data":"4ff73ec498bb3dba8446eac227a440aae9168f3fa8a10c5dffddc0d04aa9337c"} Nov 03 14:08:21.246345 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:21.246158 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-f11801-74b6fc964f-mf2fc" event={"ID":"db161f59-c4de-4fd3-bc2b-0a6262dffd38","Type":"ContainerStarted","Data":"666195638c5cc3b0458930c19ab10575a5cddc27bf267f654fb5a814a88ebdfb"} Nov 03 14:08:21.246345 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:21.246310 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-f11801-74b6fc964f-mf2fc" Nov 03 14:08:21.248052 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:21.248015 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp" event={"ID":"215d3ba5-9175-474d-9ad2-ff3cdb5ed411","Type":"ContainerStarted","Data":"1e0370196a1f27194e0d2dc6054f4e76bab990dbd4583ff9b2dc285c79ef4abe"} Nov 03 14:08:21.248188 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:21.248070 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp" event={"ID":"215d3ba5-9175-474d-9ad2-ff3cdb5ed411","Type":"ContainerStarted","Data":"3a693f6a44a742dc1fcc7f6c0b1629196bfab1872285e1a66e9dbf8e431431f8"} Nov 03 14:08:21.361494 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:21.359454 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-f11801-74b6fc964f-mf2fc" podStartSLOduration=1.359437897 podStartE2EDuration="1.359437897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:08:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:08:21.357694052 +0000 UTC m=+3783.368382313" watchObservedRunningTime="2026-11-03 14:08:21.359437897 +0000 UTC m=+3783.370126141" Nov 03 14:08:21.470983 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:21.470912 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp" podStartSLOduration=2.47089173 podStartE2EDuration="2.47089173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:08:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:08:21.468954614 +0000 UTC m=+3783.479642857" watchObservedRunningTime="2026-11-03 14:08:21.47089173 +0000 UTC m=+3783.481579974" Nov 03 14:08:22.002849 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:22.002810 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp" Nov 03 14:08:22.004363 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:22.004341 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.73:15021/healthz/ready\": dial tcp 10.134.1.73:15021: connect: connection refused" start-of-body= Nov 03 14:08:22.004448 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:22.004390 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp" podUID="215d3ba5-9175-474d-9ad2-ff3cdb5ed411" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.73:15021/healthz/ready\": dial tcp 10.134.1.73:15021: connect: connection refused" Nov 03 14:08:23.002364 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:23.002332 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.73:15021/healthz/ready\": dial tcp 10.134.1.73:15021: connect: connection refused" start-of-body= Nov 03 14:08:23.002589 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:23.002396 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp" podUID="215d3ba5-9175-474d-9ad2-ff3cdb5ed411" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.73:15021/healthz/ready\": dial tcp 10.134.1.73:15021: connect: connection refused" Nov 03 14:08:23.492672 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:23.492638 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:08:24.002896 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:24.002858 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.73:15021/healthz/ready\": dial tcp 10.134.1.73:15021: connect: connection refused" start-of-body= Nov 03 14:08:24.003116 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:24.002919 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp" podUID="215d3ba5-9175-474d-9ad2-ff3cdb5ed411" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.73:15021/healthz/ready\": dial tcp 10.134.1.73:15021: connect: connection refused" Nov 03 14:08:25.003247 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:25.003211 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.73:15021/healthz/ready\": dial tcp 10.134.1.73:15021: connect: connection refused" start-of-body= Nov 03 14:08:25.003706 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:25.003263 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp" podUID="215d3ba5-9175-474d-9ad2-ff3cdb5ed411" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.73:15021/healthz/ready\": dial tcp 10.134.1.73:15021: connect: connection refused" Nov 03 14:08:26.003201 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:26.003166 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.73:15021/healthz/ready\": dial tcp 10.134.1.73:15021: connect: connection refused" start-of-body= Nov 03 14:08:26.003389 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:26.003218 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp" podUID="215d3ba5-9175-474d-9ad2-ff3cdb5ed411" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.73:15021/healthz/ready\": dial tcp 10.134.1.73:15021: connect: connection refused" Nov 03 14:08:27.002971 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:27.002927 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.73:15021/healthz/ready\": dial tcp 10.134.1.73:15021: connect: connection refused" start-of-body= Nov 03 14:08:27.003207 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:27.002983 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp" podUID="215d3ba5-9175-474d-9ad2-ff3cdb5ed411" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.73:15021/healthz/ready\": dial tcp 10.134.1.73:15021: connect: connection refused" Nov 03 14:08:28.002624 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:28.002587 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.73:15021/healthz/ready\": dial tcp 10.134.1.73:15021: connect: connection refused" start-of-body= Nov 03 14:08:28.003154 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:28.002637 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp" podUID="215d3ba5-9175-474d-9ad2-ff3cdb5ed411" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.73:15021/healthz/ready\": dial tcp 10.134.1.73:15021: connect: connection refused" Nov 03 14:08:29.308611 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:29.308531 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp" Nov 03 14:08:29.309042 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:29.308896 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp" Nov 03 14:08:29.309705 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:29.309683 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp" Nov 03 14:08:29.919362 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:29.919325 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" podUID="d09478af-d618-425d-a4e6-e0889e0f416d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:08:32.276773 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:32.276741 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-f11801-74b6fc964f-mf2fc" Nov 03 14:08:37.235128 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:08:37.235089 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:08:39.919640 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:39.919599 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" podUID="d09478af-d618-425d-a4e6-e0889e0f416d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:08:45.895849 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:08:45.895818 5979 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-e86f9ab11580: secret "agc-cred-e86f9ab11580" not found Nov 03 14:08:45.895849 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:08:45.895842 5979 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-84dcd755bf-c9rnp: secret "agc-cred-e86f9ab11580" not found Nov 03 14:08:45.896316 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:08:45.895901 5979 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/797bcf35-fa9e-47b8-887c-bb8942af277b-provider-credentials podName:797bcf35-fa9e-47b8-887c-bb8942af277b nodeName:}" failed. No retries permitted until 2026-11-03 14:09:49.895885948 +0000 UTC m=+3871.906574169 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/797bcf35-fa9e-47b8-887c-bb8942af277b-provider-credentials") pod "payload-processing-external-model-84dcd755bf-c9rnp" (UID: "797bcf35-fa9e-47b8-887c-bb8942af277b") : secret "agc-cred-e86f9ab11580" not found Nov 03 14:08:49.919817 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:49.919772 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" podUID="d09478af-d618-425d-a4e6-e0889e0f416d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:08:59.920076 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:08:59.920038 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" podUID="d09478af-d618-425d-a4e6-e0889e0f416d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:09:00.046656 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:00.046619 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c94937d-qcd7r"] Nov 03 14:09:00.093323 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:00.093281 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c94937d-qcd7r"] Nov 03 14:09:00.109084 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:00.109042 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" podUID="d09478af-d618-425d-a4e6-e0889e0f416d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:09:00.117676 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:00.117643 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2327b6e9-03af-454b-889d-a8c565de71f4" path="/var/lib/kubelet/pods/2327b6e9-03af-454b-889d-a8c565de71f4/volumes" Nov 03 14:09:05.044777 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:05.044738 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p"] Nov 03 14:09:05.050108 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:05.050082 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p" Nov 03 14:09:05.069791 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:05.069761 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6974e682-76a0-438c-b303-9e498d41f0a0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-ngr9p\" (UID: \"6974e682-76a0-438c-b303-9e498d41f0a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p" Nov 03 14:09:05.069934 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:05.069824 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6974e682-76a0-438c-b303-9e498d41f0a0-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-ngr9p\" (UID: \"6974e682-76a0-438c-b303-9e498d41f0a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p" Nov 03 14:09:05.069934 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:05.069928 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6974e682-76a0-438c-b303-9e498d41f0a0-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-ngr9p\" (UID: \"6974e682-76a0-438c-b303-9e498d41f0a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p" Nov 03 14:09:05.070016 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:05.069965 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6974e682-76a0-438c-b303-9e498d41f0a0-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-ngr9p\" (UID: \"6974e682-76a0-438c-b303-9e498d41f0a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p" Nov 03 14:09:05.070053 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:05.070012 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6974e682-76a0-438c-b303-9e498d41f0a0-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-ngr9p\" (UID: \"6974e682-76a0-438c-b303-9e498d41f0a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p" Nov 03 14:09:05.070053 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:05.070041 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6974e682-76a0-438c-b303-9e498d41f0a0-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-ngr9p\" (UID: \"6974e682-76a0-438c-b303-9e498d41f0a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p" Nov 03 14:09:05.070122 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:05.070079 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24t9h\" (UniqueName: \"kubernetes.io/projected/6974e682-76a0-438c-b303-9e498d41f0a0-kube-api-access-24t9h\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-ngr9p\" (UID: \"6974e682-76a0-438c-b303-9e498d41f0a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p" Nov 03 14:09:05.070229 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:05.070216 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6974e682-76a0-438c-b303-9e498d41f0a0-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-ngr9p\" (UID: \"6974e682-76a0-438c-b303-9e498d41f0a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p" Nov 03 14:09:05.070268 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:05.070240 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6974e682-76a0-438c-b303-9e498d41f0a0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-ngr9p\" (UID: \"6974e682-76a0-438c-b303-9e498d41f0a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p" Nov 03 14:09:05.172124 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:05.172081 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6974e682-76a0-438c-b303-9e498d41f0a0-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-ngr9p\" (UID: \"6974e682-76a0-438c-b303-9e498d41f0a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p" Nov 03 14:09:05.172367 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:05.172151 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6974e682-76a0-438c-b303-9e498d41f0a0-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-ngr9p\" (UID: \"6974e682-76a0-438c-b303-9e498d41f0a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p" Nov 03 14:09:05.172367 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:05.172192 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6974e682-76a0-438c-b303-9e498d41f0a0-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-ngr9p\" (UID: \"6974e682-76a0-438c-b303-9e498d41f0a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p" Nov 03 14:09:05.172367 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:05.172211 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6974e682-76a0-438c-b303-9e498d41f0a0-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-ngr9p\" (UID: \"6974e682-76a0-438c-b303-9e498d41f0a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p" Nov 03 14:09:05.172367 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:05.172243 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6974e682-76a0-438c-b303-9e498d41f0a0-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-ngr9p\" (UID: \"6974e682-76a0-438c-b303-9e498d41f0a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p" Nov 03 14:09:05.172694 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:05.172352 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24t9h\" (UniqueName: \"kubernetes.io/projected/6974e682-76a0-438c-b303-9e498d41f0a0-kube-api-access-24t9h\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-ngr9p\" (UID: \"6974e682-76a0-438c-b303-9e498d41f0a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p" Nov 03 14:09:05.172694 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:05.172569 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6974e682-76a0-438c-b303-9e498d41f0a0-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-ngr9p\" (UID: \"6974e682-76a0-438c-b303-9e498d41f0a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p" Nov 03 14:09:05.172694 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:05.172609 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6974e682-76a0-438c-b303-9e498d41f0a0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-ngr9p\" (UID: \"6974e682-76a0-438c-b303-9e498d41f0a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p" Nov 03 14:09:05.172878 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:05.172701 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6974e682-76a0-438c-b303-9e498d41f0a0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-ngr9p\" (UID: \"6974e682-76a0-438c-b303-9e498d41f0a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p" Nov 03 14:09:05.172878 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:05.172705 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6974e682-76a0-438c-b303-9e498d41f0a0-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-ngr9p\" (UID: \"6974e682-76a0-438c-b303-9e498d41f0a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p" Nov 03 14:09:05.172984 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:05.172894 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6974e682-76a0-438c-b303-9e498d41f0a0-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-ngr9p\" (UID: \"6974e682-76a0-438c-b303-9e498d41f0a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p" Nov 03 14:09:05.173178 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:05.173156 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6974e682-76a0-438c-b303-9e498d41f0a0-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-ngr9p\" (UID: \"6974e682-76a0-438c-b303-9e498d41f0a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p" Nov 03 14:09:05.173294 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:05.173202 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6974e682-76a0-438c-b303-9e498d41f0a0-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-ngr9p\" (UID: \"6974e682-76a0-438c-b303-9e498d41f0a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p" Nov 03 14:09:05.173586 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:05.173539 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6974e682-76a0-438c-b303-9e498d41f0a0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-ngr9p\" (UID: \"6974e682-76a0-438c-b303-9e498d41f0a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p" Nov 03 14:09:05.175963 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:05.175931 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6974e682-76a0-438c-b303-9e498d41f0a0-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-ngr9p\" (UID: \"6974e682-76a0-438c-b303-9e498d41f0a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p" Nov 03 14:09:05.176154 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:05.175984 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6974e682-76a0-438c-b303-9e498d41f0a0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-ngr9p\" (UID: \"6974e682-76a0-438c-b303-9e498d41f0a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p" Nov 03 14:09:05.226124 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:05.226086 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24t9h\" (UniqueName: \"kubernetes.io/projected/6974e682-76a0-438c-b303-9e498d41f0a0-kube-api-access-24t9h\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-ngr9p\" (UID: \"6974e682-76a0-438c-b303-9e498d41f0a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p" Nov 03 14:09:05.226288 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:05.226130 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6974e682-76a0-438c-b303-9e498d41f0a0-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-ngr9p\" (UID: \"6974e682-76a0-438c-b303-9e498d41f0a0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p" Nov 03 14:09:05.251575 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:05.251539 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p"] Nov 03 14:09:05.364430 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:05.364336 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p" Nov 03 14:09:05.793671 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:05.793637 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p"] Nov 03 14:09:05.800079 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:09:05.800050 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6974e682_76a0_438c_b303_9e498d41f0a0.slice/crio-09fc2de535a8063d3b269537c2dcca704a3d5c5d8cda9478ea15a3b80fa4d602 WatchSource:0}: Error finding container 09fc2de535a8063d3b269537c2dcca704a3d5c5d8cda9478ea15a3b80fa4d602: Status 404 returned error can't find the container with id 09fc2de535a8063d3b269537c2dcca704a3d5c5d8cda9478ea15a3b80fa4d602 Nov 03 14:09:05.802876 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:05.802843 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:09:05.802963 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:05.802911 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:09:05.802963 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:05.802939 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:09:06.587823 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:06.587791 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p" event={"ID":"6974e682-76a0-438c-b303-9e498d41f0a0","Type":"ContainerStarted","Data":"781dcf3d32951ee0317b334db7a5d05730cff3d7310b9f4fd31ed29e67cc9f21"} Nov 03 14:09:06.588282 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:06.587840 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p" event={"ID":"6974e682-76a0-438c-b303-9e498d41f0a0","Type":"ContainerStarted","Data":"09fc2de535a8063d3b269537c2dcca704a3d5c5d8cda9478ea15a3b80fa4d602"} Nov 03 14:09:06.701729 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:06.701673 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p" podStartSLOduration=2.701658462 podStartE2EDuration="2.701658462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:09:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:09:06.700057003 +0000 UTC m=+3828.710745244" watchObservedRunningTime="2026-11-03 14:09:06.701658462 +0000 UTC m=+3828.712346704" Nov 03 14:09:07.315325 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:09:07.315278 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:09:07.368494 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:07.367626 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p" Nov 03 14:09:07.368494 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:07.367761 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" start-of-body= Nov 03 14:09:07.368494 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:07.367804 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p" podUID="6974e682-76a0-438c-b303-9e498d41f0a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" Nov 03 14:09:08.368977 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:08.368931 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" start-of-body= Nov 03 14:09:08.369419 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:08.369000 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p" podUID="6974e682-76a0-438c-b303-9e498d41f0a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" Nov 03 14:09:09.368893 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:09.368851 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" start-of-body= Nov 03 14:09:09.369115 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:09.368943 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p" podUID="6974e682-76a0-438c-b303-9e498d41f0a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" Nov 03 14:09:10.109550 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:10.109509 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" podUID="d09478af-d618-425d-a4e6-e0889e0f416d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:09:10.122723 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:10.122681 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9347bb13-zktbd"] Nov 03 14:09:10.122896 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:10.122734 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9347bb13-zktbd"] Nov 03 14:09:10.368553 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:10.365918 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" start-of-body= Nov 03 14:09:10.368553 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:10.365984 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p" podUID="6974e682-76a0-438c-b303-9e498d41f0a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" Nov 03 14:09:11.365507 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:11.364919 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" start-of-body= Nov 03 14:09:11.365507 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:11.364990 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p" podUID="6974e682-76a0-438c-b303-9e498d41f0a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" Nov 03 14:09:12.117233 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:12.117198 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db0469df-0987-4c6a-bf09-3f66ca1c70c9" path="/var/lib/kubelet/pods/db0469df-0987-4c6a-bf09-3f66ca1c70c9/volumes" Nov 03 14:09:12.373172 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:12.373080 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" start-of-body= Nov 03 14:09:12.373801 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:12.373760 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p" podUID="6974e682-76a0-438c-b303-9e498d41f0a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" Nov 03 14:09:13.365687 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:13.365645 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" start-of-body= Nov 03 14:09:13.365876 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:13.365716 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p" podUID="6974e682-76a0-438c-b303-9e498d41f0a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.75:15021/healthz/ready\": dial tcp 10.134.1.75:15021: connect: connection refused" Nov 03 14:09:14.433735 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:14.433699 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p" Nov 03 14:09:14.434826 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:14.434799 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p" Nov 03 14:09:14.435877 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:14.435854 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p" Nov 03 14:09:16.495131 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:16.495097 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:09:20.109318 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:20.109275 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" podUID="d09478af-d618-425d-a4e6-e0889e0f416d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:09:21.857831 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:21.857795 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-966a03-7cfcd9f66c-7dk75"] Nov 03 14:09:21.858244 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:21.858013 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-966a03-7cfcd9f66c-7dk75" podUID="c8f215c3-1d1d-4b0c-8868-609bb451e8ce" containerName="maas-api" containerID="cri-o://7f79a28182b5964b901b9d452212cf8af1f664fc3e19e95c73e531d1739a5c77" gracePeriod=30 Nov 03 14:09:22.167803 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:22.167775 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-966a03-7cfcd9f66c-7dk75" Nov 03 14:09:22.241620 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:22.241583 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8f215c3-1d1d-4b0c-8868-609bb451e8ce-kube-api-access-x89h5\" (UniqueName: \"kubernetes.io/projected/c8f215c3-1d1d-4b0c-8868-609bb451e8ce-kube-api-access-x89h5\") pod \"c8f215c3-1d1d-4b0c-8868-609bb451e8ce\" (UID: \"c8f215c3-1d1d-4b0c-8868-609bb451e8ce\") " Nov 03 14:09:22.241845 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:22.241718 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8f215c3-1d1d-4b0c-8868-609bb451e8ce-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8f215c3-1d1d-4b0c-8868-609bb451e8ce-metrics-certs\") pod \"c8f215c3-1d1d-4b0c-8868-609bb451e8ce\" (UID: \"c8f215c3-1d1d-4b0c-8868-609bb451e8ce\") " Nov 03 14:09:22.241845 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:22.241777 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8f215c3-1d1d-4b0c-8868-609bb451e8ce-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8f215c3-1d1d-4b0c-8868-609bb451e8ce-openshift-service-ca\") pod \"c8f215c3-1d1d-4b0c-8868-609bb451e8ce\" (UID: \"c8f215c3-1d1d-4b0c-8868-609bb451e8ce\") " Nov 03 14:09:22.241845 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:22.241820 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8f215c3-1d1d-4b0c-8868-609bb451e8ce-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8f215c3-1d1d-4b0c-8868-609bb451e8ce-maas-api-tls\") pod \"c8f215c3-1d1d-4b0c-8868-609bb451e8ce\" (UID: \"c8f215c3-1d1d-4b0c-8868-609bb451e8ce\") " Nov 03 14:09:22.242240 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:22.242204 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8f215c3-1d1d-4b0c-8868-609bb451e8ce-openshift-service-ca" pod "c8f215c3-1d1d-4b0c-8868-609bb451e8ce" (UID: "c8f215c3-1d1d-4b0c-8868-609bb451e8ce"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:09:22.242594 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:22.242573 5979 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8f215c3-1d1d-4b0c-8868-609bb451e8ce-openshift-service-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:09:22.264762 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:22.255279 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8f215c3-1d1d-4b0c-8868-609bb451e8ce-maas-api-tls" pod "c8f215c3-1d1d-4b0c-8868-609bb451e8ce" (UID: "c8f215c3-1d1d-4b0c-8868-609bb451e8ce"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:09:22.264762 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:22.259680 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8f215c3-1d1d-4b0c-8868-609bb451e8ce-kube-api-access-x89h5" pod "c8f215c3-1d1d-4b0c-8868-609bb451e8ce" (UID: "c8f215c3-1d1d-4b0c-8868-609bb451e8ce"). InnerVolumeSpecName "kube-api-access-x89h5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:09:22.271536 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:22.265773 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8f215c3-1d1d-4b0c-8868-609bb451e8ce-metrics-certs" pod "c8f215c3-1d1d-4b0c-8868-609bb451e8ce" (UID: "c8f215c3-1d1d-4b0c-8868-609bb451e8ce"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:09:22.354167 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:22.351766 5979 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8f215c3-1d1d-4b0c-8868-609bb451e8ce-metrics-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:09:22.354167 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:22.351806 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8f215c3-1d1d-4b0c-8868-609bb451e8ce-maas-api-tls\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:09:22.354167 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:22.351822 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x89h5\" (UniqueName: \"kubernetes.io/projected/c8f215c3-1d1d-4b0c-8868-609bb451e8ce-kube-api-access-x89h5\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:09:22.714773 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:22.714715 5979 generic.go:356] "Generic (PLEG): container finished" podID="c8f215c3-1d1d-4b0c-8868-609bb451e8ce" containerID="7f79a28182b5964b901b9d452212cf8af1f664fc3e19e95c73e531d1739a5c77" exitCode=0 Nov 03 14:09:22.714984 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:22.714801 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-966a03-7cfcd9f66c-7dk75" Nov 03 14:09:22.714984 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:22.714798 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-966a03-7cfcd9f66c-7dk75" event={"ID":"c8f215c3-1d1d-4b0c-8868-609bb451e8ce","Type":"ContainerDied","Data":"7f79a28182b5964b901b9d452212cf8af1f664fc3e19e95c73e531d1739a5c77"} Nov 03 14:09:22.714984 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:22.714970 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-966a03-7cfcd9f66c-7dk75" event={"ID":"c8f215c3-1d1d-4b0c-8868-609bb451e8ce","Type":"ContainerDied","Data":"796a9bbed3bbb2a067fc6c0c00b6bc0f5f5da600a5b61021fb538bde62a8c30b"} Nov 03 14:09:22.715185 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:22.715003 5979 scope.go:122] "RemoveContainer" containerID="7f79a28182b5964b901b9d452212cf8af1f664fc3e19e95c73e531d1739a5c77" Nov 03 14:09:22.725621 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:22.725602 5979 scope.go:122] "RemoveContainer" containerID="7f79a28182b5964b901b9d452212cf8af1f664fc3e19e95c73e531d1739a5c77" Nov 03 14:09:22.725895 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:09:22.725874 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f79a28182b5964b901b9d452212cf8af1f664fc3e19e95c73e531d1739a5c77\": container with ID starting with 7f79a28182b5964b901b9d452212cf8af1f664fc3e19e95c73e531d1739a5c77 not found: ID does not exist" containerID="7f79a28182b5964b901b9d452212cf8af1f664fc3e19e95c73e531d1739a5c77" Nov 03 14:09:22.725941 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:22.725903 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f79a28182b5964b901b9d452212cf8af1f664fc3e19e95c73e531d1739a5c77"} err="failed to get container status \"7f79a28182b5964b901b9d452212cf8af1f664fc3e19e95c73e531d1739a5c77\": rpc error: code = NotFound desc = could not find container \"7f79a28182b5964b901b9d452212cf8af1f664fc3e19e95c73e531d1739a5c77\": container with ID starting with 7f79a28182b5964b901b9d452212cf8af1f664fc3e19e95c73e531d1739a5c77 not found: ID does not exist" Nov 03 14:09:22.928535 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:22.918304 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-966a03-7cfcd9f66c-7dk75"] Nov 03 14:09:22.928535 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:22.925364 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-966a03-7cfcd9f66c-7dk75"] Nov 03 14:09:24.116658 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:24.116627 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8f215c3-1d1d-4b0c-8868-609bb451e8ce" path="/var/lib/kubelet/pods/c8f215c3-1d1d-4b0c-8868-609bb451e8ce/volumes" Nov 03 14:09:27.289012 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:27.288970 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2"] Nov 03 14:09:27.309771 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:27.309740 5979 cpu_manager.go:414] "containerMap: removing container" podUID="c8f215c3-1d1d-4b0c-8868-609bb451e8ce" containerName="maas-api" Nov 03 14:09:27.310253 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:27.309991 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8f215c3-1d1d-4b0c-8868-609bb451e8ce" containerName="maas-api" Nov 03 14:09:27.310717 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:27.310701 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8f215c3-1d1d-4b0c-8868-609bb451e8ce" containerName="maas-api" Nov 03 14:09:27.331539 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:27.315590 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2" Nov 03 14:09:27.342436 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:27.341882 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2"] Nov 03 14:09:27.389879 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:27.389800 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-6375e7-openshift-default-dockercfg-9q9jq\"" Nov 03 14:09:27.442019 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:27.436080 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgg9r\" (UniqueName: \"kubernetes.io/projected/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-kube-api-access-jgg9r\") pod \"e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2\" (UID: \"7bb63e83-ca20-4eb2-be94-a2e7fcca2ace\") " pod="openshift-ingress/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2" Nov 03 14:09:27.447617 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:27.442353 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-workload-socket\") pod \"e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2\" (UID: \"7bb63e83-ca20-4eb2-be94-a2e7fcca2ace\") " pod="openshift-ingress/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2" Nov 03 14:09:27.448273 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:27.448227 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-workload-certs\") pod \"e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2\" (UID: \"7bb63e83-ca20-4eb2-be94-a2e7fcca2ace\") " pod="openshift-ingress/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2" Nov 03 14:09:27.448598 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:27.448534 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-credential-socket\") pod \"e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2\" (UID: \"7bb63e83-ca20-4eb2-be94-a2e7fcca2ace\") " pod="openshift-ingress/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2" Nov 03 14:09:27.448824 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:27.448806 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-istiod-ca-cert\") pod \"e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2\" (UID: \"7bb63e83-ca20-4eb2-be94-a2e7fcca2ace\") " pod="openshift-ingress/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2" Nov 03 14:09:27.449012 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:27.448978 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-istio-token\") pod \"e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2\" (UID: \"7bb63e83-ca20-4eb2-be94-a2e7fcca2ace\") " pod="openshift-ingress/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2" Nov 03 14:09:27.449265 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:27.449157 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-istio-data\") pod \"e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2\" (UID: \"7bb63e83-ca20-4eb2-be94-a2e7fcca2ace\") " pod="openshift-ingress/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2" Nov 03 14:09:27.449697 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:27.449678 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-istio-podinfo\") pod \"e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2\" (UID: \"7bb63e83-ca20-4eb2-be94-a2e7fcca2ace\") " pod="openshift-ingress/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2" Nov 03 14:09:27.450072 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:27.450055 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-istio-envoy\") pod \"e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2\" (UID: \"7bb63e83-ca20-4eb2-be94-a2e7fcca2ace\") " pod="openshift-ingress/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2" Nov 03 14:09:27.552712 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:27.552626 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-istio-envoy\") pod \"e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2\" (UID: \"7bb63e83-ca20-4eb2-be94-a2e7fcca2ace\") " pod="openshift-ingress/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2" Nov 03 14:09:27.552888 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:27.552732 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgg9r\" (UniqueName: \"kubernetes.io/projected/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-kube-api-access-jgg9r\") pod \"e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2\" (UID: \"7bb63e83-ca20-4eb2-be94-a2e7fcca2ace\") " pod="openshift-ingress/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2" Nov 03 14:09:27.552888 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:27.552765 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-workload-socket\") pod \"e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2\" (UID: \"7bb63e83-ca20-4eb2-be94-a2e7fcca2ace\") " pod="openshift-ingress/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2" Nov 03 14:09:27.552888 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:27.552854 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-workload-certs\") pod \"e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2\" (UID: \"7bb63e83-ca20-4eb2-be94-a2e7fcca2ace\") " pod="openshift-ingress/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2" Nov 03 14:09:27.553078 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:27.552895 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-credential-socket\") pod \"e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2\" (UID: \"7bb63e83-ca20-4eb2-be94-a2e7fcca2ace\") " pod="openshift-ingress/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2" Nov 03 14:09:27.553078 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:27.552945 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-istiod-ca-cert\") pod \"e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2\" (UID: \"7bb63e83-ca20-4eb2-be94-a2e7fcca2ace\") " pod="openshift-ingress/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2" Nov 03 14:09:27.553078 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:27.552972 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-istio-token\") pod \"e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2\" (UID: \"7bb63e83-ca20-4eb2-be94-a2e7fcca2ace\") " pod="openshift-ingress/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2" Nov 03 14:09:27.553078 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:27.553002 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-istio-data\") pod \"e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2\" (UID: \"7bb63e83-ca20-4eb2-be94-a2e7fcca2ace\") " pod="openshift-ingress/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2" Nov 03 14:09:27.553305 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:27.553123 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-istio-podinfo\") pod \"e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2\" (UID: \"7bb63e83-ca20-4eb2-be94-a2e7fcca2ace\") " pod="openshift-ingress/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2" Nov 03 14:09:27.553305 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:27.553186 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-workload-socket\") pod \"e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2\" (UID: \"7bb63e83-ca20-4eb2-be94-a2e7fcca2ace\") " pod="openshift-ingress/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2" Nov 03 14:09:27.553613 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:27.553589 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-credential-socket\") pod \"e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2\" (UID: \"7bb63e83-ca20-4eb2-be94-a2e7fcca2ace\") " pod="openshift-ingress/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2" Nov 03 14:09:27.553861 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:27.553838 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-istiod-ca-cert\") pod \"e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2\" (UID: \"7bb63e83-ca20-4eb2-be94-a2e7fcca2ace\") " pod="openshift-ingress/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2" Nov 03 14:09:27.553949 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:27.553845 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-workload-certs\") pod \"e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2\" (UID: \"7bb63e83-ca20-4eb2-be94-a2e7fcca2ace\") " pod="openshift-ingress/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2" Nov 03 14:09:27.553949 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:27.553880 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-istio-data\") pod \"e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2\" (UID: \"7bb63e83-ca20-4eb2-be94-a2e7fcca2ace\") " pod="openshift-ingress/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2" Nov 03 14:09:27.555180 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:27.555158 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-istio-envoy\") pod \"e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2\" (UID: \"7bb63e83-ca20-4eb2-be94-a2e7fcca2ace\") " pod="openshift-ingress/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2" Nov 03 14:09:27.556178 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:27.556161 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-istio-podinfo\") pod \"e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2\" (UID: \"7bb63e83-ca20-4eb2-be94-a2e7fcca2ace\") " pod="openshift-ingress/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2" Nov 03 14:09:27.607527 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:27.607494 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgg9r\" (UniqueName: \"kubernetes.io/projected/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-kube-api-access-jgg9r\") pod \"e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2\" (UID: \"7bb63e83-ca20-4eb2-be94-a2e7fcca2ace\") " pod="openshift-ingress/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2" Nov 03 14:09:27.610206 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:27.610186 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-istio-token\") pod \"e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2\" (UID: \"7bb63e83-ca20-4eb2-be94-a2e7fcca2ace\") " pod="openshift-ingress/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2" Nov 03 14:09:27.641877 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:27.641855 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2" Nov 03 14:09:28.102512 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:28.102462 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2"] Nov 03 14:09:28.110650 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:09:28.110618 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bb63e83_ca20_4eb2_be94_a2e7fcca2ace.slice/crio-fc889da35abf1213c1808d5b9ec941aab86059f9f2f42929416e259dfaab11b8 WatchSource:0}: Error finding container fc889da35abf1213c1808d5b9ec941aab86059f9f2f42929416e259dfaab11b8: Status 404 returned error can't find the container with id fc889da35abf1213c1808d5b9ec941aab86059f9f2f42929416e259dfaab11b8 Nov 03 14:09:28.113402 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:28.113364 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:09:28.113549 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:28.113443 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:09:28.113549 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:28.113511 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:09:28.783695 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:28.783660 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2" event={"ID":"7bb63e83-ca20-4eb2-be94-a2e7fcca2ace","Type":"ContainerStarted","Data":"623d2e18efbf02c98355d4aab7c1c1433c17f3c7e260f2cb555ad0cf628211cc"} Nov 03 14:09:28.784112 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:28.783714 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2" event={"ID":"7bb63e83-ca20-4eb2-be94-a2e7fcca2ace","Type":"ContainerStarted","Data":"fc889da35abf1213c1808d5b9ec941aab86059f9f2f42929416e259dfaab11b8"} Nov 03 14:09:28.894986 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:28.894935 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2" podStartSLOduration=1.894920846 podStartE2EDuration="1.894920846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:09:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:09:28.893420202 +0000 UTC m=+3850.904108435" watchObservedRunningTime="2026-11-03 14:09:28.894920846 +0000 UTC m=+3850.905609151" Nov 03 14:09:29.642819 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:29.642788 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2" Nov 03 14:09:29.649059 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:29.649033 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2" Nov 03 14:09:29.794365 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:29.794335 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2" Nov 03 14:09:29.795379 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:29.795355 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2" Nov 03 14:09:30.109641 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:30.109545 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" podUID="d09478af-d618-425d-a4e6-e0889e0f416d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:09:37.235690 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:09:37.235644 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:09:38.645741 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:38.645709 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6375e7-7996cbfbbc-xsdbn"] Nov 03 14:09:38.650379 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:38.650348 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6375e7-7996cbfbbc-xsdbn" Nov 03 14:09:38.703452 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:38.703410 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6375e7-7996cbfbbc-xsdbn"] Nov 03 14:09:38.737592 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:38.737554 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a929998e-474d-47af-afb6-8c1e97702287-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6375e7-7996cbfbbc-xsdbn\" (UID: \"a929998e-474d-47af-afb6-8c1e97702287\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6375e7-7996cbfbbc-xsdbn" Nov 03 14:09:38.738120 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:38.738084 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9phbl\" (UniqueName: \"kubernetes.io/projected/a929998e-474d-47af-afb6-8c1e97702287-kube-api-access-9phbl\") pod \"payload-pre-processing-e2e-inf-a-6375e7-7996cbfbbc-xsdbn\" (UID: \"a929998e-474d-47af-afb6-8c1e97702287\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6375e7-7996cbfbbc-xsdbn" Nov 03 14:09:38.766497 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:38.755004 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-6375e7-dockercfg-vt59p\"" Nov 03 14:09:38.766497 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:38.755111 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-6375e7\"" Nov 03 14:09:38.840494 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:38.839767 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a929998e-474d-47af-afb6-8c1e97702287-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6375e7-7996cbfbbc-xsdbn\" (UID: \"a929998e-474d-47af-afb6-8c1e97702287\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6375e7-7996cbfbbc-xsdbn" Nov 03 14:09:38.840494 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:38.839981 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9phbl\" (UniqueName: \"kubernetes.io/projected/a929998e-474d-47af-afb6-8c1e97702287-kube-api-access-9phbl\") pod \"payload-pre-processing-e2e-inf-a-6375e7-7996cbfbbc-xsdbn\" (UID: \"a929998e-474d-47af-afb6-8c1e97702287\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6375e7-7996cbfbbc-xsdbn" Nov 03 14:09:38.843018 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:38.842988 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a929998e-474d-47af-afb6-8c1e97702287-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6375e7-7996cbfbbc-xsdbn\" (UID: \"a929998e-474d-47af-afb6-8c1e97702287\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6375e7-7996cbfbbc-xsdbn" Nov 03 14:09:38.918581 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:38.918456 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9phbl\" (UniqueName: \"kubernetes.io/projected/a929998e-474d-47af-afb6-8c1e97702287-kube-api-access-9phbl\") pod \"payload-pre-processing-e2e-inf-a-6375e7-7996cbfbbc-xsdbn\" (UID: \"a929998e-474d-47af-afb6-8c1e97702287\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6375e7-7996cbfbbc-xsdbn" Nov 03 14:09:38.964631 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:38.964599 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6375e7-7996cbfbbc-xsdbn" Nov 03 14:09:39.339988 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:39.339895 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6375e7-7996cbfbbc-xsdbn"] Nov 03 14:09:39.347228 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:09:39.347202 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda929998e_474d_47af_afb6_8c1e97702287.slice/crio-8c72f839ad4ae34d46a5f3a50c5022972de38e0b432543fdfe8c8dc12967d49c WatchSource:0}: Error finding container 8c72f839ad4ae34d46a5f3a50c5022972de38e0b432543fdfe8c8dc12967d49c: Status 404 returned error can't find the container with id 8c72f839ad4ae34d46a5f3a50c5022972de38e0b432543fdfe8c8dc12967d49c Nov 03 14:09:39.915710 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:39.915674 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6375e7-7996cbfbbc-xsdbn" event={"ID":"a929998e-474d-47af-afb6-8c1e97702287","Type":"ContainerStarted","Data":"4897d89fb1025deea79633693a68b4a36b1a3f0a49ef53b19b68c34f9c0d5ff1"} Nov 03 14:09:39.916129 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:39.915733 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6375e7-7996cbfbbc-xsdbn" event={"ID":"a929998e-474d-47af-afb6-8c1e97702287","Type":"ContainerStarted","Data":"8c72f839ad4ae34d46a5f3a50c5022972de38e0b432543fdfe8c8dc12967d49c"} Nov 03 14:09:39.916129 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:39.915763 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6375e7-7996cbfbbc-xsdbn" Nov 03 14:09:40.030818 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:40.030765 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6375e7-7996cbfbbc-xsdbn" podStartSLOduration=2.03075004 podStartE2EDuration="2.03075004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:09:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:09:40.027926491 +0000 UTC m=+3862.038614733" watchObservedRunningTime="2026-11-03 14:09:40.03075004 +0000 UTC m=+3862.041438281" Nov 03 14:09:40.109447 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:40.109403 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" podUID="d09478af-d618-425d-a4e6-e0889e0f416d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:09:48.988625 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:48.988543 5979 generic.go:356] "Generic (PLEG): container finished" podID="5178f75c-2f52-4a5b-bf00-c315bf7e110b" containerID="2b48c25bf1e790d5210106dc9685a1370fa259f24c4ab7508fb17bf685d3be43" exitCode=0 Nov 03 14:09:48.989037 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:48.988618 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b" event={"ID":"5178f75c-2f52-4a5b-bf00-c315bf7e110b","Type":"ContainerDied","Data":"2b48c25bf1e790d5210106dc9685a1370fa259f24c4ab7508fb17bf685d3be43"} Nov 03 14:09:48.989037 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:48.989028 5979 scope.go:122] "RemoveContainer" containerID="2b48c25bf1e790d5210106dc9685a1370fa259f24c4ab7508fb17bf685d3be43" Nov 03 14:09:48.990679 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:48.990651 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:09:48.990756 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:48.990719 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:09:48.990756 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:48.990750 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:09:49.912084 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:09:49.912045 5979 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-e86f9ab11580: secret "agc-cred-e86f9ab11580" not found Nov 03 14:09:49.912084 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:09:49.912075 5979 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-84dcd755bf-c9rnp: secret "agc-cred-e86f9ab11580" not found Nov 03 14:09:49.912325 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:09:49.912129 5979 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/797bcf35-fa9e-47b8-887c-bb8942af277b-provider-credentials podName:797bcf35-fa9e-47b8-887c-bb8942af277b nodeName:}" failed. No retries permitted until 2026-11-03 14:11:51.912113391 +0000 UTC m=+3993.922801610 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/797bcf35-fa9e-47b8-887c-bb8942af277b-provider-credentials") pod "payload-processing-external-model-84dcd755bf-c9rnp" (UID: "797bcf35-fa9e-47b8-887c-bb8942af277b") : secret "agc-cred-e86f9ab11580" not found Nov 03 14:09:50.002561 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:50.002524 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b" event={"ID":"5178f75c-2f52-4a5b-bf00-c315bf7e110b","Type":"ContainerStarted","Data":"fe00e1d456d026a140289cc8dfc5ba9abe2c6542db869016d1a5a280681db59d"} Nov 03 14:09:50.108956 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:50.108919 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" podUID="d09478af-d618-425d-a4e6-e0889e0f416d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:09:50.296565 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:50.296459 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b" Nov 03 14:09:50.297595 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:50.297566 5979 patch_prober.go:29] interesting pod/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.69:15021/healthz/ready\": dial tcp 10.134.1.69:15021: connect: connection refused" start-of-body= Nov 03 14:09:50.297750 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:50.297637 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b" podUID="5178f75c-2f52-4a5b-bf00-c315bf7e110b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.69:15021/healthz/ready\": dial tcp 10.134.1.69:15021: connect: connection refused" Nov 03 14:09:50.925874 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:50.925838 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6375e7-7996cbfbbc-xsdbn" Nov 03 14:09:51.296804 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:51.296712 5979 patch_prober.go:29] interesting pod/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.69:15021/healthz/ready\": dial tcp 10.134.1.69:15021: connect: connection refused" start-of-body= Nov 03 14:09:51.296804 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:51.296779 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b" podUID="5178f75c-2f52-4a5b-bf00-c315bf7e110b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.69:15021/healthz/ready\": dial tcp 10.134.1.69:15021: connect: connection refused" Nov 03 14:09:52.296264 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:52.296229 5979 patch_prober.go:29] interesting pod/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.69:15021/healthz/ready\": dial tcp 10.134.1.69:15021: connect: connection refused" start-of-body= Nov 03 14:09:52.296440 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:52.296289 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b" podUID="5178f75c-2f52-4a5b-bf00-c315bf7e110b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.69:15021/healthz/ready\": dial tcp 10.134.1.69:15021: connect: connection refused" Nov 03 14:09:52.941143 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:52.941106 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b" Nov 03 14:09:53.296412 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:53.296327 5979 patch_prober.go:29] interesting pod/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.69:15021/healthz/ready\": dial tcp 10.134.1.69:15021: connect: connection refused" start-of-body= Nov 03 14:09:53.296412 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:53.296393 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b" podUID="5178f75c-2f52-4a5b-bf00-c315bf7e110b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.69:15021/healthz/ready\": dial tcp 10.134.1.69:15021: connect: connection refused" Nov 03 14:09:54.295893 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:54.295854 5979 patch_prober.go:29] interesting pod/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.69:15021/healthz/ready\": dial tcp 10.134.1.69:15021: connect: connection refused" start-of-body= Nov 03 14:09:54.296314 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:54.295905 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b" podUID="5178f75c-2f52-4a5b-bf00-c315bf7e110b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.69:15021/healthz/ready\": dial tcp 10.134.1.69:15021: connect: connection refused" Nov 03 14:09:55.300968 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:55.300933 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b" Nov 03 14:09:55.302148 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:09:55.302130 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b" Nov 03 14:10:00.109771 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:10:00.109672 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" podUID="d09478af-d618-425d-a4e6-e0889e0f416d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:10:07.234990 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:10:07.234951 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:10:07.894413 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:10:07.894371 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:10:10.109663 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:10:10.109625 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" podUID="d09478af-d618-425d-a4e6-e0889e0f416d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:10:19.256790 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:10:19.256729 5979 generic.go:356] "Generic (PLEG): container finished" podID="8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130" containerID="8f530b2c5e7d54c1b2dc8044d2f741f34ed5993a3e584960e01e0546d9f28312" exitCode=0 Nov 03 14:10:19.257237 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:10:19.256948 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l" event={"ID":"8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130","Type":"ContainerDied","Data":"8f530b2c5e7d54c1b2dc8044d2f741f34ed5993a3e584960e01e0546d9f28312"} Nov 03 14:10:19.257519 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:10:19.257502 5979 scope.go:122] "RemoveContainer" containerID="8f530b2c5e7d54c1b2dc8044d2f741f34ed5993a3e584960e01e0546d9f28312" Nov 03 14:10:19.259186 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:10:19.259158 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:10:19.259260 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:10:19.259223 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:10:19.259260 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:10:19.259254 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:10:20.109865 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:10:20.109826 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" podUID="d09478af-d618-425d-a4e6-e0889e0f416d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:10:20.266032 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:10:20.265994 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l" event={"ID":"8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130","Type":"ContainerStarted","Data":"f4b54bd0c0ab7ba0133f93d78408d399f42483f9789ebf01d23beaeee9dc1760"} Nov 03 14:10:20.589217 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:10:20.589181 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l" Nov 03 14:10:20.589439 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:10:20.589401 5979 patch_prober.go:29] interesting pod/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.72:15021/healthz/ready\": dial tcp 10.134.1.72:15021: connect: connection refused" start-of-body= Nov 03 14:10:20.589524 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:10:20.589443 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l" podUID="8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.72:15021/healthz/ready\": dial tcp 10.134.1.72:15021: connect: connection refused" Nov 03 14:10:20.912058 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:10:20.912025 5979 scope.go:122] "RemoveContainer" containerID="cb77d801977d6bdd72e65351590b7e64020823ac7807f2826ee25ac65eb1df95" Nov 03 14:10:20.923167 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:10:20.923144 5979 scope.go:122] "RemoveContainer" containerID="6f77aa0d638fd47b33d7505aa060d4782a55a611aaf52c1557dfc58f09045035" Nov 03 14:10:21.589784 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:10:21.589752 5979 patch_prober.go:29] interesting pod/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.72:15021/healthz/ready\": dial tcp 10.134.1.72:15021: connect: connection refused" start-of-body= Nov 03 14:10:21.590309 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:10:21.590285 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l" podUID="8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.72:15021/healthz/ready\": dial tcp 10.134.1.72:15021: connect: connection refused" Nov 03 14:10:22.589315 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:10:22.589284 5979 patch_prober.go:29] interesting pod/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.72:15021/healthz/ready\": dial tcp 10.134.1.72:15021: connect: connection refused" start-of-body= Nov 03 14:10:22.589531 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:10:22.589336 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l" podUID="8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.72:15021/healthz/ready\": dial tcp 10.134.1.72:15021: connect: connection refused" Nov 03 14:10:23.591426 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:10:23.590937 5979 patch_prober.go:29] interesting pod/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.72:15021/healthz/ready\": dial tcp 10.134.1.72:15021: connect: connection refused" start-of-body= Nov 03 14:10:23.591426 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:10:23.590999 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l" podUID="8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.72:15021/healthz/ready\": dial tcp 10.134.1.72:15021: connect: connection refused" Nov 03 14:10:24.589176 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:10:24.589140 5979 patch_prober.go:29] interesting pod/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.72:15021/healthz/ready\": dial tcp 10.134.1.72:15021: connect: connection refused" start-of-body= Nov 03 14:10:24.589363 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:10:24.589195 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l" podUID="8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.72:15021/healthz/ready\": dial tcp 10.134.1.72:15021: connect: connection refused" Nov 03 14:10:25.589281 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:10:25.589246 5979 patch_prober.go:29] interesting pod/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.72:15021/healthz/ready\": dial tcp 10.134.1.72:15021: connect: connection refused" start-of-body= Nov 03 14:10:25.589710 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:10:25.589301 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l" podUID="8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.72:15021/healthz/ready\": dial tcp 10.134.1.72:15021: connect: connection refused" Nov 03 14:10:26.593922 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:10:26.593892 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l" Nov 03 14:10:26.594428 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:10:26.594258 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l" Nov 03 14:10:26.595071 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:10:26.595044 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l" Nov 03 14:10:27.109756 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:10:27.109723 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" podUID="d09478af-d618-425d-a4e6-e0889e0f416d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:10:37.109597 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:10:37.109561 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" podUID="d09478af-d618-425d-a4e6-e0889e0f416d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:10:37.235431 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:10:37.235389 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:10:41.532728 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:10:41.532689 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:10:47.109962 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:10:47.109921 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" podUID="d09478af-d618-425d-a4e6-e0889e0f416d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:10:57.109923 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:10:57.109870 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" podUID="d09478af-d618-425d-a4e6-e0889e0f416d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:11:00.411442 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:00.411406 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d6963e-8679d489c9-7n5xs"] Nov 03 14:11:00.527166 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:00.526897 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d6963e-cc4d8848c-bf2p7"] Nov 03 14:11:05.411750 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:05.411712 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-d6963e-8679d489c9-7n5xs" podUID="f95dd442-db70-44a1-b6ac-533be225068c" containerName="payload-processing" containerID="cri-o://9d641f843c83d02b0d6fdf82c4032024d66ba841f7b9281d576d16a40a4ac7eb" gracePeriod=25 Nov 03 14:11:05.528257 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:05.528221 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d6963e-cc4d8848c-bf2p7" podUID="8c10c218-9344-4504-a060-2d533a772408" containerName="payload-pre-processing" containerID="cri-o://4a5733889df53b6a1dd0a57aa050ce805132fdc2664195b5829345e288663e24" gracePeriod=25 Nov 03 14:11:05.653616 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:05.653588 5979 generic.go:356] "Generic (PLEG): container finished" podID="f95dd442-db70-44a1-b6ac-533be225068c" containerID="9d641f843c83d02b0d6fdf82c4032024d66ba841f7b9281d576d16a40a4ac7eb" exitCode=0 Nov 03 14:11:05.653852 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:05.653828 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d6963e-8679d489c9-7n5xs" event={"ID":"f95dd442-db70-44a1-b6ac-533be225068c","Type":"ContainerDied","Data":"9d641f843c83d02b0d6fdf82c4032024d66ba841f7b9281d576d16a40a4ac7eb"} Nov 03 14:11:05.697129 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:05.697103 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d6963e-8679d489c9-7n5xs" Nov 03 14:11:05.785019 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:05.784981 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f95dd442-db70-44a1-b6ac-533be225068c-kube-api-access-dlbdt\" (UniqueName: \"kubernetes.io/projected/f95dd442-db70-44a1-b6ac-533be225068c-kube-api-access-dlbdt\") pod \"f95dd442-db70-44a1-b6ac-533be225068c\" (UID: \"f95dd442-db70-44a1-b6ac-533be225068c\") " Nov 03 14:11:05.785205 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:05.785119 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f95dd442-db70-44a1-b6ac-533be225068c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f95dd442-db70-44a1-b6ac-533be225068c-plugins-config-volume\") pod \"f95dd442-db70-44a1-b6ac-533be225068c\" (UID: \"f95dd442-db70-44a1-b6ac-533be225068c\") " Nov 03 14:11:05.785571 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:05.785545 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f95dd442-db70-44a1-b6ac-533be225068c-plugins-config-volume" pod "f95dd442-db70-44a1-b6ac-533be225068c" (UID: "f95dd442-db70-44a1-b6ac-533be225068c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:11:05.785711 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:05.785691 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f95dd442-db70-44a1-b6ac-533be225068c-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:11:05.787185 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:05.787157 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f95dd442-db70-44a1-b6ac-533be225068c-kube-api-access-dlbdt" pod "f95dd442-db70-44a1-b6ac-533be225068c" (UID: "f95dd442-db70-44a1-b6ac-533be225068c"). InnerVolumeSpecName "kube-api-access-dlbdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:11:05.808316 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:05.808291 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d6963e-cc4d8848c-bf2p7" Nov 03 14:11:05.886821 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:05.886779 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c10c218-9344-4504-a060-2d533a772408-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c10c218-9344-4504-a060-2d533a772408-plugins-config-volume\") pod \"8c10c218-9344-4504-a060-2d533a772408\" (UID: \"8c10c218-9344-4504-a060-2d533a772408\") " Nov 03 14:11:05.887027 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:05.887006 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c10c218-9344-4504-a060-2d533a772408-kube-api-access-qv4bz\" (UniqueName: \"kubernetes.io/projected/8c10c218-9344-4504-a060-2d533a772408-kube-api-access-qv4bz\") pod \"8c10c218-9344-4504-a060-2d533a772408\" (UID: \"8c10c218-9344-4504-a060-2d533a772408\") " Nov 03 14:11:05.887156 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:05.887132 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c10c218-9344-4504-a060-2d533a772408-plugins-config-volume" pod "8c10c218-9344-4504-a060-2d533a772408" (UID: "8c10c218-9344-4504-a060-2d533a772408"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:11:05.887399 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:05.887384 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c10c218-9344-4504-a060-2d533a772408-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:11:05.887487 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:05.887401 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlbdt\" (UniqueName: \"kubernetes.io/projected/f95dd442-db70-44a1-b6ac-533be225068c-kube-api-access-dlbdt\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:11:05.889220 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:05.889196 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c10c218-9344-4504-a060-2d533a772408-kube-api-access-qv4bz" pod "8c10c218-9344-4504-a060-2d533a772408" (UID: "8c10c218-9344-4504-a060-2d533a772408"). InnerVolumeSpecName "kube-api-access-qv4bz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:11:05.988720 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:05.988638 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv4bz\" (UniqueName: \"kubernetes.io/projected/8c10c218-9344-4504-a060-2d533a772408-kube-api-access-qv4bz\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:11:06.661499 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:06.661455 5979 generic.go:356] "Generic (PLEG): container finished" podID="8c10c218-9344-4504-a060-2d533a772408" containerID="4a5733889df53b6a1dd0a57aa050ce805132fdc2664195b5829345e288663e24" exitCode=0 Nov 03 14:11:06.661944 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:06.661532 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d6963e-cc4d8848c-bf2p7" Nov 03 14:11:06.661944 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:06.661537 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d6963e-cc4d8848c-bf2p7" event={"ID":"8c10c218-9344-4504-a060-2d533a772408","Type":"ContainerDied","Data":"4a5733889df53b6a1dd0a57aa050ce805132fdc2664195b5829345e288663e24"} Nov 03 14:11:06.661944 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:06.661603 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d6963e-cc4d8848c-bf2p7" event={"ID":"8c10c218-9344-4504-a060-2d533a772408","Type":"ContainerDied","Data":"a6791350d7034a59d25352b5f4ec399008f938862b63b21aa07663abc807033b"} Nov 03 14:11:06.661944 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:06.661632 5979 scope.go:122] "RemoveContainer" containerID="4a5733889df53b6a1dd0a57aa050ce805132fdc2664195b5829345e288663e24" Nov 03 14:11:06.663102 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:06.663073 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-d6963e-8679d489c9-7n5xs" Nov 03 14:11:06.663193 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:06.663122 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-d6963e-8679d489c9-7n5xs" event={"ID":"f95dd442-db70-44a1-b6ac-533be225068c","Type":"ContainerDied","Data":"c17de2af77aa503e3336b30390502b137ee3e93cc479a27b4b03c2d32ee93526"} Nov 03 14:11:06.671998 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:06.671982 5979 scope.go:122] "RemoveContainer" containerID="4a5733889df53b6a1dd0a57aa050ce805132fdc2664195b5829345e288663e24" Nov 03 14:11:06.675708 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:11:06.675680 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a5733889df53b6a1dd0a57aa050ce805132fdc2664195b5829345e288663e24\": container with ID starting with 4a5733889df53b6a1dd0a57aa050ce805132fdc2664195b5829345e288663e24 not found: ID does not exist" containerID="4a5733889df53b6a1dd0a57aa050ce805132fdc2664195b5829345e288663e24" Nov 03 14:11:06.675890 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:06.675863 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a5733889df53b6a1dd0a57aa050ce805132fdc2664195b5829345e288663e24"} err="failed to get container status \"4a5733889df53b6a1dd0a57aa050ce805132fdc2664195b5829345e288663e24\": rpc error: code = NotFound desc = could not find container \"4a5733889df53b6a1dd0a57aa050ce805132fdc2664195b5829345e288663e24\": container with ID starting with 4a5733889df53b6a1dd0a57aa050ce805132fdc2664195b5829345e288663e24 not found: ID does not exist" Nov 03 14:11:06.675890 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:06.675888 5979 scope.go:122] "RemoveContainer" containerID="9d641f843c83d02b0d6fdf82c4032024d66ba841f7b9281d576d16a40a4ac7eb" Nov 03 14:11:06.815982 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:06.815941 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d6963e-8679d489c9-7n5xs"] Nov 03 14:11:06.862587 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:06.862548 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-d6963e-8679d489c9-7n5xs"] Nov 03 14:11:06.921753 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:06.921708 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d6963e-cc4d8848c-bf2p7"] Nov 03 14:11:06.924652 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:06.924629 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d6963e-cc4d8848c-bf2p7"] Nov 03 14:11:07.109804 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:07.109717 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" podUID="d09478af-d618-425d-a4e6-e0889e0f416d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:11:07.234790 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:11:07.234751 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:11:08.117205 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:08.117168 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c10c218-9344-4504-a060-2d533a772408" path="/var/lib/kubelet/pods/8c10c218-9344-4504-a060-2d533a772408/volumes" Nov 03 14:11:08.117666 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:08.117649 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f95dd442-db70-44a1-b6ac-533be225068c" path="/var/lib/kubelet/pods/f95dd442-db70-44a1-b6ac-533be225068c/volumes" Nov 03 14:11:17.110487 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:17.110444 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" podUID="d09478af-d618-425d-a4e6-e0889e0f416d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:11:18.614389 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:18.614346 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:11:22.041830 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:22.041794 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-f11801-74b6fc964f-mf2fc"] Nov 03 14:11:27.043038 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:27.042998 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-f11801-74b6fc964f-mf2fc" podUID="db161f59-c4de-4fd3-bc2b-0a6262dffd38" containerName="payload-processing" containerID="cri-o://4ff73ec498bb3dba8446eac227a440aae9168f3fa8a10c5dffddc0d04aa9337c" gracePeriod=25 Nov 03 14:11:27.109684 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:27.109635 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" podUID="d09478af-d618-425d-a4e6-e0889e0f416d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:11:27.341636 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:27.341604 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-f11801-74b6fc964f-mf2fc" Nov 03 14:11:27.441306 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:27.441274 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db161f59-c4de-4fd3-bc2b-0a6262dffd38-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db161f59-c4de-4fd3-bc2b-0a6262dffd38-plugins-config-volume\") pod \"db161f59-c4de-4fd3-bc2b-0a6262dffd38\" (UID: \"db161f59-c4de-4fd3-bc2b-0a6262dffd38\") " Nov 03 14:11:27.441536 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:27.441389 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db161f59-c4de-4fd3-bc2b-0a6262dffd38-kube-api-access-6rd77\" (UniqueName: \"kubernetes.io/projected/db161f59-c4de-4fd3-bc2b-0a6262dffd38-kube-api-access-6rd77\") pod \"db161f59-c4de-4fd3-bc2b-0a6262dffd38\" (UID: \"db161f59-c4de-4fd3-bc2b-0a6262dffd38\") " Nov 03 14:11:27.441767 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:27.441736 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db161f59-c4de-4fd3-bc2b-0a6262dffd38-plugins-config-volume" pod "db161f59-c4de-4fd3-bc2b-0a6262dffd38" (UID: "db161f59-c4de-4fd3-bc2b-0a6262dffd38"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:11:27.442086 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:27.442067 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db161f59-c4de-4fd3-bc2b-0a6262dffd38-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:11:27.443702 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:27.443652 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db161f59-c4de-4fd3-bc2b-0a6262dffd38-kube-api-access-6rd77" pod "db161f59-c4de-4fd3-bc2b-0a6262dffd38" (UID: "db161f59-c4de-4fd3-bc2b-0a6262dffd38"). InnerVolumeSpecName "kube-api-access-6rd77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:11:27.543380 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:27.543346 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rd77\" (UniqueName: \"kubernetes.io/projected/db161f59-c4de-4fd3-bc2b-0a6262dffd38-kube-api-access-6rd77\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:11:27.844419 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:27.844389 5979 generic.go:356] "Generic (PLEG): container finished" podID="db161f59-c4de-4fd3-bc2b-0a6262dffd38" containerID="4ff73ec498bb3dba8446eac227a440aae9168f3fa8a10c5dffddc0d04aa9337c" exitCode=0 Nov 03 14:11:27.844614 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:27.844461 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-f11801-74b6fc964f-mf2fc" Nov 03 14:11:27.844614 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:27.844492 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-f11801-74b6fc964f-mf2fc" event={"ID":"db161f59-c4de-4fd3-bc2b-0a6262dffd38","Type":"ContainerDied","Data":"4ff73ec498bb3dba8446eac227a440aae9168f3fa8a10c5dffddc0d04aa9337c"} Nov 03 14:11:27.844721 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:27.844622 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-f11801-74b6fc964f-mf2fc" event={"ID":"db161f59-c4de-4fd3-bc2b-0a6262dffd38","Type":"ContainerDied","Data":"666195638c5cc3b0458930c19ab10575a5cddc27bf267f654fb5a814a88ebdfb"} Nov 03 14:11:27.844721 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:27.844645 5979 scope.go:122] "RemoveContainer" containerID="4ff73ec498bb3dba8446eac227a440aae9168f3fa8a10c5dffddc0d04aa9337c" Nov 03 14:11:27.854912 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:27.854895 5979 scope.go:122] "RemoveContainer" containerID="4ff73ec498bb3dba8446eac227a440aae9168f3fa8a10c5dffddc0d04aa9337c" Nov 03 14:11:27.855203 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:11:27.855179 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ff73ec498bb3dba8446eac227a440aae9168f3fa8a10c5dffddc0d04aa9337c\": container with ID starting with 4ff73ec498bb3dba8446eac227a440aae9168f3fa8a10c5dffddc0d04aa9337c not found: ID does not exist" containerID="4ff73ec498bb3dba8446eac227a440aae9168f3fa8a10c5dffddc0d04aa9337c" Nov 03 14:11:27.855301 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:27.855210 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ff73ec498bb3dba8446eac227a440aae9168f3fa8a10c5dffddc0d04aa9337c"} err="failed to get container status \"4ff73ec498bb3dba8446eac227a440aae9168f3fa8a10c5dffddc0d04aa9337c\": rpc error: code = NotFound desc = could not find container \"4ff73ec498bb3dba8446eac227a440aae9168f3fa8a10c5dffddc0d04aa9337c\": container with ID starting with 4ff73ec498bb3dba8446eac227a440aae9168f3fa8a10c5dffddc0d04aa9337c not found: ID does not exist" Nov 03 14:11:27.999910 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:27.999870 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-f11801-74b6fc964f-mf2fc"] Nov 03 14:11:28.046601 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:28.046566 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-f11801-74b6fc964f-mf2fc"] Nov 03 14:11:28.118098 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:28.118015 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db161f59-c4de-4fd3-bc2b-0a6262dffd38" path="/var/lib/kubelet/pods/db161f59-c4de-4fd3-bc2b-0a6262dffd38/volumes" Nov 03 14:11:29.795575 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:29.795492 5979 patch_prober.go:29] interesting pod/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.76:15021/healthz/ready\": dial tcp 10.134.1.76:15021: connect: connection refused" start-of-body= Nov 03 14:11:29.795575 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:29.795552 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2" podUID="7bb63e83-ca20-4eb2-be94-a2e7fcca2ace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.76:15021/healthz/ready\": dial tcp 10.134.1.76:15021: connect: connection refused" Nov 03 14:11:29.860269 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:29.860185 5979 generic.go:356] "Generic (PLEG): container finished" podID="7bb63e83-ca20-4eb2-be94-a2e7fcca2ace" containerID="623d2e18efbf02c98355d4aab7c1c1433c17f3c7e260f2cb555ad0cf628211cc" exitCode=0 Nov 03 14:11:29.860430 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:29.860259 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2" event={"ID":"7bb63e83-ca20-4eb2-be94-a2e7fcca2ace","Type":"ContainerDied","Data":"623d2e18efbf02c98355d4aab7c1c1433c17f3c7e260f2cb555ad0cf628211cc"} Nov 03 14:11:29.860721 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:29.860705 5979 scope.go:122] "RemoveContainer" containerID="623d2e18efbf02c98355d4aab7c1c1433c17f3c7e260f2cb555ad0cf628211cc" Nov 03 14:11:29.862324 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:29.862299 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:11:29.862387 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:29.862362 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:11:29.862425 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:29.862394 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:11:30.878080 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:30.878041 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2" event={"ID":"7bb63e83-ca20-4eb2-be94-a2e7fcca2ace","Type":"ContainerStarted","Data":"5f56c554e07b1b399d247f41352725361a934a45e854b0bf55cdc9eeda295c7e"} Nov 03 14:11:31.109611 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:31.109571 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" podUID="d09478af-d618-425d-a4e6-e0889e0f416d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:11:31.642052 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:31.642018 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2" Nov 03 14:11:31.642286 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:31.642217 5979 patch_prober.go:29] interesting pod/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.76:15021/healthz/ready\": dial tcp 10.134.1.76:15021: connect: connection refused" start-of-body= Nov 03 14:11:31.642286 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:31.642269 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2" podUID="7bb63e83-ca20-4eb2-be94-a2e7fcca2ace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.76:15021/healthz/ready\": dial tcp 10.134.1.76:15021: connect: connection refused" Nov 03 14:11:32.643169 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:32.643132 5979 patch_prober.go:29] interesting pod/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.76:15021/healthz/ready\": dial tcp 10.134.1.76:15021: connect: connection refused" start-of-body= Nov 03 14:11:32.643637 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:32.643193 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2" podUID="7bb63e83-ca20-4eb2-be94-a2e7fcca2ace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.76:15021/healthz/ready\": dial tcp 10.134.1.76:15021: connect: connection refused" Nov 03 14:11:33.642689 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:33.642655 5979 patch_prober.go:29] interesting pod/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.76:15021/healthz/ready\": dial tcp 10.134.1.76:15021: connect: connection refused" start-of-body= Nov 03 14:11:33.642909 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:33.642710 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2" podUID="7bb63e83-ca20-4eb2-be94-a2e7fcca2ace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.76:15021/healthz/ready\": dial tcp 10.134.1.76:15021: connect: connection refused" Nov 03 14:11:34.642777 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:34.642732 5979 patch_prober.go:29] interesting pod/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.76:15021/healthz/ready\": dial tcp 10.134.1.76:15021: connect: connection refused" start-of-body= Nov 03 14:11:34.643294 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:34.642812 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2" podUID="7bb63e83-ca20-4eb2-be94-a2e7fcca2ace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.76:15021/healthz/ready\": dial tcp 10.134.1.76:15021: connect: connection refused" Nov 03 14:11:35.642698 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:35.642663 5979 patch_prober.go:29] interesting pod/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.76:15021/healthz/ready\": dial tcp 10.134.1.76:15021: connect: connection refused" start-of-body= Nov 03 14:11:35.642882 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:35.642721 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2" podUID="7bb63e83-ca20-4eb2-be94-a2e7fcca2ace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.76:15021/healthz/ready\": dial tcp 10.134.1.76:15021: connect: connection refused" Nov 03 14:11:36.647073 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:36.647039 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2" Nov 03 14:11:36.647496 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:36.647340 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2" Nov 03 14:11:36.648181 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:36.648161 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2" Nov 03 14:11:37.234701 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:11:37.234665 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:11:41.110190 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:41.110153 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" podUID="d09478af-d618-425d-a4e6-e0889e0f416d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:11:51.109605 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:51.109561 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" podUID="d09478af-d618-425d-a4e6-e0889e0f416d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:11:51.943999 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:11:51.943964 5979 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-e86f9ab11580: secret "agc-cred-e86f9ab11580" not found Nov 03 14:11:51.943999 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:11:51.943993 5979 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-84dcd755bf-c9rnp: secret "agc-cred-e86f9ab11580" not found Nov 03 14:11:51.944237 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:11:51.944044 5979 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/797bcf35-fa9e-47b8-887c-bb8942af277b-provider-credentials podName:797bcf35-fa9e-47b8-887c-bb8942af277b nodeName:}" failed. No retries permitted until 2026-11-03 14:13:53.944028701 +0000 UTC m=+4115.954716921 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/797bcf35-fa9e-47b8-887c-bb8942af277b-provider-credentials") pod "payload-processing-external-model-84dcd755bf-c9rnp" (UID: "797bcf35-fa9e-47b8-887c-bb8942af277b") : secret "agc-cred-e86f9ab11580" not found Nov 03 14:11:59.803840 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:59.803798 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b6c8c7-6f6b7ddfc6-mxvmw"] Nov 03 14:11:59.804847 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:59.804824 5979 cpu_manager.go:414] "containerMap: removing container" podUID="f95dd442-db70-44a1-b6ac-533be225068c" containerName="payload-processing" Nov 03 14:11:59.804924 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:59.804850 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f95dd442-db70-44a1-b6ac-533be225068c" containerName="payload-processing" Nov 03 14:11:59.804924 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:59.804868 5979 cpu_manager.go:414] "containerMap: removing container" podUID="db161f59-c4de-4fd3-bc2b-0a6262dffd38" containerName="payload-processing" Nov 03 14:11:59.804924 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:59.804877 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db161f59-c4de-4fd3-bc2b-0a6262dffd38" containerName="payload-processing" Nov 03 14:11:59.805055 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:59.804944 5979 cpu_manager.go:414] "containerMap: removing container" podUID="8c10c218-9344-4504-a060-2d533a772408" containerName="payload-pre-processing" Nov 03 14:11:59.805055 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:59.804953 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c10c218-9344-4504-a060-2d533a772408" containerName="payload-pre-processing" Nov 03 14:11:59.805169 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:59.805152 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c10c218-9344-4504-a060-2d533a772408" containerName="payload-pre-processing" Nov 03 14:11:59.805206 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:59.805184 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="f95dd442-db70-44a1-b6ac-533be225068c" containerName="payload-processing" Nov 03 14:11:59.805206 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:59.805196 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="db161f59-c4de-4fd3-bc2b-0a6262dffd38" containerName="payload-processing" Nov 03 14:11:59.809114 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:59.809093 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b6c8c7-6f6b7ddfc6-mxvmw" Nov 03 14:11:59.853226 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:59.853181 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jxhm\" (UniqueName: \"kubernetes.io/projected/f05f949d-904f-419e-90e3-ba5a6ff30ecf-kube-api-access-4jxhm\") pod \"maas-api-e2e-inf-a-b6c8c7-6f6b7ddfc6-mxvmw\" (UID: \"f05f949d-904f-419e-90e3-ba5a6ff30ecf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b6c8c7-6f6b7ddfc6-mxvmw" Nov 03 14:11:59.853440 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:59.853235 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f05f949d-904f-419e-90e3-ba5a6ff30ecf-maas-api-tls\") pod \"maas-api-e2e-inf-a-b6c8c7-6f6b7ddfc6-mxvmw\" (UID: \"f05f949d-904f-419e-90e3-ba5a6ff30ecf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b6c8c7-6f6b7ddfc6-mxvmw" Nov 03 14:11:59.853440 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:59.853362 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f05f949d-904f-419e-90e3-ba5a6ff30ecf-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b6c8c7-6f6b7ddfc6-mxvmw\" (UID: \"f05f949d-904f-419e-90e3-ba5a6ff30ecf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b6c8c7-6f6b7ddfc6-mxvmw" Nov 03 14:11:59.853610 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:59.853438 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f05f949d-904f-419e-90e3-ba5a6ff30ecf-metrics-certs\") pod \"maas-api-e2e-inf-a-b6c8c7-6f6b7ddfc6-mxvmw\" (UID: \"f05f949d-904f-419e-90e3-ba5a6ff30ecf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b6c8c7-6f6b7ddfc6-mxvmw" Nov 03 14:11:59.862048 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:59.862014 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b6c8c7-6f6b7ddfc6-mxvmw"] Nov 03 14:11:59.904862 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:59.904823 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-b6c8c7-serving-cert\"" Nov 03 14:11:59.955093 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:59.955049 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jxhm\" (UniqueName: \"kubernetes.io/projected/f05f949d-904f-419e-90e3-ba5a6ff30ecf-kube-api-access-4jxhm\") pod \"maas-api-e2e-inf-a-b6c8c7-6f6b7ddfc6-mxvmw\" (UID: \"f05f949d-904f-419e-90e3-ba5a6ff30ecf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b6c8c7-6f6b7ddfc6-mxvmw" Nov 03 14:11:59.955281 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:59.955102 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f05f949d-904f-419e-90e3-ba5a6ff30ecf-maas-api-tls\") pod \"maas-api-e2e-inf-a-b6c8c7-6f6b7ddfc6-mxvmw\" (UID: \"f05f949d-904f-419e-90e3-ba5a6ff30ecf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b6c8c7-6f6b7ddfc6-mxvmw" Nov 03 14:11:59.955281 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:59.955148 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f05f949d-904f-419e-90e3-ba5a6ff30ecf-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b6c8c7-6f6b7ddfc6-mxvmw\" (UID: \"f05f949d-904f-419e-90e3-ba5a6ff30ecf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b6c8c7-6f6b7ddfc6-mxvmw" Nov 03 14:11:59.955281 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:59.955218 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f05f949d-904f-419e-90e3-ba5a6ff30ecf-metrics-certs\") pod \"maas-api-e2e-inf-a-b6c8c7-6f6b7ddfc6-mxvmw\" (UID: \"f05f949d-904f-419e-90e3-ba5a6ff30ecf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b6c8c7-6f6b7ddfc6-mxvmw" Nov 03 14:11:59.955587 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:11:59.955565 5979 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-b6c8c7-serving-cert: secret "maas-api-e2e-inf-a-b6c8c7-serving-cert" not found Nov 03 14:11:59.955797 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:11:59.955783 5979 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f05f949d-904f-419e-90e3-ba5a6ff30ecf-maas-api-tls podName:f05f949d-904f-419e-90e3-ba5a6ff30ecf nodeName:}" failed. No retries permitted until 2026-11-03 14:12:00.455759823 +0000 UTC m=+4002.466448048 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f05f949d-904f-419e-90e3-ba5a6ff30ecf-maas-api-tls") pod "maas-api-e2e-inf-a-b6c8c7-6f6b7ddfc6-mxvmw" (UID: "f05f949d-904f-419e-90e3-ba5a6ff30ecf") : secret "maas-api-e2e-inf-a-b6c8c7-serving-cert" not found Nov 03 14:11:59.956427 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:59.956403 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f05f949d-904f-419e-90e3-ba5a6ff30ecf-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b6c8c7-6f6b7ddfc6-mxvmw\" (UID: \"f05f949d-904f-419e-90e3-ba5a6ff30ecf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b6c8c7-6f6b7ddfc6-mxvmw" Nov 03 14:11:59.961000 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:11:59.960977 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f05f949d-904f-419e-90e3-ba5a6ff30ecf-metrics-certs\") pod \"maas-api-e2e-inf-a-b6c8c7-6f6b7ddfc6-mxvmw\" (UID: \"f05f949d-904f-419e-90e3-ba5a6ff30ecf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b6c8c7-6f6b7ddfc6-mxvmw" Nov 03 14:12:00.011280 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:00.011245 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jxhm\" (UniqueName: \"kubernetes.io/projected/f05f949d-904f-419e-90e3-ba5a6ff30ecf-kube-api-access-4jxhm\") pod \"maas-api-e2e-inf-a-b6c8c7-6f6b7ddfc6-mxvmw\" (UID: \"f05f949d-904f-419e-90e3-ba5a6ff30ecf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b6c8c7-6f6b7ddfc6-mxvmw" Nov 03 14:12:00.482354 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:00.482317 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f05f949d-904f-419e-90e3-ba5a6ff30ecf-maas-api-tls\") pod \"maas-api-e2e-inf-a-b6c8c7-6f6b7ddfc6-mxvmw\" (UID: \"f05f949d-904f-419e-90e3-ba5a6ff30ecf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b6c8c7-6f6b7ddfc6-mxvmw" Nov 03 14:12:00.484814 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:00.484790 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f05f949d-904f-419e-90e3-ba5a6ff30ecf-maas-api-tls\") pod \"maas-api-e2e-inf-a-b6c8c7-6f6b7ddfc6-mxvmw\" (UID: \"f05f949d-904f-419e-90e3-ba5a6ff30ecf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b6c8c7-6f6b7ddfc6-mxvmw" Nov 03 14:12:00.723962 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:00.723921 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b6c8c7-6f6b7ddfc6-mxvmw" Nov 03 14:12:01.101920 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:01.101840 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b6c8c7-6f6b7ddfc6-mxvmw"] Nov 03 14:12:01.109538 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:01.109511 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" podUID="d09478af-d618-425d-a4e6-e0889e0f416d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:12:01.109674 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:12:01.109624 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf05f949d_904f_419e_90e3_ba5a6ff30ecf.slice/crio-fbb9ae7d3f4f0defc5d63351945bb43fd4702a28a091cc8eebf6d89d4b8fee49 WatchSource:0}: Error finding container fbb9ae7d3f4f0defc5d63351945bb43fd4702a28a091cc8eebf6d89d4b8fee49: Status 404 returned error can't find the container with id fbb9ae7d3f4f0defc5d63351945bb43fd4702a28a091cc8eebf6d89d4b8fee49 Nov 03 14:12:01.111445 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:01.111422 5979 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 14:12:01.126257 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:01.126231 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b6c8c7-6f6b7ddfc6-mxvmw" event={"ID":"f05f949d-904f-419e-90e3-ba5a6ff30ecf","Type":"ContainerStarted","Data":"fbb9ae7d3f4f0defc5d63351945bb43fd4702a28a091cc8eebf6d89d4b8fee49"} Nov 03 14:12:02.133905 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:02.133878 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b6c8c7-6f6b7ddfc6-mxvmw" event={"ID":"f05f949d-904f-419e-90e3-ba5a6ff30ecf","Type":"ContainerStarted","Data":"67cb04c898ae0b43cb515e51d8365a8f0ded0bcdc8039592a6085396fbcf4210"} Nov 03 14:12:02.134298 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:02.134077 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b6c8c7-6f6b7ddfc6-mxvmw" Nov 03 14:12:02.280069 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:02.280010 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b6c8c7-6f6b7ddfc6-mxvmw" podStartSLOduration=2.346187575 podStartE2EDuration="3.279993801s" totalImagesPullingTime="933.806226ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:11:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:12:02.277975767 +0000 UTC m=+4004.288664010" watchObservedRunningTime="2026-11-03 14:12:02.279993801 +0000 UTC m=+4004.290682042" Nov 03 14:12:07.044521 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:07.044463 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f945362d-vvx7f"] Nov 03 14:12:07.090605 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:07.090565 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f945362d-vvx7f"] Nov 03 14:12:07.235195 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:12:07.235158 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:12:08.117563 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:08.117531 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="798ad31e-2eea-448a-b2e8-bc02a98191c6" path="/var/lib/kubelet/pods/798ad31e-2eea-448a-b2e8-bc02a98191c6/volumes" Nov 03 14:12:08.147562 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:08.147536 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b6c8c7-6f6b7ddfc6-mxvmw" Nov 03 14:12:11.110048 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:11.110013 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" podUID="d09478af-d618-425d-a4e6-e0889e0f416d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:12:21.007975 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:21.007926 5979 scope.go:122] "RemoveContainer" containerID="a6bf3aea33b38fe5f3627446cfbd115b66e64be1dac21b1d6683e9d1511f0db2" Nov 03 14:12:21.109847 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:21.109801 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" podUID="d09478af-d618-425d-a4e6-e0889e0f416d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:12:24.390713 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:24.390676 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b"] Nov 03 14:12:24.391350 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:24.390881 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b" podUID="5178f75c-2f52-4a5b-bf00-c315bf7e110b" containerName="istio-proxy" containerID="cri-o://fe00e1d456d026a140289cc8dfc5ba9abe2c6542db869016d1a5a280681db59d" gracePeriod=30 Nov 03 14:12:24.821969 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:24.821882 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:12:25.304192 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:25.304158 5979 patch_prober.go:29] interesting pod/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 14:12:25.304370 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:25.304211 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b" podUID="5178f75c-2f52-4a5b-bf00-c315bf7e110b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 14:12:29.659198 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:29.659121 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b" Nov 03 14:12:29.773346 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:29.773312 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5178f75c-2f52-4a5b-bf00-c315bf7e110b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5178f75c-2f52-4a5b-bf00-c315bf7e110b-credential-socket\") pod \"5178f75c-2f52-4a5b-bf00-c315bf7e110b\" (UID: \"5178f75c-2f52-4a5b-bf00-c315bf7e110b\") " Nov 03 14:12:29.773346 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:29.773352 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5178f75c-2f52-4a5b-bf00-c315bf7e110b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5178f75c-2f52-4a5b-bf00-c315bf7e110b-istio-envoy\") pod \"5178f75c-2f52-4a5b-bf00-c315bf7e110b\" (UID: \"5178f75c-2f52-4a5b-bf00-c315bf7e110b\") " Nov 03 14:12:29.773646 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:29.773396 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5178f75c-2f52-4a5b-bf00-c315bf7e110b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5178f75c-2f52-4a5b-bf00-c315bf7e110b-istio-podinfo\") pod \"5178f75c-2f52-4a5b-bf00-c315bf7e110b\" (UID: \"5178f75c-2f52-4a5b-bf00-c315bf7e110b\") " Nov 03 14:12:29.773646 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:29.773421 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5178f75c-2f52-4a5b-bf00-c315bf7e110b-istio-token\" (UniqueName: \"kubernetes.io/projected/5178f75c-2f52-4a5b-bf00-c315bf7e110b-istio-token\") pod \"5178f75c-2f52-4a5b-bf00-c315bf7e110b\" (UID: \"5178f75c-2f52-4a5b-bf00-c315bf7e110b\") " Nov 03 14:12:29.773646 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:29.773456 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5178f75c-2f52-4a5b-bf00-c315bf7e110b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5178f75c-2f52-4a5b-bf00-c315bf7e110b-workload-certs\") pod \"5178f75c-2f52-4a5b-bf00-c315bf7e110b\" (UID: \"5178f75c-2f52-4a5b-bf00-c315bf7e110b\") " Nov 03 14:12:29.773646 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:29.773631 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5178f75c-2f52-4a5b-bf00-c315bf7e110b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5178f75c-2f52-4a5b-bf00-c315bf7e110b-workload-socket\") pod \"5178f75c-2f52-4a5b-bf00-c315bf7e110b\" (UID: \"5178f75c-2f52-4a5b-bf00-c315bf7e110b\") " Nov 03 14:12:29.773892 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:29.773662 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5178f75c-2f52-4a5b-bf00-c315bf7e110b-credential-socket" pod "5178f75c-2f52-4a5b-bf00-c315bf7e110b" (UID: "5178f75c-2f52-4a5b-bf00-c315bf7e110b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:12:29.773892 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:29.773678 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5178f75c-2f52-4a5b-bf00-c315bf7e110b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5178f75c-2f52-4a5b-bf00-c315bf7e110b-istio-data\") pod \"5178f75c-2f52-4a5b-bf00-c315bf7e110b\" (UID: \"5178f75c-2f52-4a5b-bf00-c315bf7e110b\") " Nov 03 14:12:29.773892 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:29.773716 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5178f75c-2f52-4a5b-bf00-c315bf7e110b-kube-api-access-cr26v\" (UniqueName: \"kubernetes.io/projected/5178f75c-2f52-4a5b-bf00-c315bf7e110b-kube-api-access-cr26v\") pod \"5178f75c-2f52-4a5b-bf00-c315bf7e110b\" (UID: \"5178f75c-2f52-4a5b-bf00-c315bf7e110b\") " Nov 03 14:12:29.773892 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:29.773777 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5178f75c-2f52-4a5b-bf00-c315bf7e110b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5178f75c-2f52-4a5b-bf00-c315bf7e110b-istiod-ca-cert\") pod \"5178f75c-2f52-4a5b-bf00-c315bf7e110b\" (UID: \"5178f75c-2f52-4a5b-bf00-c315bf7e110b\") " Nov 03 14:12:29.773892 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:29.773799 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5178f75c-2f52-4a5b-bf00-c315bf7e110b-workload-certs" pod "5178f75c-2f52-4a5b-bf00-c315bf7e110b" (UID: "5178f75c-2f52-4a5b-bf00-c315bf7e110b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:12:29.774255 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:29.773903 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5178f75c-2f52-4a5b-bf00-c315bf7e110b-workload-socket" pod "5178f75c-2f52-4a5b-bf00-c315bf7e110b" (UID: "5178f75c-2f52-4a5b-bf00-c315bf7e110b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:12:29.774255 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:29.773956 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5178f75c-2f52-4a5b-bf00-c315bf7e110b-istio-data" pod "5178f75c-2f52-4a5b-bf00-c315bf7e110b" (UID: "5178f75c-2f52-4a5b-bf00-c315bf7e110b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:12:29.774381 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:29.774279 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5178f75c-2f52-4a5b-bf00-c315bf7e110b-istiod-ca-cert" pod "5178f75c-2f52-4a5b-bf00-c315bf7e110b" (UID: "5178f75c-2f52-4a5b-bf00-c315bf7e110b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:12:29.774451 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:29.774432 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5178f75c-2f52-4a5b-bf00-c315bf7e110b-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:12:29.774451 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:29.774448 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5178f75c-2f52-4a5b-bf00-c315bf7e110b-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:12:29.774600 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:29.774458 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5178f75c-2f52-4a5b-bf00-c315bf7e110b-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:12:29.774600 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:29.774483 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5178f75c-2f52-4a5b-bf00-c315bf7e110b-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:12:29.774600 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:29.774497 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5178f75c-2f52-4a5b-bf00-c315bf7e110b-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:12:29.775686 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:29.775665 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5178f75c-2f52-4a5b-bf00-c315bf7e110b-istio-podinfo" pod "5178f75c-2f52-4a5b-bf00-c315bf7e110b" (UID: "5178f75c-2f52-4a5b-bf00-c315bf7e110b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:12:29.775795 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:29.775764 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5178f75c-2f52-4a5b-bf00-c315bf7e110b-istio-token" pod "5178f75c-2f52-4a5b-bf00-c315bf7e110b" (UID: "5178f75c-2f52-4a5b-bf00-c315bf7e110b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:12:29.776026 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:29.776002 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5178f75c-2f52-4a5b-bf00-c315bf7e110b-istio-envoy" pod "5178f75c-2f52-4a5b-bf00-c315bf7e110b" (UID: "5178f75c-2f52-4a5b-bf00-c315bf7e110b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:12:29.776090 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:29.776045 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5178f75c-2f52-4a5b-bf00-c315bf7e110b-kube-api-access-cr26v" pod "5178f75c-2f52-4a5b-bf00-c315bf7e110b" (UID: "5178f75c-2f52-4a5b-bf00-c315bf7e110b"). InnerVolumeSpecName "kube-api-access-cr26v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:12:29.875528 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:29.875463 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5178f75c-2f52-4a5b-bf00-c315bf7e110b-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:12:29.875528 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:29.875522 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5178f75c-2f52-4a5b-bf00-c315bf7e110b-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:12:29.875528 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:29.875534 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cr26v\" (UniqueName: \"kubernetes.io/projected/5178f75c-2f52-4a5b-bf00-c315bf7e110b-kube-api-access-cr26v\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:12:29.875805 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:29.875543 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5178f75c-2f52-4a5b-bf00-c315bf7e110b-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:12:30.320656 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:30.320623 5979 generic.go:356] "Generic (PLEG): container finished" podID="5178f75c-2f52-4a5b-bf00-c315bf7e110b" containerID="fe00e1d456d026a140289cc8dfc5ba9abe2c6542db869016d1a5a280681db59d" exitCode=0 Nov 03 14:12:30.320837 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:30.320673 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b" Nov 03 14:12:30.320837 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:30.320708 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b" event={"ID":"5178f75c-2f52-4a5b-bf00-c315bf7e110b","Type":"ContainerDied","Data":"fe00e1d456d026a140289cc8dfc5ba9abe2c6542db869016d1a5a280681db59d"} Nov 03 14:12:30.320837 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:30.320766 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b" event={"ID":"5178f75c-2f52-4a5b-bf00-c315bf7e110b","Type":"ContainerDied","Data":"d9c9c3a474e9943f6725ad0f08dc4677ace6187e5559acec7a1540e6dade174d"} Nov 03 14:12:30.320837 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:30.320793 5979 scope.go:122] "RemoveContainer" containerID="fe00e1d456d026a140289cc8dfc5ba9abe2c6542db869016d1a5a280681db59d" Nov 03 14:12:30.332102 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:30.332076 5979 scope.go:122] "RemoveContainer" containerID="2b48c25bf1e790d5210106dc9685a1370fa259f24c4ab7508fb17bf685d3be43" Nov 03 14:12:30.341769 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:30.341747 5979 scope.go:122] "RemoveContainer" containerID="fe00e1d456d026a140289cc8dfc5ba9abe2c6542db869016d1a5a280681db59d" Nov 03 14:12:30.342037 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:12:30.342016 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe00e1d456d026a140289cc8dfc5ba9abe2c6542db869016d1a5a280681db59d\": container with ID starting with fe00e1d456d026a140289cc8dfc5ba9abe2c6542db869016d1a5a280681db59d not found: ID does not exist" containerID="fe00e1d456d026a140289cc8dfc5ba9abe2c6542db869016d1a5a280681db59d" Nov 03 14:12:30.342116 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:30.342045 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe00e1d456d026a140289cc8dfc5ba9abe2c6542db869016d1a5a280681db59d"} err="failed to get container status \"fe00e1d456d026a140289cc8dfc5ba9abe2c6542db869016d1a5a280681db59d\": rpc error: code = NotFound desc = could not find container \"fe00e1d456d026a140289cc8dfc5ba9abe2c6542db869016d1a5a280681db59d\": container with ID starting with fe00e1d456d026a140289cc8dfc5ba9abe2c6542db869016d1a5a280681db59d not found: ID does not exist" Nov 03 14:12:30.342116 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:30.342064 5979 scope.go:122] "RemoveContainer" containerID="2b48c25bf1e790d5210106dc9685a1370fa259f24c4ab7508fb17bf685d3be43" Nov 03 14:12:30.342292 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:12:30.342277 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b48c25bf1e790d5210106dc9685a1370fa259f24c4ab7508fb17bf685d3be43\": container with ID starting with 2b48c25bf1e790d5210106dc9685a1370fa259f24c4ab7508fb17bf685d3be43 not found: ID does not exist" containerID="2b48c25bf1e790d5210106dc9685a1370fa259f24c4ab7508fb17bf685d3be43" Nov 03 14:12:30.342339 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:30.342295 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b48c25bf1e790d5210106dc9685a1370fa259f24c4ab7508fb17bf685d3be43"} err="failed to get container status \"2b48c25bf1e790d5210106dc9685a1370fa259f24c4ab7508fb17bf685d3be43\": rpc error: code = NotFound desc = could not find container \"2b48c25bf1e790d5210106dc9685a1370fa259f24c4ab7508fb17bf685d3be43\": container with ID starting with 2b48c25bf1e790d5210106dc9685a1370fa259f24c4ab7508fb17bf685d3be43 not found: ID does not exist" Nov 03 14:12:30.476392 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:30.476358 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b"] Nov 03 14:12:30.522991 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:30.522956 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-d6963e-openshift-default-868c75455c-6h65b"] Nov 03 14:12:31.109995 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:31.109959 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" podUID="d09478af-d618-425d-a4e6-e0889e0f416d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:12:32.115653 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:32.115622 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5178f75c-2f52-4a5b-bf00-c315bf7e110b" path="/var/lib/kubelet/pods/5178f75c-2f52-4a5b-bf00-c315bf7e110b/volumes" Nov 03 14:12:37.109598 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:37.109563 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" podUID="d09478af-d618-425d-a4e6-e0889e0f416d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:12:37.234754 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:12:37.234716 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:12:37.443802 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:37.443769 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6375e7-7996cbfbbc-xsdbn"] Nov 03 14:12:42.444908 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:42.444853 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6375e7-7996cbfbbc-xsdbn" podUID="a929998e-474d-47af-afb6-8c1e97702287" containerName="payload-pre-processing" containerID="cri-o://4897d89fb1025deea79633693a68b4a36b1a3f0a49ef53b19b68c34f9c0d5ff1" gracePeriod=25 Nov 03 14:12:42.715114 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:42.715054 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6375e7-7996cbfbbc-xsdbn" Nov 03 14:12:42.796266 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:42.796235 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a929998e-474d-47af-afb6-8c1e97702287-kube-api-access-9phbl\" (UniqueName: \"kubernetes.io/projected/a929998e-474d-47af-afb6-8c1e97702287-kube-api-access-9phbl\") pod \"a929998e-474d-47af-afb6-8c1e97702287\" (UID: \"a929998e-474d-47af-afb6-8c1e97702287\") " Nov 03 14:12:42.796460 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:42.796327 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a929998e-474d-47af-afb6-8c1e97702287-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a929998e-474d-47af-afb6-8c1e97702287-plugins-config-volume\") pod \"a929998e-474d-47af-afb6-8c1e97702287\" (UID: \"a929998e-474d-47af-afb6-8c1e97702287\") " Nov 03 14:12:42.796771 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:42.796747 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a929998e-474d-47af-afb6-8c1e97702287-plugins-config-volume" pod "a929998e-474d-47af-afb6-8c1e97702287" (UID: "a929998e-474d-47af-afb6-8c1e97702287"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:12:42.798499 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:42.798463 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a929998e-474d-47af-afb6-8c1e97702287-kube-api-access-9phbl" pod "a929998e-474d-47af-afb6-8c1e97702287" (UID: "a929998e-474d-47af-afb6-8c1e97702287"). InnerVolumeSpecName "kube-api-access-9phbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:12:42.898583 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:42.898545 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9phbl\" (UniqueName: \"kubernetes.io/projected/a929998e-474d-47af-afb6-8c1e97702287-kube-api-access-9phbl\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:12:42.898583 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:42.898578 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a929998e-474d-47af-afb6-8c1e97702287-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:12:43.406323 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:43.406288 5979 generic.go:356] "Generic (PLEG): container finished" podID="a929998e-474d-47af-afb6-8c1e97702287" containerID="4897d89fb1025deea79633693a68b4a36b1a3f0a49ef53b19b68c34f9c0d5ff1" exitCode=0 Nov 03 14:12:43.406545 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:43.406332 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6375e7-7996cbfbbc-xsdbn" Nov 03 14:12:43.406545 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:43.406374 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6375e7-7996cbfbbc-xsdbn" event={"ID":"a929998e-474d-47af-afb6-8c1e97702287","Type":"ContainerDied","Data":"4897d89fb1025deea79633693a68b4a36b1a3f0a49ef53b19b68c34f9c0d5ff1"} Nov 03 14:12:43.406545 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:43.406432 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6375e7-7996cbfbbc-xsdbn" event={"ID":"a929998e-474d-47af-afb6-8c1e97702287","Type":"ContainerDied","Data":"8c72f839ad4ae34d46a5f3a50c5022972de38e0b432543fdfe8c8dc12967d49c"} Nov 03 14:12:43.406545 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:43.406465 5979 scope.go:122] "RemoveContainer" containerID="4897d89fb1025deea79633693a68b4a36b1a3f0a49ef53b19b68c34f9c0d5ff1" Nov 03 14:12:43.417078 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:43.417060 5979 scope.go:122] "RemoveContainer" containerID="4897d89fb1025deea79633693a68b4a36b1a3f0a49ef53b19b68c34f9c0d5ff1" Nov 03 14:12:43.417374 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:12:43.417353 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4897d89fb1025deea79633693a68b4a36b1a3f0a49ef53b19b68c34f9c0d5ff1\": container with ID starting with 4897d89fb1025deea79633693a68b4a36b1a3f0a49ef53b19b68c34f9c0d5ff1 not found: ID does not exist" containerID="4897d89fb1025deea79633693a68b4a36b1a3f0a49ef53b19b68c34f9c0d5ff1" Nov 03 14:12:43.417455 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:43.417384 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4897d89fb1025deea79633693a68b4a36b1a3f0a49ef53b19b68c34f9c0d5ff1"} err="failed to get container status \"4897d89fb1025deea79633693a68b4a36b1a3f0a49ef53b19b68c34f9c0d5ff1\": rpc error: code = NotFound desc = could not find container \"4897d89fb1025deea79633693a68b4a36b1a3f0a49ef53b19b68c34f9c0d5ff1\": container with ID starting with 4897d89fb1025deea79633693a68b4a36b1a3f0a49ef53b19b68c34f9c0d5ff1 not found: ID does not exist" Nov 03 14:12:43.567870 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:43.567834 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6375e7-7996cbfbbc-xsdbn"] Nov 03 14:12:43.614415 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:43.614377 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6375e7-7996cbfbbc-xsdbn"] Nov 03 14:12:44.116536 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:44.116503 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a929998e-474d-47af-afb6-8c1e97702287" path="/var/lib/kubelet/pods/a929998e-474d-47af-afb6-8c1e97702287/volumes" Nov 03 14:12:44.899557 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:44.899511 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l"] Nov 03 14:12:44.900047 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:44.899710 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l" podUID="8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130" containerName="istio-proxy" containerID="cri-o://f4b54bd0c0ab7ba0133f93d78408d399f42483f9789ebf01d23beaeee9dc1760" gracePeriod=30 Nov 03 14:12:47.109913 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:47.109879 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" podUID="d09478af-d618-425d-a4e6-e0889e0f416d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:12:47.639492 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:47.632785 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx"] Nov 03 14:12:47.665982 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:47.665939 5979 cpu_manager.go:414] "containerMap: removing container" podUID="5178f75c-2f52-4a5b-bf00-c315bf7e110b" containerName="istio-proxy" Nov 03 14:12:47.666358 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:47.666329 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5178f75c-2f52-4a5b-bf00-c315bf7e110b" containerName="istio-proxy" Nov 03 14:12:47.666593 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:47.666574 5979 cpu_manager.go:414] "containerMap: removing container" podUID="a929998e-474d-47af-afb6-8c1e97702287" containerName="payload-pre-processing" Nov 03 14:12:47.666721 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:47.666705 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a929998e-474d-47af-afb6-8c1e97702287" containerName="payload-pre-processing" Nov 03 14:12:47.667253 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:47.667228 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="5178f75c-2f52-4a5b-bf00-c315bf7e110b" containerName="istio-proxy" Nov 03 14:12:47.667420 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:47.667404 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="a929998e-474d-47af-afb6-8c1e97702287" containerName="payload-pre-processing" Nov 03 14:12:47.765272 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:47.763895 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx" Nov 03 14:12:47.767570 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:47.767530 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx"] Nov 03 14:12:47.798694 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:47.798651 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43b38a5b-6579-4ce3-8bc6-1334bed0c633-istio-envoy\") pod \"e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx\" (UID: \"43b38a5b-6579-4ce3-8bc6-1334bed0c633\") " pod="openshift-ingress/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx" Nov 03 14:12:47.799060 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:47.799031 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43b38a5b-6579-4ce3-8bc6-1334bed0c633-istiod-ca-cert\") pod \"e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx\" (UID: \"43b38a5b-6579-4ce3-8bc6-1334bed0c633\") " pod="openshift-ingress/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx" Nov 03 14:12:47.799324 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:47.799303 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43b38a5b-6579-4ce3-8bc6-1334bed0c633-workload-certs\") pod \"e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx\" (UID: \"43b38a5b-6579-4ce3-8bc6-1334bed0c633\") " pod="openshift-ingress/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx" Nov 03 14:12:47.799518 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:47.799499 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43b38a5b-6579-4ce3-8bc6-1334bed0c633-istio-podinfo\") pod \"e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx\" (UID: \"43b38a5b-6579-4ce3-8bc6-1334bed0c633\") " pod="openshift-ingress/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx" Nov 03 14:12:47.799680 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:47.799660 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43b38a5b-6579-4ce3-8bc6-1334bed0c633-istio-data\") pod \"e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx\" (UID: \"43b38a5b-6579-4ce3-8bc6-1334bed0c633\") " pod="openshift-ingress/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx" Nov 03 14:12:47.799864 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:47.799845 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43b38a5b-6579-4ce3-8bc6-1334bed0c633-credential-socket\") pod \"e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx\" (UID: \"43b38a5b-6579-4ce3-8bc6-1334bed0c633\") " pod="openshift-ingress/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx" Nov 03 14:12:47.800167 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:47.800146 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43b38a5b-6579-4ce3-8bc6-1334bed0c633-istio-token\") pod \"e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx\" (UID: \"43b38a5b-6579-4ce3-8bc6-1334bed0c633\") " pod="openshift-ingress/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx" Nov 03 14:12:47.800329 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:47.800311 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccbkv\" (UniqueName: \"kubernetes.io/projected/43b38a5b-6579-4ce3-8bc6-1334bed0c633-kube-api-access-ccbkv\") pod \"e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx\" (UID: \"43b38a5b-6579-4ce3-8bc6-1334bed0c633\") " pod="openshift-ingress/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx" Nov 03 14:12:47.800537 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:47.800462 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43b38a5b-6579-4ce3-8bc6-1334bed0c633-workload-socket\") pod \"e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx\" (UID: \"43b38a5b-6579-4ce3-8bc6-1334bed0c633\") " pod="openshift-ingress/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx" Nov 03 14:12:47.907273 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:47.907153 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43b38a5b-6579-4ce3-8bc6-1334bed0c633-istio-envoy\") pod \"e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx\" (UID: \"43b38a5b-6579-4ce3-8bc6-1334bed0c633\") " pod="openshift-ingress/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx" Nov 03 14:12:47.907273 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:47.907249 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43b38a5b-6579-4ce3-8bc6-1334bed0c633-istiod-ca-cert\") pod \"e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx\" (UID: \"43b38a5b-6579-4ce3-8bc6-1334bed0c633\") " pod="openshift-ingress/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx" Nov 03 14:12:47.907430 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:47.907309 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43b38a5b-6579-4ce3-8bc6-1334bed0c633-workload-certs\") pod \"e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx\" (UID: \"43b38a5b-6579-4ce3-8bc6-1334bed0c633\") " pod="openshift-ingress/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx" Nov 03 14:12:47.907430 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:47.907374 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43b38a5b-6579-4ce3-8bc6-1334bed0c633-istio-podinfo\") pod \"e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx\" (UID: \"43b38a5b-6579-4ce3-8bc6-1334bed0c633\") " pod="openshift-ingress/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx" Nov 03 14:12:47.907430 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:47.907415 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43b38a5b-6579-4ce3-8bc6-1334bed0c633-istio-data\") pod \"e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx\" (UID: \"43b38a5b-6579-4ce3-8bc6-1334bed0c633\") " pod="openshift-ingress/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx" Nov 03 14:12:47.907631 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:47.907489 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43b38a5b-6579-4ce3-8bc6-1334bed0c633-credential-socket\") pod \"e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx\" (UID: \"43b38a5b-6579-4ce3-8bc6-1334bed0c633\") " pod="openshift-ingress/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx" Nov 03 14:12:47.907694 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:47.907632 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43b38a5b-6579-4ce3-8bc6-1334bed0c633-istio-token\") pod \"e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx\" (UID: \"43b38a5b-6579-4ce3-8bc6-1334bed0c633\") " pod="openshift-ingress/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx" Nov 03 14:12:47.907694 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:47.907679 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccbkv\" (UniqueName: \"kubernetes.io/projected/43b38a5b-6579-4ce3-8bc6-1334bed0c633-kube-api-access-ccbkv\") pod \"e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx\" (UID: \"43b38a5b-6579-4ce3-8bc6-1334bed0c633\") " pod="openshift-ingress/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx" Nov 03 14:12:47.907815 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:47.907740 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43b38a5b-6579-4ce3-8bc6-1334bed0c633-workload-socket\") pod \"e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx\" (UID: \"43b38a5b-6579-4ce3-8bc6-1334bed0c633\") " pod="openshift-ingress/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx" Nov 03 14:12:47.908525 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:47.908354 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43b38a5b-6579-4ce3-8bc6-1334bed0c633-workload-socket\") pod \"e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx\" (UID: \"43b38a5b-6579-4ce3-8bc6-1334bed0c633\") " pod="openshift-ingress/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx" Nov 03 14:12:47.909029 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:47.909009 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43b38a5b-6579-4ce3-8bc6-1334bed0c633-istio-data\") pod \"e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx\" (UID: \"43b38a5b-6579-4ce3-8bc6-1334bed0c633\") " pod="openshift-ingress/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx" Nov 03 14:12:47.909926 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:47.909902 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43b38a5b-6579-4ce3-8bc6-1334bed0c633-istiod-ca-cert\") pod \"e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx\" (UID: \"43b38a5b-6579-4ce3-8bc6-1334bed0c633\") " pod="openshift-ingress/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx" Nov 03 14:12:47.910348 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:47.910325 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43b38a5b-6579-4ce3-8bc6-1334bed0c633-workload-certs\") pod \"e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx\" (UID: \"43b38a5b-6579-4ce3-8bc6-1334bed0c633\") " pod="openshift-ingress/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx" Nov 03 14:12:47.925992 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:47.925954 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-2399b8-openshift-default-dockercfg-7zdsp\"" Nov 03 14:12:47.942574 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:47.926991 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43b38a5b-6579-4ce3-8bc6-1334bed0c633-credential-socket\") pod \"e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx\" (UID: \"43b38a5b-6579-4ce3-8bc6-1334bed0c633\") " pod="openshift-ingress/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx" Nov 03 14:12:47.942574 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:47.931286 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43b38a5b-6579-4ce3-8bc6-1334bed0c633-istio-podinfo\") pod \"e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx\" (UID: \"43b38a5b-6579-4ce3-8bc6-1334bed0c633\") " pod="openshift-ingress/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx" Nov 03 14:12:47.943787 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:47.943755 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43b38a5b-6579-4ce3-8bc6-1334bed0c633-istio-envoy\") pod \"e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx\" (UID: \"43b38a5b-6579-4ce3-8bc6-1334bed0c633\") " pod="openshift-ingress/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx" Nov 03 14:12:47.985355 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:47.985314 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43b38a5b-6579-4ce3-8bc6-1334bed0c633-istio-token\") pod \"e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx\" (UID: \"43b38a5b-6579-4ce3-8bc6-1334bed0c633\") " pod="openshift-ingress/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx" Nov 03 14:12:47.990963 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:47.990926 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccbkv\" (UniqueName: \"kubernetes.io/projected/43b38a5b-6579-4ce3-8bc6-1334bed0c633-kube-api-access-ccbkv\") pod \"e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx\" (UID: \"43b38a5b-6579-4ce3-8bc6-1334bed0c633\") " pod="openshift-ingress/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx" Nov 03 14:12:48.236640 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:48.236552 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx" Nov 03 14:12:48.706537 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:48.706494 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx"] Nov 03 14:12:48.717823 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:12:48.717792 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43b38a5b_6579_4ce3_8bc6_1334bed0c633.slice/crio-b45e0e13195bbcc0665fc06084a7a0100dc5fd56a6d87403257f3552e8472c0e WatchSource:0}: Error finding container b45e0e13195bbcc0665fc06084a7a0100dc5fd56a6d87403257f3552e8472c0e: Status 404 returned error can't find the container with id b45e0e13195bbcc0665fc06084a7a0100dc5fd56a6d87403257f3552e8472c0e Nov 03 14:12:48.721660 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:48.721617 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:12:48.721781 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:48.721708 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:12:48.721781 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:48.721756 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:12:49.496501 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:49.495449 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx" event={"ID":"43b38a5b-6579-4ce3-8bc6-1334bed0c633","Type":"ContainerStarted","Data":"803cdfa8d38fe35898e16efcbeecfedc78669413551e6edefa85b4182461e608"} Nov 03 14:12:49.496501 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:49.495543 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx" event={"ID":"43b38a5b-6579-4ce3-8bc6-1334bed0c633","Type":"ContainerStarted","Data":"b45e0e13195bbcc0665fc06084a7a0100dc5fd56a6d87403257f3552e8472c0e"} Nov 03 14:12:49.715493 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:49.713278 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx" podStartSLOduration=2.713260074 podStartE2EDuration="2.713260074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:12:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:12:49.661930457 +0000 UTC m=+4051.672618722" watchObservedRunningTime="2026-11-03 14:12:49.713260074 +0000 UTC m=+4051.723948317" Nov 03 14:12:50.238535 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:50.238455 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx" Nov 03 14:12:50.256523 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:50.256173 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx" Nov 03 14:12:50.471985 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:50.471940 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l" Nov 03 14:12:50.505237 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:50.505152 5979 generic.go:356] "Generic (PLEG): container finished" podID="8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130" containerID="f4b54bd0c0ab7ba0133f93d78408d399f42483f9789ebf01d23beaeee9dc1760" exitCode=0 Nov 03 14:12:50.505237 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:50.505207 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l" Nov 03 14:12:50.505696 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:50.505244 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l" event={"ID":"8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130","Type":"ContainerDied","Data":"f4b54bd0c0ab7ba0133f93d78408d399f42483f9789ebf01d23beaeee9dc1760"} Nov 03 14:12:50.505696 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:50.505314 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l" event={"ID":"8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130","Type":"ContainerDied","Data":"ae2698aff290b6d452c9940b1839a8e9b0d6e200bf5fe16a6670d2d27b8ca81f"} Nov 03 14:12:50.505696 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:50.505347 5979 scope.go:122] "RemoveContainer" containerID="f4b54bd0c0ab7ba0133f93d78408d399f42483f9789ebf01d23beaeee9dc1760" Nov 03 14:12:50.505696 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:50.505687 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx" Nov 03 14:12:50.508574 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:50.508547 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx" Nov 03 14:12:50.516378 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:50.516354 5979 scope.go:122] "RemoveContainer" containerID="8f530b2c5e7d54c1b2dc8044d2f741f34ed5993a3e584960e01e0546d9f28312" Nov 03 14:12:50.526064 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:50.526044 5979 scope.go:122] "RemoveContainer" containerID="f4b54bd0c0ab7ba0133f93d78408d399f42483f9789ebf01d23beaeee9dc1760" Nov 03 14:12:50.526325 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:12:50.526303 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4b54bd0c0ab7ba0133f93d78408d399f42483f9789ebf01d23beaeee9dc1760\": container with ID starting with f4b54bd0c0ab7ba0133f93d78408d399f42483f9789ebf01d23beaeee9dc1760 not found: ID does not exist" containerID="f4b54bd0c0ab7ba0133f93d78408d399f42483f9789ebf01d23beaeee9dc1760" Nov 03 14:12:50.526421 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:50.526338 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4b54bd0c0ab7ba0133f93d78408d399f42483f9789ebf01d23beaeee9dc1760"} err="failed to get container status \"f4b54bd0c0ab7ba0133f93d78408d399f42483f9789ebf01d23beaeee9dc1760\": rpc error: code = NotFound desc = could not find container \"f4b54bd0c0ab7ba0133f93d78408d399f42483f9789ebf01d23beaeee9dc1760\": container with ID starting with f4b54bd0c0ab7ba0133f93d78408d399f42483f9789ebf01d23beaeee9dc1760 not found: ID does not exist" Nov 03 14:12:50.526421 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:50.526360 5979 scope.go:122] "RemoveContainer" containerID="8f530b2c5e7d54c1b2dc8044d2f741f34ed5993a3e584960e01e0546d9f28312" Nov 03 14:12:50.526751 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:12:50.526730 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f530b2c5e7d54c1b2dc8044d2f741f34ed5993a3e584960e01e0546d9f28312\": container with ID starting with 8f530b2c5e7d54c1b2dc8044d2f741f34ed5993a3e584960e01e0546d9f28312 not found: ID does not exist" containerID="8f530b2c5e7d54c1b2dc8044d2f741f34ed5993a3e584960e01e0546d9f28312" Nov 03 14:12:50.526803 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:50.526759 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f530b2c5e7d54c1b2dc8044d2f741f34ed5993a3e584960e01e0546d9f28312"} err="failed to get container status \"8f530b2c5e7d54c1b2dc8044d2f741f34ed5993a3e584960e01e0546d9f28312\": rpc error: code = NotFound desc = could not find container \"8f530b2c5e7d54c1b2dc8044d2f741f34ed5993a3e584960e01e0546d9f28312\": container with ID starting with 8f530b2c5e7d54c1b2dc8044d2f741f34ed5993a3e584960e01e0546d9f28312 not found: ID does not exist" Nov 03 14:12:50.556596 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:50.556570 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-workload-certs\") pod \"8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130\" (UID: \"8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130\") " Nov 03 14:12:50.556758 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:50.556656 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-istiod-ca-cert\") pod \"8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130\" (UID: \"8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130\") " Nov 03 14:12:50.556758 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:50.556678 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-kube-api-access-6tpml\" (UniqueName: \"kubernetes.io/projected/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-kube-api-access-6tpml\") pod \"8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130\" (UID: \"8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130\") " Nov 03 14:12:50.556758 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:50.556735 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-istio-token\" (UniqueName: \"kubernetes.io/projected/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-istio-token\") pod \"8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130\" (UID: \"8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130\") " Nov 03 14:12:50.556758 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:50.556755 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-istio-data\") pod \"8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130\" (UID: \"8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130\") " Nov 03 14:12:50.556984 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:50.556842 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-istio-envoy\") pod \"8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130\" (UID: \"8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130\") " Nov 03 14:12:50.556984 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:50.556859 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-workload-certs" pod "8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130" (UID: "8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:12:50.556984 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:50.556937 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-credential-socket\") pod \"8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130\" (UID: \"8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130\") " Nov 03 14:12:50.556984 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:50.556965 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-istio-podinfo\") pod \"8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130\" (UID: \"8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130\") " Nov 03 14:12:50.557216 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:50.557001 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-workload-socket\") pod \"8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130\" (UID: \"8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130\") " Nov 03 14:12:50.557216 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:50.557064 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-istio-data" pod "8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130" (UID: "8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:12:50.557340 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:50.557269 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-credential-socket" pod "8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130" (UID: "8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:12:50.557750 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:50.557453 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-workload-socket" pod "8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130" (UID: "8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:12:50.557859 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:50.557761 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:12:50.557859 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:50.557783 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:12:50.557859 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:50.557799 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:12:50.557859 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:50.557814 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:12:50.559183 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:50.559156 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-istio-envoy" pod "8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130" (UID: "8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:12:50.559291 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:50.559216 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-istio-token" pod "8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130" (UID: "8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:12:50.559449 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:50.559433 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-istio-podinfo" pod "8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130" (UID: "8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:12:50.559553 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:50.559434 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-kube-api-access-6tpml" pod "8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130" (UID: "8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130"). InnerVolumeSpecName "kube-api-access-6tpml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:12:50.560295 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:50.557149 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-istiod-ca-cert" pod "8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130" (UID: "8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:12:50.659127 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:50.659096 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:12:50.659127 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:50.659124 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:12:50.659343 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:50.659137 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tpml\" (UniqueName: \"kubernetes.io/projected/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-kube-api-access-6tpml\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:12:50.659343 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:50.659146 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:12:50.659343 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:50.659158 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:12:50.927022 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:50.926987 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l"] Nov 03 14:12:50.934290 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:50.934255 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-f11801-openshift-default-6cf4c95b9-t9g9l"] Nov 03 14:12:52.115823 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:52.115790 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130" path="/var/lib/kubelet/pods/8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130/volumes" Nov 03 14:12:56.123912 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:56.123879 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2399b8-56ff84677f-mvh5j"] Nov 03 14:12:56.124716 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:56.124695 5979 cpu_manager.go:414] "containerMap: removing container" podUID="8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130" containerName="istio-proxy" Nov 03 14:12:56.124716 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:56.124717 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130" containerName="istio-proxy" Nov 03 14:12:56.124883 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:56.124732 5979 cpu_manager.go:414] "containerMap: removing container" podUID="8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130" containerName="istio-proxy" Nov 03 14:12:56.124883 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:56.124738 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130" containerName="istio-proxy" Nov 03 14:12:56.124883 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:56.124787 5979 cpu_manager.go:414] "containerMap: removing container" podUID="5178f75c-2f52-4a5b-bf00-c315bf7e110b" containerName="istio-proxy" Nov 03 14:12:56.124883 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:56.124793 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5178f75c-2f52-4a5b-bf00-c315bf7e110b" containerName="istio-proxy" Nov 03 14:12:56.125049 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:56.124933 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130" containerName="istio-proxy" Nov 03 14:12:56.125049 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:56.124942 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="5178f75c-2f52-4a5b-bf00-c315bf7e110b" containerName="istio-proxy" Nov 03 14:12:56.125049 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:56.124949 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f547ad2-b677-4e3f-bbc0-f6e1dc3ee130" containerName="istio-proxy" Nov 03 14:12:56.128540 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:56.128517 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2399b8-56ff84677f-mvh5j" Nov 03 14:12:56.181802 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:56.181770 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2399b8-56ff84677f-mvh5j"] Nov 03 14:12:56.236026 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:56.235225 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-2399b8\"" Nov 03 14:12:56.236026 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:56.235361 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-2399b8-dockercfg-62snq\"" Nov 03 14:12:56.293497 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:56.267611 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2b41401-7b37-4da6-bbde-d33fd6cd7094-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2399b8-56ff84677f-mvh5j\" (UID: \"d2b41401-7b37-4da6-bbde-d33fd6cd7094\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2399b8-56ff84677f-mvh5j" Nov 03 14:12:56.312943 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:56.312909 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmt8m\" (UniqueName: \"kubernetes.io/projected/d2b41401-7b37-4da6-bbde-d33fd6cd7094-kube-api-access-lmt8m\") pod \"payload-pre-processing-e2e-inf-a-2399b8-56ff84677f-mvh5j\" (UID: \"d2b41401-7b37-4da6-bbde-d33fd6cd7094\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2399b8-56ff84677f-mvh5j" Nov 03 14:12:56.316562 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:56.316534 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2399b8-894f4bdc9-7djnj"] Nov 03 14:12:56.329550 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:56.329517 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2399b8-894f4bdc9-7djnj" Nov 03 14:12:56.342504 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:56.342227 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2399b8-894f4bdc9-7djnj"] Nov 03 14:12:56.414632 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:56.414596 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pdhv\" (UniqueName: \"kubernetes.io/projected/e14c9532-f275-4474-a672-e76765a4d168-kube-api-access-4pdhv\") pod \"payload-processing-e2e-inf-a-2399b8-894f4bdc9-7djnj\" (UID: \"e14c9532-f275-4474-a672-e76765a4d168\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2399b8-894f4bdc9-7djnj" Nov 03 14:12:56.414808 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:56.414738 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2b41401-7b37-4da6-bbde-d33fd6cd7094-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2399b8-56ff84677f-mvh5j\" (UID: \"d2b41401-7b37-4da6-bbde-d33fd6cd7094\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2399b8-56ff84677f-mvh5j" Nov 03 14:12:56.414860 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:56.414826 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmt8m\" (UniqueName: \"kubernetes.io/projected/d2b41401-7b37-4da6-bbde-d33fd6cd7094-kube-api-access-lmt8m\") pod \"payload-pre-processing-e2e-inf-a-2399b8-56ff84677f-mvh5j\" (UID: \"d2b41401-7b37-4da6-bbde-d33fd6cd7094\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2399b8-56ff84677f-mvh5j" Nov 03 14:12:56.414921 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:56.414902 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e14c9532-f275-4474-a672-e76765a4d168-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2399b8-894f4bdc9-7djnj\" (UID: \"e14c9532-f275-4474-a672-e76765a4d168\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2399b8-894f4bdc9-7djnj" Nov 03 14:12:56.415385 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:56.415363 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2b41401-7b37-4da6-bbde-d33fd6cd7094-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2399b8-56ff84677f-mvh5j\" (UID: \"d2b41401-7b37-4da6-bbde-d33fd6cd7094\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2399b8-56ff84677f-mvh5j" Nov 03 14:12:56.468373 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:56.468348 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmt8m\" (UniqueName: \"kubernetes.io/projected/d2b41401-7b37-4da6-bbde-d33fd6cd7094-kube-api-access-lmt8m\") pod \"payload-pre-processing-e2e-inf-a-2399b8-56ff84677f-mvh5j\" (UID: \"d2b41401-7b37-4da6-bbde-d33fd6cd7094\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2399b8-56ff84677f-mvh5j" Nov 03 14:12:56.516452 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:56.516423 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e14c9532-f275-4474-a672-e76765a4d168-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2399b8-894f4bdc9-7djnj\" (UID: \"e14c9532-f275-4474-a672-e76765a4d168\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2399b8-894f4bdc9-7djnj" Nov 03 14:12:56.516640 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:56.516547 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pdhv\" (UniqueName: \"kubernetes.io/projected/e14c9532-f275-4474-a672-e76765a4d168-kube-api-access-4pdhv\") pod \"payload-processing-e2e-inf-a-2399b8-894f4bdc9-7djnj\" (UID: \"e14c9532-f275-4474-a672-e76765a4d168\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2399b8-894f4bdc9-7djnj" Nov 03 14:12:56.517140 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:56.517119 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e14c9532-f275-4474-a672-e76765a4d168-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2399b8-894f4bdc9-7djnj\" (UID: \"e14c9532-f275-4474-a672-e76765a4d168\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2399b8-894f4bdc9-7djnj" Nov 03 14:12:56.574983 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:56.574952 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pdhv\" (UniqueName: \"kubernetes.io/projected/e14c9532-f275-4474-a672-e76765a4d168-kube-api-access-4pdhv\") pod \"payload-processing-e2e-inf-a-2399b8-894f4bdc9-7djnj\" (UID: \"e14c9532-f275-4474-a672-e76765a4d168\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2399b8-894f4bdc9-7djnj" Nov 03 14:12:56.658656 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:56.658619 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2399b8-894f4bdc9-7djnj" Nov 03 14:12:56.743180 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:56.742941 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2399b8-56ff84677f-mvh5j" Nov 03 14:12:57.041027 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:57.040953 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2399b8-894f4bdc9-7djnj"] Nov 03 14:12:57.047645 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:12:57.047619 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode14c9532_f275_4474_a672_e76765a4d168.slice/crio-b5a77a1341d111f28a538dbce087c6aa5bcff02e7d37c0aa841ed45347f82abe WatchSource:0}: Error finding container b5a77a1341d111f28a538dbce087c6aa5bcff02e7d37c0aa841ed45347f82abe: Status 404 returned error can't find the container with id b5a77a1341d111f28a538dbce087c6aa5bcff02e7d37c0aa841ed45347f82abe Nov 03 14:12:57.110696 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:57.110666 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" podUID="d09478af-d618-425d-a4e6-e0889e0f416d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:12:57.154454 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:57.154423 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2399b8-56ff84677f-mvh5j"] Nov 03 14:12:57.161308 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:12:57.161283 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2b41401_7b37_4da6_bbde_d33fd6cd7094.slice/crio-eb39fd1ebbec527b447773b23eaf1af66bea9b9d454ca6ceaa9256fa45255d8e WatchSource:0}: Error finding container eb39fd1ebbec527b447773b23eaf1af66bea9b9d454ca6ceaa9256fa45255d8e: Status 404 returned error can't find the container with id eb39fd1ebbec527b447773b23eaf1af66bea9b9d454ca6ceaa9256fa45255d8e Nov 03 14:12:57.564919 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:57.564878 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2399b8-894f4bdc9-7djnj" event={"ID":"e14c9532-f275-4474-a672-e76765a4d168","Type":"ContainerStarted","Data":"05b51b00ef466a300d34fd3abba5c99be85dd4d208f8ba6d830befd7966cdc48"} Nov 03 14:12:57.565114 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:57.564972 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2399b8-894f4bdc9-7djnj" event={"ID":"e14c9532-f275-4474-a672-e76765a4d168","Type":"ContainerStarted","Data":"b5a77a1341d111f28a538dbce087c6aa5bcff02e7d37c0aa841ed45347f82abe"} Nov 03 14:12:57.565114 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:57.565038 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2399b8-894f4bdc9-7djnj" Nov 03 14:12:57.566777 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:57.566751 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2399b8-56ff84677f-mvh5j" event={"ID":"d2b41401-7b37-4da6-bbde-d33fd6cd7094","Type":"ContainerStarted","Data":"0930521fa609964d2ef759f1f8b27bfa4330af996e5306a6ca06d6f320595173"} Nov 03 14:12:57.567075 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:57.566798 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2399b8-56ff84677f-mvh5j" event={"ID":"d2b41401-7b37-4da6-bbde-d33fd6cd7094","Type":"ContainerStarted","Data":"eb39fd1ebbec527b447773b23eaf1af66bea9b9d454ca6ceaa9256fa45255d8e"} Nov 03 14:12:57.567075 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:57.566847 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2399b8-56ff84677f-mvh5j" Nov 03 14:12:57.674844 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:57.674778 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-2399b8-894f4bdc9-7djnj" podStartSLOduration=1.67476043 podStartE2EDuration="1.67476043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:12:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:12:57.6717077 +0000 UTC m=+4059.682395941" watchObservedRunningTime="2026-11-03 14:12:57.67476043 +0000 UTC m=+4059.685448675" Nov 03 14:12:57.782488 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:12:57.782423 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2399b8-56ff84677f-mvh5j" podStartSLOduration=1.782405199 podStartE2EDuration="1.782405199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:12:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:12:57.778702407 +0000 UTC m=+4059.789390649" watchObservedRunningTime="2026-11-03 14:12:57.782405199 +0000 UTC m=+4059.793093442" Nov 03 14:13:07.121546 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:13:07.118500 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" podUID="d09478af-d618-425d-a4e6-e0889e0f416d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:13:07.235777 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:13:07.235736 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:13:08.587256 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:13:08.587221 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2399b8-56ff84677f-mvh5j" Nov 03 14:13:08.587688 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:13:08.587290 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2399b8-894f4bdc9-7djnj" Nov 03 14:13:10.384488 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:13:10.384447 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:13:17.109911 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:13:17.109874 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" podUID="d09478af-d618-425d-a4e6-e0889e0f416d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:13:21.039673 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:13:21.039638 5979 scope.go:122] "RemoveContainer" containerID="c161ec0e32fb414b336a446ef932ddc1257d83a72ebb9c0ec5e358d3cc9b0818" Nov 03 14:13:21.049727 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:13:21.049702 5979 scope.go:122] "RemoveContainer" containerID="626a215e5a921b1394818f5737f54ae2be66f4e7e7938bb44e558f1cc23bc873" Nov 03 14:13:27.109757 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:13:27.109713 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" podUID="d09478af-d618-425d-a4e6-e0889e0f416d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:13:37.109730 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:13:37.109688 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" podUID="d09478af-d618-425d-a4e6-e0889e0f416d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:13:37.235277 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:13:37.235234 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:13:47.110150 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:13:47.110106 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" podUID="d09478af-d618-425d-a4e6-e0889e0f416d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:13:54.025788 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:13:54.025750 5979 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-e86f9ab11580: secret "agc-cred-e86f9ab11580" not found Nov 03 14:13:54.025788 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:13:54.025780 5979 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-84dcd755bf-c9rnp: secret "agc-cred-e86f9ab11580" not found Nov 03 14:13:54.026235 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:13:54.025838 5979 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/797bcf35-fa9e-47b8-887c-bb8942af277b-provider-credentials podName:797bcf35-fa9e-47b8-887c-bb8942af277b nodeName:}" failed. No retries permitted until 2026-11-03 14:15:56.025819405 +0000 UTC m=+4238.036507639 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/797bcf35-fa9e-47b8-887c-bb8942af277b-provider-credentials") pod "payload-processing-external-model-84dcd755bf-c9rnp" (UID: "797bcf35-fa9e-47b8-887c-bb8942af277b") : secret "agc-cred-e86f9ab11580" not found Nov 03 14:13:57.110188 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:13:57.110147 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" podUID="d09478af-d618-425d-a4e6-e0889e0f416d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:14:00.109555 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:00.109516 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" podUID="d09478af-d618-425d-a4e6-e0889e0f416d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:14:02.806485 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:02.806434 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2"] Nov 03 14:14:02.806917 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:02.806672 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2" podUID="7bb63e83-ca20-4eb2-be94-a2e7fcca2ace" containerName="istio-proxy" containerID="cri-o://5f56c554e07b1b399d247f41352725361a934a45e854b0bf55cdc9eeda295c7e" gracePeriod=30 Nov 03 14:14:06.649940 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:06.649898 5979 patch_prober.go:29] interesting pod/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 14:14:06.650419 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:06.649991 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2" podUID="7bb63e83-ca20-4eb2-be94-a2e7fcca2ace" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 14:14:07.235156 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:14:07.235120 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:14:08.084881 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:08.084802 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2" Nov 03 14:14:08.085270 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:08.085060 5979 generic.go:356] "Generic (PLEG): container finished" podID="7bb63e83-ca20-4eb2-be94-a2e7fcca2ace" containerID="5f56c554e07b1b399d247f41352725361a934a45e854b0bf55cdc9eeda295c7e" exitCode=0 Nov 03 14:14:08.085270 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:08.085162 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2" event={"ID":"7bb63e83-ca20-4eb2-be94-a2e7fcca2ace","Type":"ContainerDied","Data":"5f56c554e07b1b399d247f41352725361a934a45e854b0bf55cdc9eeda295c7e"} Nov 03 14:14:08.085270 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:08.085231 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2" event={"ID":"7bb63e83-ca20-4eb2-be94-a2e7fcca2ace","Type":"ContainerDied","Data":"fc889da35abf1213c1808d5b9ec941aab86059f9f2f42929416e259dfaab11b8"} Nov 03 14:14:08.085270 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:08.085261 5979 scope.go:122] "RemoveContainer" containerID="5f56c554e07b1b399d247f41352725361a934a45e854b0bf55cdc9eeda295c7e" Nov 03 14:14:08.095729 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:08.095710 5979 scope.go:122] "RemoveContainer" containerID="623d2e18efbf02c98355d4aab7c1c1433c17f3c7e260f2cb555ad0cf628211cc" Nov 03 14:14:08.105175 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:08.105156 5979 scope.go:122] "RemoveContainer" containerID="5f56c554e07b1b399d247f41352725361a934a45e854b0bf55cdc9eeda295c7e" Nov 03 14:14:08.105458 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:14:08.105433 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f56c554e07b1b399d247f41352725361a934a45e854b0bf55cdc9eeda295c7e\": container with ID starting with 5f56c554e07b1b399d247f41352725361a934a45e854b0bf55cdc9eeda295c7e not found: ID does not exist" containerID="5f56c554e07b1b399d247f41352725361a934a45e854b0bf55cdc9eeda295c7e" Nov 03 14:14:08.105542 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:08.105479 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f56c554e07b1b399d247f41352725361a934a45e854b0bf55cdc9eeda295c7e"} err="failed to get container status \"5f56c554e07b1b399d247f41352725361a934a45e854b0bf55cdc9eeda295c7e\": rpc error: code = NotFound desc = could not find container \"5f56c554e07b1b399d247f41352725361a934a45e854b0bf55cdc9eeda295c7e\": container with ID starting with 5f56c554e07b1b399d247f41352725361a934a45e854b0bf55cdc9eeda295c7e not found: ID does not exist" Nov 03 14:14:08.105542 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:08.105504 5979 scope.go:122] "RemoveContainer" containerID="623d2e18efbf02c98355d4aab7c1c1433c17f3c7e260f2cb555ad0cf628211cc" Nov 03 14:14:08.105799 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:14:08.105779 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"623d2e18efbf02c98355d4aab7c1c1433c17f3c7e260f2cb555ad0cf628211cc\": container with ID starting with 623d2e18efbf02c98355d4aab7c1c1433c17f3c7e260f2cb555ad0cf628211cc not found: ID does not exist" containerID="623d2e18efbf02c98355d4aab7c1c1433c17f3c7e260f2cb555ad0cf628211cc" Nov 03 14:14:08.105875 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:08.105805 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"623d2e18efbf02c98355d4aab7c1c1433c17f3c7e260f2cb555ad0cf628211cc"} err="failed to get container status \"623d2e18efbf02c98355d4aab7c1c1433c17f3c7e260f2cb555ad0cf628211cc\": rpc error: code = NotFound desc = could not find container \"623d2e18efbf02c98355d4aab7c1c1433c17f3c7e260f2cb555ad0cf628211cc\": container with ID starting with 623d2e18efbf02c98355d4aab7c1c1433c17f3c7e260f2cb555ad0cf628211cc not found: ID does not exist" Nov 03 14:14:08.134778 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:08.134751 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-credential-socket\") pod \"7bb63e83-ca20-4eb2-be94-a2e7fcca2ace\" (UID: \"7bb63e83-ca20-4eb2-be94-a2e7fcca2ace\") " Nov 03 14:14:08.134929 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:08.134796 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-istio-token\" (UniqueName: \"kubernetes.io/projected/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-istio-token\") pod \"7bb63e83-ca20-4eb2-be94-a2e7fcca2ace\" (UID: \"7bb63e83-ca20-4eb2-be94-a2e7fcca2ace\") " Nov 03 14:14:08.134929 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:08.134875 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-workload-certs\") pod \"7bb63e83-ca20-4eb2-be94-a2e7fcca2ace\" (UID: \"7bb63e83-ca20-4eb2-be94-a2e7fcca2ace\") " Nov 03 14:14:08.134929 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:08.134893 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-istio-data\") pod \"7bb63e83-ca20-4eb2-be94-a2e7fcca2ace\" (UID: \"7bb63e83-ca20-4eb2-be94-a2e7fcca2ace\") " Nov 03 14:14:08.135125 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:08.134941 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-istio-podinfo\") pod \"7bb63e83-ca20-4eb2-be94-a2e7fcca2ace\" (UID: \"7bb63e83-ca20-4eb2-be94-a2e7fcca2ace\") " Nov 03 14:14:08.135125 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:08.134974 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-istiod-ca-cert\") pod \"7bb63e83-ca20-4eb2-be94-a2e7fcca2ace\" (UID: \"7bb63e83-ca20-4eb2-be94-a2e7fcca2ace\") " Nov 03 14:14:08.135125 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:08.135026 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-credential-socket" pod "7bb63e83-ca20-4eb2-be94-a2e7fcca2ace" (UID: "7bb63e83-ca20-4eb2-be94-a2e7fcca2ace"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:14:08.135125 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:08.135089 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-istio-envoy\") pod \"7bb63e83-ca20-4eb2-be94-a2e7fcca2ace\" (UID: \"7bb63e83-ca20-4eb2-be94-a2e7fcca2ace\") " Nov 03 14:14:08.135440 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:08.135145 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-workload-socket\") pod \"7bb63e83-ca20-4eb2-be94-a2e7fcca2ace\" (UID: \"7bb63e83-ca20-4eb2-be94-a2e7fcca2ace\") " Nov 03 14:14:08.135440 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:08.135175 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-kube-api-access-jgg9r\" (UniqueName: \"kubernetes.io/projected/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-kube-api-access-jgg9r\") pod \"7bb63e83-ca20-4eb2-be94-a2e7fcca2ace\" (UID: \"7bb63e83-ca20-4eb2-be94-a2e7fcca2ace\") " Nov 03 14:14:08.135440 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:08.135189 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-istio-data" pod "7bb63e83-ca20-4eb2-be94-a2e7fcca2ace" (UID: "7bb63e83-ca20-4eb2-be94-a2e7fcca2ace"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:14:08.135440 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:08.135206 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-workload-certs" pod "7bb63e83-ca20-4eb2-be94-a2e7fcca2ace" (UID: "7bb63e83-ca20-4eb2-be94-a2e7fcca2ace"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:14:08.135440 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:08.135415 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-istiod-ca-cert" pod "7bb63e83-ca20-4eb2-be94-a2e7fcca2ace" (UID: "7bb63e83-ca20-4eb2-be94-a2e7fcca2ace"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:14:08.135768 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:08.135699 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-workload-socket" pod "7bb63e83-ca20-4eb2-be94-a2e7fcca2ace" (UID: "7bb63e83-ca20-4eb2-be94-a2e7fcca2ace"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:14:08.135907 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:08.135886 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:14:08.135997 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:08.135912 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:14:08.135997 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:08.135929 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:14:08.135997 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:08.135950 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:14:08.135997 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:08.135966 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:14:08.137364 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:08.137340 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-istio-token" pod "7bb63e83-ca20-4eb2-be94-a2e7fcca2ace" (UID: "7bb63e83-ca20-4eb2-be94-a2e7fcca2ace"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:14:08.137364 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:08.137351 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-istio-podinfo" pod "7bb63e83-ca20-4eb2-be94-a2e7fcca2ace" (UID: "7bb63e83-ca20-4eb2-be94-a2e7fcca2ace"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:14:08.137693 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:08.137674 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-kube-api-access-jgg9r" pod "7bb63e83-ca20-4eb2-be94-a2e7fcca2ace" (UID: "7bb63e83-ca20-4eb2-be94-a2e7fcca2ace"). InnerVolumeSpecName "kube-api-access-jgg9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:14:08.137763 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:08.137741 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-istio-envoy" pod "7bb63e83-ca20-4eb2-be94-a2e7fcca2ace" (UID: "7bb63e83-ca20-4eb2-be94-a2e7fcca2ace"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:14:08.237672 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:08.237631 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:14:08.237857 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:08.237685 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:14:08.237857 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:08.237702 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgg9r\" (UniqueName: \"kubernetes.io/projected/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-kube-api-access-jgg9r\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:14:08.237857 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:08.237719 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:14:09.093038 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:09.093005 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2" Nov 03 14:14:09.296300 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:09.296264 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2"] Nov 03 14:14:09.299815 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:09.299789 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6375e7-openshift-default-58c4fbfd89-mkwm2"] Nov 03 14:14:10.110029 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:10.109853 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" podUID="d09478af-d618-425d-a4e6-e0889e0f416d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:14:10.118941 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:10.118906 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bb63e83-ca20-4eb2-be94-a2e7fcca2ace" path="/var/lib/kubelet/pods/7bb63e83-ca20-4eb2-be94-a2e7fcca2ace/volumes" Nov 03 14:14:13.506203 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:13.506166 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-78f749b598-6hsx4"] Nov 03 14:14:13.508010 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:13.507928 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" podUID="d09478af-d618-425d-a4e6-e0889e0f416d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:14:13.552536 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:13.552504 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-84dcd755bf-c9rnp"] Nov 03 14:14:15.801165 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:15.801121 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:14:16.483207 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:16.483173 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh"] Nov 03 14:14:16.483877 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:16.483859 5979 cpu_manager.go:414] "containerMap: removing container" podUID="7bb63e83-ca20-4eb2-be94-a2e7fcca2ace" containerName="istio-proxy" Nov 03 14:14:16.483877 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:16.483878 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bb63e83-ca20-4eb2-be94-a2e7fcca2ace" containerName="istio-proxy" Nov 03 14:14:16.484059 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:16.483894 5979 cpu_manager.go:414] "containerMap: removing container" podUID="7bb63e83-ca20-4eb2-be94-a2e7fcca2ace" containerName="istio-proxy" Nov 03 14:14:16.484059 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:16.483902 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bb63e83-ca20-4eb2-be94-a2e7fcca2ace" containerName="istio-proxy" Nov 03 14:14:16.484059 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:16.484036 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bb63e83-ca20-4eb2-be94-a2e7fcca2ace" containerName="istio-proxy" Nov 03 14:14:16.484059 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:16.484050 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bb63e83-ca20-4eb2-be94-a2e7fcca2ace" containerName="istio-proxy" Nov 03 14:14:16.489279 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:16.489259 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh" Nov 03 14:14:16.498565 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:16.498538 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9218884f-ec19-4d1b-8fbc-33fd62abf10f-istio-podinfo\") pod \"e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh\" (UID: \"9218884f-ec19-4d1b-8fbc-33fd62abf10f\") " pod="openshift-ingress/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh" Nov 03 14:14:16.498671 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:16.498651 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9218884f-ec19-4d1b-8fbc-33fd62abf10f-credential-socket\") pod \"e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh\" (UID: \"9218884f-ec19-4d1b-8fbc-33fd62abf10f\") " pod="openshift-ingress/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh" Nov 03 14:14:16.498732 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:16.498683 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhxj8\" (UniqueName: \"kubernetes.io/projected/9218884f-ec19-4d1b-8fbc-33fd62abf10f-kube-api-access-xhxj8\") pod \"e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh\" (UID: \"9218884f-ec19-4d1b-8fbc-33fd62abf10f\") " pod="openshift-ingress/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh" Nov 03 14:14:16.498825 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:16.498805 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9218884f-ec19-4d1b-8fbc-33fd62abf10f-workload-certs\") pod \"e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh\" (UID: \"9218884f-ec19-4d1b-8fbc-33fd62abf10f\") " pod="openshift-ingress/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh" Nov 03 14:14:16.498896 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:16.498860 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9218884f-ec19-4d1b-8fbc-33fd62abf10f-workload-socket\") pod \"e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh\" (UID: \"9218884f-ec19-4d1b-8fbc-33fd62abf10f\") " pod="openshift-ingress/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh" Nov 03 14:14:16.499014 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:16.499000 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9218884f-ec19-4d1b-8fbc-33fd62abf10f-istio-token\") pod \"e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh\" (UID: \"9218884f-ec19-4d1b-8fbc-33fd62abf10f\") " pod="openshift-ingress/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh" Nov 03 14:14:16.499079 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:16.499067 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9218884f-ec19-4d1b-8fbc-33fd62abf10f-istio-data\") pod \"e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh\" (UID: \"9218884f-ec19-4d1b-8fbc-33fd62abf10f\") " pod="openshift-ingress/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh" Nov 03 14:14:16.499179 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:16.499165 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9218884f-ec19-4d1b-8fbc-33fd62abf10f-istiod-ca-cert\") pod \"e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh\" (UID: \"9218884f-ec19-4d1b-8fbc-33fd62abf10f\") " pod="openshift-ingress/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh" Nov 03 14:14:16.499255 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:16.499237 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9218884f-ec19-4d1b-8fbc-33fd62abf10f-istio-envoy\") pod \"e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh\" (UID: \"9218884f-ec19-4d1b-8fbc-33fd62abf10f\") " pod="openshift-ingress/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh" Nov 03 14:14:16.538268 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:16.538224 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-89273860-openshift-default-dockercfg-kxplp\"" Nov 03 14:14:16.600597 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:16.600514 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhxj8\" (UniqueName: \"kubernetes.io/projected/9218884f-ec19-4d1b-8fbc-33fd62abf10f-kube-api-access-xhxj8\") pod \"e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh\" (UID: \"9218884f-ec19-4d1b-8fbc-33fd62abf10f\") " pod="openshift-ingress/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh" Nov 03 14:14:16.600877 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:16.600847 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9218884f-ec19-4d1b-8fbc-33fd62abf10f-workload-certs\") pod \"e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh\" (UID: \"9218884f-ec19-4d1b-8fbc-33fd62abf10f\") " pod="openshift-ingress/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh" Nov 03 14:14:16.601020 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:16.600898 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9218884f-ec19-4d1b-8fbc-33fd62abf10f-workload-socket\") pod \"e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh\" (UID: \"9218884f-ec19-4d1b-8fbc-33fd62abf10f\") " pod="openshift-ingress/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh" Nov 03 14:14:16.601100 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:16.601065 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9218884f-ec19-4d1b-8fbc-33fd62abf10f-istio-token\") pod \"e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh\" (UID: \"9218884f-ec19-4d1b-8fbc-33fd62abf10f\") " pod="openshift-ingress/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh" Nov 03 14:14:16.601190 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:16.601157 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9218884f-ec19-4d1b-8fbc-33fd62abf10f-istio-data\") pod \"e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh\" (UID: \"9218884f-ec19-4d1b-8fbc-33fd62abf10f\") " pod="openshift-ingress/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh" Nov 03 14:14:16.601280 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:16.601263 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9218884f-ec19-4d1b-8fbc-33fd62abf10f-istiod-ca-cert\") pod \"e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh\" (UID: \"9218884f-ec19-4d1b-8fbc-33fd62abf10f\") " pod="openshift-ingress/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh" Nov 03 14:14:16.601365 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:16.601291 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9218884f-ec19-4d1b-8fbc-33fd62abf10f-workload-certs\") pod \"e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh\" (UID: \"9218884f-ec19-4d1b-8fbc-33fd62abf10f\") " pod="openshift-ingress/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh" Nov 03 14:14:16.601365 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:16.601345 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9218884f-ec19-4d1b-8fbc-33fd62abf10f-istio-envoy\") pod \"e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh\" (UID: \"9218884f-ec19-4d1b-8fbc-33fd62abf10f\") " pod="openshift-ingress/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh" Nov 03 14:14:16.601502 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:16.601395 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9218884f-ec19-4d1b-8fbc-33fd62abf10f-istio-podinfo\") pod \"e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh\" (UID: \"9218884f-ec19-4d1b-8fbc-33fd62abf10f\") " pod="openshift-ingress/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh" Nov 03 14:14:16.601502 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:16.601462 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9218884f-ec19-4d1b-8fbc-33fd62abf10f-credential-socket\") pod \"e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh\" (UID: \"9218884f-ec19-4d1b-8fbc-33fd62abf10f\") " pod="openshift-ingress/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh" Nov 03 14:14:16.601916 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:16.601893 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9218884f-ec19-4d1b-8fbc-33fd62abf10f-credential-socket\") pod \"e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh\" (UID: \"9218884f-ec19-4d1b-8fbc-33fd62abf10f\") " pod="openshift-ingress/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh" Nov 03 14:14:16.602184 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:16.602157 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9218884f-ec19-4d1b-8fbc-33fd62abf10f-istio-data\") pod \"e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh\" (UID: \"9218884f-ec19-4d1b-8fbc-33fd62abf10f\") " pod="openshift-ingress/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh" Nov 03 14:14:16.602557 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:16.602535 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9218884f-ec19-4d1b-8fbc-33fd62abf10f-istiod-ca-cert\") pod \"e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh\" (UID: \"9218884f-ec19-4d1b-8fbc-33fd62abf10f\") " pod="openshift-ingress/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh" Nov 03 14:14:16.602741 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:16.602717 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9218884f-ec19-4d1b-8fbc-33fd62abf10f-workload-socket\") pod \"e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh\" (UID: \"9218884f-ec19-4d1b-8fbc-33fd62abf10f\") " pod="openshift-ingress/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh" Nov 03 14:14:16.603345 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:16.603325 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh"] Nov 03 14:14:16.605254 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:16.605227 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9218884f-ec19-4d1b-8fbc-33fd62abf10f-istio-envoy\") pod \"e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh\" (UID: \"9218884f-ec19-4d1b-8fbc-33fd62abf10f\") " pod="openshift-ingress/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh" Nov 03 14:14:16.605416 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:16.605398 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9218884f-ec19-4d1b-8fbc-33fd62abf10f-istio-podinfo\") pod \"e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh\" (UID: \"9218884f-ec19-4d1b-8fbc-33fd62abf10f\") " pod="openshift-ingress/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh" Nov 03 14:14:16.682550 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:16.680431 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9218884f-ec19-4d1b-8fbc-33fd62abf10f-istio-token\") pod \"e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh\" (UID: \"9218884f-ec19-4d1b-8fbc-33fd62abf10f\") " pod="openshift-ingress/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh" Nov 03 14:14:16.749538 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:16.749111 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhxj8\" (UniqueName: \"kubernetes.io/projected/9218884f-ec19-4d1b-8fbc-33fd62abf10f-kube-api-access-xhxj8\") pod \"e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh\" (UID: \"9218884f-ec19-4d1b-8fbc-33fd62abf10f\") " pod="openshift-ingress/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh" Nov 03 14:14:16.805208 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:16.803927 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh" Nov 03 14:14:17.263135 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:17.263103 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh"] Nov 03 14:14:17.269714 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:14:17.269684 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9218884f_ec19_4d1b_8fbc_33fd62abf10f.slice/crio-99b0cad9dbb53f62bf408086e6c789df688e422a91253453367990461ac63b88 WatchSource:0}: Error finding container 99b0cad9dbb53f62bf408086e6c789df688e422a91253453367990461ac63b88: Status 404 returned error can't find the container with id 99b0cad9dbb53f62bf408086e6c789df688e422a91253453367990461ac63b88 Nov 03 14:14:17.272242 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:17.272206 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:14:17.272350 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:17.272275 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:14:17.272350 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:17.272305 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:14:18.152160 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:18.152121 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh" event={"ID":"9218884f-ec19-4d1b-8fbc-33fd62abf10f","Type":"ContainerStarted","Data":"99a0e937e244158693b60a6360059e1ce5d511e18de7f9143aeabf1b3cf5c25d"} Nov 03 14:14:18.152659 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:18.152179 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh" event={"ID":"9218884f-ec19-4d1b-8fbc-33fd62abf10f","Type":"ContainerStarted","Data":"99b0cad9dbb53f62bf408086e6c789df688e422a91253453367990461ac63b88"} Nov 03 14:14:18.507050 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:18.506959 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" podUID="d09478af-d618-425d-a4e6-e0889e0f416d" containerName="payload-processing" containerID="cri-o://471394e3f03a245f6eee2c31b1e7c1fa191108205c4be0d8a4d2fb1fe5214fd8" gracePeriod=25 Nov 03 14:14:18.553604 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:18.553570 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-84dcd755bf-c9rnp" podUID="797bcf35-fa9e-47b8-887c-bb8942af277b" containerName="payload-processing" containerID="cri-o://56b7486b76019cadddbbc0091d32603ae9dd4e92673b0d4e72b583b7ac271548" gracePeriod=25 Nov 03 14:14:18.806094 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:18.806061 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh" Nov 03 14:14:18.810711 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:18.810673 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" Nov 03 14:14:18.812208 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:18.812188 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh" Nov 03 14:14:18.846691 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:18.846664 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d09478af-d618-425d-a4e6-e0889e0f416d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d09478af-d618-425d-a4e6-e0889e0f416d-routing-overlay-volume\") pod \"d09478af-d618-425d-a4e6-e0889e0f416d\" (UID: \"d09478af-d618-425d-a4e6-e0889e0f416d\") " Nov 03 14:14:18.846812 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:18.846724 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d09478af-d618-425d-a4e6-e0889e0f416d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d09478af-d618-425d-a4e6-e0889e0f416d-plugins-config-volume\") pod \"d09478af-d618-425d-a4e6-e0889e0f416d\" (UID: \"d09478af-d618-425d-a4e6-e0889e0f416d\") " Nov 03 14:14:18.846994 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:18.846933 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d09478af-d618-425d-a4e6-e0889e0f416d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d09478af-d618-425d-a4e6-e0889e0f416d-provider-credentials\") pod \"d09478af-d618-425d-a4e6-e0889e0f416d\" (UID: \"d09478af-d618-425d-a4e6-e0889e0f416d\") " Nov 03 14:14:18.846994 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:18.846973 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d09478af-d618-425d-a4e6-e0889e0f416d-kube-api-access-5t82t\" (UniqueName: \"kubernetes.io/projected/d09478af-d618-425d-a4e6-e0889e0f416d-kube-api-access-5t82t\") pod \"d09478af-d618-425d-a4e6-e0889e0f416d\" (UID: \"d09478af-d618-425d-a4e6-e0889e0f416d\") " Nov 03 14:14:18.847172 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:18.847039 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d09478af-d618-425d-a4e6-e0889e0f416d-routing-overlay-volume" pod "d09478af-d618-425d-a4e6-e0889e0f416d" (UID: "d09478af-d618-425d-a4e6-e0889e0f416d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:14:18.847330 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:18.847300 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d09478af-d618-425d-a4e6-e0889e0f416d-plugins-config-volume" pod "d09478af-d618-425d-a4e6-e0889e0f416d" (UID: "d09478af-d618-425d-a4e6-e0889e0f416d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:14:18.847614 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:18.847591 5979 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d09478af-d618-425d-a4e6-e0889e0f416d-routing-overlay-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:14:18.847676 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:18.847621 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d09478af-d618-425d-a4e6-e0889e0f416d-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:14:18.849149 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:18.849123 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d09478af-d618-425d-a4e6-e0889e0f416d-provider-credentials" pod "d09478af-d618-425d-a4e6-e0889e0f416d" (UID: "d09478af-d618-425d-a4e6-e0889e0f416d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:14:18.849277 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:18.849258 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d09478af-d618-425d-a4e6-e0889e0f416d-kube-api-access-5t82t" pod "d09478af-d618-425d-a4e6-e0889e0f416d" (UID: "d09478af-d618-425d-a4e6-e0889e0f416d"). InnerVolumeSpecName "kube-api-access-5t82t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:14:18.852388 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:18.852369 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-84dcd755bf-c9rnp" Nov 03 14:14:18.920048 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:18.919993 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh" podStartSLOduration=2.919972058 podStartE2EDuration="2.919972058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:14:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:14:18.307712374 +0000 UTC m=+4140.318400616" watchObservedRunningTime="2026-11-03 14:14:18.919972058 +0000 UTC m=+4140.930660300" Nov 03 14:14:18.948906 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:18.948871 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/797bcf35-fa9e-47b8-887c-bb8942af277b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/797bcf35-fa9e-47b8-887c-bb8942af277b-provider-credentials\") pod \"797bcf35-fa9e-47b8-887c-bb8942af277b\" (UID: \"797bcf35-fa9e-47b8-887c-bb8942af277b\") " Nov 03 14:14:18.949106 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:18.948959 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/797bcf35-fa9e-47b8-887c-bb8942af277b-kube-api-access-b859h\" (UniqueName: \"kubernetes.io/projected/797bcf35-fa9e-47b8-887c-bb8942af277b-kube-api-access-b859h\") pod \"797bcf35-fa9e-47b8-887c-bb8942af277b\" (UID: \"797bcf35-fa9e-47b8-887c-bb8942af277b\") " Nov 03 14:14:18.949106 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:18.949019 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/797bcf35-fa9e-47b8-887c-bb8942af277b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/797bcf35-fa9e-47b8-887c-bb8942af277b-routing-overlay-volume\") pod \"797bcf35-fa9e-47b8-887c-bb8942af277b\" (UID: \"797bcf35-fa9e-47b8-887c-bb8942af277b\") " Nov 03 14:14:18.949106 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:18.949058 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/797bcf35-fa9e-47b8-887c-bb8942af277b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/797bcf35-fa9e-47b8-887c-bb8942af277b-plugins-config-volume\") pod \"797bcf35-fa9e-47b8-887c-bb8942af277b\" (UID: \"797bcf35-fa9e-47b8-887c-bb8942af277b\") " Nov 03 14:14:18.949618 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:18.949596 5979 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d09478af-d618-425d-a4e6-e0889e0f416d-provider-credentials\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:14:18.949618 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:18.949620 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5t82t\" (UniqueName: \"kubernetes.io/projected/d09478af-d618-425d-a4e6-e0889e0f416d-kube-api-access-5t82t\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:14:18.949847 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:18.949819 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/797bcf35-fa9e-47b8-887c-bb8942af277b-plugins-config-volume" pod "797bcf35-fa9e-47b8-887c-bb8942af277b" (UID: "797bcf35-fa9e-47b8-887c-bb8942af277b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:14:18.949921 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:18.949904 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/797bcf35-fa9e-47b8-887c-bb8942af277b-routing-overlay-volume" pod "797bcf35-fa9e-47b8-887c-bb8942af277b" (UID: "797bcf35-fa9e-47b8-887c-bb8942af277b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:14:18.951709 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:18.951687 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/797bcf35-fa9e-47b8-887c-bb8942af277b-provider-credentials" pod "797bcf35-fa9e-47b8-887c-bb8942af277b" (UID: "797bcf35-fa9e-47b8-887c-bb8942af277b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:14:18.951807 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:18.951715 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/797bcf35-fa9e-47b8-887c-bb8942af277b-kube-api-access-b859h" pod "797bcf35-fa9e-47b8-887c-bb8942af277b" (UID: "797bcf35-fa9e-47b8-887c-bb8942af277b"). InnerVolumeSpecName "kube-api-access-b859h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:14:19.050635 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:19.050559 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b859h\" (UniqueName: \"kubernetes.io/projected/797bcf35-fa9e-47b8-887c-bb8942af277b-kube-api-access-b859h\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:14:19.050635 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:19.050590 5979 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/797bcf35-fa9e-47b8-887c-bb8942af277b-routing-overlay-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:14:19.050635 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:19.050600 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/797bcf35-fa9e-47b8-887c-bb8942af277b-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:14:19.050635 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:19.050613 5979 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/797bcf35-fa9e-47b8-887c-bb8942af277b-provider-credentials\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:14:19.160295 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:19.160261 5979 generic.go:356] "Generic (PLEG): container finished" podID="d09478af-d618-425d-a4e6-e0889e0f416d" containerID="471394e3f03a245f6eee2c31b1e7c1fa191108205c4be0d8a4d2fb1fe5214fd8" exitCode=0 Nov 03 14:14:19.160782 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:19.160307 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" Nov 03 14:14:19.160782 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:19.160336 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" event={"ID":"d09478af-d618-425d-a4e6-e0889e0f416d","Type":"ContainerDied","Data":"471394e3f03a245f6eee2c31b1e7c1fa191108205c4be0d8a4d2fb1fe5214fd8"} Nov 03 14:14:19.160782 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:19.160394 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-78f749b598-6hsx4" event={"ID":"d09478af-d618-425d-a4e6-e0889e0f416d","Type":"ContainerDied","Data":"e6c8ba1640c087f8af1422528ef613b3c8e4de7e315601a3a51d48429053408d"} Nov 03 14:14:19.160782 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:19.160424 5979 scope.go:122] "RemoveContainer" containerID="471394e3f03a245f6eee2c31b1e7c1fa191108205c4be0d8a4d2fb1fe5214fd8" Nov 03 14:14:19.161787 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:19.161764 5979 generic.go:356] "Generic (PLEG): container finished" podID="797bcf35-fa9e-47b8-887c-bb8942af277b" containerID="56b7486b76019cadddbbc0091d32603ae9dd4e92673b0d4e72b583b7ac271548" exitCode=0 Nov 03 14:14:19.162977 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:19.162956 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-84dcd755bf-c9rnp" Nov 03 14:14:19.163088 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:19.163067 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-84dcd755bf-c9rnp" event={"ID":"797bcf35-fa9e-47b8-887c-bb8942af277b","Type":"ContainerDied","Data":"56b7486b76019cadddbbc0091d32603ae9dd4e92673b0d4e72b583b7ac271548"} Nov 03 14:14:19.163153 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:19.163115 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-84dcd755bf-c9rnp" event={"ID":"797bcf35-fa9e-47b8-887c-bb8942af277b","Type":"ContainerDied","Data":"cb7e262a4fb23ae615389a67eb479700bffec883797e67fe5e7297351237178d"} Nov 03 14:14:19.163153 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:19.163146 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh" Nov 03 14:14:19.169528 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:19.169507 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh" Nov 03 14:14:19.172325 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:19.172303 5979 scope.go:122] "RemoveContainer" containerID="471394e3f03a245f6eee2c31b1e7c1fa191108205c4be0d8a4d2fb1fe5214fd8" Nov 03 14:14:19.172632 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:14:19.172615 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"471394e3f03a245f6eee2c31b1e7c1fa191108205c4be0d8a4d2fb1fe5214fd8\": container with ID starting with 471394e3f03a245f6eee2c31b1e7c1fa191108205c4be0d8a4d2fb1fe5214fd8 not found: ID does not exist" containerID="471394e3f03a245f6eee2c31b1e7c1fa191108205c4be0d8a4d2fb1fe5214fd8" Nov 03 14:14:19.172719 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:19.172640 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"471394e3f03a245f6eee2c31b1e7c1fa191108205c4be0d8a4d2fb1fe5214fd8"} err="failed to get container status \"471394e3f03a245f6eee2c31b1e7c1fa191108205c4be0d8a4d2fb1fe5214fd8\": rpc error: code = NotFound desc = could not find container \"471394e3f03a245f6eee2c31b1e7c1fa191108205c4be0d8a4d2fb1fe5214fd8\": container with ID starting with 471394e3f03a245f6eee2c31b1e7c1fa191108205c4be0d8a4d2fb1fe5214fd8 not found: ID does not exist" Nov 03 14:14:19.172719 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:19.172655 5979 scope.go:122] "RemoveContainer" containerID="56b7486b76019cadddbbc0091d32603ae9dd4e92673b0d4e72b583b7ac271548" Nov 03 14:14:19.181789 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:19.181769 5979 scope.go:122] "RemoveContainer" containerID="56b7486b76019cadddbbc0091d32603ae9dd4e92673b0d4e72b583b7ac271548" Nov 03 14:14:19.182044 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:14:19.182025 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56b7486b76019cadddbbc0091d32603ae9dd4e92673b0d4e72b583b7ac271548\": container with ID starting with 56b7486b76019cadddbbc0091d32603ae9dd4e92673b0d4e72b583b7ac271548 not found: ID does not exist" containerID="56b7486b76019cadddbbc0091d32603ae9dd4e92673b0d4e72b583b7ac271548" Nov 03 14:14:19.182094 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:19.182051 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56b7486b76019cadddbbc0091d32603ae9dd4e92673b0d4e72b583b7ac271548"} err="failed to get container status \"56b7486b76019cadddbbc0091d32603ae9dd4e92673b0d4e72b583b7ac271548\": rpc error: code = NotFound desc = could not find container \"56b7486b76019cadddbbc0091d32603ae9dd4e92673b0d4e72b583b7ac271548\": container with ID starting with 56b7486b76019cadddbbc0091d32603ae9dd4e92673b0d4e72b583b7ac271548 not found: ID does not exist" Nov 03 14:14:19.318566 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:19.318450 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-84dcd755bf-c9rnp"] Nov 03 14:14:19.323299 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:19.322960 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-84dcd755bf-c9rnp"] Nov 03 14:14:19.532724 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:19.532685 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-78f749b598-6hsx4"] Nov 03 14:14:19.535207 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:19.535183 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-78f749b598-6hsx4"] Nov 03 14:14:20.116636 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:20.116549 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="797bcf35-fa9e-47b8-887c-bb8942af277b" path="/var/lib/kubelet/pods/797bcf35-fa9e-47b8-887c-bb8942af277b/volumes" Nov 03 14:14:20.117066 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:20.117050 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d09478af-d618-425d-a4e6-e0889e0f416d" path="/var/lib/kubelet/pods/d09478af-d618-425d-a4e6-e0889e0f416d/volumes" Nov 03 14:14:21.091999 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:21.091965 5979 scope.go:122] "RemoveContainer" containerID="9c39bdd958b2057337ec83a0df00aa13e6e3b1960360aefa588a518cf8a741ec" Nov 03 14:14:24.934525 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:24.934466 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-89273860-7d6d454d77-dz5mp"] Nov 03 14:14:24.936024 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:24.935996 5979 cpu_manager.go:414] "containerMap: removing container" podUID="d09478af-d618-425d-a4e6-e0889e0f416d" containerName="payload-processing" Nov 03 14:14:24.936184 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:24.936169 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d09478af-d618-425d-a4e6-e0889e0f416d" containerName="payload-processing" Nov 03 14:14:24.936302 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:24.936291 5979 cpu_manager.go:414] "containerMap: removing container" podUID="797bcf35-fa9e-47b8-887c-bb8942af277b" containerName="payload-processing" Nov 03 14:14:24.936377 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:24.936367 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="797bcf35-fa9e-47b8-887c-bb8942af277b" containerName="payload-processing" Nov 03 14:14:24.936686 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:24.936669 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="797bcf35-fa9e-47b8-887c-bb8942af277b" containerName="payload-processing" Nov 03 14:14:24.936818 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:24.936807 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="d09478af-d618-425d-a4e6-e0889e0f416d" containerName="payload-processing" Nov 03 14:14:24.966450 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:24.966417 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-89273860-7d6d454d77-dz5mp" Nov 03 14:14:24.994935 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:24.994902 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-89273860-7d6d454d77-dz5mp"] Nov 03 14:14:25.038182 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:25.038150 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-89273860\"" Nov 03 14:14:25.038348 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:25.038249 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-89273860-dockercfg-4qlzn\"" Nov 03 14:14:25.053574 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:25.053536 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtb7q\" (UniqueName: \"kubernetes.io/projected/1ee90e7b-21ab-4625-94c9-373c51dc9b73-kube-api-access-jtb7q\") pod \"payload-processing-e2e-mt-89273860-7d6d454d77-dz5mp\" (UID: \"1ee90e7b-21ab-4625-94c9-373c51dc9b73\") " pod="openshift-ingress/payload-processing-e2e-mt-89273860-7d6d454d77-dz5mp" Nov 03 14:14:25.053779 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:25.053756 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ee90e7b-21ab-4625-94c9-373c51dc9b73-plugins-config-volume\") pod \"payload-processing-e2e-mt-89273860-7d6d454d77-dz5mp\" (UID: \"1ee90e7b-21ab-4625-94c9-373c51dc9b73\") " pod="openshift-ingress/payload-processing-e2e-mt-89273860-7d6d454d77-dz5mp" Nov 03 14:14:25.154794 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:25.154760 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ee90e7b-21ab-4625-94c9-373c51dc9b73-plugins-config-volume\") pod \"payload-processing-e2e-mt-89273860-7d6d454d77-dz5mp\" (UID: \"1ee90e7b-21ab-4625-94c9-373c51dc9b73\") " pod="openshift-ingress/payload-processing-e2e-mt-89273860-7d6d454d77-dz5mp" Nov 03 14:14:25.154971 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:25.154883 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtb7q\" (UniqueName: \"kubernetes.io/projected/1ee90e7b-21ab-4625-94c9-373c51dc9b73-kube-api-access-jtb7q\") pod \"payload-processing-e2e-mt-89273860-7d6d454d77-dz5mp\" (UID: \"1ee90e7b-21ab-4625-94c9-373c51dc9b73\") " pod="openshift-ingress/payload-processing-e2e-mt-89273860-7d6d454d77-dz5mp" Nov 03 14:14:25.155512 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:25.155489 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ee90e7b-21ab-4625-94c9-373c51dc9b73-plugins-config-volume\") pod \"payload-processing-e2e-mt-89273860-7d6d454d77-dz5mp\" (UID: \"1ee90e7b-21ab-4625-94c9-373c51dc9b73\") " pod="openshift-ingress/payload-processing-e2e-mt-89273860-7d6d454d77-dz5mp" Nov 03 14:14:25.208509 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:25.208410 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtb7q\" (UniqueName: \"kubernetes.io/projected/1ee90e7b-21ab-4625-94c9-373c51dc9b73-kube-api-access-jtb7q\") pod \"payload-processing-e2e-mt-89273860-7d6d454d77-dz5mp\" (UID: \"1ee90e7b-21ab-4625-94c9-373c51dc9b73\") " pod="openshift-ingress/payload-processing-e2e-mt-89273860-7d6d454d77-dz5mp" Nov 03 14:14:25.287097 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:25.287060 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-89273860-7d6d454d77-dz5mp" Nov 03 14:14:25.658679 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:25.658650 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-89273860-7d6d454d77-dz5mp"] Nov 03 14:14:25.666658 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:14:25.666625 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ee90e7b_21ab_4625_94c9_373c51dc9b73.slice/crio-47d1a563fb22237e4a35cc7dc9298324cac1e8802cf760f750c54c36b856127b WatchSource:0}: Error finding container 47d1a563fb22237e4a35cc7dc9298324cac1e8802cf760f750c54c36b856127b: Status 404 returned error can't find the container with id 47d1a563fb22237e4a35cc7dc9298324cac1e8802cf760f750c54c36b856127b Nov 03 14:14:26.213879 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:26.213841 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-89273860-7d6d454d77-dz5mp" event={"ID":"1ee90e7b-21ab-4625-94c9-373c51dc9b73","Type":"ContainerStarted","Data":"35a1af80e53af21118063019add43901a25f9ac5eed5a1d89036d106b3090e45"} Nov 03 14:14:26.214296 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:26.213895 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-89273860-7d6d454d77-dz5mp" event={"ID":"1ee90e7b-21ab-4625-94c9-373c51dc9b73","Type":"ContainerStarted","Data":"47d1a563fb22237e4a35cc7dc9298324cac1e8802cf760f750c54c36b856127b"} Nov 03 14:14:26.214296 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:26.213921 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-89273860-7d6d454d77-dz5mp" Nov 03 14:14:26.327063 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:26.327009 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-89273860-7d6d454d77-dz5mp" podStartSLOduration=2.326995572 podStartE2EDuration="2.326995572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:14:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:14:26.323951651 +0000 UTC m=+4148.334639874" watchObservedRunningTime="2026-11-03 14:14:26.326995572 +0000 UTC m=+4148.337683813" Nov 03 14:14:37.223129 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:37.223101 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-89273860-7d6d454d77-dz5mp" Nov 03 14:14:37.235761 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:14:37.235727 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:14:51.660730 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:51.660694 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b6c8c7-6f6b7ddfc6-mxvmw"] Nov 03 14:14:51.661286 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:51.660900 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b6c8c7-6f6b7ddfc6-mxvmw" podUID="f05f949d-904f-419e-90e3-ba5a6ff30ecf" containerName="maas-api" containerID="cri-o://67cb04c898ae0b43cb515e51d8365a8f0ded0bcdc8039592a6085396fbcf4210" gracePeriod=30 Nov 03 14:14:51.965445 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:51.965420 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b6c8c7-6f6b7ddfc6-mxvmw" Nov 03 14:14:52.121813 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:52.120284 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f05f949d-904f-419e-90e3-ba5a6ff30ecf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f05f949d-904f-419e-90e3-ba5a6ff30ecf-maas-api-tls\") pod \"f05f949d-904f-419e-90e3-ba5a6ff30ecf\" (UID: \"f05f949d-904f-419e-90e3-ba5a6ff30ecf\") " Nov 03 14:14:52.121813 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:52.120353 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f05f949d-904f-419e-90e3-ba5a6ff30ecf-kube-api-access-4jxhm\" (UniqueName: \"kubernetes.io/projected/f05f949d-904f-419e-90e3-ba5a6ff30ecf-kube-api-access-4jxhm\") pod \"f05f949d-904f-419e-90e3-ba5a6ff30ecf\" (UID: \"f05f949d-904f-419e-90e3-ba5a6ff30ecf\") " Nov 03 14:14:52.121813 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:52.120585 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f05f949d-904f-419e-90e3-ba5a6ff30ecf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f05f949d-904f-419e-90e3-ba5a6ff30ecf-openshift-service-ca\") pod \"f05f949d-904f-419e-90e3-ba5a6ff30ecf\" (UID: \"f05f949d-904f-419e-90e3-ba5a6ff30ecf\") " Nov 03 14:14:52.121813 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:52.120661 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f05f949d-904f-419e-90e3-ba5a6ff30ecf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f05f949d-904f-419e-90e3-ba5a6ff30ecf-metrics-certs\") pod \"f05f949d-904f-419e-90e3-ba5a6ff30ecf\" (UID: \"f05f949d-904f-419e-90e3-ba5a6ff30ecf\") " Nov 03 14:14:52.121813 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:52.121440 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f05f949d-904f-419e-90e3-ba5a6ff30ecf-openshift-service-ca" pod "f05f949d-904f-419e-90e3-ba5a6ff30ecf" (UID: "f05f949d-904f-419e-90e3-ba5a6ff30ecf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:14:52.122420 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:52.122380 5979 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f05f949d-904f-419e-90e3-ba5a6ff30ecf-openshift-service-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:14:52.126288 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:52.126240 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f05f949d-904f-419e-90e3-ba5a6ff30ecf-kube-api-access-4jxhm" pod "f05f949d-904f-419e-90e3-ba5a6ff30ecf" (UID: "f05f949d-904f-419e-90e3-ba5a6ff30ecf"). InnerVolumeSpecName "kube-api-access-4jxhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:14:52.128679 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:52.128616 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f05f949d-904f-419e-90e3-ba5a6ff30ecf-maas-api-tls" pod "f05f949d-904f-419e-90e3-ba5a6ff30ecf" (UID: "f05f949d-904f-419e-90e3-ba5a6ff30ecf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:14:52.133536 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:52.133493 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f05f949d-904f-419e-90e3-ba5a6ff30ecf-metrics-certs" pod "f05f949d-904f-419e-90e3-ba5a6ff30ecf" (UID: "f05f949d-904f-419e-90e3-ba5a6ff30ecf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:14:52.225747 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:52.225593 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f05f949d-904f-419e-90e3-ba5a6ff30ecf-maas-api-tls\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:14:52.225747 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:52.225647 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jxhm\" (UniqueName: \"kubernetes.io/projected/f05f949d-904f-419e-90e3-ba5a6ff30ecf-kube-api-access-4jxhm\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:14:52.225747 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:52.225667 5979 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f05f949d-904f-419e-90e3-ba5a6ff30ecf-metrics-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:14:52.392862 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:52.392826 5979 generic.go:356] "Generic (PLEG): container finished" podID="f05f949d-904f-419e-90e3-ba5a6ff30ecf" containerID="67cb04c898ae0b43cb515e51d8365a8f0ded0bcdc8039592a6085396fbcf4210" exitCode=0 Nov 03 14:14:52.393041 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:52.392872 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b6c8c7-6f6b7ddfc6-mxvmw" Nov 03 14:14:52.393041 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:52.392909 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b6c8c7-6f6b7ddfc6-mxvmw" event={"ID":"f05f949d-904f-419e-90e3-ba5a6ff30ecf","Type":"ContainerDied","Data":"67cb04c898ae0b43cb515e51d8365a8f0ded0bcdc8039592a6085396fbcf4210"} Nov 03 14:14:52.393041 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:52.392972 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b6c8c7-6f6b7ddfc6-mxvmw" event={"ID":"f05f949d-904f-419e-90e3-ba5a6ff30ecf","Type":"ContainerDied","Data":"fbb9ae7d3f4f0defc5d63351945bb43fd4702a28a091cc8eebf6d89d4b8fee49"} Nov 03 14:14:52.393041 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:52.392996 5979 scope.go:122] "RemoveContainer" containerID="67cb04c898ae0b43cb515e51d8365a8f0ded0bcdc8039592a6085396fbcf4210" Nov 03 14:14:52.403381 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:52.403361 5979 scope.go:122] "RemoveContainer" containerID="67cb04c898ae0b43cb515e51d8365a8f0ded0bcdc8039592a6085396fbcf4210" Nov 03 14:14:52.403679 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:14:52.403660 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67cb04c898ae0b43cb515e51d8365a8f0ded0bcdc8039592a6085396fbcf4210\": container with ID starting with 67cb04c898ae0b43cb515e51d8365a8f0ded0bcdc8039592a6085396fbcf4210 not found: ID does not exist" containerID="67cb04c898ae0b43cb515e51d8365a8f0ded0bcdc8039592a6085396fbcf4210" Nov 03 14:14:52.403734 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:52.403689 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67cb04c898ae0b43cb515e51d8365a8f0ded0bcdc8039592a6085396fbcf4210"} err="failed to get container status \"67cb04c898ae0b43cb515e51d8365a8f0ded0bcdc8039592a6085396fbcf4210\": rpc error: code = NotFound desc = could not find container \"67cb04c898ae0b43cb515e51d8365a8f0ded0bcdc8039592a6085396fbcf4210\": container with ID starting with 67cb04c898ae0b43cb515e51d8365a8f0ded0bcdc8039592a6085396fbcf4210 not found: ID does not exist" Nov 03 14:14:52.551564 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:52.551462 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b6c8c7-6f6b7ddfc6-mxvmw"] Nov 03 14:14:52.597149 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:52.597106 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b6c8c7-6f6b7ddfc6-mxvmw"] Nov 03 14:14:54.116486 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:14:54.116447 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f05f949d-904f-419e-90e3-ba5a6ff30ecf" path="/var/lib/kubelet/pods/f05f949d-904f-419e-90e3-ba5a6ff30ecf/volumes" Nov 03 14:15:00.201496 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.197626 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895255-cg2mt"] Nov 03 14:15:00.201496 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.198836 5979 cpu_manager.go:414] "containerMap: removing container" podUID="f05f949d-904f-419e-90e3-ba5a6ff30ecf" containerName="maas-api" Nov 03 14:15:00.201496 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.198859 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f05f949d-904f-419e-90e3-ba5a6ff30ecf" containerName="maas-api" Nov 03 14:15:00.201496 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.199016 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="f05f949d-904f-419e-90e3-ba5a6ff30ecf" containerName="maas-api" Nov 03 14:15:00.211506 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.211422 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895255-cg2mt" Nov 03 14:15:00.212348 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.212321 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2399b8-29895255-wmw9v"] Nov 03 14:15:00.222603 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.218183 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2399b8-29895255-wmw9v" Nov 03 14:15:00.222603 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.221681 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895255-h595k"] Nov 03 14:15:00.227171 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.227146 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895255-h595k" Nov 03 14:15:00.233029 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.232936 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895255-hqw2d"] Nov 03 14:15:00.240284 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.240261 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-367381-29895255-spbd5"] Nov 03 14:15:00.240421 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.240394 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895255-hqw2d" Nov 03 14:15:00.262493 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.260112 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-367381-29895255-spbd5" Nov 03 14:15:00.294513 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.292711 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895255-xkfz2"] Nov 03 14:15:00.294513 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.293666 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 14:15:00.303099 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.302268 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/018cf2a9-b246-4ed8-9328-e301c3b10abc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-2399b8-29895255-wmw9v\" (UID: \"018cf2a9-b246-4ed8-9328-e301c3b10abc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2399b8-29895255-wmw9v" Nov 03 14:15:00.303099 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.302332 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmlsl\" (UniqueName: \"kubernetes.io/projected/018cf2a9-b246-4ed8-9328-e301c3b10abc-kube-api-access-qmlsl\") pod \"maas-api-key-cleanup-e2e-inf-a-2399b8-29895255-wmw9v\" (UID: \"018cf2a9-b246-4ed8-9328-e301c3b10abc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2399b8-29895255-wmw9v" Nov 03 14:15:00.303099 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.302442 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbdad710-0252-4bd1-aba0-6bd737f94bac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895255-cg2mt\" (UID: \"fbdad710-0252-4bd1-aba0-6bd737f94bac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895255-cg2mt" Nov 03 14:15:00.303099 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.302934 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5hn7\" (UniqueName: \"kubernetes.io/projected/fbdad710-0252-4bd1-aba0-6bd737f94bac-kube-api-access-t5hn7\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895255-cg2mt\" (UID: \"fbdad710-0252-4bd1-aba0-6bd737f94bac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895255-cg2mt" Nov 03 14:15:00.310116 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.310086 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895255-xkfz2" Nov 03 14:15:00.337520 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.333530 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895255-cg2mt"] Nov 03 14:15:00.337520 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.336143 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2399b8-29895255-wmw9v"] Nov 03 14:15:00.344854 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.344419 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895255-h595k"] Nov 03 14:15:00.411543 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.385075 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-367381-29895255-spbd5"] Nov 03 14:15:00.411543 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.403977 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5hn7\" (UniqueName: \"kubernetes.io/projected/fbdad710-0252-4bd1-aba0-6bd737f94bac-kube-api-access-t5hn7\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895255-cg2mt\" (UID: \"fbdad710-0252-4bd1-aba0-6bd737f94bac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895255-cg2mt" Nov 03 14:15:00.411543 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.404067 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f89b9146-737e-450c-823e-d298a93cf055-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895255-hqw2d\" (UID: \"f89b9146-737e-450c-823e-d298a93cf055\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895255-hqw2d" Nov 03 14:15:00.411543 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.404102 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/018cf2a9-b246-4ed8-9328-e301c3b10abc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-2399b8-29895255-wmw9v\" (UID: \"018cf2a9-b246-4ed8-9328-e301c3b10abc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2399b8-29895255-wmw9v" Nov 03 14:15:00.411543 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.404134 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95cebd5e-01ba-4b81-8d1d-024b4fe773e8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895255-h595k\" (UID: \"95cebd5e-01ba-4b81-8d1d-024b4fe773e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895255-h595k" Nov 03 14:15:00.411543 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.404168 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmlsl\" (UniqueName: \"kubernetes.io/projected/018cf2a9-b246-4ed8-9328-e301c3b10abc-kube-api-access-qmlsl\") pod \"maas-api-key-cleanup-e2e-inf-a-2399b8-29895255-wmw9v\" (UID: \"018cf2a9-b246-4ed8-9328-e301c3b10abc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2399b8-29895255-wmw9v" Nov 03 14:15:00.411543 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.404231 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcs9j\" (UniqueName: \"kubernetes.io/projected/f89b9146-737e-450c-823e-d298a93cf055-kube-api-access-kcs9j\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895255-hqw2d\" (UID: \"f89b9146-737e-450c-823e-d298a93cf055\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895255-hqw2d" Nov 03 14:15:00.411543 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.404258 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4a9f29a-7aaa-4d75-9fb5-e34a8fdca48c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-42c707-29895255-xkfz2\" (UID: \"e4a9f29a-7aaa-4d75-9fb5-e34a8fdca48c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895255-xkfz2" Nov 03 14:15:00.411543 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.404303 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hltnt\" (UniqueName: \"kubernetes.io/projected/e4a9f29a-7aaa-4d75-9fb5-e34a8fdca48c-kube-api-access-hltnt\") pod \"maas-api-key-cleanup-e2e-worker-w0-42c707-29895255-xkfz2\" (UID: \"e4a9f29a-7aaa-4d75-9fb5-e34a8fdca48c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895255-xkfz2" Nov 03 14:15:00.411543 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.404367 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbdad710-0252-4bd1-aba0-6bd737f94bac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895255-cg2mt\" (UID: \"fbdad710-0252-4bd1-aba0-6bd737f94bac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895255-cg2mt" Nov 03 14:15:00.411543 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.404413 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px99p\" (UniqueName: \"kubernetes.io/projected/5e7900ee-f17f-45e8-87f7-f58389e788f6-kube-api-access-px99p\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-367381-29895255-spbd5\" (UID: \"5e7900ee-f17f-45e8-87f7-f58389e788f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-367381-29895255-spbd5" Nov 03 14:15:00.417493 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.412679 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfrdh\" (UniqueName: \"kubernetes.io/projected/95cebd5e-01ba-4b81-8d1d-024b4fe773e8-kube-api-access-nfrdh\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895255-h595k\" (UID: \"95cebd5e-01ba-4b81-8d1d-024b4fe773e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895255-h595k" Nov 03 14:15:00.417493 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.412751 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e7900ee-f17f-45e8-87f7-f58389e788f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-367381-29895255-spbd5\" (UID: \"5e7900ee-f17f-45e8-87f7-f58389e788f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-367381-29895255-spbd5" Nov 03 14:15:00.419277 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.417966 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/018cf2a9-b246-4ed8-9328-e301c3b10abc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-2399b8-29895255-wmw9v\" (UID: \"018cf2a9-b246-4ed8-9328-e301c3b10abc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2399b8-29895255-wmw9v" Nov 03 14:15:00.419277 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.418720 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbdad710-0252-4bd1-aba0-6bd737f94bac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895255-cg2mt\" (UID: \"fbdad710-0252-4bd1-aba0-6bd737f94bac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895255-cg2mt" Nov 03 14:15:00.559553 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.555651 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfrdh\" (UniqueName: \"kubernetes.io/projected/95cebd5e-01ba-4b81-8d1d-024b4fe773e8-kube-api-access-nfrdh\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895255-h595k\" (UID: \"95cebd5e-01ba-4b81-8d1d-024b4fe773e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895255-h595k" Nov 03 14:15:00.559553 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.555713 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e7900ee-f17f-45e8-87f7-f58389e788f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-367381-29895255-spbd5\" (UID: \"5e7900ee-f17f-45e8-87f7-f58389e788f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-367381-29895255-spbd5" Nov 03 14:15:00.559553 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.555819 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f89b9146-737e-450c-823e-d298a93cf055-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895255-hqw2d\" (UID: \"f89b9146-737e-450c-823e-d298a93cf055\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895255-hqw2d" Nov 03 14:15:00.559553 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.555852 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95cebd5e-01ba-4b81-8d1d-024b4fe773e8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895255-h595k\" (UID: \"95cebd5e-01ba-4b81-8d1d-024b4fe773e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895255-h595k" Nov 03 14:15:00.559553 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.555927 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcs9j\" (UniqueName: \"kubernetes.io/projected/f89b9146-737e-450c-823e-d298a93cf055-kube-api-access-kcs9j\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895255-hqw2d\" (UID: \"f89b9146-737e-450c-823e-d298a93cf055\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895255-hqw2d" Nov 03 14:15:00.559553 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.555962 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4a9f29a-7aaa-4d75-9fb5-e34a8fdca48c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-42c707-29895255-xkfz2\" (UID: \"e4a9f29a-7aaa-4d75-9fb5-e34a8fdca48c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895255-xkfz2" Nov 03 14:15:00.559553 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.556004 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hltnt\" (UniqueName: \"kubernetes.io/projected/e4a9f29a-7aaa-4d75-9fb5-e34a8fdca48c-kube-api-access-hltnt\") pod \"maas-api-key-cleanup-e2e-worker-w0-42c707-29895255-xkfz2\" (UID: \"e4a9f29a-7aaa-4d75-9fb5-e34a8fdca48c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895255-xkfz2" Nov 03 14:15:00.559553 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.556089 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px99p\" (UniqueName: \"kubernetes.io/projected/5e7900ee-f17f-45e8-87f7-f58389e788f6-kube-api-access-px99p\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-367381-29895255-spbd5\" (UID: \"5e7900ee-f17f-45e8-87f7-f58389e788f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-367381-29895255-spbd5" Nov 03 14:15:00.559553 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.557336 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e7900ee-f17f-45e8-87f7-f58389e788f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-367381-29895255-spbd5\" (UID: \"5e7900ee-f17f-45e8-87f7-f58389e788f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-367381-29895255-spbd5" Nov 03 14:15:00.587497 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.584110 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95cebd5e-01ba-4b81-8d1d-024b4fe773e8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895255-h595k\" (UID: \"95cebd5e-01ba-4b81-8d1d-024b4fe773e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895255-h595k" Nov 03 14:15:00.587497 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.585090 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4a9f29a-7aaa-4d75-9fb5-e34a8fdca48c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-42c707-29895255-xkfz2\" (UID: \"e4a9f29a-7aaa-4d75-9fb5-e34a8fdca48c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895255-xkfz2" Nov 03 14:15:00.629616 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.629575 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f89b9146-737e-450c-823e-d298a93cf055-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895255-hqw2d\" (UID: \"f89b9146-737e-450c-823e-d298a93cf055\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895255-hqw2d" Nov 03 14:15:00.654493 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.651765 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px99p\" (UniqueName: \"kubernetes.io/projected/5e7900ee-f17f-45e8-87f7-f58389e788f6-kube-api-access-px99p\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-367381-29895255-spbd5\" (UID: \"5e7900ee-f17f-45e8-87f7-f58389e788f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-367381-29895255-spbd5" Nov 03 14:15:00.719541 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.708787 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895255-xkfz2"] Nov 03 14:15:00.741490 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.725850 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hltnt\" (UniqueName: \"kubernetes.io/projected/e4a9f29a-7aaa-4d75-9fb5-e34a8fdca48c-kube-api-access-hltnt\") pod \"maas-api-key-cleanup-e2e-worker-w0-42c707-29895255-xkfz2\" (UID: \"e4a9f29a-7aaa-4d75-9fb5-e34a8fdca48c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895255-xkfz2" Nov 03 14:15:00.741490 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.737918 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895255-hqw2d"] Nov 03 14:15:00.747556 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.743314 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcs9j\" (UniqueName: \"kubernetes.io/projected/f89b9146-737e-450c-823e-d298a93cf055-kube-api-access-kcs9j\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895255-hqw2d\" (UID: \"f89b9146-737e-450c-823e-d298a93cf055\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895255-hqw2d" Nov 03 14:15:00.754416 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.754152 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895255-hqw2d" Nov 03 14:15:00.759290 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.759217 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfrdh\" (UniqueName: \"kubernetes.io/projected/95cebd5e-01ba-4b81-8d1d-024b4fe773e8-kube-api-access-nfrdh\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895255-h595k\" (UID: \"95cebd5e-01ba-4b81-8d1d-024b4fe773e8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895255-h595k" Nov 03 14:15:00.823812 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.823698 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5hn7\" (UniqueName: \"kubernetes.io/projected/fbdad710-0252-4bd1-aba0-6bd737f94bac-kube-api-access-t5hn7\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895255-cg2mt\" (UID: \"fbdad710-0252-4bd1-aba0-6bd737f94bac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895255-cg2mt" Nov 03 14:15:00.824198 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.824146 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmlsl\" (UniqueName: \"kubernetes.io/projected/018cf2a9-b246-4ed8-9328-e301c3b10abc-kube-api-access-qmlsl\") pod \"maas-api-key-cleanup-e2e-inf-a-2399b8-29895255-wmw9v\" (UID: \"018cf2a9-b246-4ed8-9328-e301c3b10abc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2399b8-29895255-wmw9v" Nov 03 14:15:00.830490 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.828731 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-367381-29895255-spbd5" Nov 03 14:15:00.833703 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.833676 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895255-cg2mt" Nov 03 14:15:00.847514 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.843813 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2399b8-29895255-wmw9v" Nov 03 14:15:00.943304 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.943192 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895255-xkfz2" Nov 03 14:15:01.014593 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:00.975686 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895255-h595k" Nov 03 14:15:01.936528 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:01.936492 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895255-hqw2d"] Nov 03 14:15:01.938050 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:15:01.937994 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf89b9146_737e_450c_823e_d298a93cf055.slice/crio-622f01750a0d3562faccab46e41cd91c8d3b1d866135126503da78492bc3c0eb WatchSource:0}: Error finding container 622f01750a0d3562faccab46e41cd91c8d3b1d866135126503da78492bc3c0eb: Status 404 returned error can't find the container with id 622f01750a0d3562faccab46e41cd91c8d3b1d866135126503da78492bc3c0eb Nov 03 14:15:02.225363 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:02.195537 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-367381-29895255-spbd5"] Nov 03 14:15:02.446198 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:02.446162 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895255-cg2mt"] Nov 03 14:15:02.746307 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:02.745150 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895255-h595k"] Nov 03 14:15:02.828500 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:02.828167 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895255-hqw2d" event={"ID":"f89b9146-737e-450c-823e-d298a93cf055","Type":"ContainerStarted","Data":"2e2f4409efce05ec96b0a802b19e24a00bc94ef85ed43bda78b8d06e3a1f7f71"} Nov 03 14:15:02.828500 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:02.828220 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895255-hqw2d" event={"ID":"f89b9146-737e-450c-823e-d298a93cf055","Type":"ContainerStarted","Data":"622f01750a0d3562faccab46e41cd91c8d3b1d866135126503da78492bc3c0eb"} Nov 03 14:15:02.839908 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:02.838111 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895255-xkfz2"] Nov 03 14:15:02.839908 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:02.838803 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895255-h595k" event={"ID":"95cebd5e-01ba-4b81-8d1d-024b4fe773e8","Type":"ContainerStarted","Data":"8b46f778a58229ba19ef91b53d0ac8c2f1dbd621e5456b8560b13319f2fc86f1"} Nov 03 14:15:02.841812 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:02.841779 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-367381-29895255-spbd5" event={"ID":"5e7900ee-f17f-45e8-87f7-f58389e788f6","Type":"ContainerStarted","Data":"8289a109b806ba46ea443a565f1e459cc6178269ba650eabcf7adf32764a4c1d"} Nov 03 14:15:02.878515 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:02.865581 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895255-cg2mt" event={"ID":"fbdad710-0252-4bd1-aba0-6bd737f94bac","Type":"ContainerStarted","Data":"d755aebdd6f1dffcbd3489d47daa22e3c44ba43686dc560bd1b7e08098be7131"} Nov 03 14:15:02.941341 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:02.941217 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895255-hqw2d" podStartSLOduration=2.941194422 podStartE2EDuration="2.941194422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:15:02.937241766 +0000 UTC m=+4184.947930000" watchObservedRunningTime="2026-11-03 14:15:02.941194422 +0000 UTC m=+4184.951882664" Nov 03 14:15:02.978593 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:02.977920 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2399b8-29895255-wmw9v"] Nov 03 14:15:03.889058 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:03.888922 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895255-h595k" event={"ID":"95cebd5e-01ba-4b81-8d1d-024b4fe773e8","Type":"ContainerStarted","Data":"79620aa61579fd87257e16076e30a571f7c51ab23f04c66c95afb3d2d9d64266"} Nov 03 14:15:03.891384 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:03.891298 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-367381-29895255-spbd5" event={"ID":"5e7900ee-f17f-45e8-87f7-f58389e788f6","Type":"ContainerStarted","Data":"d1e9193aeb384aabd43bba96eb885cac9459e4d28dfb36fe94b2b30e3c748885"} Nov 03 14:15:03.891622 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:03.891488 5979 scope.go:122] "RemoveContainer" containerID="d1e9193aeb384aabd43bba96eb885cac9459e4d28dfb36fe94b2b30e3c748885" Nov 03 14:15:03.894949 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:03.894924 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895255-cg2mt" event={"ID":"fbdad710-0252-4bd1-aba0-6bd737f94bac","Type":"ContainerStarted","Data":"305c043b8602922c64e02234e4142fc8e938dc3d4f59d635baa2178988a3fe14"} Nov 03 14:15:03.895215 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:03.895198 5979 scope.go:122] "RemoveContainer" containerID="305c043b8602922c64e02234e4142fc8e938dc3d4f59d635baa2178988a3fe14" Nov 03 14:15:03.897412 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:03.897140 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895255-xkfz2" event={"ID":"e4a9f29a-7aaa-4d75-9fb5-e34a8fdca48c","Type":"ContainerStarted","Data":"f75d4d04cae47e32766d3d3775d19bc340b9f291de66cd9ca912c02a953f2cc6"} Nov 03 14:15:03.897412 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:03.897183 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895255-xkfz2" event={"ID":"e4a9f29a-7aaa-4d75-9fb5-e34a8fdca48c","Type":"ContainerStarted","Data":"a74cd6c6d906000c49338b60390d034c8128629e7e9dd3a477ebb700fec94bb7"} Nov 03 14:15:03.898995 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:03.898919 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2399b8-29895255-wmw9v" event={"ID":"018cf2a9-b246-4ed8-9328-e301c3b10abc","Type":"ContainerStarted","Data":"96c61c0ae211f13bd6c3296f519e053e1305004419bdeaf1bf291e1f1d032408"} Nov 03 14:15:03.898995 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:03.898962 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2399b8-29895255-wmw9v" event={"ID":"018cf2a9-b246-4ed8-9328-e301c3b10abc","Type":"ContainerStarted","Data":"56964862c66d9755f54e6502c27a0167f2ae39cf060073235f9ef858a35dc5d7"} Nov 03 14:15:03.901155 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:03.901130 5979 generic.go:356] "Generic (PLEG): container finished" podID="f89b9146-737e-450c-823e-d298a93cf055" containerID="2e2f4409efce05ec96b0a802b19e24a00bc94ef85ed43bda78b8d06e3a1f7f71" exitCode=60 Nov 03 14:15:03.901272 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:03.901195 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895255-hqw2d" event={"ID":"f89b9146-737e-450c-823e-d298a93cf055","Type":"ContainerDied","Data":"2e2f4409efce05ec96b0a802b19e24a00bc94ef85ed43bda78b8d06e3a1f7f71"} Nov 03 14:15:03.901692 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:03.901673 5979 scope.go:122] "RemoveContainer" containerID="2e2f4409efce05ec96b0a802b19e24a00bc94ef85ed43bda78b8d06e3a1f7f71" Nov 03 14:15:04.001650 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:04.001591 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895255-h595k" podStartSLOduration=4.001576998 podStartE2EDuration="4.001576998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:15:03.998088684 +0000 UTC m=+4186.008776926" watchObservedRunningTime="2026-11-03 14:15:04.001576998 +0000 UTC m=+4186.012265252" Nov 03 14:15:04.394960 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:04.394846 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2399b8-29895255-wmw9v" podStartSLOduration=4.394825735 podStartE2EDuration="4.394825735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:15:04.394520756 +0000 UTC m=+4186.405208998" watchObservedRunningTime="2026-11-03 14:15:04.394825735 +0000 UTC m=+4186.405513983" Nov 03 14:15:04.512908 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:04.512841 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895255-xkfz2" podStartSLOduration=4.512819834 podStartE2EDuration="4.512819834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:15:04.509299906 +0000 UTC m=+4186.519988151" watchObservedRunningTime="2026-11-03 14:15:04.512819834 +0000 UTC m=+4186.523508077" Nov 03 14:15:04.910345 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:04.910304 5979 generic.go:356] "Generic (PLEG): container finished" podID="018cf2a9-b246-4ed8-9328-e301c3b10abc" containerID="96c61c0ae211f13bd6c3296f519e053e1305004419bdeaf1bf291e1f1d032408" exitCode=60 Nov 03 14:15:04.910563 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:04.910396 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2399b8-29895255-wmw9v" event={"ID":"018cf2a9-b246-4ed8-9328-e301c3b10abc","Type":"ContainerDied","Data":"96c61c0ae211f13bd6c3296f519e053e1305004419bdeaf1bf291e1f1d032408"} Nov 03 14:15:04.910783 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:04.910763 5979 scope.go:122] "RemoveContainer" containerID="96c61c0ae211f13bd6c3296f519e053e1305004419bdeaf1bf291e1f1d032408" Nov 03 14:15:04.912597 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:04.912572 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895255-hqw2d" event={"ID":"f89b9146-737e-450c-823e-d298a93cf055","Type":"ContainerStarted","Data":"fc7e09d772357f9b5a4239ee8834ce07484e0c94292f035f1af9902dc6f36617"} Nov 03 14:15:04.914333 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:04.914228 5979 generic.go:356] "Generic (PLEG): container finished" podID="95cebd5e-01ba-4b81-8d1d-024b4fe773e8" containerID="79620aa61579fd87257e16076e30a571f7c51ab23f04c66c95afb3d2d9d64266" exitCode=60 Nov 03 14:15:04.914333 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:04.914305 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895255-h595k" event={"ID":"95cebd5e-01ba-4b81-8d1d-024b4fe773e8","Type":"ContainerDied","Data":"79620aa61579fd87257e16076e30a571f7c51ab23f04c66c95afb3d2d9d64266"} Nov 03 14:15:04.914615 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:04.914593 5979 scope.go:122] "RemoveContainer" containerID="79620aa61579fd87257e16076e30a571f7c51ab23f04c66c95afb3d2d9d64266" Nov 03 14:15:04.916291 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:04.916260 5979 generic.go:356] "Generic (PLEG): container finished" podID="5e7900ee-f17f-45e8-87f7-f58389e788f6" containerID="d1e9193aeb384aabd43bba96eb885cac9459e4d28dfb36fe94b2b30e3c748885" exitCode=60 Nov 03 14:15:04.916442 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:04.916422 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-367381-29895255-spbd5" event={"ID":"5e7900ee-f17f-45e8-87f7-f58389e788f6","Type":"ContainerDied","Data":"d1e9193aeb384aabd43bba96eb885cac9459e4d28dfb36fe94b2b30e3c748885"} Nov 03 14:15:04.916560 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:04.916492 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-367381-29895255-spbd5" event={"ID":"5e7900ee-f17f-45e8-87f7-f58389e788f6","Type":"ContainerStarted","Data":"ac80a8df455df4bedc32372fd1bee95008d9e6b1a8a5bc16dc96fda4096a5a08"} Nov 03 14:15:04.919108 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:04.919082 5979 generic.go:356] "Generic (PLEG): container finished" podID="fbdad710-0252-4bd1-aba0-6bd737f94bac" containerID="305c043b8602922c64e02234e4142fc8e938dc3d4f59d635baa2178988a3fe14" exitCode=60 Nov 03 14:15:04.919222 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:04.919178 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895255-cg2mt" event={"ID":"fbdad710-0252-4bd1-aba0-6bd737f94bac","Type":"ContainerDied","Data":"305c043b8602922c64e02234e4142fc8e938dc3d4f59d635baa2178988a3fe14"} Nov 03 14:15:04.919616 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:04.919238 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895255-cg2mt" event={"ID":"fbdad710-0252-4bd1-aba0-6bd737f94bac","Type":"ContainerStarted","Data":"3a56cd3e84d3197cc393ca080f773167ae04f1bfccf6cd665ffa7e532119b746"} Nov 03 14:15:04.922354 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:04.922330 5979 generic.go:356] "Generic (PLEG): container finished" podID="e4a9f29a-7aaa-4d75-9fb5-e34a8fdca48c" containerID="f75d4d04cae47e32766d3d3775d19bc340b9f291de66cd9ca912c02a953f2cc6" exitCode=60 Nov 03 14:15:04.922463 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:04.922400 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895255-xkfz2" event={"ID":"e4a9f29a-7aaa-4d75-9fb5-e34a8fdca48c","Type":"ContainerDied","Data":"f75d4d04cae47e32766d3d3775d19bc340b9f291de66cd9ca912c02a953f2cc6"} Nov 03 14:15:04.922651 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:04.922635 5979 scope.go:122] "RemoveContainer" containerID="f75d4d04cae47e32766d3d3775d19bc340b9f291de66cd9ca912c02a953f2cc6" Nov 03 14:15:05.137768 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:05.137712 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-367381-29895255-spbd5" podStartSLOduration=5.137697678 podStartE2EDuration="5.137697678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:15:05.135952837 +0000 UTC m=+4187.146641060" watchObservedRunningTime="2026-11-03 14:15:05.137697678 +0000 UTC m=+4187.148385973" Nov 03 14:15:05.598255 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:05.598192 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895255-cg2mt" podStartSLOduration=5.598176884 podStartE2EDuration="5.598176884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:15:05.594708224 +0000 UTC m=+4187.605396465" watchObservedRunningTime="2026-11-03 14:15:05.598176884 +0000 UTC m=+4187.608865126" Nov 03 14:15:05.932934 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:05.932895 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895255-h595k" event={"ID":"95cebd5e-01ba-4b81-8d1d-024b4fe773e8","Type":"ContainerStarted","Data":"41725f081a3c6a14fcae282f5cd4c96b414927e6a74dd2c596e55805c44018b2"} Nov 03 14:15:05.934853 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:05.934827 5979 generic.go:356] "Generic (PLEG): container finished" podID="5e7900ee-f17f-45e8-87f7-f58389e788f6" containerID="ac80a8df455df4bedc32372fd1bee95008d9e6b1a8a5bc16dc96fda4096a5a08" exitCode=60 Nov 03 14:15:05.934994 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:05.934908 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-367381-29895255-spbd5" event={"ID":"5e7900ee-f17f-45e8-87f7-f58389e788f6","Type":"ContainerDied","Data":"ac80a8df455df4bedc32372fd1bee95008d9e6b1a8a5bc16dc96fda4096a5a08"} Nov 03 14:15:05.934994 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:05.934962 5979 scope.go:122] "RemoveContainer" containerID="d1e9193aeb384aabd43bba96eb885cac9459e4d28dfb36fe94b2b30e3c748885" Nov 03 14:15:05.935154 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:05.935126 5979 scope.go:122] "RemoveContainer" containerID="ac80a8df455df4bedc32372fd1bee95008d9e6b1a8a5bc16dc96fda4096a5a08" Nov 03 14:15:05.935398 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:15:05.935369 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-367381-29895255-spbd5_odh-ai-gateway-infra(5e7900ee-f17f-45e8-87f7-f58389e788f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-367381-29895255-spbd5" podUID="5e7900ee-f17f-45e8-87f7-f58389e788f6" Nov 03 14:15:05.936951 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:05.936932 5979 generic.go:356] "Generic (PLEG): container finished" podID="fbdad710-0252-4bd1-aba0-6bd737f94bac" containerID="3a56cd3e84d3197cc393ca080f773167ae04f1bfccf6cd665ffa7e532119b746" exitCode=60 Nov 03 14:15:05.937053 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:05.937006 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895255-cg2mt" event={"ID":"fbdad710-0252-4bd1-aba0-6bd737f94bac","Type":"ContainerDied","Data":"3a56cd3e84d3197cc393ca080f773167ae04f1bfccf6cd665ffa7e532119b746"} Nov 03 14:15:05.937200 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:05.937185 5979 scope.go:122] "RemoveContainer" containerID="3a56cd3e84d3197cc393ca080f773167ae04f1bfccf6cd665ffa7e532119b746" Nov 03 14:15:05.937442 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:15:05.937407 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895255-cg2mt_odh-ai-gateway-infra(fbdad710-0252-4bd1-aba0-6bd737f94bac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895255-cg2mt" podUID="fbdad710-0252-4bd1-aba0-6bd737f94bac" Nov 03 14:15:05.938967 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:05.938941 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895255-xkfz2" event={"ID":"e4a9f29a-7aaa-4d75-9fb5-e34a8fdca48c","Type":"ContainerStarted","Data":"da59147df9654e80021179060caa4eae7b68eb19d511d8b4478f9c2570cb870d"} Nov 03 14:15:05.940767 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:05.940743 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2399b8-29895255-wmw9v" event={"ID":"018cf2a9-b246-4ed8-9328-e301c3b10abc","Type":"ContainerStarted","Data":"78cdab2aa9edfa83846cc30697ca91546b9c4e1ace81558a088371aa0c290282"} Nov 03 14:15:05.942894 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:05.942868 5979 generic.go:356] "Generic (PLEG): container finished" podID="f89b9146-737e-450c-823e-d298a93cf055" containerID="fc7e09d772357f9b5a4239ee8834ce07484e0c94292f035f1af9902dc6f36617" exitCode=60 Nov 03 14:15:05.942984 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:05.942905 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895255-hqw2d" event={"ID":"f89b9146-737e-450c-823e-d298a93cf055","Type":"ContainerDied","Data":"fc7e09d772357f9b5a4239ee8834ce07484e0c94292f035f1af9902dc6f36617"} Nov 03 14:15:05.943161 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:05.943139 5979 scope.go:122] "RemoveContainer" containerID="fc7e09d772357f9b5a4239ee8834ce07484e0c94292f035f1af9902dc6f36617" Nov 03 14:15:05.943375 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:15:05.943356 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895255-hqw2d_odh-ai-gateway-infra(f89b9146-737e-450c-823e-d298a93cf055)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895255-hqw2d" podUID="f89b9146-737e-450c-823e-d298a93cf055" Nov 03 14:15:05.949197 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:05.949177 5979 scope.go:122] "RemoveContainer" containerID="305c043b8602922c64e02234e4142fc8e938dc3d4f59d635baa2178988a3fe14" Nov 03 14:15:05.963219 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:05.963201 5979 scope.go:122] "RemoveContainer" containerID="2e2f4409efce05ec96b0a802b19e24a00bc94ef85ed43bda78b8d06e3a1f7f71" Nov 03 14:15:06.960729 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:06.960684 5979 generic.go:356] "Generic (PLEG): container finished" podID="95cebd5e-01ba-4b81-8d1d-024b4fe773e8" containerID="41725f081a3c6a14fcae282f5cd4c96b414927e6a74dd2c596e55805c44018b2" exitCode=60 Nov 03 14:15:06.961248 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:06.960787 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895255-h595k" event={"ID":"95cebd5e-01ba-4b81-8d1d-024b4fe773e8","Type":"ContainerDied","Data":"41725f081a3c6a14fcae282f5cd4c96b414927e6a74dd2c596e55805c44018b2"} Nov 03 14:15:06.961248 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:06.960944 5979 scope.go:122] "RemoveContainer" containerID="79620aa61579fd87257e16076e30a571f7c51ab23f04c66c95afb3d2d9d64266" Nov 03 14:15:06.961248 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:06.961103 5979 scope.go:122] "RemoveContainer" containerID="41725f081a3c6a14fcae282f5cd4c96b414927e6a74dd2c596e55805c44018b2" Nov 03 14:15:06.961511 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:15:06.961452 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895255-h595k_odh-ai-gateway-infra(95cebd5e-01ba-4b81-8d1d-024b4fe773e8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895255-h595k" podUID="95cebd5e-01ba-4b81-8d1d-024b4fe773e8" Nov 03 14:15:06.963406 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:06.963251 5979 scope.go:122] "RemoveContainer" containerID="ac80a8df455df4bedc32372fd1bee95008d9e6b1a8a5bc16dc96fda4096a5a08" Nov 03 14:15:06.963599 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:15:06.963577 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-367381-29895255-spbd5_odh-ai-gateway-infra(5e7900ee-f17f-45e8-87f7-f58389e788f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-367381-29895255-spbd5" podUID="5e7900ee-f17f-45e8-87f7-f58389e788f6" Nov 03 14:15:06.969063 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:06.965365 5979 scope.go:122] "RemoveContainer" containerID="3a56cd3e84d3197cc393ca080f773167ae04f1bfccf6cd665ffa7e532119b746" Nov 03 14:15:06.969063 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:15:06.965760 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895255-cg2mt_odh-ai-gateway-infra(fbdad710-0252-4bd1-aba0-6bd737f94bac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895255-cg2mt" podUID="fbdad710-0252-4bd1-aba0-6bd737f94bac" Nov 03 14:15:06.969063 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:06.967060 5979 generic.go:356] "Generic (PLEG): container finished" podID="e4a9f29a-7aaa-4d75-9fb5-e34a8fdca48c" containerID="da59147df9654e80021179060caa4eae7b68eb19d511d8b4478f9c2570cb870d" exitCode=60 Nov 03 14:15:06.969063 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:06.967141 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895255-xkfz2" event={"ID":"e4a9f29a-7aaa-4d75-9fb5-e34a8fdca48c","Type":"ContainerDied","Data":"da59147df9654e80021179060caa4eae7b68eb19d511d8b4478f9c2570cb870d"} Nov 03 14:15:06.969063 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:06.967304 5979 scope.go:122] "RemoveContainer" containerID="da59147df9654e80021179060caa4eae7b68eb19d511d8b4478f9c2570cb870d" Nov 03 14:15:06.969063 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:15:06.967545 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-42c707-29895255-xkfz2_odh-ai-gateway-infra(e4a9f29a-7aaa-4d75-9fb5-e34a8fdca48c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895255-xkfz2" podUID="e4a9f29a-7aaa-4d75-9fb5-e34a8fdca48c" Nov 03 14:15:06.969343 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:06.969096 5979 generic.go:356] "Generic (PLEG): container finished" podID="018cf2a9-b246-4ed8-9328-e301c3b10abc" containerID="78cdab2aa9edfa83846cc30697ca91546b9c4e1ace81558a088371aa0c290282" exitCode=60 Nov 03 14:15:06.969343 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:06.969316 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2399b8-29895255-wmw9v" event={"ID":"018cf2a9-b246-4ed8-9328-e301c3b10abc","Type":"ContainerDied","Data":"78cdab2aa9edfa83846cc30697ca91546b9c4e1ace81558a088371aa0c290282"} Nov 03 14:15:06.969442 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:06.969432 5979 scope.go:122] "RemoveContainer" containerID="78cdab2aa9edfa83846cc30697ca91546b9c4e1ace81558a088371aa0c290282" Nov 03 14:15:06.969867 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:15:06.969708 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-2399b8-29895255-wmw9v_odh-ai-gateway-infra(018cf2a9-b246-4ed8-9328-e301c3b10abc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2399b8-29895255-wmw9v" podUID="018cf2a9-b246-4ed8-9328-e301c3b10abc" Nov 03 14:15:06.971283 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:06.971267 5979 scope.go:122] "RemoveContainer" containerID="fc7e09d772357f9b5a4239ee8834ce07484e0c94292f035f1af9902dc6f36617" Nov 03 14:15:06.971458 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:15:06.971442 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895255-hqw2d_odh-ai-gateway-infra(f89b9146-737e-450c-823e-d298a93cf055)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895255-hqw2d" podUID="f89b9146-737e-450c-823e-d298a93cf055" Nov 03 14:15:06.975174 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:06.975137 5979 scope.go:122] "RemoveContainer" containerID="f75d4d04cae47e32766d3d3775d19bc340b9f291de66cd9ca912c02a953f2cc6" Nov 03 14:15:06.995408 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:06.994731 5979 scope.go:122] "RemoveContainer" containerID="96c61c0ae211f13bd6c3296f519e053e1305004419bdeaf1bf291e1f1d032408" Nov 03 14:15:07.235919 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:15:07.235840 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:15:07.980925 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:07.980892 5979 scope.go:122] "RemoveContainer" containerID="78cdab2aa9edfa83846cc30697ca91546b9c4e1ace81558a088371aa0c290282" Nov 03 14:15:07.981379 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:15:07.981266 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-2399b8-29895255-wmw9v_odh-ai-gateway-infra(018cf2a9-b246-4ed8-9328-e301c3b10abc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2399b8-29895255-wmw9v" podUID="018cf2a9-b246-4ed8-9328-e301c3b10abc" Nov 03 14:15:07.982638 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:07.982618 5979 scope.go:122] "RemoveContainer" containerID="41725f081a3c6a14fcae282f5cd4c96b414927e6a74dd2c596e55805c44018b2" Nov 03 14:15:07.982938 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:15:07.982913 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895255-h595k_odh-ai-gateway-infra(95cebd5e-01ba-4b81-8d1d-024b4fe773e8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895255-h595k" podUID="95cebd5e-01ba-4b81-8d1d-024b4fe773e8" Nov 03 14:15:07.984260 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:07.984242 5979 scope.go:122] "RemoveContainer" containerID="da59147df9654e80021179060caa4eae7b68eb19d511d8b4478f9c2570cb870d" Nov 03 14:15:07.984515 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:15:07.984494 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-42c707-29895255-xkfz2_odh-ai-gateway-infra(e4a9f29a-7aaa-4d75-9fb5-e34a8fdca48c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895255-xkfz2" podUID="e4a9f29a-7aaa-4d75-9fb5-e34a8fdca48c" Nov 03 14:15:10.028062 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:10.028013 5979 generic.go:356] "Generic (PLEG): container finished" podID="43b38a5b-6579-4ce3-8bc6-1334bed0c633" containerID="803cdfa8d38fe35898e16efcbeecfedc78669413551e6edefa85b4182461e608" exitCode=0 Nov 03 14:15:10.028605 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:10.028125 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx" event={"ID":"43b38a5b-6579-4ce3-8bc6-1334bed0c633","Type":"ContainerDied","Data":"803cdfa8d38fe35898e16efcbeecfedc78669413551e6edefa85b4182461e608"} Nov 03 14:15:10.028605 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:10.028571 5979 scope.go:122] "RemoveContainer" containerID="803cdfa8d38fe35898e16efcbeecfedc78669413551e6edefa85b4182461e608" Nov 03 14:15:10.030590 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:10.030558 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:15:10.030823 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:10.030797 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:15:10.030932 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:10.030843 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:15:11.037089 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:11.037051 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx" event={"ID":"43b38a5b-6579-4ce3-8bc6-1334bed0c633","Type":"ContainerStarted","Data":"d6b42d4b7c65ebb2df3d6e402c7a86e292cf5f12c1ce393d2c812accc608a5cf"} Nov 03 14:15:11.237426 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:11.237396 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx" Nov 03 14:15:11.237655 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:11.237634 5979 patch_prober.go:29] interesting pod/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.79:15021/healthz/ready\": dial tcp 10.134.1.79:15021: connect: connection refused" start-of-body= Nov 03 14:15:11.237712 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:11.237679 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx" podUID="43b38a5b-6579-4ce3-8bc6-1334bed0c633" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.79:15021/healthz/ready\": dial tcp 10.134.1.79:15021: connect: connection refused" Nov 03 14:15:12.237801 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:12.237752 5979 patch_prober.go:29] interesting pod/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.79:15021/healthz/ready\": dial tcp 10.134.1.79:15021: connect: connection refused" start-of-body= Nov 03 14:15:12.238228 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:12.237828 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx" podUID="43b38a5b-6579-4ce3-8bc6-1334bed0c633" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.79:15021/healthz/ready\": dial tcp 10.134.1.79:15021: connect: connection refused" Nov 03 14:15:13.237493 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:13.237431 5979 patch_prober.go:29] interesting pod/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.79:15021/healthz/ready\": dial tcp 10.134.1.79:15021: connect: connection refused" start-of-body= Nov 03 14:15:13.237671 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:13.237519 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx" podUID="43b38a5b-6579-4ce3-8bc6-1334bed0c633" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.79:15021/healthz/ready\": dial tcp 10.134.1.79:15021: connect: connection refused" Nov 03 14:15:14.237375 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:14.237338 5979 patch_prober.go:29] interesting pod/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.79:15021/healthz/ready\": dial tcp 10.134.1.79:15021: connect: connection refused" start-of-body= Nov 03 14:15:14.237821 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:14.237396 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx" podUID="43b38a5b-6579-4ce3-8bc6-1334bed0c633" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.79:15021/healthz/ready\": dial tcp 10.134.1.79:15021: connect: connection refused" Nov 03 14:15:15.247517 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:15.247010 5979 patch_prober.go:29] interesting pod/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.79:15021/healthz/ready\": dial tcp 10.134.1.79:15021: connect: connection refused" start-of-body= Nov 03 14:15:15.247517 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:15.247070 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx" podUID="43b38a5b-6579-4ce3-8bc6-1334bed0c633" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.79:15021/healthz/ready\": dial tcp 10.134.1.79:15021: connect: connection refused" Nov 03 14:15:16.108297 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:16.108263 5979 scope.go:122] "RemoveContainer" containerID="3a56cd3e84d3197cc393ca080f773167ae04f1bfccf6cd665ffa7e532119b746" Nov 03 14:15:16.108563 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:16.108370 5979 scope.go:122] "RemoveContainer" containerID="ac80a8df455df4bedc32372fd1bee95008d9e6b1a8a5bc16dc96fda4096a5a08" Nov 03 14:15:16.237399 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:16.237366 5979 patch_prober.go:29] interesting pod/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.79:15021/healthz/ready\": dial tcp 10.134.1.79:15021: connect: connection refused" start-of-body= Nov 03 14:15:16.237570 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:16.237420 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx" podUID="43b38a5b-6579-4ce3-8bc6-1334bed0c633" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.79:15021/healthz/ready\": dial tcp 10.134.1.79:15021: connect: connection refused" Nov 03 14:15:17.099515 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:17.099446 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-367381-29895255-spbd5" event={"ID":"5e7900ee-f17f-45e8-87f7-f58389e788f6","Type":"ContainerStarted","Data":"117091d83128273e88a1ec51f300b5fbe8bce76e9e62ee92fa10e2299fab9a96"} Nov 03 14:15:17.101246 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:17.101215 5979 generic.go:356] "Generic (PLEG): container finished" podID="fbdad710-0252-4bd1-aba0-6bd737f94bac" containerID="254c95c56411e613e0ee866eb43405266e65bdf2db815b34a34b416f6734b9a2" exitCode=60 Nov 03 14:15:17.101400 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:17.101386 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895255-cg2mt" event={"ID":"fbdad710-0252-4bd1-aba0-6bd737f94bac","Type":"ContainerDied","Data":"254c95c56411e613e0ee866eb43405266e65bdf2db815b34a34b416f6734b9a2"} Nov 03 14:15:17.101444 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:17.101434 5979 scope.go:122] "RemoveContainer" containerID="3a56cd3e84d3197cc393ca080f773167ae04f1bfccf6cd665ffa7e532119b746" Nov 03 14:15:17.101914 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:17.101899 5979 scope.go:122] "RemoveContainer" containerID="254c95c56411e613e0ee866eb43405266e65bdf2db815b34a34b416f6734b9a2" Nov 03 14:15:17.102230 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:15:17.102210 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895255-cg2mt_odh-ai-gateway-infra(fbdad710-0252-4bd1-aba0-6bd737f94bac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895255-cg2mt" podUID="fbdad710-0252-4bd1-aba0-6bd737f94bac" Nov 03 14:15:17.108008 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:17.107985 5979 scope.go:122] "RemoveContainer" containerID="fc7e09d772357f9b5a4239ee8834ce07484e0c94292f035f1af9902dc6f36617" Nov 03 14:15:17.241685 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:17.241658 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx" Nov 03 14:15:17.241970 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:17.241949 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx" Nov 03 14:15:17.242642 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:17.242624 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx" Nov 03 14:15:18.111157 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:18.111120 5979 generic.go:356] "Generic (PLEG): container finished" podID="5e7900ee-f17f-45e8-87f7-f58389e788f6" containerID="117091d83128273e88a1ec51f300b5fbe8bce76e9e62ee92fa10e2299fab9a96" exitCode=60 Nov 03 14:15:18.115762 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:18.115732 5979 generic.go:356] "Generic (PLEG): container finished" podID="f89b9146-737e-450c-823e-d298a93cf055" containerID="ff34961068d9da6f853aa8144aa6d316369bbb35a5dedfdb6d020b1156a9cf9e" exitCode=60 Nov 03 14:15:18.118925 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:18.118892 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-367381-29895255-spbd5" event={"ID":"5e7900ee-f17f-45e8-87f7-f58389e788f6","Type":"ContainerDied","Data":"117091d83128273e88a1ec51f300b5fbe8bce76e9e62ee92fa10e2299fab9a96"} Nov 03 14:15:18.119086 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:18.118989 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895255-hqw2d" event={"ID":"f89b9146-737e-450c-823e-d298a93cf055","Type":"ContainerDied","Data":"ff34961068d9da6f853aa8144aa6d316369bbb35a5dedfdb6d020b1156a9cf9e"} Nov 03 14:15:18.119086 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:18.119031 5979 scope.go:122] "RemoveContainer" containerID="ac80a8df455df4bedc32372fd1bee95008d9e6b1a8a5bc16dc96fda4096a5a08" Nov 03 14:15:18.119398 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:18.119378 5979 scope.go:122] "RemoveContainer" containerID="117091d83128273e88a1ec51f300b5fbe8bce76e9e62ee92fa10e2299fab9a96" Nov 03 14:15:18.119582 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:18.119557 5979 scope.go:122] "RemoveContainer" containerID="78cdab2aa9edfa83846cc30697ca91546b9c4e1ace81558a088371aa0c290282" Nov 03 14:15:18.119867 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:15:18.119841 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-367381-29895255-spbd5_odh-ai-gateway-infra(5e7900ee-f17f-45e8-87f7-f58389e788f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-367381-29895255-spbd5" podUID="5e7900ee-f17f-45e8-87f7-f58389e788f6" Nov 03 14:15:18.119962 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:18.119866 5979 scope.go:122] "RemoveContainer" containerID="ff34961068d9da6f853aa8144aa6d316369bbb35a5dedfdb6d020b1156a9cf9e" Nov 03 14:15:18.120097 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:15:18.120059 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895255-hqw2d_odh-ai-gateway-infra(f89b9146-737e-450c-823e-d298a93cf055)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895255-hqw2d" podUID="f89b9146-737e-450c-823e-d298a93cf055" Nov 03 14:15:18.132732 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:18.132707 5979 scope.go:122] "RemoveContainer" containerID="fc7e09d772357f9b5a4239ee8834ce07484e0c94292f035f1af9902dc6f36617" Nov 03 14:15:18.220807 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:18.220728 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-367381-29895255-spbd5"] Nov 03 14:15:18.337815 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:18.337783 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895255-cg2mt"] Nov 03 14:15:18.503179 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:18.503153 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895255-cg2mt" Nov 03 14:15:18.550430 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:18.550401 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbdad710-0252-4bd1-aba0-6bd737f94bac-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbdad710-0252-4bd1-aba0-6bd737f94bac-maas-api-service-ca\") pod \"fbdad710-0252-4bd1-aba0-6bd737f94bac\" (UID: \"fbdad710-0252-4bd1-aba0-6bd737f94bac\") " Nov 03 14:15:18.550626 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:18.550511 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbdad710-0252-4bd1-aba0-6bd737f94bac-kube-api-access-t5hn7\" (UniqueName: \"kubernetes.io/projected/fbdad710-0252-4bd1-aba0-6bd737f94bac-kube-api-access-t5hn7\") pod \"fbdad710-0252-4bd1-aba0-6bd737f94bac\" (UID: \"fbdad710-0252-4bd1-aba0-6bd737f94bac\") " Nov 03 14:15:18.550831 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:18.550803 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbdad710-0252-4bd1-aba0-6bd737f94bac-maas-api-service-ca" pod "fbdad710-0252-4bd1-aba0-6bd737f94bac" (UID: "fbdad710-0252-4bd1-aba0-6bd737f94bac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:15:18.551143 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:18.551126 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbdad710-0252-4bd1-aba0-6bd737f94bac-maas-api-service-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:15:18.553025 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:18.552991 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbdad710-0252-4bd1-aba0-6bd737f94bac-kube-api-access-t5hn7" pod "fbdad710-0252-4bd1-aba0-6bd737f94bac" (UID: "fbdad710-0252-4bd1-aba0-6bd737f94bac"). InnerVolumeSpecName "kube-api-access-t5hn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:15:18.652711 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:18.652680 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5hn7\" (UniqueName: \"kubernetes.io/projected/fbdad710-0252-4bd1-aba0-6bd737f94bac-kube-api-access-t5hn7\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:15:19.108426 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:19.108396 5979 scope.go:122] "RemoveContainer" containerID="41725f081a3c6a14fcae282f5cd4c96b414927e6a74dd2c596e55805c44018b2" Nov 03 14:15:19.126317 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:19.126295 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895255-cg2mt" Nov 03 14:15:19.126690 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:19.126337 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895255-cg2mt" event={"ID":"fbdad710-0252-4bd1-aba0-6bd737f94bac","Type":"ContainerDied","Data":"d755aebdd6f1dffcbd3489d47daa22e3c44ba43686dc560bd1b7e08098be7131"} Nov 03 14:15:19.126690 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:19.126407 5979 scope.go:122] "RemoveContainer" containerID="254c95c56411e613e0ee866eb43405266e65bdf2db815b34a34b416f6734b9a2" Nov 03 14:15:19.129009 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:19.128983 5979 generic.go:356] "Generic (PLEG): container finished" podID="018cf2a9-b246-4ed8-9328-e301c3b10abc" containerID="efcdff1bd2c2471ce8a57b8bdd3c71bda39a409df57d612d248430b797f0ade2" exitCode=60 Nov 03 14:15:19.129171 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:19.129067 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2399b8-29895255-wmw9v" event={"ID":"018cf2a9-b246-4ed8-9328-e301c3b10abc","Type":"ContainerDied","Data":"efcdff1bd2c2471ce8a57b8bdd3c71bda39a409df57d612d248430b797f0ade2"} Nov 03 14:15:19.129556 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:19.129533 5979 scope.go:122] "RemoveContainer" containerID="efcdff1bd2c2471ce8a57b8bdd3c71bda39a409df57d612d248430b797f0ade2" Nov 03 14:15:19.129879 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:15:19.129856 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-2399b8-29895255-wmw9v_odh-ai-gateway-infra(018cf2a9-b246-4ed8-9328-e301c3b10abc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2399b8-29895255-wmw9v" podUID="018cf2a9-b246-4ed8-9328-e301c3b10abc" Nov 03 14:15:19.166525 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:19.166500 5979 scope.go:122] "RemoveContainer" containerID="78cdab2aa9edfa83846cc30697ca91546b9c4e1ace81558a088371aa0c290282" Nov 03 14:15:19.277533 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:19.277507 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-367381-29895255-spbd5" Nov 03 14:15:19.338434 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:19.338358 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895255-cg2mt"] Nov 03 14:15:19.343668 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:19.343640 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-d0231d-29895255-cg2mt"] Nov 03 14:15:19.360860 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:19.360831 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e7900ee-f17f-45e8-87f7-f58389e788f6-kube-api-access-px99p\" (UniqueName: \"kubernetes.io/projected/5e7900ee-f17f-45e8-87f7-f58389e788f6-kube-api-access-px99p\") pod \"5e7900ee-f17f-45e8-87f7-f58389e788f6\" (UID: \"5e7900ee-f17f-45e8-87f7-f58389e788f6\") " Nov 03 14:15:19.361002 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:19.360923 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e7900ee-f17f-45e8-87f7-f58389e788f6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e7900ee-f17f-45e8-87f7-f58389e788f6-maas-api-service-ca\") pod \"5e7900ee-f17f-45e8-87f7-f58389e788f6\" (UID: \"5e7900ee-f17f-45e8-87f7-f58389e788f6\") " Nov 03 14:15:19.361600 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:19.361562 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e7900ee-f17f-45e8-87f7-f58389e788f6-maas-api-service-ca" pod "5e7900ee-f17f-45e8-87f7-f58389e788f6" (UID: "5e7900ee-f17f-45e8-87f7-f58389e788f6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:15:19.362190 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:19.362163 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e7900ee-f17f-45e8-87f7-f58389e788f6-maas-api-service-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:15:19.373158 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:19.373123 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e7900ee-f17f-45e8-87f7-f58389e788f6-kube-api-access-px99p" pod "5e7900ee-f17f-45e8-87f7-f58389e788f6" (UID: "5e7900ee-f17f-45e8-87f7-f58389e788f6"). InnerVolumeSpecName "kube-api-access-px99p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:15:19.382571 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:19.382546 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895255-hqw2d"] Nov 03 14:15:19.464030 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:19.463991 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px99p\" (UniqueName: \"kubernetes.io/projected/5e7900ee-f17f-45e8-87f7-f58389e788f6-kube-api-access-px99p\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:15:19.539968 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:19.539934 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895255-hqw2d" Nov 03 14:15:19.564583 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:19.564559 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f89b9146-737e-450c-823e-d298a93cf055-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f89b9146-737e-450c-823e-d298a93cf055-maas-api-service-ca\") pod \"f89b9146-737e-450c-823e-d298a93cf055\" (UID: \"f89b9146-737e-450c-823e-d298a93cf055\") " Nov 03 14:15:19.564735 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:19.564694 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f89b9146-737e-450c-823e-d298a93cf055-kube-api-access-kcs9j\" (UniqueName: \"kubernetes.io/projected/f89b9146-737e-450c-823e-d298a93cf055-kube-api-access-kcs9j\") pod \"f89b9146-737e-450c-823e-d298a93cf055\" (UID: \"f89b9146-737e-450c-823e-d298a93cf055\") " Nov 03 14:15:19.564921 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:19.564898 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f89b9146-737e-450c-823e-d298a93cf055-maas-api-service-ca" pod "f89b9146-737e-450c-823e-d298a93cf055" (UID: "f89b9146-737e-450c-823e-d298a93cf055"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:15:19.565211 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:19.565193 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f89b9146-737e-450c-823e-d298a93cf055-maas-api-service-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:15:19.566803 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:19.566780 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f89b9146-737e-450c-823e-d298a93cf055-kube-api-access-kcs9j" pod "f89b9146-737e-450c-823e-d298a93cf055" (UID: "f89b9146-737e-450c-823e-d298a93cf055"). InnerVolumeSpecName "kube-api-access-kcs9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:15:19.666566 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:19.666531 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcs9j\" (UniqueName: \"kubernetes.io/projected/f89b9146-737e-450c-823e-d298a93cf055-kube-api-access-kcs9j\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:15:20.109020 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:20.108932 5979 scope.go:122] "RemoveContainer" containerID="da59147df9654e80021179060caa4eae7b68eb19d511d8b4478f9c2570cb870d" Nov 03 14:15:20.118521 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:20.118384 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbdad710-0252-4bd1-aba0-6bd737f94bac" path="/var/lib/kubelet/pods/fbdad710-0252-4bd1-aba0-6bd737f94bac/volumes" Nov 03 14:15:20.142773 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:20.142755 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895255-hqw2d" Nov 03 14:15:20.143238 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:20.142780 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895255-hqw2d" event={"ID":"f89b9146-737e-450c-823e-d298a93cf055","Type":"ContainerDied","Data":"622f01750a0d3562faccab46e41cd91c8d3b1d866135126503da78492bc3c0eb"} Nov 03 14:15:20.143238 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:20.142836 5979 scope.go:122] "RemoveContainer" containerID="ff34961068d9da6f853aa8144aa6d316369bbb35a5dedfdb6d020b1156a9cf9e" Nov 03 14:15:20.145140 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:20.145111 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895255-h595k" event={"ID":"95cebd5e-01ba-4b81-8d1d-024b4fe773e8","Type":"ContainerStarted","Data":"4669f9313d23128933d01ddafcdc2f75200db83aa51aeb091ec1403b63755a88"} Nov 03 14:15:20.148019 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:20.147992 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-367381-29895255-spbd5" event={"ID":"5e7900ee-f17f-45e8-87f7-f58389e788f6","Type":"ContainerDied","Data":"8289a109b806ba46ea443a565f1e459cc6178269ba650eabcf7adf32764a4c1d"} Nov 03 14:15:20.148121 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:20.148015 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-367381-29895255-spbd5" Nov 03 14:15:20.311591 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:20.311561 5979 scope.go:122] "RemoveContainer" containerID="117091d83128273e88a1ec51f300b5fbe8bce76e9e62ee92fa10e2299fab9a96" Nov 03 14:15:20.461098 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:20.459313 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2399b8-29895255-wmw9v"] Nov 03 14:15:20.507352 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:20.507308 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895255-hqw2d"] Nov 03 14:15:20.510406 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:20.510378 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895255-hqw2d"] Nov 03 14:15:20.612078 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:20.612050 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-367381-29895255-spbd5"] Nov 03 14:15:20.615360 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:20.615333 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-367381-29895255-spbd5"] Nov 03 14:15:20.629358 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:20.629336 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2399b8-29895255-wmw9v" Nov 03 14:15:20.704980 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:20.704889 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/018cf2a9-b246-4ed8-9328-e301c3b10abc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/018cf2a9-b246-4ed8-9328-e301c3b10abc-maas-api-service-ca\") pod \"018cf2a9-b246-4ed8-9328-e301c3b10abc\" (UID: \"018cf2a9-b246-4ed8-9328-e301c3b10abc\") " Nov 03 14:15:20.704980 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:20.704936 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/018cf2a9-b246-4ed8-9328-e301c3b10abc-kube-api-access-qmlsl\" (UniqueName: \"kubernetes.io/projected/018cf2a9-b246-4ed8-9328-e301c3b10abc-kube-api-access-qmlsl\") pod \"018cf2a9-b246-4ed8-9328-e301c3b10abc\" (UID: \"018cf2a9-b246-4ed8-9328-e301c3b10abc\") " Nov 03 14:15:20.705232 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:20.705212 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/018cf2a9-b246-4ed8-9328-e301c3b10abc-maas-api-service-ca" pod "018cf2a9-b246-4ed8-9328-e301c3b10abc" (UID: "018cf2a9-b246-4ed8-9328-e301c3b10abc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:15:20.705338 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:20.705315 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/018cf2a9-b246-4ed8-9328-e301c3b10abc-maas-api-service-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:15:20.707092 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:20.707070 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/018cf2a9-b246-4ed8-9328-e301c3b10abc-kube-api-access-qmlsl" pod "018cf2a9-b246-4ed8-9328-e301c3b10abc" (UID: "018cf2a9-b246-4ed8-9328-e301c3b10abc"). InnerVolumeSpecName "kube-api-access-qmlsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:15:20.806558 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:20.806464 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmlsl\" (UniqueName: \"kubernetes.io/projected/018cf2a9-b246-4ed8-9328-e301c3b10abc-kube-api-access-qmlsl\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:15:21.157621 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:21.157592 5979 generic.go:356] "Generic (PLEG): container finished" podID="95cebd5e-01ba-4b81-8d1d-024b4fe773e8" containerID="4669f9313d23128933d01ddafcdc2f75200db83aa51aeb091ec1403b63755a88" exitCode=60 Nov 03 14:15:21.157971 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:21.157675 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895255-h595k" event={"ID":"95cebd5e-01ba-4b81-8d1d-024b4fe773e8","Type":"ContainerDied","Data":"4669f9313d23128933d01ddafcdc2f75200db83aa51aeb091ec1403b63755a88"} Nov 03 14:15:21.157971 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:21.157744 5979 scope.go:122] "RemoveContainer" containerID="41725f081a3c6a14fcae282f5cd4c96b414927e6a74dd2c596e55805c44018b2" Nov 03 14:15:21.158112 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:21.158092 5979 scope.go:122] "RemoveContainer" containerID="4669f9313d23128933d01ddafcdc2f75200db83aa51aeb091ec1403b63755a88" Nov 03 14:15:21.158518 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:15:21.158488 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895255-h595k_odh-ai-gateway-infra(95cebd5e-01ba-4b81-8d1d-024b4fe773e8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895255-h595k" podUID="95cebd5e-01ba-4b81-8d1d-024b4fe773e8" Nov 03 14:15:21.160533 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:21.160512 5979 generic.go:356] "Generic (PLEG): container finished" podID="e4a9f29a-7aaa-4d75-9fb5-e34a8fdca48c" containerID="19116cd94c3d559317e8c40827a79f2ef8020e0c6f73ae1cc4755fae4219befc" exitCode=60 Nov 03 14:15:21.160620 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:21.160582 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895255-xkfz2" event={"ID":"e4a9f29a-7aaa-4d75-9fb5-e34a8fdca48c","Type":"ContainerDied","Data":"19116cd94c3d559317e8c40827a79f2ef8020e0c6f73ae1cc4755fae4219befc"} Nov 03 14:15:21.161025 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:21.161009 5979 scope.go:122] "RemoveContainer" containerID="19116cd94c3d559317e8c40827a79f2ef8020e0c6f73ae1cc4755fae4219befc" Nov 03 14:15:21.161323 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:15:21.161300 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-42c707-29895255-xkfz2_odh-ai-gateway-infra(e4a9f29a-7aaa-4d75-9fb5-e34a8fdca48c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895255-xkfz2" podUID="e4a9f29a-7aaa-4d75-9fb5-e34a8fdca48c" Nov 03 14:15:21.161981 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:21.161961 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2399b8-29895255-wmw9v" event={"ID":"018cf2a9-b246-4ed8-9328-e301c3b10abc","Type":"ContainerDied","Data":"56964862c66d9755f54e6502c27a0167f2ae39cf060073235f9ef858a35dc5d7"} Nov 03 14:15:21.162062 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:21.161963 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2399b8-29895255-wmw9v" Nov 03 14:15:21.174542 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:21.174524 5979 scope.go:122] "RemoveContainer" containerID="da59147df9654e80021179060caa4eae7b68eb19d511d8b4478f9c2570cb870d" Nov 03 14:15:21.186566 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:21.186550 5979 scope.go:122] "RemoveContainer" containerID="efcdff1bd2c2471ce8a57b8bdd3c71bda39a409df57d612d248430b797f0ade2" Nov 03 14:15:21.362348 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:21.362263 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895255-h595k"] Nov 03 14:15:21.369962 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:21.369698 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2399b8-29895255-wmw9v"] Nov 03 14:15:21.374646 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:21.374620 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2399b8-29895255-wmw9v"] Nov 03 14:15:22.117445 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:22.117402 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="018cf2a9-b246-4ed8-9328-e301c3b10abc" path="/var/lib/kubelet/pods/018cf2a9-b246-4ed8-9328-e301c3b10abc/volumes" Nov 03 14:15:22.117971 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:22.117947 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e7900ee-f17f-45e8-87f7-f58389e788f6" path="/var/lib/kubelet/pods/5e7900ee-f17f-45e8-87f7-f58389e788f6/volumes" Nov 03 14:15:22.118405 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:22.118386 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f89b9146-737e-450c-823e-d298a93cf055" path="/var/lib/kubelet/pods/f89b9146-737e-450c-823e-d298a93cf055/volumes" Nov 03 14:15:22.336635 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:22.336608 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895255-h595k" Nov 03 14:15:22.424690 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:22.424654 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95cebd5e-01ba-4b81-8d1d-024b4fe773e8-kube-api-access-nfrdh\" (UniqueName: \"kubernetes.io/projected/95cebd5e-01ba-4b81-8d1d-024b4fe773e8-kube-api-access-nfrdh\") pod \"95cebd5e-01ba-4b81-8d1d-024b4fe773e8\" (UID: \"95cebd5e-01ba-4b81-8d1d-024b4fe773e8\") " Nov 03 14:15:22.424876 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:22.424830 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95cebd5e-01ba-4b81-8d1d-024b4fe773e8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95cebd5e-01ba-4b81-8d1d-024b4fe773e8-maas-api-service-ca\") pod \"95cebd5e-01ba-4b81-8d1d-024b4fe773e8\" (UID: \"95cebd5e-01ba-4b81-8d1d-024b4fe773e8\") " Nov 03 14:15:22.425156 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:22.425129 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95cebd5e-01ba-4b81-8d1d-024b4fe773e8-maas-api-service-ca" pod "95cebd5e-01ba-4b81-8d1d-024b4fe773e8" (UID: "95cebd5e-01ba-4b81-8d1d-024b4fe773e8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:15:22.425285 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:22.425273 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95cebd5e-01ba-4b81-8d1d-024b4fe773e8-maas-api-service-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:15:22.426770 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:22.426748 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95cebd5e-01ba-4b81-8d1d-024b4fe773e8-kube-api-access-nfrdh" pod "95cebd5e-01ba-4b81-8d1d-024b4fe773e8" (UID: "95cebd5e-01ba-4b81-8d1d-024b4fe773e8"). InnerVolumeSpecName "kube-api-access-nfrdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:15:22.490905 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:22.490856 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895255-xkfz2"] Nov 03 14:15:22.526969 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:22.526926 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfrdh\" (UniqueName: \"kubernetes.io/projected/95cebd5e-01ba-4b81-8d1d-024b4fe773e8-kube-api-access-nfrdh\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:15:22.656683 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:22.656649 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895255-xkfz2" Nov 03 14:15:22.728786 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:22.728710 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4a9f29a-7aaa-4d75-9fb5-e34a8fdca48c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4a9f29a-7aaa-4d75-9fb5-e34a8fdca48c-maas-api-service-ca\") pod \"e4a9f29a-7aaa-4d75-9fb5-e34a8fdca48c\" (UID: \"e4a9f29a-7aaa-4d75-9fb5-e34a8fdca48c\") " Nov 03 14:15:22.729013 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:22.728790 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4a9f29a-7aaa-4d75-9fb5-e34a8fdca48c-kube-api-access-hltnt\" (UniqueName: \"kubernetes.io/projected/e4a9f29a-7aaa-4d75-9fb5-e34a8fdca48c-kube-api-access-hltnt\") pod \"e4a9f29a-7aaa-4d75-9fb5-e34a8fdca48c\" (UID: \"e4a9f29a-7aaa-4d75-9fb5-e34a8fdca48c\") " Nov 03 14:15:22.729071 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:22.729042 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4a9f29a-7aaa-4d75-9fb5-e34a8fdca48c-maas-api-service-ca" pod "e4a9f29a-7aaa-4d75-9fb5-e34a8fdca48c" (UID: "e4a9f29a-7aaa-4d75-9fb5-e34a8fdca48c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:15:22.729161 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:22.729145 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4a9f29a-7aaa-4d75-9fb5-e34a8fdca48c-maas-api-service-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:15:22.730815 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:22.730792 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4a9f29a-7aaa-4d75-9fb5-e34a8fdca48c-kube-api-access-hltnt" pod "e4a9f29a-7aaa-4d75-9fb5-e34a8fdca48c" (UID: "e4a9f29a-7aaa-4d75-9fb5-e34a8fdca48c"). InnerVolumeSpecName "kube-api-access-hltnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:15:22.830755 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:22.830721 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hltnt\" (UniqueName: \"kubernetes.io/projected/e4a9f29a-7aaa-4d75-9fb5-e34a8fdca48c-kube-api-access-hltnt\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:15:23.180457 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:23.180426 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895255-xkfz2" Nov 03 14:15:23.180457 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:23.180446 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895255-xkfz2" event={"ID":"e4a9f29a-7aaa-4d75-9fb5-e34a8fdca48c","Type":"ContainerDied","Data":"a74cd6c6d906000c49338b60390d034c8128629e7e9dd3a477ebb700fec94bb7"} Nov 03 14:15:23.180766 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:23.180529 5979 scope.go:122] "RemoveContainer" containerID="19116cd94c3d559317e8c40827a79f2ef8020e0c6f73ae1cc4755fae4219befc" Nov 03 14:15:23.182183 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:23.182162 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895255-h595k" Nov 03 14:15:23.182299 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:23.182168 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895255-h595k" event={"ID":"95cebd5e-01ba-4b81-8d1d-024b4fe773e8","Type":"ContainerDied","Data":"8b46f778a58229ba19ef91b53d0ac8c2f1dbd621e5456b8560b13319f2fc86f1"} Nov 03 14:15:23.191798 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:23.191777 5979 scope.go:122] "RemoveContainer" containerID="4669f9313d23128933d01ddafcdc2f75200db83aa51aeb091ec1403b63755a88" Nov 03 14:15:23.333577 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:23.333548 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895255-h595k"] Nov 03 14:15:23.338968 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:23.338939 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-5ae63d-29895255-h595k"] Nov 03 14:15:23.440569 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:23.440488 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895255-xkfz2"] Nov 03 14:15:23.445079 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:23.445054 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-42c707-29895255-xkfz2"] Nov 03 14:15:24.116199 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:24.116166 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95cebd5e-01ba-4b81-8d1d-024b4fe773e8" path="/var/lib/kubelet/pods/95cebd5e-01ba-4b81-8d1d-024b4fe773e8/volumes" Nov 03 14:15:24.116647 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:24.116632 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4a9f29a-7aaa-4d75-9fb5-e34a8fdca48c" path="/var/lib/kubelet/pods/e4a9f29a-7aaa-4d75-9fb5-e34a8fdca48c/volumes" Nov 03 14:15:36.532036 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:36.531996 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:15:37.234898 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:15:37.234853 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:15:42.254437 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.254401 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm"] Nov 03 14:15:42.255056 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.255031 5979 cpu_manager.go:414] "containerMap: removing container" podUID="fbdad710-0252-4bd1-aba0-6bd737f94bac" containerName="cleanup" Nov 03 14:15:42.255056 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.255055 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbdad710-0252-4bd1-aba0-6bd737f94bac" containerName="cleanup" Nov 03 14:15:42.255265 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.255076 5979 cpu_manager.go:414] "containerMap: removing container" podUID="f89b9146-737e-450c-823e-d298a93cf055" containerName="cleanup" Nov 03 14:15:42.255265 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.255083 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f89b9146-737e-450c-823e-d298a93cf055" containerName="cleanup" Nov 03 14:15:42.255265 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.255094 5979 cpu_manager.go:414] "containerMap: removing container" podUID="f89b9146-737e-450c-823e-d298a93cf055" containerName="cleanup" Nov 03 14:15:42.255265 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.255101 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f89b9146-737e-450c-823e-d298a93cf055" containerName="cleanup" Nov 03 14:15:42.255265 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.255113 5979 cpu_manager.go:414] "containerMap: removing container" podUID="95cebd5e-01ba-4b81-8d1d-024b4fe773e8" containerName="cleanup" Nov 03 14:15:42.255265 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.255121 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95cebd5e-01ba-4b81-8d1d-024b4fe773e8" containerName="cleanup" Nov 03 14:15:42.255265 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.255141 5979 cpu_manager.go:414] "containerMap: removing container" podUID="018cf2a9-b246-4ed8-9328-e301c3b10abc" containerName="cleanup" Nov 03 14:15:42.255265 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.255146 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="018cf2a9-b246-4ed8-9328-e301c3b10abc" containerName="cleanup" Nov 03 14:15:42.255265 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.255157 5979 cpu_manager.go:414] "containerMap: removing container" podUID="5e7900ee-f17f-45e8-87f7-f58389e788f6" containerName="cleanup" Nov 03 14:15:42.255265 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.255161 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e7900ee-f17f-45e8-87f7-f58389e788f6" containerName="cleanup" Nov 03 14:15:42.255265 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.255171 5979 cpu_manager.go:414] "containerMap: removing container" podUID="5e7900ee-f17f-45e8-87f7-f58389e788f6" containerName="cleanup" Nov 03 14:15:42.255265 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.255176 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e7900ee-f17f-45e8-87f7-f58389e788f6" containerName="cleanup" Nov 03 14:15:42.255265 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.255186 5979 cpu_manager.go:414] "containerMap: removing container" podUID="fbdad710-0252-4bd1-aba0-6bd737f94bac" containerName="cleanup" Nov 03 14:15:42.255265 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.255194 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbdad710-0252-4bd1-aba0-6bd737f94bac" containerName="cleanup" Nov 03 14:15:42.255265 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.255200 5979 cpu_manager.go:414] "containerMap: removing container" podUID="e4a9f29a-7aaa-4d75-9fb5-e34a8fdca48c" containerName="cleanup" Nov 03 14:15:42.255265 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.255204 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4a9f29a-7aaa-4d75-9fb5-e34a8fdca48c" containerName="cleanup" Nov 03 14:15:42.255265 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.255214 5979 cpu_manager.go:414] "containerMap: removing container" podUID="95cebd5e-01ba-4b81-8d1d-024b4fe773e8" containerName="cleanup" Nov 03 14:15:42.255265 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.255223 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95cebd5e-01ba-4b81-8d1d-024b4fe773e8" containerName="cleanup" Nov 03 14:15:42.255265 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.255251 5979 cpu_manager.go:414] "containerMap: removing container" podUID="e4a9f29a-7aaa-4d75-9fb5-e34a8fdca48c" containerName="cleanup" Nov 03 14:15:42.255265 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.255255 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4a9f29a-7aaa-4d75-9fb5-e34a8fdca48c" containerName="cleanup" Nov 03 14:15:42.255265 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.255262 5979 cpu_manager.go:414] "containerMap: removing container" podUID="018cf2a9-b246-4ed8-9328-e301c3b10abc" containerName="cleanup" Nov 03 14:15:42.255265 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.255268 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="018cf2a9-b246-4ed8-9328-e301c3b10abc" containerName="cleanup" Nov 03 14:15:42.256107 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.255381 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbdad710-0252-4bd1-aba0-6bd737f94bac" containerName="cleanup" Nov 03 14:15:42.256107 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.255391 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="f89b9146-737e-450c-823e-d298a93cf055" containerName="cleanup" Nov 03 14:15:42.256107 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.255397 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="f89b9146-737e-450c-823e-d298a93cf055" containerName="cleanup" Nov 03 14:15:42.256107 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.255403 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="95cebd5e-01ba-4b81-8d1d-024b4fe773e8" containerName="cleanup" Nov 03 14:15:42.256107 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.255412 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="018cf2a9-b246-4ed8-9328-e301c3b10abc" containerName="cleanup" Nov 03 14:15:42.256107 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.255420 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e7900ee-f17f-45e8-87f7-f58389e788f6" containerName="cleanup" Nov 03 14:15:42.256107 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.255430 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e7900ee-f17f-45e8-87f7-f58389e788f6" containerName="cleanup" Nov 03 14:15:42.256107 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.255438 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbdad710-0252-4bd1-aba0-6bd737f94bac" containerName="cleanup" Nov 03 14:15:42.256107 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.255446 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4a9f29a-7aaa-4d75-9fb5-e34a8fdca48c" containerName="cleanup" Nov 03 14:15:42.256107 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.255454 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="95cebd5e-01ba-4b81-8d1d-024b4fe773e8" containerName="cleanup" Nov 03 14:15:42.256107 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.255480 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="f89b9146-737e-450c-823e-d298a93cf055" containerName="cleanup" Nov 03 14:15:42.256107 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.255492 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="018cf2a9-b246-4ed8-9328-e301c3b10abc" containerName="cleanup" Nov 03 14:15:42.256107 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.255500 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e7900ee-f17f-45e8-87f7-f58389e788f6" containerName="cleanup" Nov 03 14:15:42.256107 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.255506 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4a9f29a-7aaa-4d75-9fb5-e34a8fdca48c" containerName="cleanup" Nov 03 14:15:42.256107 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.255512 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="018cf2a9-b246-4ed8-9328-e301c3b10abc" containerName="cleanup" Nov 03 14:15:42.259155 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.259136 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm" Nov 03 14:15:42.378895 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.378857 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-b5daee-openshift-default-dockercfg-njsrn\"" Nov 03 14:15:42.384206 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.384165 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm"] Nov 03 14:15:42.418846 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.415913 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9650f53f-0ff8-49f0-ba84-1186f4753807-istio-podinfo\") pod \"e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm\" (UID: \"9650f53f-0ff8-49f0-ba84-1186f4753807\") " pod="openshift-ingress/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm" Nov 03 14:15:42.418846 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.415988 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9650f53f-0ff8-49f0-ba84-1186f4753807-istio-token\") pod \"e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm\" (UID: \"9650f53f-0ff8-49f0-ba84-1186f4753807\") " pod="openshift-ingress/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm" Nov 03 14:15:42.418846 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.416018 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9650f53f-0ff8-49f0-ba84-1186f4753807-workload-socket\") pod \"e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm\" (UID: \"9650f53f-0ff8-49f0-ba84-1186f4753807\") " pod="openshift-ingress/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm" Nov 03 14:15:42.418846 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.416061 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9650f53f-0ff8-49f0-ba84-1186f4753807-workload-certs\") pod \"e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm\" (UID: \"9650f53f-0ff8-49f0-ba84-1186f4753807\") " pod="openshift-ingress/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm" Nov 03 14:15:42.418846 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.416118 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9650f53f-0ff8-49f0-ba84-1186f4753807-istio-envoy\") pod \"e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm\" (UID: \"9650f53f-0ff8-49f0-ba84-1186f4753807\") " pod="openshift-ingress/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm" Nov 03 14:15:42.418846 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.416144 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9650f53f-0ff8-49f0-ba84-1186f4753807-credential-socket\") pod \"e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm\" (UID: \"9650f53f-0ff8-49f0-ba84-1186f4753807\") " pod="openshift-ingress/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm" Nov 03 14:15:42.418846 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.416239 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9650f53f-0ff8-49f0-ba84-1186f4753807-istiod-ca-cert\") pod \"e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm\" (UID: \"9650f53f-0ff8-49f0-ba84-1186f4753807\") " pod="openshift-ingress/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm" Nov 03 14:15:42.418846 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.416312 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4q8p\" (UniqueName: \"kubernetes.io/projected/9650f53f-0ff8-49f0-ba84-1186f4753807-kube-api-access-z4q8p\") pod \"e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm\" (UID: \"9650f53f-0ff8-49f0-ba84-1186f4753807\") " pod="openshift-ingress/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm" Nov 03 14:15:42.418846 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.416379 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9650f53f-0ff8-49f0-ba84-1186f4753807-istio-data\") pod \"e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm\" (UID: \"9650f53f-0ff8-49f0-ba84-1186f4753807\") " pod="openshift-ingress/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm" Nov 03 14:15:42.517117 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.517037 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9650f53f-0ff8-49f0-ba84-1186f4753807-istiod-ca-cert\") pod \"e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm\" (UID: \"9650f53f-0ff8-49f0-ba84-1186f4753807\") " pod="openshift-ingress/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm" Nov 03 14:15:42.517117 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.517098 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4q8p\" (UniqueName: \"kubernetes.io/projected/9650f53f-0ff8-49f0-ba84-1186f4753807-kube-api-access-z4q8p\") pod \"e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm\" (UID: \"9650f53f-0ff8-49f0-ba84-1186f4753807\") " pod="openshift-ingress/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm" Nov 03 14:15:42.517321 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.517154 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9650f53f-0ff8-49f0-ba84-1186f4753807-istio-data\") pod \"e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm\" (UID: \"9650f53f-0ff8-49f0-ba84-1186f4753807\") " pod="openshift-ingress/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm" Nov 03 14:15:42.517321 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.517202 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9650f53f-0ff8-49f0-ba84-1186f4753807-istio-podinfo\") pod \"e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm\" (UID: \"9650f53f-0ff8-49f0-ba84-1186f4753807\") " pod="openshift-ingress/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm" Nov 03 14:15:42.517321 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.517248 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9650f53f-0ff8-49f0-ba84-1186f4753807-istio-token\") pod \"e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm\" (UID: \"9650f53f-0ff8-49f0-ba84-1186f4753807\") " pod="openshift-ingress/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm" Nov 03 14:15:42.517321 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.517274 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9650f53f-0ff8-49f0-ba84-1186f4753807-workload-socket\") pod \"e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm\" (UID: \"9650f53f-0ff8-49f0-ba84-1186f4753807\") " pod="openshift-ingress/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm" Nov 03 14:15:42.517321 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.517313 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9650f53f-0ff8-49f0-ba84-1186f4753807-workload-certs\") pod \"e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm\" (UID: \"9650f53f-0ff8-49f0-ba84-1186f4753807\") " pod="openshift-ingress/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm" Nov 03 14:15:42.517521 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.517373 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9650f53f-0ff8-49f0-ba84-1186f4753807-istio-envoy\") pod \"e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm\" (UID: \"9650f53f-0ff8-49f0-ba84-1186f4753807\") " pod="openshift-ingress/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm" Nov 03 14:15:42.517521 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.517398 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9650f53f-0ff8-49f0-ba84-1186f4753807-credential-socket\") pod \"e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm\" (UID: \"9650f53f-0ff8-49f0-ba84-1186f4753807\") " pod="openshift-ingress/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm" Nov 03 14:15:42.517646 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.517620 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9650f53f-0ff8-49f0-ba84-1186f4753807-istio-data\") pod \"e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm\" (UID: \"9650f53f-0ff8-49f0-ba84-1186f4753807\") " pod="openshift-ingress/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm" Nov 03 14:15:42.517729 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.517704 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9650f53f-0ff8-49f0-ba84-1186f4753807-workload-socket\") pod \"e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm\" (UID: \"9650f53f-0ff8-49f0-ba84-1186f4753807\") " pod="openshift-ingress/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm" Nov 03 14:15:42.517810 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.517776 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9650f53f-0ff8-49f0-ba84-1186f4753807-credential-socket\") pod \"e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm\" (UID: \"9650f53f-0ff8-49f0-ba84-1186f4753807\") " pod="openshift-ingress/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm" Nov 03 14:15:42.517895 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.517871 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9650f53f-0ff8-49f0-ba84-1186f4753807-workload-certs\") pod \"e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm\" (UID: \"9650f53f-0ff8-49f0-ba84-1186f4753807\") " pod="openshift-ingress/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm" Nov 03 14:15:42.517939 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.517901 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9650f53f-0ff8-49f0-ba84-1186f4753807-istiod-ca-cert\") pod \"e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm\" (UID: \"9650f53f-0ff8-49f0-ba84-1186f4753807\") " pod="openshift-ingress/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm" Nov 03 14:15:42.519653 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.519631 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9650f53f-0ff8-49f0-ba84-1186f4753807-istio-envoy\") pod \"e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm\" (UID: \"9650f53f-0ff8-49f0-ba84-1186f4753807\") " pod="openshift-ingress/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm" Nov 03 14:15:42.519797 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.519780 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9650f53f-0ff8-49f0-ba84-1186f4753807-istio-podinfo\") pod \"e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm\" (UID: \"9650f53f-0ff8-49f0-ba84-1186f4753807\") " pod="openshift-ingress/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm" Nov 03 14:15:42.576523 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.576341 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9650f53f-0ff8-49f0-ba84-1186f4753807-istio-token\") pod \"e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm\" (UID: \"9650f53f-0ff8-49f0-ba84-1186f4753807\") " pod="openshift-ingress/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm" Nov 03 14:15:42.576735 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.576704 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4q8p\" (UniqueName: \"kubernetes.io/projected/9650f53f-0ff8-49f0-ba84-1186f4753807-kube-api-access-z4q8p\") pod \"e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm\" (UID: \"9650f53f-0ff8-49f0-ba84-1186f4753807\") " pod="openshift-ingress/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm" Nov 03 14:15:42.872618 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:42.872521 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm" Nov 03 14:15:43.304668 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:43.304633 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm"] Nov 03 14:15:43.311398 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:15:43.311366 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9650f53f_0ff8_49f0_ba84_1186f4753807.slice/crio-214e3c8de1b69bb5931f0203341b17e64359684779b33b475f1e5433dc34e777 WatchSource:0}: Error finding container 214e3c8de1b69bb5931f0203341b17e64359684779b33b475f1e5433dc34e777: Status 404 returned error can't find the container with id 214e3c8de1b69bb5931f0203341b17e64359684779b33b475f1e5433dc34e777 Nov 03 14:15:43.314117 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:43.314078 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:15:43.314230 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:43.314168 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:15:43.314230 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:43.314213 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:15:43.401338 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:43.401290 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm" event={"ID":"9650f53f-0ff8-49f0-ba84-1186f4753807","Type":"ContainerStarted","Data":"51380ace269523c2098117a40084ef8dced7c5e9b09ab75ffbd43e81e869b668"} Nov 03 14:15:43.401526 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:43.401356 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm" event={"ID":"9650f53f-0ff8-49f0-ba84-1186f4753807","Type":"ContainerStarted","Data":"214e3c8de1b69bb5931f0203341b17e64359684779b33b475f1e5433dc34e777"} Nov 03 14:15:43.513082 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:43.513031 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm" podStartSLOduration=1.513014247 podStartE2EDuration="1.513014247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:15:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:15:43.512551127 +0000 UTC m=+4225.523239369" watchObservedRunningTime="2026-11-03 14:15:43.513014247 +0000 UTC m=+4225.523702490" Nov 03 14:15:43.872800 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:43.872763 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm" Nov 03 14:15:44.878795 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:44.878761 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm" Nov 03 14:15:45.413532 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:45.413498 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm" Nov 03 14:15:45.414914 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:45.414890 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm" Nov 03 14:15:50.830299 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:50.827831 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b5daee-5fcb8897cf-kdkph"] Nov 03 14:15:50.830299 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:50.828896 5979 cpu_manager.go:414] "containerMap: removing container" podUID="5e7900ee-f17f-45e8-87f7-f58389e788f6" containerName="cleanup" Nov 03 14:15:50.830299 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:50.828920 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e7900ee-f17f-45e8-87f7-f58389e788f6" containerName="cleanup" Nov 03 14:15:50.830299 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:50.828934 5979 cpu_manager.go:414] "containerMap: removing container" podUID="fbdad710-0252-4bd1-aba0-6bd737f94bac" containerName="cleanup" Nov 03 14:15:50.830299 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:50.828942 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbdad710-0252-4bd1-aba0-6bd737f94bac" containerName="cleanup" Nov 03 14:15:50.830299 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:50.828999 5979 cpu_manager.go:414] "containerMap: removing container" podUID="e4a9f29a-7aaa-4d75-9fb5-e34a8fdca48c" containerName="cleanup" Nov 03 14:15:50.830299 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:50.829007 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4a9f29a-7aaa-4d75-9fb5-e34a8fdca48c" containerName="cleanup" Nov 03 14:15:50.830299 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:50.829022 5979 cpu_manager.go:414] "containerMap: removing container" podUID="018cf2a9-b246-4ed8-9328-e301c3b10abc" containerName="cleanup" Nov 03 14:15:50.830299 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:50.829029 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="018cf2a9-b246-4ed8-9328-e301c3b10abc" containerName="cleanup" Nov 03 14:15:50.830299 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:50.829060 5979 cpu_manager.go:414] "containerMap: removing container" podUID="f89b9146-737e-450c-823e-d298a93cf055" containerName="cleanup" Nov 03 14:15:50.830299 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:50.829067 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f89b9146-737e-450c-823e-d298a93cf055" containerName="cleanup" Nov 03 14:15:50.830299 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:50.829081 5979 cpu_manager.go:414] "containerMap: removing container" podUID="95cebd5e-01ba-4b81-8d1d-024b4fe773e8" containerName="cleanup" Nov 03 14:15:50.830299 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:50.829090 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95cebd5e-01ba-4b81-8d1d-024b4fe773e8" containerName="cleanup" Nov 03 14:15:50.830299 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:50.829252 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbdad710-0252-4bd1-aba0-6bd737f94bac" containerName="cleanup" Nov 03 14:15:50.830299 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:50.829264 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4a9f29a-7aaa-4d75-9fb5-e34a8fdca48c" containerName="cleanup" Nov 03 14:15:50.830299 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:50.829274 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="95cebd5e-01ba-4b81-8d1d-024b4fe773e8" containerName="cleanup" Nov 03 14:15:50.852607 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:50.852578 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b5daee-5fcb8897cf-kdkph" Nov 03 14:15:50.879345 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:50.877947 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b5daee-5fcb8897cf-kdkph"] Nov 03 14:15:50.920227 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:50.919993 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-b5daee\"" Nov 03 14:15:50.920227 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:50.920175 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-b5daee-dockercfg-rm8bl\"" Nov 03 14:15:50.980593 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:50.980546 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npxq6\" (UniqueName: \"kubernetes.io/projected/f0d0e2f4-b051-47a1-8118-88694a09dddc-kube-api-access-npxq6\") pod \"payload-processing-e2e-inf-b-b5daee-5fcb8897cf-kdkph\" (UID: \"f0d0e2f4-b051-47a1-8118-88694a09dddc\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b5daee-5fcb8897cf-kdkph" Nov 03 14:15:50.980801 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:50.980778 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0d0e2f4-b051-47a1-8118-88694a09dddc-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b5daee-5fcb8897cf-kdkph\" (UID: \"f0d0e2f4-b051-47a1-8118-88694a09dddc\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b5daee-5fcb8897cf-kdkph" Nov 03 14:15:51.018512 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:51.018438 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b5daee-797cb4f7f-ljgr2"] Nov 03 14:15:51.023003 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:51.022980 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b5daee-797cb4f7f-ljgr2" Nov 03 14:15:51.077432 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:51.077399 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b5daee-797cb4f7f-ljgr2"] Nov 03 14:15:51.082246 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:51.082160 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npxq6\" (UniqueName: \"kubernetes.io/projected/f0d0e2f4-b051-47a1-8118-88694a09dddc-kube-api-access-npxq6\") pod \"payload-processing-e2e-inf-b-b5daee-5fcb8897cf-kdkph\" (UID: \"f0d0e2f4-b051-47a1-8118-88694a09dddc\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b5daee-5fcb8897cf-kdkph" Nov 03 14:15:51.082401 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:51.082303 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0d0e2f4-b051-47a1-8118-88694a09dddc-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b5daee-5fcb8897cf-kdkph\" (UID: \"f0d0e2f4-b051-47a1-8118-88694a09dddc\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b5daee-5fcb8897cf-kdkph" Nov 03 14:15:51.083001 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:51.082977 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0d0e2f4-b051-47a1-8118-88694a09dddc-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b5daee-5fcb8897cf-kdkph\" (UID: \"f0d0e2f4-b051-47a1-8118-88694a09dddc\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b5daee-5fcb8897cf-kdkph" Nov 03 14:15:51.136768 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:51.136736 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npxq6\" (UniqueName: \"kubernetes.io/projected/f0d0e2f4-b051-47a1-8118-88694a09dddc-kube-api-access-npxq6\") pod \"payload-processing-e2e-inf-b-b5daee-5fcb8897cf-kdkph\" (UID: \"f0d0e2f4-b051-47a1-8118-88694a09dddc\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b5daee-5fcb8897cf-kdkph" Nov 03 14:15:51.183988 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:51.183954 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzjn4\" (UniqueName: \"kubernetes.io/projected/97fe4000-c797-4c39-8872-38308ffdf700-kube-api-access-xzjn4\") pod \"payload-pre-processing-e2e-inf-b-b5daee-797cb4f7f-ljgr2\" (UID: \"97fe4000-c797-4c39-8872-38308ffdf700\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b5daee-797cb4f7f-ljgr2" Nov 03 14:15:51.184155 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:51.184015 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97fe4000-c797-4c39-8872-38308ffdf700-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b5daee-797cb4f7f-ljgr2\" (UID: \"97fe4000-c797-4c39-8872-38308ffdf700\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b5daee-797cb4f7f-ljgr2" Nov 03 14:15:51.195830 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:51.195805 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b5daee-5fcb8897cf-kdkph" Nov 03 14:15:51.285987 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:51.285784 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzjn4\" (UniqueName: \"kubernetes.io/projected/97fe4000-c797-4c39-8872-38308ffdf700-kube-api-access-xzjn4\") pod \"payload-pre-processing-e2e-inf-b-b5daee-797cb4f7f-ljgr2\" (UID: \"97fe4000-c797-4c39-8872-38308ffdf700\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b5daee-797cb4f7f-ljgr2" Nov 03 14:15:51.285987 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:51.285862 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97fe4000-c797-4c39-8872-38308ffdf700-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b5daee-797cb4f7f-ljgr2\" (UID: \"97fe4000-c797-4c39-8872-38308ffdf700\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b5daee-797cb4f7f-ljgr2" Nov 03 14:15:51.287559 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:51.287534 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97fe4000-c797-4c39-8872-38308ffdf700-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b5daee-797cb4f7f-ljgr2\" (UID: \"97fe4000-c797-4c39-8872-38308ffdf700\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b5daee-797cb4f7f-ljgr2" Nov 03 14:15:51.340302 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:51.340237 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzjn4\" (UniqueName: \"kubernetes.io/projected/97fe4000-c797-4c39-8872-38308ffdf700-kube-api-access-xzjn4\") pod \"payload-pre-processing-e2e-inf-b-b5daee-797cb4f7f-ljgr2\" (UID: \"97fe4000-c797-4c39-8872-38308ffdf700\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b5daee-797cb4f7f-ljgr2" Nov 03 14:15:51.569063 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:51.569023 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b5daee-5fcb8897cf-kdkph"] Nov 03 14:15:51.576493 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:15:51.575709 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0d0e2f4_b051_47a1_8118_88694a09dddc.slice/crio-5861eb5d5de14af26d6593b07d6af04096792f4ac64fa697a2cb426dbdecbfb5 WatchSource:0}: Error finding container 5861eb5d5de14af26d6593b07d6af04096792f4ac64fa697a2cb426dbdecbfb5: Status 404 returned error can't find the container with id 5861eb5d5de14af26d6593b07d6af04096792f4ac64fa697a2cb426dbdecbfb5 Nov 03 14:15:51.636405 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:51.636377 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b5daee-797cb4f7f-ljgr2" Nov 03 14:15:52.017452 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:52.017419 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b5daee-797cb4f7f-ljgr2"] Nov 03 14:15:52.024512 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:15:52.024482 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97fe4000_c797_4c39_8872_38308ffdf700.slice/crio-5b04ee55cec1c1152c553d8e751d732523002dbdad48d8ae55565dc939ac1c17 WatchSource:0}: Error finding container 5b04ee55cec1c1152c553d8e751d732523002dbdad48d8ae55565dc939ac1c17: Status 404 returned error can't find the container with id 5b04ee55cec1c1152c553d8e751d732523002dbdad48d8ae55565dc939ac1c17 Nov 03 14:15:52.470329 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:52.470291 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b5daee-797cb4f7f-ljgr2" event={"ID":"97fe4000-c797-4c39-8872-38308ffdf700","Type":"ContainerStarted","Data":"8a802fe4ca0c79bf8310918e639ab746257b6612611e5d227be9e48278b826cd"} Nov 03 14:15:52.470601 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:52.470344 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b5daee-797cb4f7f-ljgr2" event={"ID":"97fe4000-c797-4c39-8872-38308ffdf700","Type":"ContainerStarted","Data":"5b04ee55cec1c1152c553d8e751d732523002dbdad48d8ae55565dc939ac1c17"} Nov 03 14:15:52.470601 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:52.470388 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b5daee-797cb4f7f-ljgr2" Nov 03 14:15:52.472095 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:52.472071 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b5daee-5fcb8897cf-kdkph" event={"ID":"f0d0e2f4-b051-47a1-8118-88694a09dddc","Type":"ContainerStarted","Data":"c8c75f62c1211c923954989848838a280c4d2947d703bd3adac5a3354778ae19"} Nov 03 14:15:52.472219 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:52.472110 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b5daee-5fcb8897cf-kdkph" event={"ID":"f0d0e2f4-b051-47a1-8118-88694a09dddc","Type":"ContainerStarted","Data":"5861eb5d5de14af26d6593b07d6af04096792f4ac64fa697a2cb426dbdecbfb5"} Nov 03 14:15:52.472219 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:52.472140 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b5daee-5fcb8897cf-kdkph" Nov 03 14:15:52.580964 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:52.580900 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b5daee-797cb4f7f-ljgr2" podStartSLOduration=2.580882671 podStartE2EDuration="2.580882671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:15:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:15:52.576403363 +0000 UTC m=+4234.587091605" watchObservedRunningTime="2026-11-03 14:15:52.580882671 +0000 UTC m=+4234.591570913" Nov 03 14:15:52.700609 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:15:52.690523 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-b5daee-5fcb8897cf-kdkph" podStartSLOduration=2.690504194 podStartE2EDuration="2.690504194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:15:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:15:52.687577054 +0000 UTC m=+4234.698265311" watchObservedRunningTime="2026-11-03 14:15:52.690504194 +0000 UTC m=+4234.701192437" Nov 03 14:16:03.482539 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:16:03.482448 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b5daee-797cb4f7f-ljgr2" Nov 03 14:16:03.483094 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:16:03.482676 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b5daee-5fcb8897cf-kdkph" Nov 03 14:16:07.235186 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:16:07.235148 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:16:13.637315 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:16:13.637275 5979 generic.go:356] "Generic (PLEG): container finished" podID="9218884f-ec19-4d1b-8fbc-33fd62abf10f" containerID="99a0e937e244158693b60a6360059e1ce5d511e18de7f9143aeabf1b3cf5c25d" exitCode=0 Nov 03 14:16:13.637777 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:16:13.637343 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh" event={"ID":"9218884f-ec19-4d1b-8fbc-33fd62abf10f","Type":"ContainerDied","Data":"99a0e937e244158693b60a6360059e1ce5d511e18de7f9143aeabf1b3cf5c25d"} Nov 03 14:16:13.637824 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:16:13.637782 5979 scope.go:122] "RemoveContainer" containerID="99a0e937e244158693b60a6360059e1ce5d511e18de7f9143aeabf1b3cf5c25d" Nov 03 14:16:13.639326 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:16:13.639299 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:16:13.639386 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:16:13.639365 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:16:13.639431 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:16:13.639397 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:16:14.646283 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:16:14.646247 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh" event={"ID":"9218884f-ec19-4d1b-8fbc-33fd62abf10f","Type":"ContainerStarted","Data":"d91eae83e07dfd5c8fe50930eefb6c14bbc19078da4880da0d91cbb87db2b608"} Nov 03 14:16:14.805844 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:16:14.805805 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh" Nov 03 14:16:14.807263 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:16:14.807232 5979 patch_prober.go:29] interesting pod/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.82:15021/healthz/ready\": dial tcp 10.134.1.82:15021: connect: connection refused" start-of-body= Nov 03 14:16:14.807403 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:16:14.807302 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh" podUID="9218884f-ec19-4d1b-8fbc-33fd62abf10f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.82:15021/healthz/ready\": dial tcp 10.134.1.82:15021: connect: connection refused" Nov 03 14:16:15.806632 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:16:15.806598 5979 patch_prober.go:29] interesting pod/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.82:15021/healthz/ready\": dial tcp 10.134.1.82:15021: connect: connection refused" start-of-body= Nov 03 14:16:15.807055 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:16:15.806663 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh" podUID="9218884f-ec19-4d1b-8fbc-33fd62abf10f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.82:15021/healthz/ready\": dial tcp 10.134.1.82:15021: connect: connection refused" Nov 03 14:16:16.806721 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:16:16.806683 5979 patch_prober.go:29] interesting pod/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.82:15021/healthz/ready\": dial tcp 10.134.1.82:15021: connect: connection refused" start-of-body= Nov 03 14:16:16.807153 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:16:16.806744 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh" podUID="9218884f-ec19-4d1b-8fbc-33fd62abf10f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.82:15021/healthz/ready\": dial tcp 10.134.1.82:15021: connect: connection refused" Nov 03 14:16:17.805847 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:16:17.805809 5979 patch_prober.go:29] interesting pod/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.82:15021/healthz/ready\": dial tcp 10.134.1.82:15021: connect: connection refused" start-of-body= Nov 03 14:16:17.806033 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:16:17.805859 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh" podUID="9218884f-ec19-4d1b-8fbc-33fd62abf10f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.82:15021/healthz/ready\": dial tcp 10.134.1.82:15021: connect: connection refused" Nov 03 14:16:18.806731 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:16:18.806698 5979 patch_prober.go:29] interesting pod/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.82:15021/healthz/ready\": dial tcp 10.134.1.82:15021: connect: connection refused" start-of-body= Nov 03 14:16:18.807142 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:16:18.806745 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh" podUID="9218884f-ec19-4d1b-8fbc-33fd62abf10f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.82:15021/healthz/ready\": dial tcp 10.134.1.82:15021: connect: connection refused" Nov 03 14:16:19.169495 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:16:19.168601 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh" Nov 03 14:16:19.806286 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:16:19.806244 5979 patch_prober.go:29] interesting pod/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.82:15021/healthz/ready\": dial tcp 10.134.1.82:15021: connect: connection refused" start-of-body= Nov 03 14:16:19.806523 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:16:19.806305 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh" podUID="9218884f-ec19-4d1b-8fbc-33fd62abf10f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.82:15021/healthz/ready\": dial tcp 10.134.1.82:15021: connect: connection refused" Nov 03 14:16:20.811193 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:16:20.811163 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh" Nov 03 14:16:20.812113 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:16:20.812093 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh" Nov 03 14:16:37.235115 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:16:37.235075 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:16:47.577964 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:16:47.577925 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:17:07.277763 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:17:07.277723 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:17:28.529597 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:17:28.529557 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-89273860-7d6d454d77-dz5mp"] Nov 03 14:17:33.530364 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:17:33.530311 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-89273860-7d6d454d77-dz5mp" podUID="1ee90e7b-21ab-4625-94c9-373c51dc9b73" containerName="payload-processing" containerID="cri-o://35a1af80e53af21118063019add43901a25f9ac5eed5a1d89036d106b3090e45" gracePeriod=25 Nov 03 14:17:33.825849 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:17:33.825815 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-89273860-7d6d454d77-dz5mp" Nov 03 14:17:33.841762 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:17:33.841730 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ee90e7b-21ab-4625-94c9-373c51dc9b73-kube-api-access-jtb7q\" (UniqueName: \"kubernetes.io/projected/1ee90e7b-21ab-4625-94c9-373c51dc9b73-kube-api-access-jtb7q\") pod \"1ee90e7b-21ab-4625-94c9-373c51dc9b73\" (UID: \"1ee90e7b-21ab-4625-94c9-373c51dc9b73\") " Nov 03 14:17:33.841908 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:17:33.841825 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ee90e7b-21ab-4625-94c9-373c51dc9b73-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ee90e7b-21ab-4625-94c9-373c51dc9b73-plugins-config-volume\") pod \"1ee90e7b-21ab-4625-94c9-373c51dc9b73\" (UID: \"1ee90e7b-21ab-4625-94c9-373c51dc9b73\") " Nov 03 14:17:33.842174 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:17:33.842150 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ee90e7b-21ab-4625-94c9-373c51dc9b73-plugins-config-volume" pod "1ee90e7b-21ab-4625-94c9-373c51dc9b73" (UID: "1ee90e7b-21ab-4625-94c9-373c51dc9b73"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:17:33.843795 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:17:33.843772 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ee90e7b-21ab-4625-94c9-373c51dc9b73-kube-api-access-jtb7q" pod "1ee90e7b-21ab-4625-94c9-373c51dc9b73" (UID: "1ee90e7b-21ab-4625-94c9-373c51dc9b73"). InnerVolumeSpecName "kube-api-access-jtb7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:17:33.943647 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:17:33.943604 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtb7q\" (UniqueName: \"kubernetes.io/projected/1ee90e7b-21ab-4625-94c9-373c51dc9b73-kube-api-access-jtb7q\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:17:33.943647 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:17:33.943647 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ee90e7b-21ab-4625-94c9-373c51dc9b73-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:17:34.231290 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:17:34.231254 5979 generic.go:356] "Generic (PLEG): container finished" podID="1ee90e7b-21ab-4625-94c9-373c51dc9b73" containerID="35a1af80e53af21118063019add43901a25f9ac5eed5a1d89036d106b3090e45" exitCode=0 Nov 03 14:17:34.231509 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:17:34.231317 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-89273860-7d6d454d77-dz5mp" Nov 03 14:17:34.231509 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:17:34.231317 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-89273860-7d6d454d77-dz5mp" event={"ID":"1ee90e7b-21ab-4625-94c9-373c51dc9b73","Type":"ContainerDied","Data":"35a1af80e53af21118063019add43901a25f9ac5eed5a1d89036d106b3090e45"} Nov 03 14:17:34.231509 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:17:34.231457 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-89273860-7d6d454d77-dz5mp" event={"ID":"1ee90e7b-21ab-4625-94c9-373c51dc9b73","Type":"ContainerDied","Data":"47d1a563fb22237e4a35cc7dc9298324cac1e8802cf760f750c54c36b856127b"} Nov 03 14:17:34.231713 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:17:34.231520 5979 scope.go:122] "RemoveContainer" containerID="35a1af80e53af21118063019add43901a25f9ac5eed5a1d89036d106b3090e45" Nov 03 14:17:34.241360 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:17:34.241338 5979 scope.go:122] "RemoveContainer" containerID="35a1af80e53af21118063019add43901a25f9ac5eed5a1d89036d106b3090e45" Nov 03 14:17:34.241672 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:17:34.241649 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35a1af80e53af21118063019add43901a25f9ac5eed5a1d89036d106b3090e45\": container with ID starting with 35a1af80e53af21118063019add43901a25f9ac5eed5a1d89036d106b3090e45 not found: ID does not exist" containerID="35a1af80e53af21118063019add43901a25f9ac5eed5a1d89036d106b3090e45" Nov 03 14:17:34.241765 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:17:34.241681 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35a1af80e53af21118063019add43901a25f9ac5eed5a1d89036d106b3090e45"} err="failed to get container status \"35a1af80e53af21118063019add43901a25f9ac5eed5a1d89036d106b3090e45\": rpc error: code = NotFound desc = could not find container \"35a1af80e53af21118063019add43901a25f9ac5eed5a1d89036d106b3090e45\": container with ID starting with 35a1af80e53af21118063019add43901a25f9ac5eed5a1d89036d106b3090e45 not found: ID does not exist" Nov 03 14:17:34.386184 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:17:34.386148 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-89273860-7d6d454d77-dz5mp"] Nov 03 14:17:34.392161 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:17:34.392133 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-89273860-7d6d454d77-dz5mp"] Nov 03 14:17:36.116219 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:17:36.116182 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ee90e7b-21ab-4625-94c9-373c51dc9b73" path="/var/lib/kubelet/pods/1ee90e7b-21ab-4625-94c9-373c51dc9b73/volumes" Nov 03 14:17:36.253257 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:17:36.253222 5979 generic.go:356] "Generic (PLEG): container finished" podID="9650f53f-0ff8-49f0-ba84-1186f4753807" containerID="51380ace269523c2098117a40084ef8dced7c5e9b09ab75ffbd43e81e869b668" exitCode=0 Nov 03 14:17:36.253443 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:17:36.253293 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm" event={"ID":"9650f53f-0ff8-49f0-ba84-1186f4753807","Type":"ContainerDied","Data":"51380ace269523c2098117a40084ef8dced7c5e9b09ab75ffbd43e81e869b668"} Nov 03 14:17:36.253809 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:17:36.253788 5979 scope.go:122] "RemoveContainer" containerID="51380ace269523c2098117a40084ef8dced7c5e9b09ab75ffbd43e81e869b668" Nov 03 14:17:36.255573 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:17:36.255544 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:17:36.255699 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:17:36.255624 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:17:36.255699 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:17:36.255672 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:17:37.234970 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:17:37.234931 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:17:37.262693 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:17:37.262658 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm" event={"ID":"9650f53f-0ff8-49f0-ba84-1186f4753807","Type":"ContainerStarted","Data":"02f12789787e4198f74c4d0b0129c7475038d135f0ba2fdb4fcf607e1d730a6e"} Nov 03 14:17:37.873043 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:17:37.872998 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm" Nov 03 14:17:37.873282 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:17:37.873199 5979 patch_prober.go:29] interesting pod/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.90:15021/healthz/ready\": dial tcp 10.134.1.90:15021: connect: connection refused" start-of-body= Nov 03 14:17:37.873282 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:17:37.873241 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm" podUID="9650f53f-0ff8-49f0-ba84-1186f4753807" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.90:15021/healthz/ready\": dial tcp 10.134.1.90:15021: connect: connection refused" Nov 03 14:17:38.872927 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:17:38.872891 5979 patch_prober.go:29] interesting pod/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.90:15021/healthz/ready\": dial tcp 10.134.1.90:15021: connect: connection refused" start-of-body= Nov 03 14:17:38.873424 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:17:38.872950 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm" podUID="9650f53f-0ff8-49f0-ba84-1186f4753807" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.90:15021/healthz/ready\": dial tcp 10.134.1.90:15021: connect: connection refused" Nov 03 14:17:39.873649 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:17:39.873617 5979 patch_prober.go:29] interesting pod/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.90:15021/healthz/ready\": dial tcp 10.134.1.90:15021: connect: connection refused" start-of-body= Nov 03 14:17:39.874075 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:17:39.873679 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm" podUID="9650f53f-0ff8-49f0-ba84-1186f4753807" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.90:15021/healthz/ready\": dial tcp 10.134.1.90:15021: connect: connection refused" Nov 03 14:17:40.872896 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:17:40.872857 5979 patch_prober.go:29] interesting pod/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.90:15021/healthz/ready\": dial tcp 10.134.1.90:15021: connect: connection refused" start-of-body= Nov 03 14:17:40.873087 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:17:40.872913 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm" podUID="9650f53f-0ff8-49f0-ba84-1186f4753807" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.90:15021/healthz/ready\": dial tcp 10.134.1.90:15021: connect: connection refused" Nov 03 14:17:41.873130 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:17:41.873090 5979 patch_prober.go:29] interesting pod/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.90:15021/healthz/ready\": dial tcp 10.134.1.90:15021: connect: connection refused" start-of-body= Nov 03 14:17:41.873610 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:17:41.873167 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm" podUID="9650f53f-0ff8-49f0-ba84-1186f4753807" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.90:15021/healthz/ready\": dial tcp 10.134.1.90:15021: connect: connection refused" Nov 03 14:17:42.873722 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:17:42.873686 5979 patch_prober.go:29] interesting pod/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.90:15021/healthz/ready\": dial tcp 10.134.1.90:15021: connect: connection refused" start-of-body= Nov 03 14:17:42.874160 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:17:42.873745 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm" podUID="9650f53f-0ff8-49f0-ba84-1186f4753807" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.90:15021/healthz/ready\": dial tcp 10.134.1.90:15021: connect: connection refused" Nov 03 14:17:43.877897 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:17:43.877865 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm" Nov 03 14:17:43.878344 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:17:43.878331 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm" Nov 03 14:17:43.878993 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:17:43.878975 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm" Nov 03 14:18:06.320252 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:06.320211 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:18:07.235311 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:18:07.235268 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:18:36.159886 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:36.159851 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh"] Nov 03 14:18:36.160358 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:36.160093 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh" podUID="9218884f-ec19-4d1b-8fbc-33fd62abf10f" containerName="istio-proxy" containerID="cri-o://d91eae83e07dfd5c8fe50930eefb6c14bbc19078da4880da0d91cbb87db2b608" gracePeriod=30 Nov 03 14:18:37.235350 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:18:37.235304 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:18:41.449555 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:41.449466 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh" Nov 03 14:18:41.537791 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:41.537758 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9218884f-ec19-4d1b-8fbc-33fd62abf10f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9218884f-ec19-4d1b-8fbc-33fd62abf10f-credential-socket\") pod \"9218884f-ec19-4d1b-8fbc-33fd62abf10f\" (UID: \"9218884f-ec19-4d1b-8fbc-33fd62abf10f\") " Nov 03 14:18:41.538000 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:41.537801 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9218884f-ec19-4d1b-8fbc-33fd62abf10f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9218884f-ec19-4d1b-8fbc-33fd62abf10f-istio-data\") pod \"9218884f-ec19-4d1b-8fbc-33fd62abf10f\" (UID: \"9218884f-ec19-4d1b-8fbc-33fd62abf10f\") " Nov 03 14:18:41.538000 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:41.537831 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9218884f-ec19-4d1b-8fbc-33fd62abf10f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9218884f-ec19-4d1b-8fbc-33fd62abf10f-workload-certs\") pod \"9218884f-ec19-4d1b-8fbc-33fd62abf10f\" (UID: \"9218884f-ec19-4d1b-8fbc-33fd62abf10f\") " Nov 03 14:18:41.538000 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:41.537948 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9218884f-ec19-4d1b-8fbc-33fd62abf10f-kube-api-access-xhxj8\" (UniqueName: \"kubernetes.io/projected/9218884f-ec19-4d1b-8fbc-33fd62abf10f-kube-api-access-xhxj8\") pod \"9218884f-ec19-4d1b-8fbc-33fd62abf10f\" (UID: \"9218884f-ec19-4d1b-8fbc-33fd62abf10f\") " Nov 03 14:18:41.538000 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:41.537980 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9218884f-ec19-4d1b-8fbc-33fd62abf10f-istio-token\" (UniqueName: \"kubernetes.io/projected/9218884f-ec19-4d1b-8fbc-33fd62abf10f-istio-token\") pod \"9218884f-ec19-4d1b-8fbc-33fd62abf10f\" (UID: \"9218884f-ec19-4d1b-8fbc-33fd62abf10f\") " Nov 03 14:18:41.538229 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:41.538091 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9218884f-ec19-4d1b-8fbc-33fd62abf10f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9218884f-ec19-4d1b-8fbc-33fd62abf10f-workload-socket\") pod \"9218884f-ec19-4d1b-8fbc-33fd62abf10f\" (UID: \"9218884f-ec19-4d1b-8fbc-33fd62abf10f\") " Nov 03 14:18:41.538229 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:41.538089 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9218884f-ec19-4d1b-8fbc-33fd62abf10f-istio-data" pod "9218884f-ec19-4d1b-8fbc-33fd62abf10f" (UID: "9218884f-ec19-4d1b-8fbc-33fd62abf10f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:18:41.538229 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:41.538099 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9218884f-ec19-4d1b-8fbc-33fd62abf10f-credential-socket" pod "9218884f-ec19-4d1b-8fbc-33fd62abf10f" (UID: "9218884f-ec19-4d1b-8fbc-33fd62abf10f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:18:41.538229 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:41.538168 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9218884f-ec19-4d1b-8fbc-33fd62abf10f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9218884f-ec19-4d1b-8fbc-33fd62abf10f-istio-podinfo\") pod \"9218884f-ec19-4d1b-8fbc-33fd62abf10f\" (UID: \"9218884f-ec19-4d1b-8fbc-33fd62abf10f\") " Nov 03 14:18:41.538452 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:41.538235 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9218884f-ec19-4d1b-8fbc-33fd62abf10f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9218884f-ec19-4d1b-8fbc-33fd62abf10f-istio-envoy\") pod \"9218884f-ec19-4d1b-8fbc-33fd62abf10f\" (UID: \"9218884f-ec19-4d1b-8fbc-33fd62abf10f\") " Nov 03 14:18:41.538452 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:41.538289 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9218884f-ec19-4d1b-8fbc-33fd62abf10f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9218884f-ec19-4d1b-8fbc-33fd62abf10f-istiod-ca-cert\") pod \"9218884f-ec19-4d1b-8fbc-33fd62abf10f\" (UID: \"9218884f-ec19-4d1b-8fbc-33fd62abf10f\") " Nov 03 14:18:41.538452 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:41.538380 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9218884f-ec19-4d1b-8fbc-33fd62abf10f-workload-socket" pod "9218884f-ec19-4d1b-8fbc-33fd62abf10f" (UID: "9218884f-ec19-4d1b-8fbc-33fd62abf10f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:18:41.538635 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:41.538592 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9218884f-ec19-4d1b-8fbc-33fd62abf10f-workload-certs" pod "9218884f-ec19-4d1b-8fbc-33fd62abf10f" (UID: "9218884f-ec19-4d1b-8fbc-33fd62abf10f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:18:41.539108 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:41.538779 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9218884f-ec19-4d1b-8fbc-33fd62abf10f-istiod-ca-cert" pod "9218884f-ec19-4d1b-8fbc-33fd62abf10f" (UID: "9218884f-ec19-4d1b-8fbc-33fd62abf10f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:18:41.539108 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:41.539015 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9218884f-ec19-4d1b-8fbc-33fd62abf10f-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:18:41.539108 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:41.539034 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9218884f-ec19-4d1b-8fbc-33fd62abf10f-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:18:41.539108 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:41.539051 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9218884f-ec19-4d1b-8fbc-33fd62abf10f-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:18:41.539108 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:41.539065 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9218884f-ec19-4d1b-8fbc-33fd62abf10f-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:18:41.539108 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:41.539079 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9218884f-ec19-4d1b-8fbc-33fd62abf10f-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:18:41.540947 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:41.540825 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9218884f-ec19-4d1b-8fbc-33fd62abf10f-istio-envoy" pod "9218884f-ec19-4d1b-8fbc-33fd62abf10f" (UID: "9218884f-ec19-4d1b-8fbc-33fd62abf10f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:18:41.540947 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:41.540922 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9218884f-ec19-4d1b-8fbc-33fd62abf10f-istio-podinfo" pod "9218884f-ec19-4d1b-8fbc-33fd62abf10f" (UID: "9218884f-ec19-4d1b-8fbc-33fd62abf10f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:18:41.541137 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:41.540990 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9218884f-ec19-4d1b-8fbc-33fd62abf10f-istio-token" pod "9218884f-ec19-4d1b-8fbc-33fd62abf10f" (UID: "9218884f-ec19-4d1b-8fbc-33fd62abf10f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:18:41.541366 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:41.541346 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9218884f-ec19-4d1b-8fbc-33fd62abf10f-kube-api-access-xhxj8" pod "9218884f-ec19-4d1b-8fbc-33fd62abf10f" (UID: "9218884f-ec19-4d1b-8fbc-33fd62abf10f"). InnerVolumeSpecName "kube-api-access-xhxj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:18:41.640877 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:41.640841 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9218884f-ec19-4d1b-8fbc-33fd62abf10f-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:18:41.640877 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:41.640872 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9218884f-ec19-4d1b-8fbc-33fd62abf10f-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:18:41.640877 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:41.640882 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhxj8\" (UniqueName: \"kubernetes.io/projected/9218884f-ec19-4d1b-8fbc-33fd62abf10f-kube-api-access-xhxj8\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:18:41.641132 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:41.640891 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9218884f-ec19-4d1b-8fbc-33fd62abf10f-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:18:41.708960 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:41.708868 5979 generic.go:356] "Generic (PLEG): container finished" podID="9218884f-ec19-4d1b-8fbc-33fd62abf10f" containerID="d91eae83e07dfd5c8fe50930eefb6c14bbc19078da4880da0d91cbb87db2b608" exitCode=0 Nov 03 14:18:41.708960 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:41.708909 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh" event={"ID":"9218884f-ec19-4d1b-8fbc-33fd62abf10f","Type":"ContainerDied","Data":"d91eae83e07dfd5c8fe50930eefb6c14bbc19078da4880da0d91cbb87db2b608"} Nov 03 14:18:41.708960 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:41.708949 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh" Nov 03 14:18:41.709332 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:41.709006 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh" event={"ID":"9218884f-ec19-4d1b-8fbc-33fd62abf10f","Type":"ContainerDied","Data":"99b0cad9dbb53f62bf408086e6c789df688e422a91253453367990461ac63b88"} Nov 03 14:18:41.709332 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:41.709031 5979 scope.go:122] "RemoveContainer" containerID="d91eae83e07dfd5c8fe50930eefb6c14bbc19078da4880da0d91cbb87db2b608" Nov 03 14:18:41.721969 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:41.720777 5979 scope.go:122] "RemoveContainer" containerID="99a0e937e244158693b60a6360059e1ce5d511e18de7f9143aeabf1b3cf5c25d" Nov 03 14:18:41.731430 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:41.731412 5979 scope.go:122] "RemoveContainer" containerID="d91eae83e07dfd5c8fe50930eefb6c14bbc19078da4880da0d91cbb87db2b608" Nov 03 14:18:41.731775 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:18:41.731757 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d91eae83e07dfd5c8fe50930eefb6c14bbc19078da4880da0d91cbb87db2b608\": container with ID starting with d91eae83e07dfd5c8fe50930eefb6c14bbc19078da4880da0d91cbb87db2b608 not found: ID does not exist" containerID="d91eae83e07dfd5c8fe50930eefb6c14bbc19078da4880da0d91cbb87db2b608" Nov 03 14:18:41.731827 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:41.731784 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d91eae83e07dfd5c8fe50930eefb6c14bbc19078da4880da0d91cbb87db2b608"} err="failed to get container status \"d91eae83e07dfd5c8fe50930eefb6c14bbc19078da4880da0d91cbb87db2b608\": rpc error: code = NotFound desc = could not find container \"d91eae83e07dfd5c8fe50930eefb6c14bbc19078da4880da0d91cbb87db2b608\": container with ID starting with d91eae83e07dfd5c8fe50930eefb6c14bbc19078da4880da0d91cbb87db2b608 not found: ID does not exist" Nov 03 14:18:41.731827 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:41.731801 5979 scope.go:122] "RemoveContainer" containerID="99a0e937e244158693b60a6360059e1ce5d511e18de7f9143aeabf1b3cf5c25d" Nov 03 14:18:41.732021 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:18:41.732004 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99a0e937e244158693b60a6360059e1ce5d511e18de7f9143aeabf1b3cf5c25d\": container with ID starting with 99a0e937e244158693b60a6360059e1ce5d511e18de7f9143aeabf1b3cf5c25d not found: ID does not exist" containerID="99a0e937e244158693b60a6360059e1ce5d511e18de7f9143aeabf1b3cf5c25d" Nov 03 14:18:41.732066 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:41.732024 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99a0e937e244158693b60a6360059e1ce5d511e18de7f9143aeabf1b3cf5c25d"} err="failed to get container status \"99a0e937e244158693b60a6360059e1ce5d511e18de7f9143aeabf1b3cf5c25d\": rpc error: code = NotFound desc = could not find container \"99a0e937e244158693b60a6360059e1ce5d511e18de7f9143aeabf1b3cf5c25d\": container with ID starting with 99a0e937e244158693b60a6360059e1ce5d511e18de7f9143aeabf1b3cf5c25d not found: ID does not exist" Nov 03 14:18:41.863783 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:41.863747 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh"] Nov 03 14:18:41.869263 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:41.869230 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-89273860-openshift-default-5f5768f46f-gq8wh"] Nov 03 14:18:42.116747 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:42.116671 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9218884f-ec19-4d1b-8fbc-33fd62abf10f" path="/var/lib/kubelet/pods/9218884f-ec19-4d1b-8fbc-33fd62abf10f/volumes" Nov 03 14:18:43.757660 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:43.757628 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e745c4d5-bwwdl"] Nov 03 14:18:43.758348 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:43.758330 5979 cpu_manager.go:414] "containerMap: removing container" podUID="9218884f-ec19-4d1b-8fbc-33fd62abf10f" containerName="istio-proxy" Nov 03 14:18:43.758402 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:43.758349 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9218884f-ec19-4d1b-8fbc-33fd62abf10f" containerName="istio-proxy" Nov 03 14:18:43.758402 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:43.758399 5979 cpu_manager.go:414] "containerMap: removing container" podUID="9218884f-ec19-4d1b-8fbc-33fd62abf10f" containerName="istio-proxy" Nov 03 14:18:43.758494 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:43.758405 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9218884f-ec19-4d1b-8fbc-33fd62abf10f" containerName="istio-proxy" Nov 03 14:18:43.758494 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:43.758414 5979 cpu_manager.go:414] "containerMap: removing container" podUID="1ee90e7b-21ab-4625-94c9-373c51dc9b73" containerName="payload-processing" Nov 03 14:18:43.758494 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:43.758419 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ee90e7b-21ab-4625-94c9-373c51dc9b73" containerName="payload-processing" Nov 03 14:18:43.758609 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:43.758568 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="9218884f-ec19-4d1b-8fbc-33fd62abf10f" containerName="istio-proxy" Nov 03 14:18:43.758609 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:43.758581 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ee90e7b-21ab-4625-94c9-373c51dc9b73" containerName="payload-processing" Nov 03 14:18:43.763662 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:43.763642 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e745c4d5-bwwdl" Nov 03 14:18:43.816619 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:43.816565 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e745c4d5-bwwdl"] Nov 03 14:18:43.838815 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:43.838776 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:18:43.858898 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:43.858866 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 14:18:43.865541 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:43.865517 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2048118-7dbd-46d4-9f4b-fec408439ed6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e745c4d5-bwwdl\" (UID: \"b2048118-7dbd-46d4-9f4b-fec408439ed6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e745c4d5-bwwdl" Nov 03 14:18:43.865729 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:43.865711 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2048118-7dbd-46d4-9f4b-fec408439ed6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e745c4d5-bwwdl\" (UID: \"b2048118-7dbd-46d4-9f4b-fec408439ed6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e745c4d5-bwwdl" Nov 03 14:18:43.865773 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:43.865739 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzjwv\" (UniqueName: \"kubernetes.io/projected/b2048118-7dbd-46d4-9f4b-fec408439ed6-kube-api-access-hzjwv\") pod \"maas-api-revoke-sub-e745c4d5-bwwdl\" (UID: \"b2048118-7dbd-46d4-9f4b-fec408439ed6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e745c4d5-bwwdl" Nov 03 14:18:43.966974 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:43.966939 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2048118-7dbd-46d4-9f4b-fec408439ed6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e745c4d5-bwwdl\" (UID: \"b2048118-7dbd-46d4-9f4b-fec408439ed6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e745c4d5-bwwdl" Nov 03 14:18:43.966974 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:43.966974 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzjwv\" (UniqueName: \"kubernetes.io/projected/b2048118-7dbd-46d4-9f4b-fec408439ed6-kube-api-access-hzjwv\") pod \"maas-api-revoke-sub-e745c4d5-bwwdl\" (UID: \"b2048118-7dbd-46d4-9f4b-fec408439ed6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e745c4d5-bwwdl" Nov 03 14:18:43.967195 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:43.967069 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2048118-7dbd-46d4-9f4b-fec408439ed6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e745c4d5-bwwdl\" (UID: \"b2048118-7dbd-46d4-9f4b-fec408439ed6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e745c4d5-bwwdl" Nov 03 14:18:43.967823 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:43.967798 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2048118-7dbd-46d4-9f4b-fec408439ed6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e745c4d5-bwwdl\" (UID: \"b2048118-7dbd-46d4-9f4b-fec408439ed6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e745c4d5-bwwdl" Nov 03 14:18:43.969429 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:43.969410 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2048118-7dbd-46d4-9f4b-fec408439ed6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e745c4d5-bwwdl\" (UID: \"b2048118-7dbd-46d4-9f4b-fec408439ed6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e745c4d5-bwwdl" Nov 03 14:18:44.021010 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:44.020935 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzjwv\" (UniqueName: \"kubernetes.io/projected/b2048118-7dbd-46d4-9f4b-fec408439ed6-kube-api-access-hzjwv\") pod \"maas-api-revoke-sub-e745c4d5-bwwdl\" (UID: \"b2048118-7dbd-46d4-9f4b-fec408439ed6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e745c4d5-bwwdl" Nov 03 14:18:44.077906 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:44.077868 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e745c4d5-bwwdl" Nov 03 14:18:44.454028 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:44.453996 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e745c4d5-bwwdl"] Nov 03 14:18:44.460865 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:18:44.460831 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2048118_7dbd_46d4_9f4b_fec408439ed6.slice/crio-87469b8631f37d746232432dd3e2dd7b9f3d7fca2e36cf568be5de83de5b744c WatchSource:0}: Error finding container 87469b8631f37d746232432dd3e2dd7b9f3d7fca2e36cf568be5de83de5b744c: Status 404 returned error can't find the container with id 87469b8631f37d746232432dd3e2dd7b9f3d7fca2e36cf568be5de83de5b744c Nov 03 14:18:44.732178 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:44.732094 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e745c4d5-bwwdl" event={"ID":"b2048118-7dbd-46d4-9f4b-fec408439ed6","Type":"ContainerStarted","Data":"87469b8631f37d746232432dd3e2dd7b9f3d7fca2e36cf568be5de83de5b744c"} Nov 03 14:18:45.741647 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:45.741561 5979 generic.go:356] "Generic (PLEG): container finished" podID="b2048118-7dbd-46d4-9f4b-fec408439ed6" containerID="cd91282891faefe16536226f153f88d31daa75c15f4c140493065992678cc1b8" exitCode=0 Nov 03 14:18:45.742013 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:45.741645 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e745c4d5-bwwdl" event={"ID":"b2048118-7dbd-46d4-9f4b-fec408439ed6","Type":"ContainerDied","Data":"cd91282891faefe16536226f153f88d31daa75c15f4c140493065992678cc1b8"} Nov 03 14:18:46.904757 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:46.904727 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e745c4d5-bwwdl" Nov 03 14:18:47.002302 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:47.002210 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2048118-7dbd-46d4-9f4b-fec408439ed6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2048118-7dbd-46d4-9f4b-fec408439ed6-maas-api-openshift-ca\") pod \"b2048118-7dbd-46d4-9f4b-fec408439ed6\" (UID: \"b2048118-7dbd-46d4-9f4b-fec408439ed6\") " Nov 03 14:18:47.002522 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:47.002300 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2048118-7dbd-46d4-9f4b-fec408439ed6-kube-api-access-hzjwv\" (UniqueName: \"kubernetes.io/projected/b2048118-7dbd-46d4-9f4b-fec408439ed6-kube-api-access-hzjwv\") pod \"b2048118-7dbd-46d4-9f4b-fec408439ed6\" (UID: \"b2048118-7dbd-46d4-9f4b-fec408439ed6\") " Nov 03 14:18:47.002522 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:47.002326 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2048118-7dbd-46d4-9f4b-fec408439ed6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2048118-7dbd-46d4-9f4b-fec408439ed6-maas-api-xks-ca\") pod \"b2048118-7dbd-46d4-9f4b-fec408439ed6\" (UID: \"b2048118-7dbd-46d4-9f4b-fec408439ed6\") " Nov 03 14:18:47.002669 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:47.002619 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2048118-7dbd-46d4-9f4b-fec408439ed6-maas-api-openshift-ca" pod "b2048118-7dbd-46d4-9f4b-fec408439ed6" (UID: "b2048118-7dbd-46d4-9f4b-fec408439ed6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:18:47.002886 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:47.002865 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2048118-7dbd-46d4-9f4b-fec408439ed6-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:18:47.004482 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:47.004450 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2048118-7dbd-46d4-9f4b-fec408439ed6-maas-api-xks-ca" pod "b2048118-7dbd-46d4-9f4b-fec408439ed6" (UID: "b2048118-7dbd-46d4-9f4b-fec408439ed6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:18:47.004564 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:47.004488 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2048118-7dbd-46d4-9f4b-fec408439ed6-kube-api-access-hzjwv" pod "b2048118-7dbd-46d4-9f4b-fec408439ed6" (UID: "b2048118-7dbd-46d4-9f4b-fec408439ed6"). InnerVolumeSpecName "kube-api-access-hzjwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:18:47.104579 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:47.104521 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzjwv\" (UniqueName: \"kubernetes.io/projected/b2048118-7dbd-46d4-9f4b-fec408439ed6-kube-api-access-hzjwv\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:18:47.104579 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:47.104561 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2048118-7dbd-46d4-9f4b-fec408439ed6-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:18:47.757659 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:47.757607 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e745c4d5-bwwdl" Nov 03 14:18:47.757884 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:47.757599 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e745c4d5-bwwdl" event={"ID":"b2048118-7dbd-46d4-9f4b-fec408439ed6","Type":"ContainerDied","Data":"87469b8631f37d746232432dd3e2dd7b9f3d7fca2e36cf568be5de83de5b744c"} Nov 03 14:18:47.757884 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:47.757768 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87469b8631f37d746232432dd3e2dd7b9f3d7fca2e36cf568be5de83de5b744c" Nov 03 14:18:48.054910 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:48.054830 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5bb463/distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw"] Nov 03 14:18:48.078530 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:48.078490 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5bb463/premium-w3-5bb463-kserve-5f6ff6795c-jxmsr"] Nov 03 14:18:56.452564 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:56.452530 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dceac349-7nzfc"] Nov 03 14:18:56.453166 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:56.453150 5979 cpu_manager.go:414] "containerMap: removing container" podUID="b2048118-7dbd-46d4-9f4b-fec408439ed6" containerName="revoke-keys" Nov 03 14:18:56.453213 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:56.453167 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2048118-7dbd-46d4-9f4b-fec408439ed6" containerName="revoke-keys" Nov 03 14:18:56.453285 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:56.453276 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2048118-7dbd-46d4-9f4b-fec408439ed6" containerName="revoke-keys" Nov 03 14:18:56.453323 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:56.453290 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="9218884f-ec19-4d1b-8fbc-33fd62abf10f" containerName="istio-proxy" Nov 03 14:18:56.456959 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:56.456938 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dceac349-7nzfc" Nov 03 14:18:56.507953 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:56.507917 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:18:56.528843 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:56.528808 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k25z\" (UniqueName: \"kubernetes.io/projected/a29e24d0-3285-4676-8171-622e9dedacd4-kube-api-access-8k25z\") pod \"maas-api-revoke-sub-dceac349-7nzfc\" (UID: \"a29e24d0-3285-4676-8171-622e9dedacd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dceac349-7nzfc" Nov 03 14:18:56.529007 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:56.528895 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a29e24d0-3285-4676-8171-622e9dedacd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dceac349-7nzfc\" (UID: \"a29e24d0-3285-4676-8171-622e9dedacd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dceac349-7nzfc" Nov 03 14:18:56.529054 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:56.529003 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a29e24d0-3285-4676-8171-622e9dedacd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dceac349-7nzfc\" (UID: \"a29e24d0-3285-4676-8171-622e9dedacd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dceac349-7nzfc" Nov 03 14:18:56.554490 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:56.554449 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 14:18:56.557218 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:56.557195 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dceac349-7nzfc"] Nov 03 14:18:56.630037 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:56.629998 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a29e24d0-3285-4676-8171-622e9dedacd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dceac349-7nzfc\" (UID: \"a29e24d0-3285-4676-8171-622e9dedacd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dceac349-7nzfc" Nov 03 14:18:56.630217 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:56.630054 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a29e24d0-3285-4676-8171-622e9dedacd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dceac349-7nzfc\" (UID: \"a29e24d0-3285-4676-8171-622e9dedacd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dceac349-7nzfc" Nov 03 14:18:56.630297 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:56.630279 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k25z\" (UniqueName: \"kubernetes.io/projected/a29e24d0-3285-4676-8171-622e9dedacd4-kube-api-access-8k25z\") pod \"maas-api-revoke-sub-dceac349-7nzfc\" (UID: \"a29e24d0-3285-4676-8171-622e9dedacd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dceac349-7nzfc" Nov 03 14:18:56.630760 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:56.630741 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a29e24d0-3285-4676-8171-622e9dedacd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dceac349-7nzfc\" (UID: \"a29e24d0-3285-4676-8171-622e9dedacd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dceac349-7nzfc" Nov 03 14:18:56.632543 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:56.632511 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a29e24d0-3285-4676-8171-622e9dedacd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dceac349-7nzfc\" (UID: \"a29e24d0-3285-4676-8171-622e9dedacd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dceac349-7nzfc" Nov 03 14:18:56.683967 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:56.683930 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k25z\" (UniqueName: \"kubernetes.io/projected/a29e24d0-3285-4676-8171-622e9dedacd4-kube-api-access-8k25z\") pod \"maas-api-revoke-sub-dceac349-7nzfc\" (UID: \"a29e24d0-3285-4676-8171-622e9dedacd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dceac349-7nzfc" Nov 03 14:18:56.771358 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:56.771274 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dceac349-7nzfc" Nov 03 14:18:57.147004 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:57.146961 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dceac349-7nzfc"] Nov 03 14:18:57.153936 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:18:57.153905 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda29e24d0_3285_4676_8171_622e9dedacd4.slice/crio-c0ad1761888b71fda01551ff0d5f2441abe4ca6d95df8ca7d091196a6e033750 WatchSource:0}: Error finding container c0ad1761888b71fda01551ff0d5f2441abe4ca6d95df8ca7d091196a6e033750: Status 404 returned error can't find the container with id c0ad1761888b71fda01551ff0d5f2441abe4ca6d95df8ca7d091196a6e033750 Nov 03 14:18:57.823784 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:57.823694 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dceac349-7nzfc" event={"ID":"a29e24d0-3285-4676-8171-622e9dedacd4","Type":"ContainerStarted","Data":"b65eb00a04749cce05c7b560ee88e7584bf04f040e5185da89fa8ce35294a721"} Nov 03 14:18:57.823784 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:57.823755 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dceac349-7nzfc" event={"ID":"a29e24d0-3285-4676-8171-622e9dedacd4","Type":"ContainerStarted","Data":"c0ad1761888b71fda01551ff0d5f2441abe4ca6d95df8ca7d091196a6e033750"} Nov 03 14:18:57.935316 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:57.935255 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dceac349-7nzfc" podStartSLOduration=1.935241167 podStartE2EDuration="1.935241167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:18:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:18:57.932686863 +0000 UTC m=+4419.943375105" watchObservedRunningTime="2026-11-03 14:18:57.935241167 +0000 UTC m=+4419.945929408" Nov 03 14:18:58.835384 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:58.835160 5979 generic.go:356] "Generic (PLEG): container finished" podID="a29e24d0-3285-4676-8171-622e9dedacd4" containerID="b65eb00a04749cce05c7b560ee88e7584bf04f040e5185da89fa8ce35294a721" exitCode=0 Nov 03 14:18:58.835384 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:58.835256 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dceac349-7nzfc" event={"ID":"a29e24d0-3285-4676-8171-622e9dedacd4","Type":"ContainerDied","Data":"b65eb00a04749cce05c7b560ee88e7584bf04f040e5185da89fa8ce35294a721"} Nov 03 14:18:59.994864 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:18:59.994830 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dceac349-7nzfc" Nov 03 14:19:00.081823 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:00.081785 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a29e24d0-3285-4676-8171-622e9dedacd4-kube-api-access-8k25z\" (UniqueName: \"kubernetes.io/projected/a29e24d0-3285-4676-8171-622e9dedacd4-kube-api-access-8k25z\") pod \"a29e24d0-3285-4676-8171-622e9dedacd4\" (UID: \"a29e24d0-3285-4676-8171-622e9dedacd4\") " Nov 03 14:19:00.082032 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:00.081839 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a29e24d0-3285-4676-8171-622e9dedacd4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a29e24d0-3285-4676-8171-622e9dedacd4-maas-api-xks-ca\") pod \"a29e24d0-3285-4676-8171-622e9dedacd4\" (UID: \"a29e24d0-3285-4676-8171-622e9dedacd4\") " Nov 03 14:19:00.082032 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:00.081893 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a29e24d0-3285-4676-8171-622e9dedacd4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a29e24d0-3285-4676-8171-622e9dedacd4-maas-api-openshift-ca\") pod \"a29e24d0-3285-4676-8171-622e9dedacd4\" (UID: \"a29e24d0-3285-4676-8171-622e9dedacd4\") " Nov 03 14:19:00.082448 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:00.082418 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a29e24d0-3285-4676-8171-622e9dedacd4-maas-api-openshift-ca" pod "a29e24d0-3285-4676-8171-622e9dedacd4" (UID: "a29e24d0-3285-4676-8171-622e9dedacd4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:19:00.084156 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:00.084131 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a29e24d0-3285-4676-8171-622e9dedacd4-maas-api-xks-ca" pod "a29e24d0-3285-4676-8171-622e9dedacd4" (UID: "a29e24d0-3285-4676-8171-622e9dedacd4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:19:00.084375 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:00.084353 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a29e24d0-3285-4676-8171-622e9dedacd4-kube-api-access-8k25z" pod "a29e24d0-3285-4676-8171-622e9dedacd4" (UID: "a29e24d0-3285-4676-8171-622e9dedacd4"). InnerVolumeSpecName "kube-api-access-8k25z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:19:00.085132 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:00.085107 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:19:00.183732 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:00.183638 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8k25z\" (UniqueName: \"kubernetes.io/projected/a29e24d0-3285-4676-8171-622e9dedacd4-kube-api-access-8k25z\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:19:00.183732 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:00.183675 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a29e24d0-3285-4676-8171-622e9dedacd4-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:19:00.183732 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:00.183692 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a29e24d0-3285-4676-8171-622e9dedacd4-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:19:00.885640 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:00.885595 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dceac349-7nzfc" event={"ID":"a29e24d0-3285-4676-8171-622e9dedacd4","Type":"ContainerDied","Data":"c0ad1761888b71fda01551ff0d5f2441abe4ca6d95df8ca7d091196a6e033750"} Nov 03 14:19:00.885861 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:00.885667 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0ad1761888b71fda01551ff0d5f2441abe4ca6d95df8ca7d091196a6e033750" Nov 03 14:19:00.885861 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:00.885744 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dceac349-7nzfc" Nov 03 14:19:03.076432 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:03.076367 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-5bb463/distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw" podUID="f7c171f9-4569-431e-a891-5cae3328ce47" containerName="main" containerID="cri-o://55ca2f708f84e253317eb6c95027c22de16c5ee1e2fabbefd12f2809d16ee760" gracePeriod=45 Nov 03 14:19:03.099959 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:03.099911 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-5bb463/premium-w3-5bb463-kserve-5f6ff6795c-jxmsr" podUID="8ee26f55-5c0c-4bd8-ad9b-fb8c03641698" containerName="main" containerID="cri-o://7874929dd2fd364bf24c1a522a68cf1174b153b65c56d3b7a377566e71e3f23b" gracePeriod=45 Nov 03 14:19:03.386020 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:03.385987 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5bb463/distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw" Nov 03 14:19:03.407360 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:03.407334 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5bb463/premium-w3-5bb463-kserve-5f6ff6795c-jxmsr" Nov 03 14:19:03.429540 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:03.429506 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7c171f9-4569-431e-a891-5cae3328ce47-kube-api-access-9g9rx\" (UniqueName: \"kubernetes.io/projected/f7c171f9-4569-431e-a891-5cae3328ce47-kube-api-access-9g9rx\") pod \"f7c171f9-4569-431e-a891-5cae3328ce47\" (UID: \"f7c171f9-4569-431e-a891-5cae3328ce47\") " Nov 03 14:19:03.429679 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:03.429602 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7c171f9-4569-431e-a891-5cae3328ce47-home\" (UniqueName: \"kubernetes.io/empty-dir/f7c171f9-4569-431e-a891-5cae3328ce47-home\") pod \"f7c171f9-4569-431e-a891-5cae3328ce47\" (UID: \"f7c171f9-4569-431e-a891-5cae3328ce47\") " Nov 03 14:19:03.429679 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:03.429637 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7c171f9-4569-431e-a891-5cae3328ce47-tls-certs\" (UniqueName: \"kubernetes.io/secret/f7c171f9-4569-431e-a891-5cae3328ce47-tls-certs\") pod \"f7c171f9-4569-431e-a891-5cae3328ce47\" (UID: \"f7c171f9-4569-431e-a891-5cae3328ce47\") " Nov 03 14:19:03.429679 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:03.429660 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7c171f9-4569-431e-a891-5cae3328ce47-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7c171f9-4569-431e-a891-5cae3328ce47-model-cache\") pod \"f7c171f9-4569-431e-a891-5cae3328ce47\" (UID: \"f7c171f9-4569-431e-a891-5cae3328ce47\") " Nov 03 14:19:03.429851 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:03.429812 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7c171f9-4569-431e-a891-5cae3328ce47-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7c171f9-4569-431e-a891-5cae3328ce47-tmp-dir\") pod \"f7c171f9-4569-431e-a891-5cae3328ce47\" (UID: \"f7c171f9-4569-431e-a891-5cae3328ce47\") " Nov 03 14:19:03.429909 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:03.429855 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7c171f9-4569-431e-a891-5cae3328ce47-home" pod "f7c171f9-4569-431e-a891-5cae3328ce47" (UID: "f7c171f9-4569-431e-a891-5cae3328ce47"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:19:03.429967 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:03.429935 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7c171f9-4569-431e-a891-5cae3328ce47-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7c171f9-4569-431e-a891-5cae3328ce47-dshm\") pod \"f7c171f9-4569-431e-a891-5cae3328ce47\" (UID: \"f7c171f9-4569-431e-a891-5cae3328ce47\") " Nov 03 14:19:03.430033 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:03.429960 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7c171f9-4569-431e-a891-5cae3328ce47-model-cache" pod "f7c171f9-4569-431e-a891-5cae3328ce47" (UID: "f7c171f9-4569-431e-a891-5cae3328ce47"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:19:03.430091 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:03.430049 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7c171f9-4569-431e-a891-5cae3328ce47-tmp-dir" pod "f7c171f9-4569-431e-a891-5cae3328ce47" (UID: "f7c171f9-4569-431e-a891-5cae3328ce47"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:19:03.430426 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:03.430406 5979 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7c171f9-4569-431e-a891-5cae3328ce47-home\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:19:03.430538 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:03.430430 5979 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7c171f9-4569-431e-a891-5cae3328ce47-model-cache\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:19:03.430538 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:03.430445 5979 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7c171f9-4569-431e-a891-5cae3328ce47-tmp-dir\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:19:03.431782 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:03.431755 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7c171f9-4569-431e-a891-5cae3328ce47-tls-certs" pod "f7c171f9-4569-431e-a891-5cae3328ce47" (UID: "f7c171f9-4569-431e-a891-5cae3328ce47"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:19:03.431866 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:03.431809 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7c171f9-4569-431e-a891-5cae3328ce47-dshm" pod "f7c171f9-4569-431e-a891-5cae3328ce47" (UID: "f7c171f9-4569-431e-a891-5cae3328ce47"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:19:03.431918 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:03.431865 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7c171f9-4569-431e-a891-5cae3328ce47-kube-api-access-9g9rx" pod "f7c171f9-4569-431e-a891-5cae3328ce47" (UID: "f7c171f9-4569-431e-a891-5cae3328ce47"). InnerVolumeSpecName "kube-api-access-9g9rx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:19:03.531601 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:03.531565 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ee26f55-5c0c-4bd8-ad9b-fb8c03641698-tls-certs\" (UniqueName: \"kubernetes.io/secret/8ee26f55-5c0c-4bd8-ad9b-fb8c03641698-tls-certs\") pod \"8ee26f55-5c0c-4bd8-ad9b-fb8c03641698\" (UID: \"8ee26f55-5c0c-4bd8-ad9b-fb8c03641698\") " Nov 03 14:19:03.531787 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:03.531617 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ee26f55-5c0c-4bd8-ad9b-fb8c03641698-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ee26f55-5c0c-4bd8-ad9b-fb8c03641698-model-cache\") pod \"8ee26f55-5c0c-4bd8-ad9b-fb8c03641698\" (UID: \"8ee26f55-5c0c-4bd8-ad9b-fb8c03641698\") " Nov 03 14:19:03.531787 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:03.531674 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ee26f55-5c0c-4bd8-ad9b-fb8c03641698-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ee26f55-5c0c-4bd8-ad9b-fb8c03641698-dshm\") pod \"8ee26f55-5c0c-4bd8-ad9b-fb8c03641698\" (UID: \"8ee26f55-5c0c-4bd8-ad9b-fb8c03641698\") " Nov 03 14:19:03.531787 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:03.531711 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ee26f55-5c0c-4bd8-ad9b-fb8c03641698-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ee26f55-5c0c-4bd8-ad9b-fb8c03641698-tmp-dir\") pod \"8ee26f55-5c0c-4bd8-ad9b-fb8c03641698\" (UID: \"8ee26f55-5c0c-4bd8-ad9b-fb8c03641698\") " Nov 03 14:19:03.532004 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:03.531866 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ee26f55-5c0c-4bd8-ad9b-fb8c03641698-home\" (UniqueName: \"kubernetes.io/empty-dir/8ee26f55-5c0c-4bd8-ad9b-fb8c03641698-home\") pod \"8ee26f55-5c0c-4bd8-ad9b-fb8c03641698\" (UID: \"8ee26f55-5c0c-4bd8-ad9b-fb8c03641698\") " Nov 03 14:19:03.532004 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:03.531936 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ee26f55-5c0c-4bd8-ad9b-fb8c03641698-kube-api-access-9fvfp\" (UniqueName: \"kubernetes.io/projected/8ee26f55-5c0c-4bd8-ad9b-fb8c03641698-kube-api-access-9fvfp\") pod \"8ee26f55-5c0c-4bd8-ad9b-fb8c03641698\" (UID: \"8ee26f55-5c0c-4bd8-ad9b-fb8c03641698\") " Nov 03 14:19:03.532133 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:03.532036 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ee26f55-5c0c-4bd8-ad9b-fb8c03641698-model-cache" pod "8ee26f55-5c0c-4bd8-ad9b-fb8c03641698" (UID: "8ee26f55-5c0c-4bd8-ad9b-fb8c03641698"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:19:03.532340 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:03.532310 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ee26f55-5c0c-4bd8-ad9b-fb8c03641698-tmp-dir" pod "8ee26f55-5c0c-4bd8-ad9b-fb8c03641698" (UID: "8ee26f55-5c0c-4bd8-ad9b-fb8c03641698"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:19:03.533050 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:03.532795 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ee26f55-5c0c-4bd8-ad9b-fb8c03641698-home" pod "8ee26f55-5c0c-4bd8-ad9b-fb8c03641698" (UID: "8ee26f55-5c0c-4bd8-ad9b-fb8c03641698"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:19:03.533664 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:03.533637 5979 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7c171f9-4569-431e-a891-5cae3328ce47-dshm\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:19:03.533833 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:03.533667 5979 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ee26f55-5c0c-4bd8-ad9b-fb8c03641698-home\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:19:03.533833 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:03.533685 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g9rx\" (UniqueName: \"kubernetes.io/projected/f7c171f9-4569-431e-a891-5cae3328ce47-kube-api-access-9g9rx\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:19:03.533833 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:03.533700 5979 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7c171f9-4569-431e-a891-5cae3328ce47-tls-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:19:03.533833 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:03.533714 5979 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ee26f55-5c0c-4bd8-ad9b-fb8c03641698-model-cache\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:19:03.533833 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:03.533727 5979 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ee26f55-5c0c-4bd8-ad9b-fb8c03641698-tmp-dir\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:19:03.534675 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:03.534640 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ee26f55-5c0c-4bd8-ad9b-fb8c03641698-tls-certs" pod "8ee26f55-5c0c-4bd8-ad9b-fb8c03641698" (UID: "8ee26f55-5c0c-4bd8-ad9b-fb8c03641698"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:19:03.535372 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:03.535335 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ee26f55-5c0c-4bd8-ad9b-fb8c03641698-kube-api-access-9fvfp" pod "8ee26f55-5c0c-4bd8-ad9b-fb8c03641698" (UID: "8ee26f55-5c0c-4bd8-ad9b-fb8c03641698"). InnerVolumeSpecName "kube-api-access-9fvfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:19:03.535544 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:03.535520 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ee26f55-5c0c-4bd8-ad9b-fb8c03641698-dshm" pod "8ee26f55-5c0c-4bd8-ad9b-fb8c03641698" (UID: "8ee26f55-5c0c-4bd8-ad9b-fb8c03641698"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:19:03.635534 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:03.635497 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fvfp\" (UniqueName: \"kubernetes.io/projected/8ee26f55-5c0c-4bd8-ad9b-fb8c03641698-kube-api-access-9fvfp\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:19:03.635747 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:03.635537 5979 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ee26f55-5c0c-4bd8-ad9b-fb8c03641698-tls-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:19:03.635747 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:03.635550 5979 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ee26f55-5c0c-4bd8-ad9b-fb8c03641698-dshm\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:19:03.915976 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:03.915880 5979 generic.go:356] "Generic (PLEG): container finished" podID="8ee26f55-5c0c-4bd8-ad9b-fb8c03641698" containerID="7874929dd2fd364bf24c1a522a68cf1174b153b65c56d3b7a377566e71e3f23b" exitCode=0 Nov 03 14:19:03.915976 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:03.915922 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5bb463/premium-w3-5bb463-kserve-5f6ff6795c-jxmsr" Nov 03 14:19:03.916191 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:03.915975 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5bb463/premium-w3-5bb463-kserve-5f6ff6795c-jxmsr" event={"ID":"8ee26f55-5c0c-4bd8-ad9b-fb8c03641698","Type":"ContainerDied","Data":"7874929dd2fd364bf24c1a522a68cf1174b153b65c56d3b7a377566e71e3f23b"} Nov 03 14:19:03.916191 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:03.916019 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5bb463/premium-w3-5bb463-kserve-5f6ff6795c-jxmsr" event={"ID":"8ee26f55-5c0c-4bd8-ad9b-fb8c03641698","Type":"ContainerDied","Data":"22623d2a1cc1a758a4387af732b8ec448c13884b57608e57a084490b95d67c20"} Nov 03 14:19:03.916191 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:03.916044 5979 scope.go:122] "RemoveContainer" containerID="7874929dd2fd364bf24c1a522a68cf1174b153b65c56d3b7a377566e71e3f23b" Nov 03 14:19:03.917574 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:03.917548 5979 generic.go:356] "Generic (PLEG): container finished" podID="f7c171f9-4569-431e-a891-5cae3328ce47" containerID="55ca2f708f84e253317eb6c95027c22de16c5ee1e2fabbefd12f2809d16ee760" exitCode=0 Nov 03 14:19:03.917702 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:03.917590 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5bb463/distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw" Nov 03 14:19:03.917702 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:03.917648 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5bb463/distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw" event={"ID":"f7c171f9-4569-431e-a891-5cae3328ce47","Type":"ContainerDied","Data":"55ca2f708f84e253317eb6c95027c22de16c5ee1e2fabbefd12f2809d16ee760"} Nov 03 14:19:03.917702 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:03.917690 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5bb463/distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw" event={"ID":"f7c171f9-4569-431e-a891-5cae3328ce47","Type":"ContainerDied","Data":"b6e0d36f4ab41004c2ded0c880395833444566b2929954c0be1a488a43c28ee1"} Nov 03 14:19:03.927894 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:03.927856 5979 scope.go:122] "RemoveContainer" containerID="7874929dd2fd364bf24c1a522a68cf1174b153b65c56d3b7a377566e71e3f23b" Nov 03 14:19:03.928177 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:19:03.928151 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7874929dd2fd364bf24c1a522a68cf1174b153b65c56d3b7a377566e71e3f23b\": container with ID starting with 7874929dd2fd364bf24c1a522a68cf1174b153b65c56d3b7a377566e71e3f23b not found: ID does not exist" containerID="7874929dd2fd364bf24c1a522a68cf1174b153b65c56d3b7a377566e71e3f23b" Nov 03 14:19:03.928294 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:03.928182 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7874929dd2fd364bf24c1a522a68cf1174b153b65c56d3b7a377566e71e3f23b"} err="failed to get container status \"7874929dd2fd364bf24c1a522a68cf1174b153b65c56d3b7a377566e71e3f23b\": rpc error: code = NotFound desc = could not find container \"7874929dd2fd364bf24c1a522a68cf1174b153b65c56d3b7a377566e71e3f23b\": container with ID starting with 7874929dd2fd364bf24c1a522a68cf1174b153b65c56d3b7a377566e71e3f23b not found: ID does not exist" Nov 03 14:19:03.928294 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:03.928200 5979 scope.go:122] "RemoveContainer" containerID="55ca2f708f84e253317eb6c95027c22de16c5ee1e2fabbefd12f2809d16ee760" Nov 03 14:19:03.937224 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:03.937200 5979 scope.go:122] "RemoveContainer" containerID="55ca2f708f84e253317eb6c95027c22de16c5ee1e2fabbefd12f2809d16ee760" Nov 03 14:19:03.937612 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:19:03.937587 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55ca2f708f84e253317eb6c95027c22de16c5ee1e2fabbefd12f2809d16ee760\": container with ID starting with 55ca2f708f84e253317eb6c95027c22de16c5ee1e2fabbefd12f2809d16ee760 not found: ID does not exist" containerID="55ca2f708f84e253317eb6c95027c22de16c5ee1e2fabbefd12f2809d16ee760" Nov 03 14:19:03.937700 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:03.937620 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55ca2f708f84e253317eb6c95027c22de16c5ee1e2fabbefd12f2809d16ee760"} err="failed to get container status \"55ca2f708f84e253317eb6c95027c22de16c5ee1e2fabbefd12f2809d16ee760\": rpc error: code = NotFound desc = could not find container \"55ca2f708f84e253317eb6c95027c22de16c5ee1e2fabbefd12f2809d16ee760\": container with ID starting with 55ca2f708f84e253317eb6c95027c22de16c5ee1e2fabbefd12f2809d16ee760 not found: ID does not exist" Nov 03 14:19:04.070235 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:04.070201 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5bb463/distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw"] Nov 03 14:19:04.074436 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:04.074407 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-5bb463/distinct-2-w3-5bb463-kserve-95dbf8767-hrmfw"] Nov 03 14:19:04.116988 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:04.116957 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7c171f9-4569-431e-a891-5cae3328ce47" path="/var/lib/kubelet/pods/f7c171f9-4569-431e-a891-5cae3328ce47/volumes" Nov 03 14:19:04.174494 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:04.174377 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5bb463/premium-w3-5bb463-kserve-5f6ff6795c-jxmsr"] Nov 03 14:19:04.178699 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:04.178672 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-5bb463/premium-w3-5bb463-kserve-5f6ff6795c-jxmsr"] Nov 03 14:19:06.116683 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:06.116650 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ee26f55-5c0c-4bd8-ad9b-fb8c03641698" path="/var/lib/kubelet/pods/8ee26f55-5c0c-4bd8-ad9b-fb8c03641698/volumes" Nov 03 14:19:07.234695 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:19:07.234656 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:19:21.364494 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:21.360599 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-dkw27 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:19:21.364494 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:21.360701 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dkw27" podUID="f313510b-02d8-4c68-accd-920343219917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:19:37.241802 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:19:37.241752 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:19:47.384817 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:47.384777 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj"] Nov 03 14:19:47.385251 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:47.385049 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj" podUID="a6729077-e216-4dd9-a68d-69ce28c4c103" containerName="istio-proxy" containerID="cri-o://6f45e00950225ed37bdd646c0aac73b693265fd2fa7f93812e644f2152ff77c2" gracePeriod=30 Nov 03 14:19:52.664210 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:52.664150 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj" Nov 03 14:19:52.709397 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:52.709367 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6729077-e216-4dd9-a68d-69ce28c4c103-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6729077-e216-4dd9-a68d-69ce28c4c103-istio-data\") pod \"a6729077-e216-4dd9-a68d-69ce28c4c103\" (UID: \"a6729077-e216-4dd9-a68d-69ce28c4c103\") " Nov 03 14:19:52.709576 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:52.709415 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a6729077-e216-4dd9-a68d-69ce28c4c103-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6729077-e216-4dd9-a68d-69ce28c4c103-istio-podinfo\") pod \"a6729077-e216-4dd9-a68d-69ce28c4c103\" (UID: \"a6729077-e216-4dd9-a68d-69ce28c4c103\") " Nov 03 14:19:52.709576 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:52.709467 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6729077-e216-4dd9-a68d-69ce28c4c103-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6729077-e216-4dd9-a68d-69ce28c4c103-istio-envoy\") pod \"a6729077-e216-4dd9-a68d-69ce28c4c103\" (UID: \"a6729077-e216-4dd9-a68d-69ce28c4c103\") " Nov 03 14:19:52.709576 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:52.709548 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6729077-e216-4dd9-a68d-69ce28c4c103-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6729077-e216-4dd9-a68d-69ce28c4c103-workload-socket\") pod \"a6729077-e216-4dd9-a68d-69ce28c4c103\" (UID: \"a6729077-e216-4dd9-a68d-69ce28c4c103\") " Nov 03 14:19:52.709576 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:52.709568 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6729077-e216-4dd9-a68d-69ce28c4c103-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6729077-e216-4dd9-a68d-69ce28c4c103-istiod-ca-cert\") pod \"a6729077-e216-4dd9-a68d-69ce28c4c103\" (UID: \"a6729077-e216-4dd9-a68d-69ce28c4c103\") " Nov 03 14:19:52.709773 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:52.709651 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6729077-e216-4dd9-a68d-69ce28c4c103-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6729077-e216-4dd9-a68d-69ce28c4c103-workload-certs\") pod \"a6729077-e216-4dd9-a68d-69ce28c4c103\" (UID: \"a6729077-e216-4dd9-a68d-69ce28c4c103\") " Nov 03 14:19:52.709773 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:52.709651 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6729077-e216-4dd9-a68d-69ce28c4c103-istio-data" pod "a6729077-e216-4dd9-a68d-69ce28c4c103" (UID: "a6729077-e216-4dd9-a68d-69ce28c4c103"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:19:52.709773 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:52.709693 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6729077-e216-4dd9-a68d-69ce28c4c103-kube-api-access-jth4k\" (UniqueName: \"kubernetes.io/projected/a6729077-e216-4dd9-a68d-69ce28c4c103-kube-api-access-jth4k\") pod \"a6729077-e216-4dd9-a68d-69ce28c4c103\" (UID: \"a6729077-e216-4dd9-a68d-69ce28c4c103\") " Nov 03 14:19:52.709773 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:52.709730 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6729077-e216-4dd9-a68d-69ce28c4c103-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6729077-e216-4dd9-a68d-69ce28c4c103-credential-socket\") pod \"a6729077-e216-4dd9-a68d-69ce28c4c103\" (UID: \"a6729077-e216-4dd9-a68d-69ce28c4c103\") " Nov 03 14:19:52.710007 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:52.709772 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6729077-e216-4dd9-a68d-69ce28c4c103-istio-token\" (UniqueName: \"kubernetes.io/projected/a6729077-e216-4dd9-a68d-69ce28c4c103-istio-token\") pod \"a6729077-e216-4dd9-a68d-69ce28c4c103\" (UID: \"a6729077-e216-4dd9-a68d-69ce28c4c103\") " Nov 03 14:19:52.710007 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:52.709837 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6729077-e216-4dd9-a68d-69ce28c4c103-workload-socket" pod "a6729077-e216-4dd9-a68d-69ce28c4c103" (UID: "a6729077-e216-4dd9-a68d-69ce28c4c103"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:19:52.710132 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:52.710067 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6729077-e216-4dd9-a68d-69ce28c4c103-istiod-ca-cert" pod "a6729077-e216-4dd9-a68d-69ce28c4c103" (UID: "a6729077-e216-4dd9-a68d-69ce28c4c103"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:19:52.710258 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:52.710229 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6729077-e216-4dd9-a68d-69ce28c4c103-credential-socket" pod "a6729077-e216-4dd9-a68d-69ce28c4c103" (UID: "a6729077-e216-4dd9-a68d-69ce28c4c103"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:19:52.710506 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:52.710428 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6729077-e216-4dd9-a68d-69ce28c4c103-workload-certs" pod "a6729077-e216-4dd9-a68d-69ce28c4c103" (UID: "a6729077-e216-4dd9-a68d-69ce28c4c103"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:19:52.710706 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:52.710682 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6729077-e216-4dd9-a68d-69ce28c4c103-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:19:52.710796 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:52.710713 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6729077-e216-4dd9-a68d-69ce28c4c103-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:19:52.710796 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:52.710727 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6729077-e216-4dd9-a68d-69ce28c4c103-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:19:52.710796 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:52.710740 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6729077-e216-4dd9-a68d-69ce28c4c103-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:19:52.710796 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:52.710752 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6729077-e216-4dd9-a68d-69ce28c4c103-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:19:52.712028 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:52.712001 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6729077-e216-4dd9-a68d-69ce28c4c103-istio-podinfo" pod "a6729077-e216-4dd9-a68d-69ce28c4c103" (UID: "a6729077-e216-4dd9-a68d-69ce28c4c103"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:19:52.712141 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:52.712117 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6729077-e216-4dd9-a68d-69ce28c4c103-kube-api-access-jth4k" pod "a6729077-e216-4dd9-a68d-69ce28c4c103" (UID: "a6729077-e216-4dd9-a68d-69ce28c4c103"). InnerVolumeSpecName "kube-api-access-jth4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:19:52.712192 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:52.712173 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6729077-e216-4dd9-a68d-69ce28c4c103-istio-envoy" pod "a6729077-e216-4dd9-a68d-69ce28c4c103" (UID: "a6729077-e216-4dd9-a68d-69ce28c4c103"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:19:52.712303 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:52.712283 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6729077-e216-4dd9-a68d-69ce28c4c103-istio-token" pod "a6729077-e216-4dd9-a68d-69ce28c4c103" (UID: "a6729077-e216-4dd9-a68d-69ce28c4c103"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:19:52.812419 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:52.812384 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6729077-e216-4dd9-a68d-69ce28c4c103-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:19:52.812419 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:52.812415 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jth4k\" (UniqueName: \"kubernetes.io/projected/a6729077-e216-4dd9-a68d-69ce28c4c103-kube-api-access-jth4k\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:19:52.812419 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:52.812428 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6729077-e216-4dd9-a68d-69ce28c4c103-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:19:52.812709 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:52.812437 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6729077-e216-4dd9-a68d-69ce28c4c103-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:19:53.331016 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:53.330983 5979 generic.go:356] "Generic (PLEG): container finished" podID="a6729077-e216-4dd9-a68d-69ce28c4c103" containerID="6f45e00950225ed37bdd646c0aac73b693265fd2fa7f93812e644f2152ff77c2" exitCode=0 Nov 03 14:19:53.331206 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:53.331036 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj" Nov 03 14:19:53.331206 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:53.331070 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj" event={"ID":"a6729077-e216-4dd9-a68d-69ce28c4c103","Type":"ContainerDied","Data":"6f45e00950225ed37bdd646c0aac73b693265fd2fa7f93812e644f2152ff77c2"} Nov 03 14:19:53.331206 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:53.331119 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj" event={"ID":"a6729077-e216-4dd9-a68d-69ce28c4c103","Type":"ContainerDied","Data":"2721fbe30d7f870d13430df63a0faa2058aaa383947e38c08e2cfa944c2d946f"} Nov 03 14:19:53.331206 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:53.331146 5979 scope.go:122] "RemoveContainer" containerID="6f45e00950225ed37bdd646c0aac73b693265fd2fa7f93812e644f2152ff77c2" Nov 03 14:19:53.342015 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:53.342000 5979 scope.go:122] "RemoveContainer" containerID="4523c2499303c3864c2b430ca1044c2ffec208f7c8d94ebf83c16da3cb6e5c5e" Nov 03 14:19:53.350992 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:53.350973 5979 scope.go:122] "RemoveContainer" containerID="6f45e00950225ed37bdd646c0aac73b693265fd2fa7f93812e644f2152ff77c2" Nov 03 14:19:53.351266 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:19:53.351247 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f45e00950225ed37bdd646c0aac73b693265fd2fa7f93812e644f2152ff77c2\": container with ID starting with 6f45e00950225ed37bdd646c0aac73b693265fd2fa7f93812e644f2152ff77c2 not found: ID does not exist" containerID="6f45e00950225ed37bdd646c0aac73b693265fd2fa7f93812e644f2152ff77c2" Nov 03 14:19:53.351318 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:53.351275 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f45e00950225ed37bdd646c0aac73b693265fd2fa7f93812e644f2152ff77c2"} err="failed to get container status \"6f45e00950225ed37bdd646c0aac73b693265fd2fa7f93812e644f2152ff77c2\": rpc error: code = NotFound desc = could not find container \"6f45e00950225ed37bdd646c0aac73b693265fd2fa7f93812e644f2152ff77c2\": container with ID starting with 6f45e00950225ed37bdd646c0aac73b693265fd2fa7f93812e644f2152ff77c2 not found: ID does not exist" Nov 03 14:19:53.351318 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:53.351295 5979 scope.go:122] "RemoveContainer" containerID="4523c2499303c3864c2b430ca1044c2ffec208f7c8d94ebf83c16da3cb6e5c5e" Nov 03 14:19:53.351617 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:19:53.351586 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4523c2499303c3864c2b430ca1044c2ffec208f7c8d94ebf83c16da3cb6e5c5e\": container with ID starting with 4523c2499303c3864c2b430ca1044c2ffec208f7c8d94ebf83c16da3cb6e5c5e not found: ID does not exist" containerID="4523c2499303c3864c2b430ca1044c2ffec208f7c8d94ebf83c16da3cb6e5c5e" Nov 03 14:19:53.351716 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:53.351625 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4523c2499303c3864c2b430ca1044c2ffec208f7c8d94ebf83c16da3cb6e5c5e"} err="failed to get container status \"4523c2499303c3864c2b430ca1044c2ffec208f7c8d94ebf83c16da3cb6e5c5e\": rpc error: code = NotFound desc = could not find container \"4523c2499303c3864c2b430ca1044c2ffec208f7c8d94ebf83c16da3cb6e5c5e\": container with ID starting with 4523c2499303c3864c2b430ca1044c2ffec208f7c8d94ebf83c16da3cb6e5c5e not found: ID does not exist" Nov 03 14:19:53.485640 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:53.485596 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj"] Nov 03 14:19:53.531797 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:53.531751 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5bb463-openshift-default-7f7666545f-czrwj"] Nov 03 14:19:54.119059 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:54.119017 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6729077-e216-4dd9-a68d-69ce28c4c103" path="/var/lib/kubelet/pods/a6729077-e216-4dd9-a68d-69ce28c4c103/volumes" Nov 03 14:19:58.655225 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:19:58.655190 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:20:07.233879 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:20:07.233837 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:20:37.235261 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:20:37.235217 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:20:38.029973 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:20:38.029935 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-037f380d-kfk4l"] Nov 03 14:20:38.030575 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:20:38.030559 5979 cpu_manager.go:414] "containerMap: removing container" podUID="a29e24d0-3285-4676-8171-622e9dedacd4" containerName="revoke-keys" Nov 03 14:20:38.030635 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:20:38.030576 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a29e24d0-3285-4676-8171-622e9dedacd4" containerName="revoke-keys" Nov 03 14:20:38.030635 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:20:38.030586 5979 cpu_manager.go:414] "containerMap: removing container" podUID="a6729077-e216-4dd9-a68d-69ce28c4c103" containerName="istio-proxy" Nov 03 14:20:38.030635 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:20:38.030591 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6729077-e216-4dd9-a68d-69ce28c4c103" containerName="istio-proxy" Nov 03 14:20:38.030635 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:20:38.030611 5979 cpu_manager.go:414] "containerMap: removing container" podUID="8ee26f55-5c0c-4bd8-ad9b-fb8c03641698" containerName="main" Nov 03 14:20:38.030635 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:20:38.030616 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ee26f55-5c0c-4bd8-ad9b-fb8c03641698" containerName="main" Nov 03 14:20:38.030635 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:20:38.030626 5979 cpu_manager.go:414] "containerMap: removing container" podUID="a6729077-e216-4dd9-a68d-69ce28c4c103" containerName="istio-proxy" Nov 03 14:20:38.030635 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:20:38.030631 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6729077-e216-4dd9-a68d-69ce28c4c103" containerName="istio-proxy" Nov 03 14:20:38.030869 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:20:38.030649 5979 cpu_manager.go:414] "containerMap: removing container" podUID="f7c171f9-4569-431e-a891-5cae3328ce47" containerName="main" Nov 03 14:20:38.030869 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:20:38.030654 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7c171f9-4569-431e-a891-5cae3328ce47" containerName="main" Nov 03 14:20:38.030869 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:20:38.030743 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ee26f55-5c0c-4bd8-ad9b-fb8c03641698" containerName="main" Nov 03 14:20:38.030869 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:20:38.030750 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6729077-e216-4dd9-a68d-69ce28c4c103" containerName="istio-proxy" Nov 03 14:20:38.030869 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:20:38.030760 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7c171f9-4569-431e-a891-5cae3328ce47" containerName="main" Nov 03 14:20:38.030869 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:20:38.030770 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="a29e24d0-3285-4676-8171-622e9dedacd4" containerName="revoke-keys" Nov 03 14:20:38.034611 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:20:38.034589 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-037f380d-kfk4l" Nov 03 14:20:38.085268 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:20:38.085238 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:20:38.119086 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:20:38.119046 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkbjj\" (UniqueName: \"kubernetes.io/projected/15b07cd0-6863-442d-8f1c-3015333f708d-kube-api-access-nkbjj\") pod \"maas-api-revoke-sub-037f380d-kfk4l\" (UID: \"15b07cd0-6863-442d-8f1c-3015333f708d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-037f380d-kfk4l" Nov 03 14:20:38.119277 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:20:38.119155 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15b07cd0-6863-442d-8f1c-3015333f708d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-037f380d-kfk4l\" (UID: \"15b07cd0-6863-442d-8f1c-3015333f708d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-037f380d-kfk4l" Nov 03 14:20:38.119277 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:20:38.119213 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15b07cd0-6863-442d-8f1c-3015333f708d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-037f380d-kfk4l\" (UID: \"15b07cd0-6863-442d-8f1c-3015333f708d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-037f380d-kfk4l" Nov 03 14:20:38.131935 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:20:38.131904 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 14:20:38.134704 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:20:38.134677 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-037f380d-kfk4l"] Nov 03 14:20:38.220924 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:20:38.220889 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkbjj\" (UniqueName: \"kubernetes.io/projected/15b07cd0-6863-442d-8f1c-3015333f708d-kube-api-access-nkbjj\") pod \"maas-api-revoke-sub-037f380d-kfk4l\" (UID: \"15b07cd0-6863-442d-8f1c-3015333f708d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-037f380d-kfk4l" Nov 03 14:20:38.221146 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:20:38.220957 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15b07cd0-6863-442d-8f1c-3015333f708d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-037f380d-kfk4l\" (UID: \"15b07cd0-6863-442d-8f1c-3015333f708d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-037f380d-kfk4l" Nov 03 14:20:38.221146 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:20:38.221004 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15b07cd0-6863-442d-8f1c-3015333f708d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-037f380d-kfk4l\" (UID: \"15b07cd0-6863-442d-8f1c-3015333f708d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-037f380d-kfk4l" Nov 03 14:20:38.221755 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:20:38.221730 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15b07cd0-6863-442d-8f1c-3015333f708d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-037f380d-kfk4l\" (UID: \"15b07cd0-6863-442d-8f1c-3015333f708d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-037f380d-kfk4l" Nov 03 14:20:38.223500 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:20:38.223460 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15b07cd0-6863-442d-8f1c-3015333f708d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-037f380d-kfk4l\" (UID: \"15b07cd0-6863-442d-8f1c-3015333f708d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-037f380d-kfk4l" Nov 03 14:20:38.275200 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:20:38.275168 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkbjj\" (UniqueName: \"kubernetes.io/projected/15b07cd0-6863-442d-8f1c-3015333f708d-kube-api-access-nkbjj\") pod \"maas-api-revoke-sub-037f380d-kfk4l\" (UID: \"15b07cd0-6863-442d-8f1c-3015333f708d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-037f380d-kfk4l" Nov 03 14:20:38.348688 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:20:38.348602 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-037f380d-kfk4l" Nov 03 14:20:38.721997 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:20:38.721961 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-037f380d-kfk4l"] Nov 03 14:20:38.728947 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:20:38.728919 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15b07cd0_6863_442d_8f1c_3015333f708d.slice/crio-9f1dd87dcde8bc2a59e225e233ffc36d7b473979c99e58e00bd504b4e1644bb1 WatchSource:0}: Error finding container 9f1dd87dcde8bc2a59e225e233ffc36d7b473979c99e58e00bd504b4e1644bb1: Status 404 returned error can't find the container with id 9f1dd87dcde8bc2a59e225e233ffc36d7b473979c99e58e00bd504b4e1644bb1 Nov 03 14:20:39.669347 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:20:39.669307 5979 generic.go:356] "Generic (PLEG): container finished" podID="15b07cd0-6863-442d-8f1c-3015333f708d" containerID="7a03dc6ed3aa868c481ebba0f8d3f98455f75cbd98a20ce34233115ffe89aa86" exitCode=0 Nov 03 14:20:39.669909 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:20:39.669378 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-037f380d-kfk4l" event={"ID":"15b07cd0-6863-442d-8f1c-3015333f708d","Type":"ContainerDied","Data":"7a03dc6ed3aa868c481ebba0f8d3f98455f75cbd98a20ce34233115ffe89aa86"} Nov 03 14:20:39.669909 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:20:39.669430 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-037f380d-kfk4l" event={"ID":"15b07cd0-6863-442d-8f1c-3015333f708d","Type":"ContainerStarted","Data":"9f1dd87dcde8bc2a59e225e233ffc36d7b473979c99e58e00bd504b4e1644bb1"} Nov 03 14:20:40.815050 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:20:40.815021 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-037f380d-kfk4l" Nov 03 14:20:40.960769 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:20:40.960676 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15b07cd0-6863-442d-8f1c-3015333f708d-kube-api-access-nkbjj\" (UniqueName: \"kubernetes.io/projected/15b07cd0-6863-442d-8f1c-3015333f708d-kube-api-access-nkbjj\") pod \"15b07cd0-6863-442d-8f1c-3015333f708d\" (UID: \"15b07cd0-6863-442d-8f1c-3015333f708d\") " Nov 03 14:20:40.960769 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:20:40.960725 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15b07cd0-6863-442d-8f1c-3015333f708d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15b07cd0-6863-442d-8f1c-3015333f708d-maas-api-xks-ca\") pod \"15b07cd0-6863-442d-8f1c-3015333f708d\" (UID: \"15b07cd0-6863-442d-8f1c-3015333f708d\") " Nov 03 14:20:40.961020 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:20:40.960820 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15b07cd0-6863-442d-8f1c-3015333f708d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15b07cd0-6863-442d-8f1c-3015333f708d-maas-api-openshift-ca\") pod \"15b07cd0-6863-442d-8f1c-3015333f708d\" (UID: \"15b07cd0-6863-442d-8f1c-3015333f708d\") " Nov 03 14:20:40.961263 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:20:40.961232 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15b07cd0-6863-442d-8f1c-3015333f708d-maas-api-openshift-ca" pod "15b07cd0-6863-442d-8f1c-3015333f708d" (UID: "15b07cd0-6863-442d-8f1c-3015333f708d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:20:40.961453 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:20:40.961431 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15b07cd0-6863-442d-8f1c-3015333f708d-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:20:40.962888 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:20:40.962861 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15b07cd0-6863-442d-8f1c-3015333f708d-maas-api-xks-ca" pod "15b07cd0-6863-442d-8f1c-3015333f708d" (UID: "15b07cd0-6863-442d-8f1c-3015333f708d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:20:40.962888 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:20:40.962876 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15b07cd0-6863-442d-8f1c-3015333f708d-kube-api-access-nkbjj" pod "15b07cd0-6863-442d-8f1c-3015333f708d" (UID: "15b07cd0-6863-442d-8f1c-3015333f708d"). InnerVolumeSpecName "kube-api-access-nkbjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:20:41.063065 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:20:41.063029 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkbjj\" (UniqueName: \"kubernetes.io/projected/15b07cd0-6863-442d-8f1c-3015333f708d-kube-api-access-nkbjj\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:20:41.063065 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:20:41.063060 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15b07cd0-6863-442d-8f1c-3015333f708d-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:20:41.682933 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:20:41.682894 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-037f380d-kfk4l" Nov 03 14:20:41.682933 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:20:41.682909 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-037f380d-kfk4l" event={"ID":"15b07cd0-6863-442d-8f1c-3015333f708d","Type":"ContainerDied","Data":"9f1dd87dcde8bc2a59e225e233ffc36d7b473979c99e58e00bd504b4e1644bb1"} Nov 03 14:20:41.683159 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:20:41.682957 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f1dd87dcde8bc2a59e225e233ffc36d7b473979c99e58e00bd504b4e1644bb1" Nov 03 14:21:07.234562 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:21:07.234510 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:21:10.595283 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:21:10.595242 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:21:37.233935 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:21:37.233897 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:21:58.855555 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:21:58.855465 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:22:07.235397 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:22:07.235358 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:22:09.657221 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:09.657183 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ef29fb8-lfd8x"] Nov 03 14:22:09.657863 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:09.657849 5979 cpu_manager.go:414] "containerMap: removing container" podUID="15b07cd0-6863-442d-8f1c-3015333f708d" containerName="revoke-keys" Nov 03 14:22:09.658020 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:09.657864 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15b07cd0-6863-442d-8f1c-3015333f708d" containerName="revoke-keys" Nov 03 14:22:09.658020 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:09.657965 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="15b07cd0-6863-442d-8f1c-3015333f708d" containerName="revoke-keys" Nov 03 14:22:09.658020 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:09.657978 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6729077-e216-4dd9-a68d-69ce28c4c103" containerName="istio-proxy" Nov 03 14:22:09.661562 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:09.661541 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef29fb8-lfd8x" Nov 03 14:22:09.715123 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:09.715076 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ef29fb8-lfd8x"] Nov 03 14:22:09.758282 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:09.758245 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:22:09.758462 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:09.758297 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 14:22:09.796384 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:09.796338 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73ac05ea-aa66-4656-8e64-c53dfa42eed9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ef29fb8-lfd8x\" (UID: \"73ac05ea-aa66-4656-8e64-c53dfa42eed9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef29fb8-lfd8x" Nov 03 14:22:09.796605 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:09.796564 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73ac05ea-aa66-4656-8e64-c53dfa42eed9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ef29fb8-lfd8x\" (UID: \"73ac05ea-aa66-4656-8e64-c53dfa42eed9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef29fb8-lfd8x" Nov 03 14:22:09.796781 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:09.796754 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gtwb\" (UniqueName: \"kubernetes.io/projected/73ac05ea-aa66-4656-8e64-c53dfa42eed9-kube-api-access-2gtwb\") pod \"maas-api-revoke-sub-1ef29fb8-lfd8x\" (UID: \"73ac05ea-aa66-4656-8e64-c53dfa42eed9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef29fb8-lfd8x" Nov 03 14:22:09.897747 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:09.897712 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73ac05ea-aa66-4656-8e64-c53dfa42eed9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ef29fb8-lfd8x\" (UID: \"73ac05ea-aa66-4656-8e64-c53dfa42eed9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef29fb8-lfd8x" Nov 03 14:22:09.897953 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:09.897833 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gtwb\" (UniqueName: \"kubernetes.io/projected/73ac05ea-aa66-4656-8e64-c53dfa42eed9-kube-api-access-2gtwb\") pod \"maas-api-revoke-sub-1ef29fb8-lfd8x\" (UID: \"73ac05ea-aa66-4656-8e64-c53dfa42eed9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef29fb8-lfd8x" Nov 03 14:22:09.897953 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:09.897886 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73ac05ea-aa66-4656-8e64-c53dfa42eed9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ef29fb8-lfd8x\" (UID: \"73ac05ea-aa66-4656-8e64-c53dfa42eed9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef29fb8-lfd8x" Nov 03 14:22:09.898520 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:09.898460 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73ac05ea-aa66-4656-8e64-c53dfa42eed9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ef29fb8-lfd8x\" (UID: \"73ac05ea-aa66-4656-8e64-c53dfa42eed9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef29fb8-lfd8x" Nov 03 14:22:09.900116 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:09.900094 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73ac05ea-aa66-4656-8e64-c53dfa42eed9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ef29fb8-lfd8x\" (UID: \"73ac05ea-aa66-4656-8e64-c53dfa42eed9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef29fb8-lfd8x" Nov 03 14:22:09.951762 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:09.951689 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gtwb\" (UniqueName: \"kubernetes.io/projected/73ac05ea-aa66-4656-8e64-c53dfa42eed9-kube-api-access-2gtwb\") pod \"maas-api-revoke-sub-1ef29fb8-lfd8x\" (UID: \"73ac05ea-aa66-4656-8e64-c53dfa42eed9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef29fb8-lfd8x" Nov 03 14:22:09.974978 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:09.974947 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef29fb8-lfd8x" Nov 03 14:22:10.348620 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:10.348534 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ef29fb8-lfd8x"] Nov 03 14:22:10.355812 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:22:10.355785 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73ac05ea_aa66_4656_8e64_c53dfa42eed9.slice/crio-2c5d70f4ced0e0edcc2e6409dc297fb39fbacfe480880d1367aaca20e75448bd WatchSource:0}: Error finding container 2c5d70f4ced0e0edcc2e6409dc297fb39fbacfe480880d1367aaca20e75448bd: Status 404 returned error can't find the container with id 2c5d70f4ced0e0edcc2e6409dc297fb39fbacfe480880d1367aaca20e75448bd Nov 03 14:22:11.337548 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:11.337511 5979 generic.go:356] "Generic (PLEG): container finished" podID="73ac05ea-aa66-4656-8e64-c53dfa42eed9" containerID="2c6e1257f013a9afc4ce63864437e1dc365a9ace095d28eaa206b0e5e2525243" exitCode=0 Nov 03 14:22:11.337957 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:11.337601 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef29fb8-lfd8x" event={"ID":"73ac05ea-aa66-4656-8e64-c53dfa42eed9","Type":"ContainerDied","Data":"2c6e1257f013a9afc4ce63864437e1dc365a9ace095d28eaa206b0e5e2525243"} Nov 03 14:22:11.337957 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:11.337652 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef29fb8-lfd8x" event={"ID":"73ac05ea-aa66-4656-8e64-c53dfa42eed9","Type":"ContainerStarted","Data":"2c5d70f4ced0e0edcc2e6409dc297fb39fbacfe480880d1367aaca20e75448bd"} Nov 03 14:22:12.483746 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:12.483721 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef29fb8-lfd8x" Nov 03 14:22:12.527945 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:12.527907 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73ac05ea-aa66-4656-8e64-c53dfa42eed9-kube-api-access-2gtwb\" (UniqueName: \"kubernetes.io/projected/73ac05ea-aa66-4656-8e64-c53dfa42eed9-kube-api-access-2gtwb\") pod \"73ac05ea-aa66-4656-8e64-c53dfa42eed9\" (UID: \"73ac05ea-aa66-4656-8e64-c53dfa42eed9\") " Nov 03 14:22:12.528116 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:12.527979 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73ac05ea-aa66-4656-8e64-c53dfa42eed9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73ac05ea-aa66-4656-8e64-c53dfa42eed9-maas-api-openshift-ca\") pod \"73ac05ea-aa66-4656-8e64-c53dfa42eed9\" (UID: \"73ac05ea-aa66-4656-8e64-c53dfa42eed9\") " Nov 03 14:22:12.528116 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:12.528029 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73ac05ea-aa66-4656-8e64-c53dfa42eed9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73ac05ea-aa66-4656-8e64-c53dfa42eed9-maas-api-xks-ca\") pod \"73ac05ea-aa66-4656-8e64-c53dfa42eed9\" (UID: \"73ac05ea-aa66-4656-8e64-c53dfa42eed9\") " Nov 03 14:22:12.528413 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:12.528382 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73ac05ea-aa66-4656-8e64-c53dfa42eed9-maas-api-openshift-ca" pod "73ac05ea-aa66-4656-8e64-c53dfa42eed9" (UID: "73ac05ea-aa66-4656-8e64-c53dfa42eed9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:22:12.529996 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:12.529977 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73ac05ea-aa66-4656-8e64-c53dfa42eed9-maas-api-xks-ca" pod "73ac05ea-aa66-4656-8e64-c53dfa42eed9" (UID: "73ac05ea-aa66-4656-8e64-c53dfa42eed9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:22:12.530112 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:12.530090 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73ac05ea-aa66-4656-8e64-c53dfa42eed9-kube-api-access-2gtwb" pod "73ac05ea-aa66-4656-8e64-c53dfa42eed9" (UID: "73ac05ea-aa66-4656-8e64-c53dfa42eed9"). InnerVolumeSpecName "kube-api-access-2gtwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:22:12.630531 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:12.630427 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73ac05ea-aa66-4656-8e64-c53dfa42eed9-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:22:12.630531 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:12.630459 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73ac05ea-aa66-4656-8e64-c53dfa42eed9-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:22:12.630531 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:12.630488 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gtwb\" (UniqueName: \"kubernetes.io/projected/73ac05ea-aa66-4656-8e64-c53dfa42eed9-kube-api-access-2gtwb\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:22:13.352420 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:13.352377 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef29fb8-lfd8x" event={"ID":"73ac05ea-aa66-4656-8e64-c53dfa42eed9","Type":"ContainerDied","Data":"2c5d70f4ced0e0edcc2e6409dc297fb39fbacfe480880d1367aaca20e75448bd"} Nov 03 14:22:13.352420 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:13.352408 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ef29fb8-lfd8x" Nov 03 14:22:13.352683 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:13.352441 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c5d70f4ced0e0edcc2e6409dc297fb39fbacfe480880d1367aaca20e75448bd" Nov 03 14:22:20.352607 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:20.352574 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4771b897-wrdwm"] Nov 03 14:22:20.353224 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:20.353209 5979 cpu_manager.go:414] "containerMap: removing container" podUID="73ac05ea-aa66-4656-8e64-c53dfa42eed9" containerName="revoke-keys" Nov 03 14:22:20.353275 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:20.353226 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73ac05ea-aa66-4656-8e64-c53dfa42eed9" containerName="revoke-keys" Nov 03 14:22:20.353340 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:20.353330 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="73ac05ea-aa66-4656-8e64-c53dfa42eed9" containerName="revoke-keys" Nov 03 14:22:20.357160 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:20.357139 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4771b897-wrdwm" Nov 03 14:22:20.406775 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:20.406743 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:22:20.409974 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:20.409944 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4771b897-wrdwm"] Nov 03 14:22:20.426610 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:20.426569 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12b107b5-7b5c-4db7-99d3-570163320ef3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4771b897-wrdwm\" (UID: \"12b107b5-7b5c-4db7-99d3-570163320ef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4771b897-wrdwm" Nov 03 14:22:20.426768 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:20.426635 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj6tp\" (UniqueName: \"kubernetes.io/projected/12b107b5-7b5c-4db7-99d3-570163320ef3-kube-api-access-sj6tp\") pod \"maas-api-revoke-sub-4771b897-wrdwm\" (UID: \"12b107b5-7b5c-4db7-99d3-570163320ef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4771b897-wrdwm" Nov 03 14:22:20.426768 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:20.426742 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12b107b5-7b5c-4db7-99d3-570163320ef3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4771b897-wrdwm\" (UID: \"12b107b5-7b5c-4db7-99d3-570163320ef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4771b897-wrdwm" Nov 03 14:22:20.453101 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:20.453074 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 14:22:20.528027 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:20.527978 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12b107b5-7b5c-4db7-99d3-570163320ef3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4771b897-wrdwm\" (UID: \"12b107b5-7b5c-4db7-99d3-570163320ef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4771b897-wrdwm" Nov 03 14:22:20.528188 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:20.528166 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12b107b5-7b5c-4db7-99d3-570163320ef3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4771b897-wrdwm\" (UID: \"12b107b5-7b5c-4db7-99d3-570163320ef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4771b897-wrdwm" Nov 03 14:22:20.528275 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:20.528203 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj6tp\" (UniqueName: \"kubernetes.io/projected/12b107b5-7b5c-4db7-99d3-570163320ef3-kube-api-access-sj6tp\") pod \"maas-api-revoke-sub-4771b897-wrdwm\" (UID: \"12b107b5-7b5c-4db7-99d3-570163320ef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4771b897-wrdwm" Nov 03 14:22:20.528865 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:20.528841 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12b107b5-7b5c-4db7-99d3-570163320ef3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4771b897-wrdwm\" (UID: \"12b107b5-7b5c-4db7-99d3-570163320ef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4771b897-wrdwm" Nov 03 14:22:20.530394 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:20.530375 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12b107b5-7b5c-4db7-99d3-570163320ef3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4771b897-wrdwm\" (UID: \"12b107b5-7b5c-4db7-99d3-570163320ef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4771b897-wrdwm" Nov 03 14:22:20.581847 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:20.581821 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj6tp\" (UniqueName: \"kubernetes.io/projected/12b107b5-7b5c-4db7-99d3-570163320ef3-kube-api-access-sj6tp\") pod \"maas-api-revoke-sub-4771b897-wrdwm\" (UID: \"12b107b5-7b5c-4db7-99d3-570163320ef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4771b897-wrdwm" Nov 03 14:22:20.670548 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:20.670509 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4771b897-wrdwm" Nov 03 14:22:21.045735 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:21.045648 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4771b897-wrdwm"] Nov 03 14:22:21.052666 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:22:21.052626 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12b107b5_7b5c_4db7_99d3_570163320ef3.slice/crio-8b7e79da2d4b244b72632874298486f571f6041bddc574a71f59457427f15538 WatchSource:0}: Error finding container 8b7e79da2d4b244b72632874298486f571f6041bddc574a71f59457427f15538: Status 404 returned error can't find the container with id 8b7e79da2d4b244b72632874298486f571f6041bddc574a71f59457427f15538 Nov 03 14:22:21.405081 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:21.405047 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4771b897-wrdwm" event={"ID":"12b107b5-7b5c-4db7-99d3-570163320ef3","Type":"ContainerStarted","Data":"a01bab3feca0310a084d0c6ae0e7efde51057678e7717a05c6fb1a448e997fe6"} Nov 03 14:22:21.405514 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:21.405102 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4771b897-wrdwm" event={"ID":"12b107b5-7b5c-4db7-99d3-570163320ef3","Type":"ContainerStarted","Data":"8b7e79da2d4b244b72632874298486f571f6041bddc574a71f59457427f15538"} Nov 03 14:22:21.516166 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:21.516116 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4771b897-wrdwm" podStartSLOduration=1.516102957 podStartE2EDuration="1.516102957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:22:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:22:21.51377529 +0000 UTC m=+4623.524463531" watchObservedRunningTime="2026-11-03 14:22:21.516102957 +0000 UTC m=+4623.526791198" Nov 03 14:22:22.412932 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:22.412896 5979 generic.go:356] "Generic (PLEG): container finished" podID="12b107b5-7b5c-4db7-99d3-570163320ef3" containerID="a01bab3feca0310a084d0c6ae0e7efde51057678e7717a05c6fb1a448e997fe6" exitCode=0 Nov 03 14:22:22.413398 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:22.412983 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4771b897-wrdwm" event={"ID":"12b107b5-7b5c-4db7-99d3-570163320ef3","Type":"ContainerDied","Data":"a01bab3feca0310a084d0c6ae0e7efde51057678e7717a05c6fb1a448e997fe6"} Nov 03 14:22:23.579198 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:23.579171 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4771b897-wrdwm" Nov 03 14:22:23.669966 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:23.669885 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12b107b5-7b5c-4db7-99d3-570163320ef3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12b107b5-7b5c-4db7-99d3-570163320ef3-maas-api-xks-ca\") pod \"12b107b5-7b5c-4db7-99d3-570163320ef3\" (UID: \"12b107b5-7b5c-4db7-99d3-570163320ef3\") " Nov 03 14:22:23.669966 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:23.669946 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12b107b5-7b5c-4db7-99d3-570163320ef3-kube-api-access-sj6tp\" (UniqueName: \"kubernetes.io/projected/12b107b5-7b5c-4db7-99d3-570163320ef3-kube-api-access-sj6tp\") pod \"12b107b5-7b5c-4db7-99d3-570163320ef3\" (UID: \"12b107b5-7b5c-4db7-99d3-570163320ef3\") " Nov 03 14:22:23.670178 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:23.670055 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12b107b5-7b5c-4db7-99d3-570163320ef3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12b107b5-7b5c-4db7-99d3-570163320ef3-maas-api-openshift-ca\") pod \"12b107b5-7b5c-4db7-99d3-570163320ef3\" (UID: \"12b107b5-7b5c-4db7-99d3-570163320ef3\") " Nov 03 14:22:23.670486 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:23.670438 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12b107b5-7b5c-4db7-99d3-570163320ef3-maas-api-openshift-ca" pod "12b107b5-7b5c-4db7-99d3-570163320ef3" (UID: "12b107b5-7b5c-4db7-99d3-570163320ef3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:22:23.670633 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:23.670616 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12b107b5-7b5c-4db7-99d3-570163320ef3-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:22:23.672032 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:23.672010 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12b107b5-7b5c-4db7-99d3-570163320ef3-kube-api-access-sj6tp" pod "12b107b5-7b5c-4db7-99d3-570163320ef3" (UID: "12b107b5-7b5c-4db7-99d3-570163320ef3"). InnerVolumeSpecName "kube-api-access-sj6tp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:22:23.672092 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:23.672023 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12b107b5-7b5c-4db7-99d3-570163320ef3-maas-api-xks-ca" pod "12b107b5-7b5c-4db7-99d3-570163320ef3" (UID: "12b107b5-7b5c-4db7-99d3-570163320ef3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:22:23.771483 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:23.771434 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12b107b5-7b5c-4db7-99d3-570163320ef3-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:22:23.771483 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:23.771464 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sj6tp\" (UniqueName: \"kubernetes.io/projected/12b107b5-7b5c-4db7-99d3-570163320ef3-kube-api-access-sj6tp\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:22:24.427266 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:24.427234 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4771b897-wrdwm" Nov 03 14:22:24.427266 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:24.427249 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4771b897-wrdwm" event={"ID":"12b107b5-7b5c-4db7-99d3-570163320ef3","Type":"ContainerDied","Data":"8b7e79da2d4b244b72632874298486f571f6041bddc574a71f59457427f15538"} Nov 03 14:22:24.427588 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:24.427302 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b7e79da2d4b244b72632874298486f571f6041bddc574a71f59457427f15538" Nov 03 14:22:25.019484 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:25.019432 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5faf5d-764fdf77d7-tm6xj"] Nov 03 14:22:30.024404 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:30.024340 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5faf5d-764fdf77d7-tm6xj" podUID="6e7dbc4b-d67d-4f15-bddc-088c23fdee66" containerName="payload-pre-processing" containerID="cri-o://bd872d66863ba45799ba82fda949726b450ba561baa4ad85cb810c6d59b208f5" gracePeriod=25 Nov 03 14:22:31.869125 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:31.869081 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5faf5d-764fdf77d7-tm6xj" podUID="6e7dbc4b-d67d-4f15-bddc-088c23fdee66" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:22:37.236036 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:22:37.235996 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:22:41.869370 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:41.869327 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5faf5d-764fdf77d7-tm6xj" podUID="6e7dbc4b-d67d-4f15-bddc-088c23fdee66" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:22:47.229383 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:47.229337 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:22:50.294886 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:50.294791 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5faf5d-764fdf77d7-tm6xj" Nov 03 14:22:50.440503 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:50.440430 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e7dbc4b-d67d-4f15-bddc-088c23fdee66-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e7dbc4b-d67d-4f15-bddc-088c23fdee66-plugins-config-volume\") pod \"6e7dbc4b-d67d-4f15-bddc-088c23fdee66\" (UID: \"6e7dbc4b-d67d-4f15-bddc-088c23fdee66\") " Nov 03 14:22:50.440503 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:50.440497 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e7dbc4b-d67d-4f15-bddc-088c23fdee66-kube-api-access-8sxtk\" (UniqueName: \"kubernetes.io/projected/6e7dbc4b-d67d-4f15-bddc-088c23fdee66-kube-api-access-8sxtk\") pod \"6e7dbc4b-d67d-4f15-bddc-088c23fdee66\" (UID: \"6e7dbc4b-d67d-4f15-bddc-088c23fdee66\") " Nov 03 14:22:50.440861 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:50.440829 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e7dbc4b-d67d-4f15-bddc-088c23fdee66-plugins-config-volume" pod "6e7dbc4b-d67d-4f15-bddc-088c23fdee66" (UID: "6e7dbc4b-d67d-4f15-bddc-088c23fdee66"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:22:50.441188 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:50.441162 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e7dbc4b-d67d-4f15-bddc-088c23fdee66-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:22:50.442969 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:50.442939 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e7dbc4b-d67d-4f15-bddc-088c23fdee66-kube-api-access-8sxtk" pod "6e7dbc4b-d67d-4f15-bddc-088c23fdee66" (UID: "6e7dbc4b-d67d-4f15-bddc-088c23fdee66"). InnerVolumeSpecName "kube-api-access-8sxtk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:22:50.542324 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:50.542291 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sxtk\" (UniqueName: \"kubernetes.io/projected/6e7dbc4b-d67d-4f15-bddc-088c23fdee66-kube-api-access-8sxtk\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:22:50.640277 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:50.640243 5979 generic.go:356] "Generic (PLEG): container finished" podID="6e7dbc4b-d67d-4f15-bddc-088c23fdee66" containerID="bd872d66863ba45799ba82fda949726b450ba561baa4ad85cb810c6d59b208f5" exitCode=0 Nov 03 14:22:50.640511 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:50.640292 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5faf5d-764fdf77d7-tm6xj" Nov 03 14:22:50.640511 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:50.640329 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5faf5d-764fdf77d7-tm6xj" event={"ID":"6e7dbc4b-d67d-4f15-bddc-088c23fdee66","Type":"ContainerDied","Data":"bd872d66863ba45799ba82fda949726b450ba561baa4ad85cb810c6d59b208f5"} Nov 03 14:22:50.640511 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:50.640392 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5faf5d-764fdf77d7-tm6xj" event={"ID":"6e7dbc4b-d67d-4f15-bddc-088c23fdee66","Type":"ContainerDied","Data":"44b1f888a95cc4615fef8525ad1601ee7519c2014473b5c4881bcb9fb81c6dc5"} Nov 03 14:22:50.640511 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:50.640420 5979 scope.go:122] "RemoveContainer" containerID="bd872d66863ba45799ba82fda949726b450ba561baa4ad85cb810c6d59b208f5" Nov 03 14:22:50.652896 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:50.652869 5979 scope.go:122] "RemoveContainer" containerID="bd872d66863ba45799ba82fda949726b450ba561baa4ad85cb810c6d59b208f5" Nov 03 14:22:50.653181 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:22:50.653161 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd872d66863ba45799ba82fda949726b450ba561baa4ad85cb810c6d59b208f5\": container with ID starting with bd872d66863ba45799ba82fda949726b450ba561baa4ad85cb810c6d59b208f5 not found: ID does not exist" containerID="bd872d66863ba45799ba82fda949726b450ba561baa4ad85cb810c6d59b208f5" Nov 03 14:22:50.653239 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:50.653190 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd872d66863ba45799ba82fda949726b450ba561baa4ad85cb810c6d59b208f5"} err="failed to get container status \"bd872d66863ba45799ba82fda949726b450ba561baa4ad85cb810c6d59b208f5\": rpc error: code = NotFound desc = could not find container \"bd872d66863ba45799ba82fda949726b450ba561baa4ad85cb810c6d59b208f5\": container with ID starting with bd872d66863ba45799ba82fda949726b450ba561baa4ad85cb810c6d59b208f5 not found: ID does not exist" Nov 03 14:22:50.794090 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:50.794056 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5faf5d-764fdf77d7-tm6xj"] Nov 03 14:22:50.798932 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:50.798901 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-5faf5d-764fdf77d7-tm6xj"] Nov 03 14:22:52.116049 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:22:52.116013 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e7dbc4b-d67d-4f15-bddc-088c23fdee66" path="/var/lib/kubelet/pods/6e7dbc4b-d67d-4f15-bddc-088c23fdee66/volumes" Nov 03 14:23:07.234051 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:23:07.234007 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:23:22.386108 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:22.386071 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5faf5d-669fbdddd8-ms8tf"] Nov 03 14:23:22.386607 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:22.386285 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5faf5d-669fbdddd8-ms8tf" podUID="49140ce2-70a3-43bc-8ad4-5c0ac5dff963" containerName="maas-api" containerID="cri-o://ab99e76474aaf8e48b3e986b05f74495aa8ce275bb53e96bdb1d48cb1dd176c9" gracePeriod=30 Nov 03 14:23:22.690335 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:22.690304 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5faf5d-669fbdddd8-ms8tf" Nov 03 14:23:22.864141 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:22.864101 5979 generic.go:356] "Generic (PLEG): container finished" podID="49140ce2-70a3-43bc-8ad4-5c0ac5dff963" containerID="ab99e76474aaf8e48b3e986b05f74495aa8ce275bb53e96bdb1d48cb1dd176c9" exitCode=0 Nov 03 14:23:22.864409 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:22.864165 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5faf5d-669fbdddd8-ms8tf" Nov 03 14:23:22.864409 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:22.864206 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5faf5d-669fbdddd8-ms8tf" event={"ID":"49140ce2-70a3-43bc-8ad4-5c0ac5dff963","Type":"ContainerDied","Data":"ab99e76474aaf8e48b3e986b05f74495aa8ce275bb53e96bdb1d48cb1dd176c9"} Nov 03 14:23:22.864409 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:22.864382 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5faf5d-669fbdddd8-ms8tf" event={"ID":"49140ce2-70a3-43bc-8ad4-5c0ac5dff963","Type":"ContainerDied","Data":"2963f3405bbadca32cf5a0b12cfb3513cc6fbc8f2eebc4e987841580b7d9306b"} Nov 03 14:23:22.864614 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:22.864437 5979 scope.go:122] "RemoveContainer" containerID="ab99e76474aaf8e48b3e986b05f74495aa8ce275bb53e96bdb1d48cb1dd176c9" Nov 03 14:23:22.866908 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:22.866878 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49140ce2-70a3-43bc-8ad4-5c0ac5dff963-metrics-certs\" (UniqueName: \"kubernetes.io/secret/49140ce2-70a3-43bc-8ad4-5c0ac5dff963-metrics-certs\") pod \"49140ce2-70a3-43bc-8ad4-5c0ac5dff963\" (UID: \"49140ce2-70a3-43bc-8ad4-5c0ac5dff963\") " Nov 03 14:23:22.867021 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:22.867001 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49140ce2-70a3-43bc-8ad4-5c0ac5dff963-kube-api-access-4rq8z\" (UniqueName: \"kubernetes.io/projected/49140ce2-70a3-43bc-8ad4-5c0ac5dff963-kube-api-access-4rq8z\") pod \"49140ce2-70a3-43bc-8ad4-5c0ac5dff963\" (UID: \"49140ce2-70a3-43bc-8ad4-5c0ac5dff963\") " Nov 03 14:23:22.867080 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:22.867037 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49140ce2-70a3-43bc-8ad4-5c0ac5dff963-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49140ce2-70a3-43bc-8ad4-5c0ac5dff963-openshift-service-ca\") pod \"49140ce2-70a3-43bc-8ad4-5c0ac5dff963\" (UID: \"49140ce2-70a3-43bc-8ad4-5c0ac5dff963\") " Nov 03 14:23:22.867269 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:22.867246 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49140ce2-70a3-43bc-8ad4-5c0ac5dff963-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49140ce2-70a3-43bc-8ad4-5c0ac5dff963-maas-api-tls\") pod \"49140ce2-70a3-43bc-8ad4-5c0ac5dff963\" (UID: \"49140ce2-70a3-43bc-8ad4-5c0ac5dff963\") " Nov 03 14:23:22.867722 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:22.867503 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49140ce2-70a3-43bc-8ad4-5c0ac5dff963-openshift-service-ca" pod "49140ce2-70a3-43bc-8ad4-5c0ac5dff963" (UID: "49140ce2-70a3-43bc-8ad4-5c0ac5dff963"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:23:22.867883 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:22.867848 5979 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49140ce2-70a3-43bc-8ad4-5c0ac5dff963-openshift-service-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:23:22.869740 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:22.869706 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49140ce2-70a3-43bc-8ad4-5c0ac5dff963-kube-api-access-4rq8z" pod "49140ce2-70a3-43bc-8ad4-5c0ac5dff963" (UID: "49140ce2-70a3-43bc-8ad4-5c0ac5dff963"). InnerVolumeSpecName "kube-api-access-4rq8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:23:22.869833 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:22.869754 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49140ce2-70a3-43bc-8ad4-5c0ac5dff963-metrics-certs" pod "49140ce2-70a3-43bc-8ad4-5c0ac5dff963" (UID: "49140ce2-70a3-43bc-8ad4-5c0ac5dff963"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:23:22.871172 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:22.871145 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49140ce2-70a3-43bc-8ad4-5c0ac5dff963-maas-api-tls" pod "49140ce2-70a3-43bc-8ad4-5c0ac5dff963" (UID: "49140ce2-70a3-43bc-8ad4-5c0ac5dff963"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:23:22.897586 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:22.897465 5979 scope.go:122] "RemoveContainer" containerID="ab99e76474aaf8e48b3e986b05f74495aa8ce275bb53e96bdb1d48cb1dd176c9" Nov 03 14:23:22.897920 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:23:22.897897 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab99e76474aaf8e48b3e986b05f74495aa8ce275bb53e96bdb1d48cb1dd176c9\": container with ID starting with ab99e76474aaf8e48b3e986b05f74495aa8ce275bb53e96bdb1d48cb1dd176c9 not found: ID does not exist" containerID="ab99e76474aaf8e48b3e986b05f74495aa8ce275bb53e96bdb1d48cb1dd176c9" Nov 03 14:23:22.897977 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:22.897928 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab99e76474aaf8e48b3e986b05f74495aa8ce275bb53e96bdb1d48cb1dd176c9"} err="failed to get container status \"ab99e76474aaf8e48b3e986b05f74495aa8ce275bb53e96bdb1d48cb1dd176c9\": rpc error: code = NotFound desc = could not find container \"ab99e76474aaf8e48b3e986b05f74495aa8ce275bb53e96bdb1d48cb1dd176c9\": container with ID starting with ab99e76474aaf8e48b3e986b05f74495aa8ce275bb53e96bdb1d48cb1dd176c9 not found: ID does not exist" Nov 03 14:23:22.974036 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:22.973934 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49140ce2-70a3-43bc-8ad4-5c0ac5dff963-maas-api-tls\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:23:22.974036 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:22.973972 5979 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49140ce2-70a3-43bc-8ad4-5c0ac5dff963-metrics-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:23:22.974036 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:22.973996 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rq8z\" (UniqueName: \"kubernetes.io/projected/49140ce2-70a3-43bc-8ad4-5c0ac5dff963-kube-api-access-4rq8z\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:23:23.277609 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:23.277527 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5faf5d-669fbdddd8-ms8tf"] Nov 03 14:23:23.280036 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:23.280010 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5faf5d-669fbdddd8-ms8tf"] Nov 03 14:23:24.115676 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:24.115642 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49140ce2-70a3-43bc-8ad4-5c0ac5dff963" path="/var/lib/kubelet/pods/49140ce2-70a3-43bc-8ad4-5c0ac5dff963/volumes" Nov 03 14:23:32.015446 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:32.015404 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-60261a/sub-test-model-60261a-kserve-5cddbb6dd5-skpws"] Nov 03 14:23:32.016189 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:32.016169 5979 cpu_manager.go:414] "containerMap: removing container" podUID="49140ce2-70a3-43bc-8ad4-5c0ac5dff963" containerName="maas-api" Nov 03 14:23:32.016247 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:32.016190 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49140ce2-70a3-43bc-8ad4-5c0ac5dff963" containerName="maas-api" Nov 03 14:23:32.016247 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:32.016210 5979 cpu_manager.go:414] "containerMap: removing container" podUID="6e7dbc4b-d67d-4f15-bddc-088c23fdee66" containerName="payload-pre-processing" Nov 03 14:23:32.016247 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:32.016216 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e7dbc4b-d67d-4f15-bddc-088c23fdee66" containerName="payload-pre-processing" Nov 03 14:23:32.016247 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:32.016238 5979 cpu_manager.go:414] "containerMap: removing container" podUID="12b107b5-7b5c-4db7-99d3-570163320ef3" containerName="revoke-keys" Nov 03 14:23:32.016247 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:32.016243 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12b107b5-7b5c-4db7-99d3-570163320ef3" containerName="revoke-keys" Nov 03 14:23:32.016415 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:32.016349 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="49140ce2-70a3-43bc-8ad4-5c0ac5dff963" containerName="maas-api" Nov 03 14:23:32.016415 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:32.016359 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e7dbc4b-d67d-4f15-bddc-088c23fdee66" containerName="payload-pre-processing" Nov 03 14:23:32.016415 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:32.016367 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="12b107b5-7b5c-4db7-99d3-570163320ef3" containerName="revoke-keys" Nov 03 14:23:32.019852 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:32.019833 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-60261a/sub-test-model-60261a-kserve-5cddbb6dd5-skpws" Nov 03 14:23:32.072817 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:32.072786 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-60261a/sub-test-model-60261a-kserve-5cddbb6dd5-skpws"] Nov 03 14:23:32.087440 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:32.087405 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d1bbe2d-c09b-4caa-86ec-a0a64465e419-dshm\") pod \"sub-test-model-60261a-kserve-5cddbb6dd5-skpws\" (UID: \"5d1bbe2d-c09b-4caa-86ec-a0a64465e419\") " pod="ai-tenant-e2e-shared-a-w1-60261a/sub-test-model-60261a-kserve-5cddbb6dd5-skpws" Nov 03 14:23:32.087646 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:32.087625 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d1bbe2d-c09b-4caa-86ec-a0a64465e419-model-cache\") pod \"sub-test-model-60261a-kserve-5cddbb6dd5-skpws\" (UID: \"5d1bbe2d-c09b-4caa-86ec-a0a64465e419\") " pod="ai-tenant-e2e-shared-a-w1-60261a/sub-test-model-60261a-kserve-5cddbb6dd5-skpws" Nov 03 14:23:32.087706 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:32.087665 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwx8s\" (UniqueName: \"kubernetes.io/projected/5d1bbe2d-c09b-4caa-86ec-a0a64465e419-kube-api-access-mwx8s\") pod \"sub-test-model-60261a-kserve-5cddbb6dd5-skpws\" (UID: \"5d1bbe2d-c09b-4caa-86ec-a0a64465e419\") " pod="ai-tenant-e2e-shared-a-w1-60261a/sub-test-model-60261a-kserve-5cddbb6dd5-skpws" Nov 03 14:23:32.087790 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:32.087762 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d1bbe2d-c09b-4caa-86ec-a0a64465e419-tmp-dir\") pod \"sub-test-model-60261a-kserve-5cddbb6dd5-skpws\" (UID: \"5d1bbe2d-c09b-4caa-86ec-a0a64465e419\") " pod="ai-tenant-e2e-shared-a-w1-60261a/sub-test-model-60261a-kserve-5cddbb6dd5-skpws" Nov 03 14:23:32.087857 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:32.087836 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d1bbe2d-c09b-4caa-86ec-a0a64465e419-home\") pod \"sub-test-model-60261a-kserve-5cddbb6dd5-skpws\" (UID: \"5d1bbe2d-c09b-4caa-86ec-a0a64465e419\") " pod="ai-tenant-e2e-shared-a-w1-60261a/sub-test-model-60261a-kserve-5cddbb6dd5-skpws" Nov 03 14:23:32.087918 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:32.087900 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d1bbe2d-c09b-4caa-86ec-a0a64465e419-tls-certs\") pod \"sub-test-model-60261a-kserve-5cddbb6dd5-skpws\" (UID: \"5d1bbe2d-c09b-4caa-86ec-a0a64465e419\") " pod="ai-tenant-e2e-shared-a-w1-60261a/sub-test-model-60261a-kserve-5cddbb6dd5-skpws" Nov 03 14:23:32.115524 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:32.115487 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-60261a\"/\"openshift-service-ca.crt\"" Nov 03 14:23:32.115715 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:32.115633 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-60261a\"/\"default-dockercfg-4n97f\"" Nov 03 14:23:32.115715 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:32.115659 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-60261a\"/\"sub-test-model-60261a-kserve-self-signed-certs\"" Nov 03 14:23:32.115715 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:32.115692 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-60261a\"/\"kube-root-ca.crt\"" Nov 03 14:23:32.189942 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:32.189904 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d1bbe2d-c09b-4caa-86ec-a0a64465e419-home\") pod \"sub-test-model-60261a-kserve-5cddbb6dd5-skpws\" (UID: \"5d1bbe2d-c09b-4caa-86ec-a0a64465e419\") " pod="ai-tenant-e2e-shared-a-w1-60261a/sub-test-model-60261a-kserve-5cddbb6dd5-skpws" Nov 03 14:23:32.190213 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:32.190193 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d1bbe2d-c09b-4caa-86ec-a0a64465e419-tls-certs\") pod \"sub-test-model-60261a-kserve-5cddbb6dd5-skpws\" (UID: \"5d1bbe2d-c09b-4caa-86ec-a0a64465e419\") " pod="ai-tenant-e2e-shared-a-w1-60261a/sub-test-model-60261a-kserve-5cddbb6dd5-skpws" Nov 03 14:23:32.190449 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:32.190434 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d1bbe2d-c09b-4caa-86ec-a0a64465e419-dshm\") pod \"sub-test-model-60261a-kserve-5cddbb6dd5-skpws\" (UID: \"5d1bbe2d-c09b-4caa-86ec-a0a64465e419\") " pod="ai-tenant-e2e-shared-a-w1-60261a/sub-test-model-60261a-kserve-5cddbb6dd5-skpws" Nov 03 14:23:32.190760 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:32.190738 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d1bbe2d-c09b-4caa-86ec-a0a64465e419-model-cache\") pod \"sub-test-model-60261a-kserve-5cddbb6dd5-skpws\" (UID: \"5d1bbe2d-c09b-4caa-86ec-a0a64465e419\") " pod="ai-tenant-e2e-shared-a-w1-60261a/sub-test-model-60261a-kserve-5cddbb6dd5-skpws" Nov 03 14:23:32.190902 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:32.190883 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwx8s\" (UniqueName: \"kubernetes.io/projected/5d1bbe2d-c09b-4caa-86ec-a0a64465e419-kube-api-access-mwx8s\") pod \"sub-test-model-60261a-kserve-5cddbb6dd5-skpws\" (UID: \"5d1bbe2d-c09b-4caa-86ec-a0a64465e419\") " pod="ai-tenant-e2e-shared-a-w1-60261a/sub-test-model-60261a-kserve-5cddbb6dd5-skpws" Nov 03 14:23:32.191023 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:32.191008 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d1bbe2d-c09b-4caa-86ec-a0a64465e419-tmp-dir\") pod \"sub-test-model-60261a-kserve-5cddbb6dd5-skpws\" (UID: \"5d1bbe2d-c09b-4caa-86ec-a0a64465e419\") " pod="ai-tenant-e2e-shared-a-w1-60261a/sub-test-model-60261a-kserve-5cddbb6dd5-skpws" Nov 03 14:23:32.191633 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:32.191609 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d1bbe2d-c09b-4caa-86ec-a0a64465e419-tmp-dir\") pod \"sub-test-model-60261a-kserve-5cddbb6dd5-skpws\" (UID: \"5d1bbe2d-c09b-4caa-86ec-a0a64465e419\") " pod="ai-tenant-e2e-shared-a-w1-60261a/sub-test-model-60261a-kserve-5cddbb6dd5-skpws" Nov 03 14:23:32.192047 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:32.192024 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d1bbe2d-c09b-4caa-86ec-a0a64465e419-model-cache\") pod \"sub-test-model-60261a-kserve-5cddbb6dd5-skpws\" (UID: \"5d1bbe2d-c09b-4caa-86ec-a0a64465e419\") " pod="ai-tenant-e2e-shared-a-w1-60261a/sub-test-model-60261a-kserve-5cddbb6dd5-skpws" Nov 03 14:23:32.197747 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:32.192923 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d1bbe2d-c09b-4caa-86ec-a0a64465e419-home\") pod \"sub-test-model-60261a-kserve-5cddbb6dd5-skpws\" (UID: \"5d1bbe2d-c09b-4caa-86ec-a0a64465e419\") " pod="ai-tenant-e2e-shared-a-w1-60261a/sub-test-model-60261a-kserve-5cddbb6dd5-skpws" Nov 03 14:23:32.200555 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:32.198249 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d1bbe2d-c09b-4caa-86ec-a0a64465e419-dshm\") pod \"sub-test-model-60261a-kserve-5cddbb6dd5-skpws\" (UID: \"5d1bbe2d-c09b-4caa-86ec-a0a64465e419\") " pod="ai-tenant-e2e-shared-a-w1-60261a/sub-test-model-60261a-kserve-5cddbb6dd5-skpws" Nov 03 14:23:32.200555 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:32.199655 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d1bbe2d-c09b-4caa-86ec-a0a64465e419-tls-certs\") pod \"sub-test-model-60261a-kserve-5cddbb6dd5-skpws\" (UID: \"5d1bbe2d-c09b-4caa-86ec-a0a64465e419\") " pod="ai-tenant-e2e-shared-a-w1-60261a/sub-test-model-60261a-kserve-5cddbb6dd5-skpws" Nov 03 14:23:32.247166 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:32.247132 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwx8s\" (UniqueName: \"kubernetes.io/projected/5d1bbe2d-c09b-4caa-86ec-a0a64465e419-kube-api-access-mwx8s\") pod \"sub-test-model-60261a-kserve-5cddbb6dd5-skpws\" (UID: \"5d1bbe2d-c09b-4caa-86ec-a0a64465e419\") " pod="ai-tenant-e2e-shared-a-w1-60261a/sub-test-model-60261a-kserve-5cddbb6dd5-skpws" Nov 03 14:23:32.332609 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:32.332524 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-60261a/sub-test-model-60261a-kserve-5cddbb6dd5-skpws" Nov 03 14:23:32.709338 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:32.709307 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-60261a/sub-test-model-60261a-kserve-5cddbb6dd5-skpws"] Nov 03 14:23:32.716662 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:23:32.716632 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d1bbe2d_c09b_4caa_86ec_a0a64465e419.slice/crio-e2a9398a33a0562afcd15e7b89c8325e13bfeb3b0ebea2c3154228f04673f5fe WatchSource:0}: Error finding container e2a9398a33a0562afcd15e7b89c8325e13bfeb3b0ebea2c3154228f04673f5fe: Status 404 returned error can't find the container with id e2a9398a33a0562afcd15e7b89c8325e13bfeb3b0ebea2c3154228f04673f5fe Nov 03 14:23:32.931845 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:32.931806 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-60261a/sub-test-model-60261a-kserve-5cddbb6dd5-skpws" event={"ID":"5d1bbe2d-c09b-4caa-86ec-a0a64465e419","Type":"ContainerStarted","Data":"543360af1b7ca39f7373e4ad819fe6033ca5eb02fe8fae0fe163df2b6b8ff415"} Nov 03 14:23:32.932036 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:32.931852 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-60261a/sub-test-model-60261a-kserve-5cddbb6dd5-skpws" event={"ID":"5d1bbe2d-c09b-4caa-86ec-a0a64465e419","Type":"ContainerStarted","Data":"e2a9398a33a0562afcd15e7b89c8325e13bfeb3b0ebea2c3154228f04673f5fe"} Nov 03 14:23:33.333497 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:33.333438 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-60261a/sub-test-model-60261a-kserve-5cddbb6dd5-skpws" Nov 03 14:23:33.557454 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:33.557403 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-60261a/sub-test-model-60261a-kserve-5cddbb6dd5-skpws" podStartSLOduration=2.557387989 podStartE2EDuration="2.557387989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:23:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:23:33.045807712 +0000 UTC m=+4695.056495955" watchObservedRunningTime="2026-11-03 14:23:33.557387989 +0000 UTC m=+4695.568076276" Nov 03 14:23:33.559123 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:33.559105 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:23:37.235389 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:23:37.235349 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:23:38.687552 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:38.687514 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-456f842c-rgb2l"] Nov 03 14:23:38.692136 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:38.692111 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-456f842c-rgb2l" Nov 03 14:23:38.787976 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:38.787540 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 14:23:38.787976 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:38.787791 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:23:38.792094 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:38.792070 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-456f842c-rgb2l"] Nov 03 14:23:38.803576 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:38.803554 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bb75ba9-1a58-4a3a-8013-a85739105276-maas-api-xks-ca\") pod \"maas-api-revoke-sub-456f842c-rgb2l\" (UID: \"0bb75ba9-1a58-4a3a-8013-a85739105276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-456f842c-rgb2l" Nov 03 14:23:38.803684 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:38.803598 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bb75ba9-1a58-4a3a-8013-a85739105276-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-456f842c-rgb2l\" (UID: \"0bb75ba9-1a58-4a3a-8013-a85739105276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-456f842c-rgb2l" Nov 03 14:23:38.803756 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:38.803744 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhbpc\" (UniqueName: \"kubernetes.io/projected/0bb75ba9-1a58-4a3a-8013-a85739105276-kube-api-access-mhbpc\") pod \"maas-api-revoke-sub-456f842c-rgb2l\" (UID: \"0bb75ba9-1a58-4a3a-8013-a85739105276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-456f842c-rgb2l" Nov 03 14:23:38.905114 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:38.905077 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhbpc\" (UniqueName: \"kubernetes.io/projected/0bb75ba9-1a58-4a3a-8013-a85739105276-kube-api-access-mhbpc\") pod \"maas-api-revoke-sub-456f842c-rgb2l\" (UID: \"0bb75ba9-1a58-4a3a-8013-a85739105276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-456f842c-rgb2l" Nov 03 14:23:38.905300 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:38.905208 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bb75ba9-1a58-4a3a-8013-a85739105276-maas-api-xks-ca\") pod \"maas-api-revoke-sub-456f842c-rgb2l\" (UID: \"0bb75ba9-1a58-4a3a-8013-a85739105276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-456f842c-rgb2l" Nov 03 14:23:38.905300 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:38.905243 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bb75ba9-1a58-4a3a-8013-a85739105276-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-456f842c-rgb2l\" (UID: \"0bb75ba9-1a58-4a3a-8013-a85739105276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-456f842c-rgb2l" Nov 03 14:23:38.905870 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:38.905849 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bb75ba9-1a58-4a3a-8013-a85739105276-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-456f842c-rgb2l\" (UID: \"0bb75ba9-1a58-4a3a-8013-a85739105276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-456f842c-rgb2l" Nov 03 14:23:38.907627 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:38.907608 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bb75ba9-1a58-4a3a-8013-a85739105276-maas-api-xks-ca\") pod \"maas-api-revoke-sub-456f842c-rgb2l\" (UID: \"0bb75ba9-1a58-4a3a-8013-a85739105276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-456f842c-rgb2l" Nov 03 14:23:38.958430 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:38.958363 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhbpc\" (UniqueName: \"kubernetes.io/projected/0bb75ba9-1a58-4a3a-8013-a85739105276-kube-api-access-mhbpc\") pod \"maas-api-revoke-sub-456f842c-rgb2l\" (UID: \"0bb75ba9-1a58-4a3a-8013-a85739105276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-456f842c-rgb2l" Nov 03 14:23:39.006319 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:39.006289 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-456f842c-rgb2l" Nov 03 14:23:39.383323 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:39.383293 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-456f842c-rgb2l"] Nov 03 14:23:39.389835 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:23:39.389807 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bb75ba9_1a58_4a3a_8013_a85739105276.slice/crio-e03fc773741012d81ddbec551dfaf88e8ec79ebadf8227d68ccebabd03437c4e WatchSource:0}: Error finding container e03fc773741012d81ddbec551dfaf88e8ec79ebadf8227d68ccebabd03437c4e: Status 404 returned error can't find the container with id e03fc773741012d81ddbec551dfaf88e8ec79ebadf8227d68ccebabd03437c4e Nov 03 14:23:39.990085 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:39.990049 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-456f842c-rgb2l" event={"ID":"0bb75ba9-1a58-4a3a-8013-a85739105276","Type":"ContainerStarted","Data":"2bf8a76175d2954784636ee7c41c6e80dce1e34cbd5972432c0c1e74b9f23009"} Nov 03 14:23:39.990505 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:39.990102 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-456f842c-rgb2l" event={"ID":"0bb75ba9-1a58-4a3a-8013-a85739105276","Type":"ContainerStarted","Data":"e03fc773741012d81ddbec551dfaf88e8ec79ebadf8227d68ccebabd03437c4e"} Nov 03 14:23:40.098560 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:40.098510 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-456f842c-rgb2l" podStartSLOduration=2.098496691 podStartE2EDuration="2.098496691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:23:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:23:40.096680922 +0000 UTC m=+4702.107369164" watchObservedRunningTime="2026-11-03 14:23:40.098496691 +0000 UTC m=+4702.109184929" Nov 03 14:23:40.968709 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:40.968362 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d99263c-lsvgn"] Nov 03 14:23:40.980506 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:40.975692 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d99263c-lsvgn" Nov 03 14:23:41.001032 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:41.000999 5979 generic.go:356] "Generic (PLEG): container finished" podID="0bb75ba9-1a58-4a3a-8013-a85739105276" containerID="2bf8a76175d2954784636ee7c41c6e80dce1e34cbd5972432c0c1e74b9f23009" exitCode=0 Nov 03 14:23:41.001382 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:41.001040 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-456f842c-rgb2l" event={"ID":"0bb75ba9-1a58-4a3a-8013-a85739105276","Type":"ContainerDied","Data":"2bf8a76175d2954784636ee7c41c6e80dce1e34cbd5972432c0c1e74b9f23009"} Nov 03 14:23:41.021856 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:41.021823 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d99263c-lsvgn"] Nov 03 14:23:41.041414 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:41.041381 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d7217b4-7e79-4f5a-b05e-9eb0ad53e9da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d99263c-lsvgn\" (UID: \"0d7217b4-7e79-4f5a-b05e-9eb0ad53e9da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d99263c-lsvgn" Nov 03 14:23:41.041611 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:41.041535 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d7217b4-7e79-4f5a-b05e-9eb0ad53e9da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d99263c-lsvgn\" (UID: \"0d7217b4-7e79-4f5a-b05e-9eb0ad53e9da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d99263c-lsvgn" Nov 03 14:23:41.041735 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:41.041718 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dngt7\" (UniqueName: \"kubernetes.io/projected/0d7217b4-7e79-4f5a-b05e-9eb0ad53e9da-kube-api-access-dngt7\") pod \"maas-api-revoke-sub-2d99263c-lsvgn\" (UID: \"0d7217b4-7e79-4f5a-b05e-9eb0ad53e9da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d99263c-lsvgn" Nov 03 14:23:41.143175 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:41.143137 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d7217b4-7e79-4f5a-b05e-9eb0ad53e9da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d99263c-lsvgn\" (UID: \"0d7217b4-7e79-4f5a-b05e-9eb0ad53e9da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d99263c-lsvgn" Nov 03 14:23:41.143376 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:41.143222 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d7217b4-7e79-4f5a-b05e-9eb0ad53e9da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d99263c-lsvgn\" (UID: \"0d7217b4-7e79-4f5a-b05e-9eb0ad53e9da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d99263c-lsvgn" Nov 03 14:23:41.143376 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:41.143322 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dngt7\" (UniqueName: \"kubernetes.io/projected/0d7217b4-7e79-4f5a-b05e-9eb0ad53e9da-kube-api-access-dngt7\") pod \"maas-api-revoke-sub-2d99263c-lsvgn\" (UID: \"0d7217b4-7e79-4f5a-b05e-9eb0ad53e9da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d99263c-lsvgn" Nov 03 14:23:41.143884 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:41.143846 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d7217b4-7e79-4f5a-b05e-9eb0ad53e9da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d99263c-lsvgn\" (UID: \"0d7217b4-7e79-4f5a-b05e-9eb0ad53e9da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d99263c-lsvgn" Nov 03 14:23:41.145682 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:41.145659 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d7217b4-7e79-4f5a-b05e-9eb0ad53e9da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d99263c-lsvgn\" (UID: \"0d7217b4-7e79-4f5a-b05e-9eb0ad53e9da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d99263c-lsvgn" Nov 03 14:23:41.197040 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:41.197007 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dngt7\" (UniqueName: \"kubernetes.io/projected/0d7217b4-7e79-4f5a-b05e-9eb0ad53e9da-kube-api-access-dngt7\") pod \"maas-api-revoke-sub-2d99263c-lsvgn\" (UID: \"0d7217b4-7e79-4f5a-b05e-9eb0ad53e9da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d99263c-lsvgn" Nov 03 14:23:41.302172 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:41.302087 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d99263c-lsvgn" Nov 03 14:23:41.676814 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:41.676769 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d99263c-lsvgn"] Nov 03 14:23:41.684115 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:23:41.684085 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d7217b4_7e79_4f5a_b05e_9eb0ad53e9da.slice/crio-07afbf387c8015975c649f9dfb92d35f94782448bf012ce06d6c9f22e3f7049b WatchSource:0}: Error finding container 07afbf387c8015975c649f9dfb92d35f94782448bf012ce06d6c9f22e3f7049b: Status 404 returned error can't find the container with id 07afbf387c8015975c649f9dfb92d35f94782448bf012ce06d6c9f22e3f7049b Nov 03 14:23:42.008619 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:42.008579 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d99263c-lsvgn" event={"ID":"0d7217b4-7e79-4f5a-b05e-9eb0ad53e9da","Type":"ContainerStarted","Data":"9f93770d223351f867177c52585cafd9b5e342493f0429489a6ba7ef8ea6144a"} Nov 03 14:23:42.009005 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:42.008638 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d99263c-lsvgn" event={"ID":"0d7217b4-7e79-4f5a-b05e-9eb0ad53e9da","Type":"ContainerStarted","Data":"07afbf387c8015975c649f9dfb92d35f94782448bf012ce06d6c9f22e3f7049b"} Nov 03 14:23:42.125322 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:42.125266 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d99263c-lsvgn" podStartSLOduration=2.125253159 podStartE2EDuration="2.125253159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:23:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:23:42.124762653 +0000 UTC m=+4704.135450896" watchObservedRunningTime="2026-11-03 14:23:42.125253159 +0000 UTC m=+4704.135941401" Nov 03 14:23:42.142249 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:42.142221 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-456f842c-rgb2l" Nov 03 14:23:42.257584 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:42.257501 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0bb75ba9-1a58-4a3a-8013-a85739105276-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bb75ba9-1a58-4a3a-8013-a85739105276-maas-api-xks-ca\") pod \"0bb75ba9-1a58-4a3a-8013-a85739105276\" (UID: \"0bb75ba9-1a58-4a3a-8013-a85739105276\") " Nov 03 14:23:42.257758 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:42.257674 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bb75ba9-1a58-4a3a-8013-a85739105276-kube-api-access-mhbpc\" (UniqueName: \"kubernetes.io/projected/0bb75ba9-1a58-4a3a-8013-a85739105276-kube-api-access-mhbpc\") pod \"0bb75ba9-1a58-4a3a-8013-a85739105276\" (UID: \"0bb75ba9-1a58-4a3a-8013-a85739105276\") " Nov 03 14:23:42.257852 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:42.257757 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bb75ba9-1a58-4a3a-8013-a85739105276-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bb75ba9-1a58-4a3a-8013-a85739105276-maas-api-openshift-ca\") pod \"0bb75ba9-1a58-4a3a-8013-a85739105276\" (UID: \"0bb75ba9-1a58-4a3a-8013-a85739105276\") " Nov 03 14:23:42.259736 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:42.259706 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bb75ba9-1a58-4a3a-8013-a85739105276-maas-api-openshift-ca" pod "0bb75ba9-1a58-4a3a-8013-a85739105276" (UID: "0bb75ba9-1a58-4a3a-8013-a85739105276"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:23:42.260501 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:42.260445 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bb75ba9-1a58-4a3a-8013-a85739105276-maas-api-xks-ca" pod "0bb75ba9-1a58-4a3a-8013-a85739105276" (UID: "0bb75ba9-1a58-4a3a-8013-a85739105276"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:23:42.260784 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:42.260603 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bb75ba9-1a58-4a3a-8013-a85739105276-kube-api-access-mhbpc" pod "0bb75ba9-1a58-4a3a-8013-a85739105276" (UID: "0bb75ba9-1a58-4a3a-8013-a85739105276"). InnerVolumeSpecName "kube-api-access-mhbpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:23:42.333573 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:42.333543 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-60261a/sub-test-model-60261a-kserve-5cddbb6dd5-skpws" Nov 03 14:23:42.346538 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:42.346511 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-60261a/sub-test-model-60261a-kserve-5cddbb6dd5-skpws" Nov 03 14:23:42.359346 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:42.359317 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhbpc\" (UniqueName: \"kubernetes.io/projected/0bb75ba9-1a58-4a3a-8013-a85739105276-kube-api-access-mhbpc\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:23:42.359346 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:42.359343 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bb75ba9-1a58-4a3a-8013-a85739105276-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:23:42.359531 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:42.359356 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bb75ba9-1a58-4a3a-8013-a85739105276-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:23:43.016824 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:43.016786 5979 generic.go:356] "Generic (PLEG): container finished" podID="0d7217b4-7e79-4f5a-b05e-9eb0ad53e9da" containerID="9f93770d223351f867177c52585cafd9b5e342493f0429489a6ba7ef8ea6144a" exitCode=0 Nov 03 14:23:43.017335 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:43.016880 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d99263c-lsvgn" event={"ID":"0d7217b4-7e79-4f5a-b05e-9eb0ad53e9da","Type":"ContainerDied","Data":"9f93770d223351f867177c52585cafd9b5e342493f0429489a6ba7ef8ea6144a"} Nov 03 14:23:43.018382 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:43.018364 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-456f842c-rgb2l" Nov 03 14:23:43.018510 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:43.018399 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-456f842c-rgb2l" event={"ID":"0bb75ba9-1a58-4a3a-8013-a85739105276","Type":"ContainerDied","Data":"e03fc773741012d81ddbec551dfaf88e8ec79ebadf8227d68ccebabd03437c4e"} Nov 03 14:23:43.018510 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:43.018439 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e03fc773741012d81ddbec551dfaf88e8ec79ebadf8227d68ccebabd03437c4e" Nov 03 14:23:43.030053 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:43.030031 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-60261a/sub-test-model-60261a-kserve-5cddbb6dd5-skpws" Nov 03 14:23:44.186934 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:44.186904 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d99263c-lsvgn" Nov 03 14:23:44.284363 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:44.284284 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d7217b4-7e79-4f5a-b05e-9eb0ad53e9da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d7217b4-7e79-4f5a-b05e-9eb0ad53e9da-maas-api-openshift-ca\") pod \"0d7217b4-7e79-4f5a-b05e-9eb0ad53e9da\" (UID: \"0d7217b4-7e79-4f5a-b05e-9eb0ad53e9da\") " Nov 03 14:23:44.284544 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:44.284364 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d7217b4-7e79-4f5a-b05e-9eb0ad53e9da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d7217b4-7e79-4f5a-b05e-9eb0ad53e9da-maas-api-xks-ca\") pod \"0d7217b4-7e79-4f5a-b05e-9eb0ad53e9da\" (UID: \"0d7217b4-7e79-4f5a-b05e-9eb0ad53e9da\") " Nov 03 14:23:44.284606 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:44.284565 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d7217b4-7e79-4f5a-b05e-9eb0ad53e9da-kube-api-access-dngt7\" (UniqueName: \"kubernetes.io/projected/0d7217b4-7e79-4f5a-b05e-9eb0ad53e9da-kube-api-access-dngt7\") pod \"0d7217b4-7e79-4f5a-b05e-9eb0ad53e9da\" (UID: \"0d7217b4-7e79-4f5a-b05e-9eb0ad53e9da\") " Nov 03 14:23:44.284682 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:44.284659 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d7217b4-7e79-4f5a-b05e-9eb0ad53e9da-maas-api-openshift-ca" pod "0d7217b4-7e79-4f5a-b05e-9eb0ad53e9da" (UID: "0d7217b4-7e79-4f5a-b05e-9eb0ad53e9da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:23:44.284907 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:44.284892 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d7217b4-7e79-4f5a-b05e-9eb0ad53e9da-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:23:44.286404 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:44.286379 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d7217b4-7e79-4f5a-b05e-9eb0ad53e9da-maas-api-xks-ca" pod "0d7217b4-7e79-4f5a-b05e-9eb0ad53e9da" (UID: "0d7217b4-7e79-4f5a-b05e-9eb0ad53e9da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:23:44.286596 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:44.286576 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d7217b4-7e79-4f5a-b05e-9eb0ad53e9da-kube-api-access-dngt7" pod "0d7217b4-7e79-4f5a-b05e-9eb0ad53e9da" (UID: "0d7217b4-7e79-4f5a-b05e-9eb0ad53e9da"). InnerVolumeSpecName "kube-api-access-dngt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:23:44.386682 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:44.386643 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dngt7\" (UniqueName: \"kubernetes.io/projected/0d7217b4-7e79-4f5a-b05e-9eb0ad53e9da-kube-api-access-dngt7\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:23:44.386682 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:44.386678 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d7217b4-7e79-4f5a-b05e-9eb0ad53e9da-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:23:45.034277 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:45.034247 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d99263c-lsvgn" Nov 03 14:23:45.034489 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:45.034273 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d99263c-lsvgn" event={"ID":"0d7217b4-7e79-4f5a-b05e-9eb0ad53e9da","Type":"ContainerDied","Data":"07afbf387c8015975c649f9dfb92d35f94782448bf012ce06d6c9f22e3f7049b"} Nov 03 14:23:45.034489 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:45.034326 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07afbf387c8015975c649f9dfb92d35f94782448bf012ce06d6c9f22e3f7049b" Nov 03 14:23:48.046846 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:48.046810 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e745c4d5-bwwdl"] Nov 03 14:23:48.093484 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:48.093443 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e745c4d5-bwwdl"] Nov 03 14:23:48.118072 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:48.118033 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2048118-7dbd-46d4-9f4b-fec408439ed6" path="/var/lib/kubelet/pods/b2048118-7dbd-46d4-9f4b-fec408439ed6/volumes" Nov 03 14:23:49.388731 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:49.388696 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87e0a1a4-kmz8f"] Nov 03 14:23:49.389303 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:49.389288 5979 cpu_manager.go:414] "containerMap: removing container" podUID="0bb75ba9-1a58-4a3a-8013-a85739105276" containerName="revoke-keys" Nov 03 14:23:49.389359 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:49.389303 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bb75ba9-1a58-4a3a-8013-a85739105276" containerName="revoke-keys" Nov 03 14:23:49.389359 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:49.389318 5979 cpu_manager.go:414] "containerMap: removing container" podUID="0d7217b4-7e79-4f5a-b05e-9eb0ad53e9da" containerName="revoke-keys" Nov 03 14:23:49.389359 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:49.389324 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d7217b4-7e79-4f5a-b05e-9eb0ad53e9da" containerName="revoke-keys" Nov 03 14:23:49.389461 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:49.389425 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bb75ba9-1a58-4a3a-8013-a85739105276" containerName="revoke-keys" Nov 03 14:23:49.389461 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:49.389433 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d7217b4-7e79-4f5a-b05e-9eb0ad53e9da" containerName="revoke-keys" Nov 03 14:23:49.393512 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:49.393460 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e0a1a4-kmz8f" Nov 03 14:23:49.489733 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:49.489700 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:23:49.489970 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:49.489746 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 14:23:49.493016 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:49.492987 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87e0a1a4-kmz8f"] Nov 03 14:23:49.553399 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:49.553366 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w72s\" (UniqueName: \"kubernetes.io/projected/ab455ae0-3c29-4b6e-9caf-2c705e2a9d4b-kube-api-access-4w72s\") pod \"maas-api-revoke-sub-87e0a1a4-kmz8f\" (UID: \"ab455ae0-3c29-4b6e-9caf-2c705e2a9d4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e0a1a4-kmz8f" Nov 03 14:23:49.553605 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:49.553416 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab455ae0-3c29-4b6e-9caf-2c705e2a9d4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87e0a1a4-kmz8f\" (UID: \"ab455ae0-3c29-4b6e-9caf-2c705e2a9d4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e0a1a4-kmz8f" Nov 03 14:23:49.553605 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:49.553556 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab455ae0-3c29-4b6e-9caf-2c705e2a9d4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87e0a1a4-kmz8f\" (UID: \"ab455ae0-3c29-4b6e-9caf-2c705e2a9d4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e0a1a4-kmz8f" Nov 03 14:23:49.654838 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:49.654747 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w72s\" (UniqueName: \"kubernetes.io/projected/ab455ae0-3c29-4b6e-9caf-2c705e2a9d4b-kube-api-access-4w72s\") pod \"maas-api-revoke-sub-87e0a1a4-kmz8f\" (UID: \"ab455ae0-3c29-4b6e-9caf-2c705e2a9d4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e0a1a4-kmz8f" Nov 03 14:23:49.654838 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:49.654803 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab455ae0-3c29-4b6e-9caf-2c705e2a9d4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87e0a1a4-kmz8f\" (UID: \"ab455ae0-3c29-4b6e-9caf-2c705e2a9d4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e0a1a4-kmz8f" Nov 03 14:23:49.655082 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:49.654856 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab455ae0-3c29-4b6e-9caf-2c705e2a9d4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87e0a1a4-kmz8f\" (UID: \"ab455ae0-3c29-4b6e-9caf-2c705e2a9d4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e0a1a4-kmz8f" Nov 03 14:23:49.655504 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:49.655461 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab455ae0-3c29-4b6e-9caf-2c705e2a9d4b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87e0a1a4-kmz8f\" (UID: \"ab455ae0-3c29-4b6e-9caf-2c705e2a9d4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e0a1a4-kmz8f" Nov 03 14:23:49.657284 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:49.657262 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab455ae0-3c29-4b6e-9caf-2c705e2a9d4b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87e0a1a4-kmz8f\" (UID: \"ab455ae0-3c29-4b6e-9caf-2c705e2a9d4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e0a1a4-kmz8f" Nov 03 14:23:49.709053 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:49.709019 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w72s\" (UniqueName: \"kubernetes.io/projected/ab455ae0-3c29-4b6e-9caf-2c705e2a9d4b-kube-api-access-4w72s\") pod \"maas-api-revoke-sub-87e0a1a4-kmz8f\" (UID: \"ab455ae0-3c29-4b6e-9caf-2c705e2a9d4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e0a1a4-kmz8f" Nov 03 14:23:50.007277 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:50.007198 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e0a1a4-kmz8f" Nov 03 14:23:50.383029 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:50.382983 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87e0a1a4-kmz8f"] Nov 03 14:23:50.392377 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:23:50.392350 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab455ae0_3c29_4b6e_9caf_2c705e2a9d4b.slice/crio-561a27bdba5cd794f3fab7c51533bca4179119c28ada8fe50dd833d33ecc484c WatchSource:0}: Error finding container 561a27bdba5cd794f3fab7c51533bca4179119c28ada8fe50dd833d33ecc484c: Status 404 returned error can't find the container with id 561a27bdba5cd794f3fab7c51533bca4179119c28ada8fe50dd833d33ecc484c Nov 03 14:23:51.078156 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:51.078075 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e0a1a4-kmz8f" event={"ID":"ab455ae0-3c29-4b6e-9caf-2c705e2a9d4b","Type":"ContainerStarted","Data":"8f65d5867c2cfe608b77a613457f667af16a99f3d984522d5180781de70e65e8"} Nov 03 14:23:51.078312 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:51.078156 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e0a1a4-kmz8f" event={"ID":"ab455ae0-3c29-4b6e-9caf-2c705e2a9d4b","Type":"ContainerStarted","Data":"561a27bdba5cd794f3fab7c51533bca4179119c28ada8fe50dd833d33ecc484c"} Nov 03 14:23:51.232885 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:51.232705 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e0a1a4-kmz8f" podStartSLOduration=2.232688609 podStartE2EDuration="2.232688609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:23:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:23:51.183758738 +0000 UTC m=+4713.194446980" watchObservedRunningTime="2026-11-03 14:23:51.232688609 +0000 UTC m=+4713.243376850" Nov 03 14:23:52.087176 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:52.087138 5979 generic.go:356] "Generic (PLEG): container finished" podID="ab455ae0-3c29-4b6e-9caf-2c705e2a9d4b" containerID="8f65d5867c2cfe608b77a613457f667af16a99f3d984522d5180781de70e65e8" exitCode=0 Nov 03 14:23:52.087682 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:52.087211 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e0a1a4-kmz8f" event={"ID":"ab455ae0-3c29-4b6e-9caf-2c705e2a9d4b","Type":"ContainerDied","Data":"8f65d5867c2cfe608b77a613457f667af16a99f3d984522d5180781de70e65e8"} Nov 03 14:23:53.242896 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:53.242870 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e0a1a4-kmz8f" Nov 03 14:23:53.303074 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:53.303044 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab455ae0-3c29-4b6e-9caf-2c705e2a9d4b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab455ae0-3c29-4b6e-9caf-2c705e2a9d4b-maas-api-openshift-ca\") pod \"ab455ae0-3c29-4b6e-9caf-2c705e2a9d4b\" (UID: \"ab455ae0-3c29-4b6e-9caf-2c705e2a9d4b\") " Nov 03 14:23:53.303248 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:53.303158 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab455ae0-3c29-4b6e-9caf-2c705e2a9d4b-kube-api-access-4w72s\" (UniqueName: \"kubernetes.io/projected/ab455ae0-3c29-4b6e-9caf-2c705e2a9d4b-kube-api-access-4w72s\") pod \"ab455ae0-3c29-4b6e-9caf-2c705e2a9d4b\" (UID: \"ab455ae0-3c29-4b6e-9caf-2c705e2a9d4b\") " Nov 03 14:23:53.303314 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:53.303282 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab455ae0-3c29-4b6e-9caf-2c705e2a9d4b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab455ae0-3c29-4b6e-9caf-2c705e2a9d4b-maas-api-xks-ca\") pod \"ab455ae0-3c29-4b6e-9caf-2c705e2a9d4b\" (UID: \"ab455ae0-3c29-4b6e-9caf-2c705e2a9d4b\") " Nov 03 14:23:53.303407 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:53.303383 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab455ae0-3c29-4b6e-9caf-2c705e2a9d4b-maas-api-openshift-ca" pod "ab455ae0-3c29-4b6e-9caf-2c705e2a9d4b" (UID: "ab455ae0-3c29-4b6e-9caf-2c705e2a9d4b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:23:53.303655 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:53.303640 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab455ae0-3c29-4b6e-9caf-2c705e2a9d4b-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:23:53.305356 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:53.305331 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab455ae0-3c29-4b6e-9caf-2c705e2a9d4b-maas-api-xks-ca" pod "ab455ae0-3c29-4b6e-9caf-2c705e2a9d4b" (UID: "ab455ae0-3c29-4b6e-9caf-2c705e2a9d4b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:23:53.305459 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:53.305368 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab455ae0-3c29-4b6e-9caf-2c705e2a9d4b-kube-api-access-4w72s" pod "ab455ae0-3c29-4b6e-9caf-2c705e2a9d4b" (UID: "ab455ae0-3c29-4b6e-9caf-2c705e2a9d4b"). InnerVolumeSpecName "kube-api-access-4w72s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:23:53.405017 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:53.404977 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w72s\" (UniqueName: \"kubernetes.io/projected/ab455ae0-3c29-4b6e-9caf-2c705e2a9d4b-kube-api-access-4w72s\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:23:53.405017 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:53.405014 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab455ae0-3c29-4b6e-9caf-2c705e2a9d4b-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:23:53.929764 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:53.929728 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4683a8dc-jmf9m"] Nov 03 14:23:53.930358 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:53.930339 5979 cpu_manager.go:414] "containerMap: removing container" podUID="ab455ae0-3c29-4b6e-9caf-2c705e2a9d4b" containerName="revoke-keys" Nov 03 14:23:53.930405 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:53.930360 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab455ae0-3c29-4b6e-9caf-2c705e2a9d4b" containerName="revoke-keys" Nov 03 14:23:53.930480 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:53.930461 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab455ae0-3c29-4b6e-9caf-2c705e2a9d4b" containerName="revoke-keys" Nov 03 14:23:53.934322 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:53.934303 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4683a8dc-jmf9m" Nov 03 14:23:53.988050 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:53.988014 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4683a8dc-jmf9m"] Nov 03 14:23:54.012804 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:54.012764 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bf41e61-5fac-4e9a-9287-46bf62a5bcfa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4683a8dc-jmf9m\" (UID: \"9bf41e61-5fac-4e9a-9287-46bf62a5bcfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4683a8dc-jmf9m" Nov 03 14:23:54.012988 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:54.012945 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bf41e61-5fac-4e9a-9287-46bf62a5bcfa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4683a8dc-jmf9m\" (UID: \"9bf41e61-5fac-4e9a-9287-46bf62a5bcfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4683a8dc-jmf9m" Nov 03 14:23:54.013057 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:54.013014 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjcln\" (UniqueName: \"kubernetes.io/projected/9bf41e61-5fac-4e9a-9287-46bf62a5bcfa-kube-api-access-bjcln\") pod \"maas-api-revoke-sub-4683a8dc-jmf9m\" (UID: \"9bf41e61-5fac-4e9a-9287-46bf62a5bcfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4683a8dc-jmf9m" Nov 03 14:23:54.104220 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:54.104179 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e0a1a4-kmz8f" Nov 03 14:23:54.104408 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:54.104223 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87e0a1a4-kmz8f" event={"ID":"ab455ae0-3c29-4b6e-9caf-2c705e2a9d4b","Type":"ContainerDied","Data":"561a27bdba5cd794f3fab7c51533bca4179119c28ada8fe50dd833d33ecc484c"} Nov 03 14:23:54.104408 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:54.104288 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="561a27bdba5cd794f3fab7c51533bca4179119c28ada8fe50dd833d33ecc484c" Nov 03 14:23:54.113981 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:54.113791 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bf41e61-5fac-4e9a-9287-46bf62a5bcfa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4683a8dc-jmf9m\" (UID: \"9bf41e61-5fac-4e9a-9287-46bf62a5bcfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4683a8dc-jmf9m" Nov 03 14:23:54.113981 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:54.113851 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjcln\" (UniqueName: \"kubernetes.io/projected/9bf41e61-5fac-4e9a-9287-46bf62a5bcfa-kube-api-access-bjcln\") pod \"maas-api-revoke-sub-4683a8dc-jmf9m\" (UID: \"9bf41e61-5fac-4e9a-9287-46bf62a5bcfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4683a8dc-jmf9m" Nov 03 14:23:54.114139 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:54.114023 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bf41e61-5fac-4e9a-9287-46bf62a5bcfa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4683a8dc-jmf9m\" (UID: \"9bf41e61-5fac-4e9a-9287-46bf62a5bcfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4683a8dc-jmf9m" Nov 03 14:23:54.114692 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:54.114668 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bf41e61-5fac-4e9a-9287-46bf62a5bcfa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4683a8dc-jmf9m\" (UID: \"9bf41e61-5fac-4e9a-9287-46bf62a5bcfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4683a8dc-jmf9m" Nov 03 14:23:54.116508 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:54.116487 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bf41e61-5fac-4e9a-9287-46bf62a5bcfa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4683a8dc-jmf9m\" (UID: \"9bf41e61-5fac-4e9a-9287-46bf62a5bcfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4683a8dc-jmf9m" Nov 03 14:23:54.169453 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:54.169419 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjcln\" (UniqueName: \"kubernetes.io/projected/9bf41e61-5fac-4e9a-9287-46bf62a5bcfa-kube-api-access-bjcln\") pod \"maas-api-revoke-sub-4683a8dc-jmf9m\" (UID: \"9bf41e61-5fac-4e9a-9287-46bf62a5bcfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4683a8dc-jmf9m" Nov 03 14:23:54.249369 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:54.249293 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4683a8dc-jmf9m" Nov 03 14:23:54.629256 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:54.629173 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4683a8dc-jmf9m"] Nov 03 14:23:54.636461 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:23:54.636434 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bf41e61_5fac_4e9a_9287_46bf62a5bcfa.slice/crio-d9d6ba7a8dd77627e77fa05fdb39ac6a27e7aedc9480947e5cd0648b44968db9 WatchSource:0}: Error finding container d9d6ba7a8dd77627e77fa05fdb39ac6a27e7aedc9480947e5cd0648b44968db9: Status 404 returned error can't find the container with id d9d6ba7a8dd77627e77fa05fdb39ac6a27e7aedc9480947e5cd0648b44968db9 Nov 03 14:23:55.112891 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:55.112849 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4683a8dc-jmf9m" event={"ID":"9bf41e61-5fac-4e9a-9287-46bf62a5bcfa","Type":"ContainerStarted","Data":"cd92d9fcbd6f18d0b2912cb9d9eba675dea71acead1a8455846476eb444850b7"} Nov 03 14:23:55.113074 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:55.112905 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4683a8dc-jmf9m" event={"ID":"9bf41e61-5fac-4e9a-9287-46bf62a5bcfa","Type":"ContainerStarted","Data":"d9d6ba7a8dd77627e77fa05fdb39ac6a27e7aedc9480947e5cd0648b44968db9"} Nov 03 14:23:55.255328 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:55.255217 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4683a8dc-jmf9m" podStartSLOduration=2.255198902 podStartE2EDuration="2.255198902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:23:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:23:55.254436649 +0000 UTC m=+4717.265124892" watchObservedRunningTime="2026-11-03 14:23:55.255198902 +0000 UTC m=+4717.265887166" Nov 03 14:23:56.121305 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:56.121265 5979 generic.go:356] "Generic (PLEG): container finished" podID="9bf41e61-5fac-4e9a-9287-46bf62a5bcfa" containerID="cd92d9fcbd6f18d0b2912cb9d9eba675dea71acead1a8455846476eb444850b7" exitCode=0 Nov 03 14:23:56.121502 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:56.121306 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4683a8dc-jmf9m" event={"ID":"9bf41e61-5fac-4e9a-9287-46bf62a5bcfa","Type":"ContainerDied","Data":"cd92d9fcbd6f18d0b2912cb9d9eba675dea71acead1a8455846476eb444850b7"} Nov 03 14:23:57.269182 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:57.269157 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4683a8dc-jmf9m" Nov 03 14:23:57.370762 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:57.370724 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bf41e61-5fac-4e9a-9287-46bf62a5bcfa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bf41e61-5fac-4e9a-9287-46bf62a5bcfa-maas-api-openshift-ca\") pod \"9bf41e61-5fac-4e9a-9287-46bf62a5bcfa\" (UID: \"9bf41e61-5fac-4e9a-9287-46bf62a5bcfa\") " Nov 03 14:23:57.370955 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:57.370822 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bf41e61-5fac-4e9a-9287-46bf62a5bcfa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bf41e61-5fac-4e9a-9287-46bf62a5bcfa-maas-api-xks-ca\") pod \"9bf41e61-5fac-4e9a-9287-46bf62a5bcfa\" (UID: \"9bf41e61-5fac-4e9a-9287-46bf62a5bcfa\") " Nov 03 14:23:57.370955 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:57.370913 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bf41e61-5fac-4e9a-9287-46bf62a5bcfa-kube-api-access-bjcln\" (UniqueName: \"kubernetes.io/projected/9bf41e61-5fac-4e9a-9287-46bf62a5bcfa-kube-api-access-bjcln\") pod \"9bf41e61-5fac-4e9a-9287-46bf62a5bcfa\" (UID: \"9bf41e61-5fac-4e9a-9287-46bf62a5bcfa\") " Nov 03 14:23:57.371102 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:57.371066 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bf41e61-5fac-4e9a-9287-46bf62a5bcfa-maas-api-openshift-ca" pod "9bf41e61-5fac-4e9a-9287-46bf62a5bcfa" (UID: "9bf41e61-5fac-4e9a-9287-46bf62a5bcfa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:23:57.371359 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:57.371343 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bf41e61-5fac-4e9a-9287-46bf62a5bcfa-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:23:57.372998 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:57.372972 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bf41e61-5fac-4e9a-9287-46bf62a5bcfa-maas-api-xks-ca" pod "9bf41e61-5fac-4e9a-9287-46bf62a5bcfa" (UID: "9bf41e61-5fac-4e9a-9287-46bf62a5bcfa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:23:57.373114 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:57.373004 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bf41e61-5fac-4e9a-9287-46bf62a5bcfa-kube-api-access-bjcln" pod "9bf41e61-5fac-4e9a-9287-46bf62a5bcfa" (UID: "9bf41e61-5fac-4e9a-9287-46bf62a5bcfa"). InnerVolumeSpecName "kube-api-access-bjcln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:23:57.472791 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:57.472752 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bf41e61-5fac-4e9a-9287-46bf62a5bcfa-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:23:57.472791 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:57.472800 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjcln\" (UniqueName: \"kubernetes.io/projected/9bf41e61-5fac-4e9a-9287-46bf62a5bcfa-kube-api-access-bjcln\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:23:58.135482 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:58.135447 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4683a8dc-jmf9m" Nov 03 14:23:58.135634 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:58.135498 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4683a8dc-jmf9m" event={"ID":"9bf41e61-5fac-4e9a-9287-46bf62a5bcfa","Type":"ContainerDied","Data":"d9d6ba7a8dd77627e77fa05fdb39ac6a27e7aedc9480947e5cd0648b44968db9"} Nov 03 14:23:58.135634 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:23:58.135540 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9d6ba7a8dd77627e77fa05fdb39ac6a27e7aedc9480947e5cd0648b44968db9" Nov 03 14:24:00.051187 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:00.051143 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dceac349-7nzfc"] Nov 03 14:24:00.097812 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:00.097768 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dceac349-7nzfc"] Nov 03 14:24:00.117347 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:00.117315 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a29e24d0-3285-4676-8171-622e9dedacd4" path="/var/lib/kubelet/pods/a29e24d0-3285-4676-8171-622e9dedacd4/volumes" Nov 03 14:24:04.383361 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:04.383316 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:24:07.234680 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:24:07.234645 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:24:25.422512 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:25.422459 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2399b8-894f4bdc9-7djnj"] Nov 03 14:24:25.541549 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:25.537292 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2399b8-56ff84677f-mvh5j"] Nov 03 14:24:30.423102 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:30.423059 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-2399b8-894f4bdc9-7djnj" podUID="e14c9532-f275-4474-a672-e76765a4d168" containerName="payload-processing" containerID="cri-o://05b51b00ef466a300d34fd3abba5c99be85dd4d208f8ba6d830befd7966cdc48" gracePeriod=25 Nov 03 14:24:30.538338 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:30.538285 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2399b8-56ff84677f-mvh5j" podUID="d2b41401-7b37-4da6-bbde-d33fd6cd7094" containerName="payload-pre-processing" containerID="cri-o://0930521fa609964d2ef759f1f8b27bfa4330af996e5306a6ca06d6f320595173" gracePeriod=25 Nov 03 14:24:30.691686 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:30.691617 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2399b8-894f4bdc9-7djnj" Nov 03 14:24:30.746657 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:30.746622 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e14c9532-f275-4474-a672-e76765a4d168-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e14c9532-f275-4474-a672-e76765a4d168-plugins-config-volume\") pod \"e14c9532-f275-4474-a672-e76765a4d168\" (UID: \"e14c9532-f275-4474-a672-e76765a4d168\") " Nov 03 14:24:30.746847 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:30.746827 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e14c9532-f275-4474-a672-e76765a4d168-kube-api-access-4pdhv\" (UniqueName: \"kubernetes.io/projected/e14c9532-f275-4474-a672-e76765a4d168-kube-api-access-4pdhv\") pod \"e14c9532-f275-4474-a672-e76765a4d168\" (UID: \"e14c9532-f275-4474-a672-e76765a4d168\") " Nov 03 14:24:30.747150 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:30.747126 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e14c9532-f275-4474-a672-e76765a4d168-plugins-config-volume" pod "e14c9532-f275-4474-a672-e76765a4d168" (UID: "e14c9532-f275-4474-a672-e76765a4d168"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:24:30.747315 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:30.747300 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e14c9532-f275-4474-a672-e76765a4d168-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:24:30.749018 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:30.748986 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e14c9532-f275-4474-a672-e76765a4d168-kube-api-access-4pdhv" pod "e14c9532-f275-4474-a672-e76765a4d168" (UID: "e14c9532-f275-4474-a672-e76765a4d168"). InnerVolumeSpecName "kube-api-access-4pdhv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:24:30.849087 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:30.849043 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pdhv\" (UniqueName: \"kubernetes.io/projected/e14c9532-f275-4474-a672-e76765a4d168-kube-api-access-4pdhv\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:24:31.355803 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:31.355770 5979 generic.go:356] "Generic (PLEG): container finished" podID="e14c9532-f275-4474-a672-e76765a4d168" containerID="05b51b00ef466a300d34fd3abba5c99be85dd4d208f8ba6d830befd7966cdc48" exitCode=0 Nov 03 14:24:31.356034 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:31.355822 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2399b8-894f4bdc9-7djnj" Nov 03 14:24:31.356034 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:31.355849 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2399b8-894f4bdc9-7djnj" event={"ID":"e14c9532-f275-4474-a672-e76765a4d168","Type":"ContainerDied","Data":"05b51b00ef466a300d34fd3abba5c99be85dd4d208f8ba6d830befd7966cdc48"} Nov 03 14:24:31.356034 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:31.355900 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2399b8-894f4bdc9-7djnj" event={"ID":"e14c9532-f275-4474-a672-e76765a4d168","Type":"ContainerDied","Data":"b5a77a1341d111f28a538dbce087c6aa5bcff02e7d37c0aa841ed45347f82abe"} Nov 03 14:24:31.356034 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:31.355923 5979 scope.go:122] "RemoveContainer" containerID="05b51b00ef466a300d34fd3abba5c99be85dd4d208f8ba6d830befd7966cdc48" Nov 03 14:24:31.366323 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:31.366304 5979 scope.go:122] "RemoveContainer" containerID="05b51b00ef466a300d34fd3abba5c99be85dd4d208f8ba6d830befd7966cdc48" Nov 03 14:24:31.366608 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:24:31.366589 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05b51b00ef466a300d34fd3abba5c99be85dd4d208f8ba6d830befd7966cdc48\": container with ID starting with 05b51b00ef466a300d34fd3abba5c99be85dd4d208f8ba6d830befd7966cdc48 not found: ID does not exist" containerID="05b51b00ef466a300d34fd3abba5c99be85dd4d208f8ba6d830befd7966cdc48" Nov 03 14:24:31.366668 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:31.366615 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05b51b00ef466a300d34fd3abba5c99be85dd4d208f8ba6d830befd7966cdc48"} err="failed to get container status \"05b51b00ef466a300d34fd3abba5c99be85dd4d208f8ba6d830befd7966cdc48\": rpc error: code = NotFound desc = could not find container \"05b51b00ef466a300d34fd3abba5c99be85dd4d208f8ba6d830befd7966cdc48\": container with ID starting with 05b51b00ef466a300d34fd3abba5c99be85dd4d208f8ba6d830befd7966cdc48 not found: ID does not exist" Nov 03 14:24:31.510401 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:31.510368 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2399b8-894f4bdc9-7djnj"] Nov 03 14:24:31.556904 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:31.556871 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2399b8-894f4bdc9-7djnj"] Nov 03 14:24:32.117271 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:32.117237 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e14c9532-f275-4474-a672-e76765a4d168" path="/var/lib/kubelet/pods/e14c9532-f275-4474-a672-e76765a4d168/volumes" Nov 03 14:24:37.234320 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:24:37.234277 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:24:38.585553 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:38.585516 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2399b8-56ff84677f-mvh5j" podUID="d2b41401-7b37-4da6-bbde-d33fd6cd7094" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:24:42.444066 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:42.444034 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d0231d-6c8bcc59bb-z4jqb"] Nov 03 14:24:42.444587 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:42.444240 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d0231d-6c8bcc59bb-z4jqb" podUID="e63fc009-1095-4586-96b7-ddae424d47ec" containerName="maas-api" containerID="cri-o://cb460a419f06f2ccd18a152c2691970e17efbfaffbd77029609b65472b5c59c4" gracePeriod=30 Nov 03 14:24:42.739100 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:42.739073 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d0231d-6c8bcc59bb-z4jqb" Nov 03 14:24:42.919341 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:42.919304 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e63fc009-1095-4586-96b7-ddae424d47ec-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e63fc009-1095-4586-96b7-ddae424d47ec-openshift-service-ca\") pod \"e63fc009-1095-4586-96b7-ddae424d47ec\" (UID: \"e63fc009-1095-4586-96b7-ddae424d47ec\") " Nov 03 14:24:42.919565 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:42.919387 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e63fc009-1095-4586-96b7-ddae424d47ec-kube-api-access-56fjv\" (UniqueName: \"kubernetes.io/projected/e63fc009-1095-4586-96b7-ddae424d47ec-kube-api-access-56fjv\") pod \"e63fc009-1095-4586-96b7-ddae424d47ec\" (UID: \"e63fc009-1095-4586-96b7-ddae424d47ec\") " Nov 03 14:24:42.919565 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:42.919464 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e63fc009-1095-4586-96b7-ddae424d47ec-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e63fc009-1095-4586-96b7-ddae424d47ec-maas-api-tls\") pod \"e63fc009-1095-4586-96b7-ddae424d47ec\" (UID: \"e63fc009-1095-4586-96b7-ddae424d47ec\") " Nov 03 14:24:42.919722 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:42.919601 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e63fc009-1095-4586-96b7-ddae424d47ec-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e63fc009-1095-4586-96b7-ddae424d47ec-metrics-certs\") pod \"e63fc009-1095-4586-96b7-ddae424d47ec\" (UID: \"e63fc009-1095-4586-96b7-ddae424d47ec\") " Nov 03 14:24:42.920510 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:42.919812 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e63fc009-1095-4586-96b7-ddae424d47ec-openshift-service-ca" pod "e63fc009-1095-4586-96b7-ddae424d47ec" (UID: "e63fc009-1095-4586-96b7-ddae424d47ec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:24:42.920510 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:42.920139 5979 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e63fc009-1095-4586-96b7-ddae424d47ec-openshift-service-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:24:42.921775 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:42.921749 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e63fc009-1095-4586-96b7-ddae424d47ec-maas-api-tls" pod "e63fc009-1095-4586-96b7-ddae424d47ec" (UID: "e63fc009-1095-4586-96b7-ddae424d47ec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:24:42.922093 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:42.922072 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e63fc009-1095-4586-96b7-ddae424d47ec-metrics-certs" pod "e63fc009-1095-4586-96b7-ddae424d47ec" (UID: "e63fc009-1095-4586-96b7-ddae424d47ec"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:24:42.922146 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:42.922120 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e63fc009-1095-4586-96b7-ddae424d47ec-kube-api-access-56fjv" pod "e63fc009-1095-4586-96b7-ddae424d47ec" (UID: "e63fc009-1095-4586-96b7-ddae424d47ec"). InnerVolumeSpecName "kube-api-access-56fjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:24:43.021219 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:43.021131 5979 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e63fc009-1095-4586-96b7-ddae424d47ec-metrics-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:24:43.021219 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:43.021164 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56fjv\" (UniqueName: \"kubernetes.io/projected/e63fc009-1095-4586-96b7-ddae424d47ec-kube-api-access-56fjv\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:24:43.021219 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:43.021177 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e63fc009-1095-4586-96b7-ddae424d47ec-maas-api-tls\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:24:43.443184 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:43.443143 5979 generic.go:356] "Generic (PLEG): container finished" podID="e63fc009-1095-4586-96b7-ddae424d47ec" containerID="cb460a419f06f2ccd18a152c2691970e17efbfaffbd77029609b65472b5c59c4" exitCode=0 Nov 03 14:24:43.446379 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:43.446352 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d0231d-6c8bcc59bb-z4jqb" Nov 03 14:24:43.453888 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:43.453544 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d0231d-6c8bcc59bb-z4jqb" event={"ID":"e63fc009-1095-4586-96b7-ddae424d47ec","Type":"ContainerDied","Data":"cb460a419f06f2ccd18a152c2691970e17efbfaffbd77029609b65472b5c59c4"} Nov 03 14:24:43.453888 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:43.453620 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d0231d-6c8bcc59bb-z4jqb" event={"ID":"e63fc009-1095-4586-96b7-ddae424d47ec","Type":"ContainerDied","Data":"8f735aca9243b8bb4aee45db995ee57c3d609c9272fbacb5fdc5b15e808f674f"} Nov 03 14:24:43.453888 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:43.453653 5979 scope.go:122] "RemoveContainer" containerID="cb460a419f06f2ccd18a152c2691970e17efbfaffbd77029609b65472b5c59c4" Nov 03 14:24:43.506288 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:43.506267 5979 scope.go:122] "RemoveContainer" containerID="cb460a419f06f2ccd18a152c2691970e17efbfaffbd77029609b65472b5c59c4" Nov 03 14:24:43.506631 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:24:43.506606 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb460a419f06f2ccd18a152c2691970e17efbfaffbd77029609b65472b5c59c4\": container with ID starting with cb460a419f06f2ccd18a152c2691970e17efbfaffbd77029609b65472b5c59c4 not found: ID does not exist" containerID="cb460a419f06f2ccd18a152c2691970e17efbfaffbd77029609b65472b5c59c4" Nov 03 14:24:43.506700 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:43.506639 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb460a419f06f2ccd18a152c2691970e17efbfaffbd77029609b65472b5c59c4"} err="failed to get container status \"cb460a419f06f2ccd18a152c2691970e17efbfaffbd77029609b65472b5c59c4\": rpc error: code = NotFound desc = could not find container \"cb460a419f06f2ccd18a152c2691970e17efbfaffbd77029609b65472b5c59c4\": container with ID starting with cb460a419f06f2ccd18a152c2691970e17efbfaffbd77029609b65472b5c59c4 not found: ID does not exist" Nov 03 14:24:43.611848 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:43.611812 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d0231d-6c8bcc59bb-z4jqb"] Nov 03 14:24:43.658991 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:43.658954 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-d0231d-6c8bcc59bb-z4jqb"] Nov 03 14:24:44.116091 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:44.116057 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e63fc009-1095-4586-96b7-ddae424d47ec" path="/var/lib/kubelet/pods/e63fc009-1095-4586-96b7-ddae424d47ec/volumes" Nov 03 14:24:48.585355 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:48.585317 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2399b8-56ff84677f-mvh5j" podUID="d2b41401-7b37-4da6-bbde-d33fd6cd7094" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:24:49.365057 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:49.365007 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:24:49.843892 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:49.843855 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx"] Nov 03 14:24:49.844295 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:49.844113 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx" podUID="43b38a5b-6579-4ce3-8bc6-1334bed0c633" containerName="istio-proxy" containerID="cri-o://d6b42d4b7c65ebb2df3d6e402c7a86e292cf5f12c1ce393d2c812accc608a5cf" gracePeriod=30 Nov 03 14:24:50.806056 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:50.805998 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2399b8-56ff84677f-mvh5j" Nov 03 14:24:50.845120 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:50.845091 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2b41401-7b37-4da6-bbde-d33fd6cd7094-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2b41401-7b37-4da6-bbde-d33fd6cd7094-plugins-config-volume\") pod \"d2b41401-7b37-4da6-bbde-d33fd6cd7094\" (UID: \"d2b41401-7b37-4da6-bbde-d33fd6cd7094\") " Nov 03 14:24:50.845613 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:50.845179 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2b41401-7b37-4da6-bbde-d33fd6cd7094-kube-api-access-lmt8m\" (UniqueName: \"kubernetes.io/projected/d2b41401-7b37-4da6-bbde-d33fd6cd7094-kube-api-access-lmt8m\") pod \"d2b41401-7b37-4da6-bbde-d33fd6cd7094\" (UID: \"d2b41401-7b37-4da6-bbde-d33fd6cd7094\") " Nov 03 14:24:50.845613 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:50.845452 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2b41401-7b37-4da6-bbde-d33fd6cd7094-plugins-config-volume" pod "d2b41401-7b37-4da6-bbde-d33fd6cd7094" (UID: "d2b41401-7b37-4da6-bbde-d33fd6cd7094"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:24:50.845711 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:50.845626 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2b41401-7b37-4da6-bbde-d33fd6cd7094-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:24:50.847384 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:50.847364 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2b41401-7b37-4da6-bbde-d33fd6cd7094-kube-api-access-lmt8m" pod "d2b41401-7b37-4da6-bbde-d33fd6cd7094" (UID: "d2b41401-7b37-4da6-bbde-d33fd6cd7094"). InnerVolumeSpecName "kube-api-access-lmt8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:24:50.947003 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:50.946964 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmt8m\" (UniqueName: \"kubernetes.io/projected/d2b41401-7b37-4da6-bbde-d33fd6cd7094-kube-api-access-lmt8m\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:24:51.510712 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:51.510678 5979 generic.go:356] "Generic (PLEG): container finished" podID="d2b41401-7b37-4da6-bbde-d33fd6cd7094" containerID="0930521fa609964d2ef759f1f8b27bfa4330af996e5306a6ca06d6f320595173" exitCode=0 Nov 03 14:24:51.510904 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:51.510736 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2399b8-56ff84677f-mvh5j" Nov 03 14:24:51.510904 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:51.510742 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2399b8-56ff84677f-mvh5j" event={"ID":"d2b41401-7b37-4da6-bbde-d33fd6cd7094","Type":"ContainerDied","Data":"0930521fa609964d2ef759f1f8b27bfa4330af996e5306a6ca06d6f320595173"} Nov 03 14:24:51.510904 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:51.510791 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2399b8-56ff84677f-mvh5j" event={"ID":"d2b41401-7b37-4da6-bbde-d33fd6cd7094","Type":"ContainerDied","Data":"eb39fd1ebbec527b447773b23eaf1af66bea9b9d454ca6ceaa9256fa45255d8e"} Nov 03 14:24:51.510904 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:51.510813 5979 scope.go:122] "RemoveContainer" containerID="0930521fa609964d2ef759f1f8b27bfa4330af996e5306a6ca06d6f320595173" Nov 03 14:24:51.521721 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:51.521701 5979 scope.go:122] "RemoveContainer" containerID="0930521fa609964d2ef759f1f8b27bfa4330af996e5306a6ca06d6f320595173" Nov 03 14:24:51.521992 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:24:51.521971 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0930521fa609964d2ef759f1f8b27bfa4330af996e5306a6ca06d6f320595173\": container with ID starting with 0930521fa609964d2ef759f1f8b27bfa4330af996e5306a6ca06d6f320595173 not found: ID does not exist" containerID="0930521fa609964d2ef759f1f8b27bfa4330af996e5306a6ca06d6f320595173" Nov 03 14:24:51.522049 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:51.522002 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0930521fa609964d2ef759f1f8b27bfa4330af996e5306a6ca06d6f320595173"} err="failed to get container status \"0930521fa609964d2ef759f1f8b27bfa4330af996e5306a6ca06d6f320595173\": rpc error: code = NotFound desc = could not find container \"0930521fa609964d2ef759f1f8b27bfa4330af996e5306a6ca06d6f320595173\": container with ID starting with 0930521fa609964d2ef759f1f8b27bfa4330af996e5306a6ca06d6f320595173 not found: ID does not exist" Nov 03 14:24:51.665073 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:51.665022 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2399b8-56ff84677f-mvh5j"] Nov 03 14:24:51.711146 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:51.711114 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2399b8-56ff84677f-mvh5j"] Nov 03 14:24:52.115955 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:52.115909 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2b41401-7b37-4da6-bbde-d33fd6cd7094" path="/var/lib/kubelet/pods/d2b41401-7b37-4da6-bbde-d33fd6cd7094/volumes" Nov 03 14:24:55.121268 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:55.121241 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx" Nov 03 14:24:55.201906 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:55.201864 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43b38a5b-6579-4ce3-8bc6-1334bed0c633-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43b38a5b-6579-4ce3-8bc6-1334bed0c633-istiod-ca-cert\") pod \"43b38a5b-6579-4ce3-8bc6-1334bed0c633\" (UID: \"43b38a5b-6579-4ce3-8bc6-1334bed0c633\") " Nov 03 14:24:55.202133 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:55.201918 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/43b38a5b-6579-4ce3-8bc6-1334bed0c633-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43b38a5b-6579-4ce3-8bc6-1334bed0c633-istio-podinfo\") pod \"43b38a5b-6579-4ce3-8bc6-1334bed0c633\" (UID: \"43b38a5b-6579-4ce3-8bc6-1334bed0c633\") " Nov 03 14:24:55.202133 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:55.201950 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43b38a5b-6579-4ce3-8bc6-1334bed0c633-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43b38a5b-6579-4ce3-8bc6-1334bed0c633-workload-socket\") pod \"43b38a5b-6579-4ce3-8bc6-1334bed0c633\" (UID: \"43b38a5b-6579-4ce3-8bc6-1334bed0c633\") " Nov 03 14:24:55.202133 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:55.202030 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43b38a5b-6579-4ce3-8bc6-1334bed0c633-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43b38a5b-6579-4ce3-8bc6-1334bed0c633-istio-data\") pod \"43b38a5b-6579-4ce3-8bc6-1334bed0c633\" (UID: \"43b38a5b-6579-4ce3-8bc6-1334bed0c633\") " Nov 03 14:24:55.202133 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:55.202083 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43b38a5b-6579-4ce3-8bc6-1334bed0c633-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43b38a5b-6579-4ce3-8bc6-1334bed0c633-credential-socket\") pod \"43b38a5b-6579-4ce3-8bc6-1334bed0c633\" (UID: \"43b38a5b-6579-4ce3-8bc6-1334bed0c633\") " Nov 03 14:24:55.202133 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:55.202134 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43b38a5b-6579-4ce3-8bc6-1334bed0c633-istio-token\" (UniqueName: \"kubernetes.io/projected/43b38a5b-6579-4ce3-8bc6-1334bed0c633-istio-token\") pod \"43b38a5b-6579-4ce3-8bc6-1334bed0c633\" (UID: \"43b38a5b-6579-4ce3-8bc6-1334bed0c633\") " Nov 03 14:24:55.202494 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:55.202204 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43b38a5b-6579-4ce3-8bc6-1334bed0c633-kube-api-access-ccbkv\" (UniqueName: \"kubernetes.io/projected/43b38a5b-6579-4ce3-8bc6-1334bed0c633-kube-api-access-ccbkv\") pod \"43b38a5b-6579-4ce3-8bc6-1334bed0c633\" (UID: \"43b38a5b-6579-4ce3-8bc6-1334bed0c633\") " Nov 03 14:24:55.202494 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:55.202245 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43b38a5b-6579-4ce3-8bc6-1334bed0c633-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43b38a5b-6579-4ce3-8bc6-1334bed0c633-istio-envoy\") pod \"43b38a5b-6579-4ce3-8bc6-1334bed0c633\" (UID: \"43b38a5b-6579-4ce3-8bc6-1334bed0c633\") " Nov 03 14:24:55.202494 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:55.202339 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43b38a5b-6579-4ce3-8bc6-1334bed0c633-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43b38a5b-6579-4ce3-8bc6-1334bed0c633-workload-certs\") pod \"43b38a5b-6579-4ce3-8bc6-1334bed0c633\" (UID: \"43b38a5b-6579-4ce3-8bc6-1334bed0c633\") " Nov 03 14:24:55.203442 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:55.203365 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43b38a5b-6579-4ce3-8bc6-1334bed0c633-workload-certs" pod "43b38a5b-6579-4ce3-8bc6-1334bed0c633" (UID: "43b38a5b-6579-4ce3-8bc6-1334bed0c633"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:24:55.204055 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:55.203992 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43b38a5b-6579-4ce3-8bc6-1334bed0c633-istiod-ca-cert" pod "43b38a5b-6579-4ce3-8bc6-1334bed0c633" (UID: "43b38a5b-6579-4ce3-8bc6-1334bed0c633"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:24:55.205103 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:55.204667 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43b38a5b-6579-4ce3-8bc6-1334bed0c633-credential-socket" pod "43b38a5b-6579-4ce3-8bc6-1334bed0c633" (UID: "43b38a5b-6579-4ce3-8bc6-1334bed0c633"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:24:55.205103 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:55.204843 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43b38a5b-6579-4ce3-8bc6-1334bed0c633-workload-socket" pod "43b38a5b-6579-4ce3-8bc6-1334bed0c633" (UID: "43b38a5b-6579-4ce3-8bc6-1334bed0c633"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:24:55.205103 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:55.204994 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43b38a5b-6579-4ce3-8bc6-1334bed0c633-istio-data" pod "43b38a5b-6579-4ce3-8bc6-1334bed0c633" (UID: "43b38a5b-6579-4ce3-8bc6-1334bed0c633"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:24:55.209004 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:55.208964 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43b38a5b-6579-4ce3-8bc6-1334bed0c633-istio-token" pod "43b38a5b-6579-4ce3-8bc6-1334bed0c633" (UID: "43b38a5b-6579-4ce3-8bc6-1334bed0c633"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:24:55.210062 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:55.210034 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43b38a5b-6579-4ce3-8bc6-1334bed0c633-kube-api-access-ccbkv" pod "43b38a5b-6579-4ce3-8bc6-1334bed0c633" (UID: "43b38a5b-6579-4ce3-8bc6-1334bed0c633"). InnerVolumeSpecName "kube-api-access-ccbkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:24:55.210255 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:55.210200 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43b38a5b-6579-4ce3-8bc6-1334bed0c633-istio-envoy" pod "43b38a5b-6579-4ce3-8bc6-1334bed0c633" (UID: "43b38a5b-6579-4ce3-8bc6-1334bed0c633"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:24:55.210330 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:55.210265 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43b38a5b-6579-4ce3-8bc6-1334bed0c633-istio-podinfo" pod "43b38a5b-6579-4ce3-8bc6-1334bed0c633" (UID: "43b38a5b-6579-4ce3-8bc6-1334bed0c633"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:24:55.303863 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:55.303824 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43b38a5b-6579-4ce3-8bc6-1334bed0c633-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:24:55.303863 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:55.303855 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43b38a5b-6579-4ce3-8bc6-1334bed0c633-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:24:55.303863 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:55.303868 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43b38a5b-6579-4ce3-8bc6-1334bed0c633-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:24:55.304176 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:55.303881 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43b38a5b-6579-4ce3-8bc6-1334bed0c633-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:24:55.304176 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:55.303890 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43b38a5b-6579-4ce3-8bc6-1334bed0c633-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:24:55.304176 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:55.303900 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43b38a5b-6579-4ce3-8bc6-1334bed0c633-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:24:55.304176 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:55.303914 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccbkv\" (UniqueName: \"kubernetes.io/projected/43b38a5b-6579-4ce3-8bc6-1334bed0c633-kube-api-access-ccbkv\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:24:55.304176 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:55.303935 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43b38a5b-6579-4ce3-8bc6-1334bed0c633-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:24:55.304176 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:55.303946 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43b38a5b-6579-4ce3-8bc6-1334bed0c633-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:24:55.579577 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:55.579499 5979 generic.go:356] "Generic (PLEG): container finished" podID="43b38a5b-6579-4ce3-8bc6-1334bed0c633" containerID="d6b42d4b7c65ebb2df3d6e402c7a86e292cf5f12c1ce393d2c812accc608a5cf" exitCode=0 Nov 03 14:24:55.579942 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:55.579591 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx" Nov 03 14:24:55.580804 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:55.579612 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx" event={"ID":"43b38a5b-6579-4ce3-8bc6-1334bed0c633","Type":"ContainerDied","Data":"d6b42d4b7c65ebb2df3d6e402c7a86e292cf5f12c1ce393d2c812accc608a5cf"} Nov 03 14:24:55.580944 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:55.580871 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx" event={"ID":"43b38a5b-6579-4ce3-8bc6-1334bed0c633","Type":"ContainerDied","Data":"b45e0e13195bbcc0665fc06084a7a0100dc5fd56a6d87403257f3552e8472c0e"} Nov 03 14:24:55.580944 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:55.580909 5979 scope.go:122] "RemoveContainer" containerID="d6b42d4b7c65ebb2df3d6e402c7a86e292cf5f12c1ce393d2c812accc608a5cf" Nov 03 14:24:55.593609 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:55.593588 5979 scope.go:122] "RemoveContainer" containerID="803cdfa8d38fe35898e16efcbeecfedc78669413551e6edefa85b4182461e608" Nov 03 14:24:55.602771 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:55.602751 5979 scope.go:122] "RemoveContainer" containerID="d6b42d4b7c65ebb2df3d6e402c7a86e292cf5f12c1ce393d2c812accc608a5cf" Nov 03 14:24:55.603066 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:24:55.603041 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6b42d4b7c65ebb2df3d6e402c7a86e292cf5f12c1ce393d2c812accc608a5cf\": container with ID starting with d6b42d4b7c65ebb2df3d6e402c7a86e292cf5f12c1ce393d2c812accc608a5cf not found: ID does not exist" containerID="d6b42d4b7c65ebb2df3d6e402c7a86e292cf5f12c1ce393d2c812accc608a5cf" Nov 03 14:24:55.603151 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:55.603077 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6b42d4b7c65ebb2df3d6e402c7a86e292cf5f12c1ce393d2c812accc608a5cf"} err="failed to get container status \"d6b42d4b7c65ebb2df3d6e402c7a86e292cf5f12c1ce393d2c812accc608a5cf\": rpc error: code = NotFound desc = could not find container \"d6b42d4b7c65ebb2df3d6e402c7a86e292cf5f12c1ce393d2c812accc608a5cf\": container with ID starting with d6b42d4b7c65ebb2df3d6e402c7a86e292cf5f12c1ce393d2c812accc608a5cf not found: ID does not exist" Nov 03 14:24:55.603151 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:55.603103 5979 scope.go:122] "RemoveContainer" containerID="803cdfa8d38fe35898e16efcbeecfedc78669413551e6edefa85b4182461e608" Nov 03 14:24:55.603417 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:24:55.603394 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"803cdfa8d38fe35898e16efcbeecfedc78669413551e6edefa85b4182461e608\": container with ID starting with 803cdfa8d38fe35898e16efcbeecfedc78669413551e6edefa85b4182461e608 not found: ID does not exist" containerID="803cdfa8d38fe35898e16efcbeecfedc78669413551e6edefa85b4182461e608" Nov 03 14:24:55.603478 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:55.603425 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"803cdfa8d38fe35898e16efcbeecfedc78669413551e6edefa85b4182461e608"} err="failed to get container status \"803cdfa8d38fe35898e16efcbeecfedc78669413551e6edefa85b4182461e608\": rpc error: code = NotFound desc = could not find container \"803cdfa8d38fe35898e16efcbeecfedc78669413551e6edefa85b4182461e608\": container with ID starting with 803cdfa8d38fe35898e16efcbeecfedc78669413551e6edefa85b4182461e608 not found: ID does not exist" Nov 03 14:24:55.734708 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:55.734669 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx"] Nov 03 14:24:55.781278 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:55.781249 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-2399b8-openshift-default-78fc6b98bf-8clcx"] Nov 03 14:24:56.115572 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:56.115538 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43b38a5b-6579-4ce3-8bc6-1334bed0c633" path="/var/lib/kubelet/pods/43b38a5b-6579-4ce3-8bc6-1334bed0c633/volumes" Nov 03 14:24:56.419652 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:56.419610 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8359049-kb54l"] Nov 03 14:24:56.420408 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:56.420384 5979 cpu_manager.go:414] "containerMap: removing container" podUID="9bf41e61-5fac-4e9a-9287-46bf62a5bcfa" containerName="revoke-keys" Nov 03 14:24:56.420544 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:56.420408 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bf41e61-5fac-4e9a-9287-46bf62a5bcfa" containerName="revoke-keys" Nov 03 14:24:56.420544 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:56.420429 5979 cpu_manager.go:414] "containerMap: removing container" podUID="d2b41401-7b37-4da6-bbde-d33fd6cd7094" containerName="payload-pre-processing" Nov 03 14:24:56.420544 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:56.420436 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2b41401-7b37-4da6-bbde-d33fd6cd7094" containerName="payload-pre-processing" Nov 03 14:24:56.420544 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:56.420453 5979 cpu_manager.go:414] "containerMap: removing container" podUID="e63fc009-1095-4586-96b7-ddae424d47ec" containerName="maas-api" Nov 03 14:24:56.420544 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:56.420460 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e63fc009-1095-4586-96b7-ddae424d47ec" containerName="maas-api" Nov 03 14:24:56.420544 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:56.420491 5979 cpu_manager.go:414] "containerMap: removing container" podUID="e14c9532-f275-4474-a672-e76765a4d168" containerName="payload-processing" Nov 03 14:24:56.420544 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:56.420500 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e14c9532-f275-4474-a672-e76765a4d168" containerName="payload-processing" Nov 03 14:24:56.420544 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:56.420515 5979 cpu_manager.go:414] "containerMap: removing container" podUID="43b38a5b-6579-4ce3-8bc6-1334bed0c633" containerName="istio-proxy" Nov 03 14:24:56.420544 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:56.420522 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43b38a5b-6579-4ce3-8bc6-1334bed0c633" containerName="istio-proxy" Nov 03 14:24:56.420544 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:56.420547 5979 cpu_manager.go:414] "containerMap: removing container" podUID="43b38a5b-6579-4ce3-8bc6-1334bed0c633" containerName="istio-proxy" Nov 03 14:24:56.421088 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:56.420554 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43b38a5b-6579-4ce3-8bc6-1334bed0c633" containerName="istio-proxy" Nov 03 14:24:56.421088 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:56.420730 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2b41401-7b37-4da6-bbde-d33fd6cd7094" containerName="payload-pre-processing" Nov 03 14:24:56.421088 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:56.420745 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="e63fc009-1095-4586-96b7-ddae424d47ec" containerName="maas-api" Nov 03 14:24:56.421088 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:56.420754 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="e14c9532-f275-4474-a672-e76765a4d168" containerName="payload-processing" Nov 03 14:24:56.421088 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:56.420768 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="43b38a5b-6579-4ce3-8bc6-1334bed0c633" containerName="istio-proxy" Nov 03 14:24:56.421088 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:56.420781 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="43b38a5b-6579-4ce3-8bc6-1334bed0c633" containerName="istio-proxy" Nov 03 14:24:56.421088 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:56.420800 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bf41e61-5fac-4e9a-9287-46bf62a5bcfa" containerName="revoke-keys" Nov 03 14:24:56.426179 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:56.426155 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8359049-kb54l" Nov 03 14:24:56.477090 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:56.477056 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:24:56.479765 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:56.479742 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8359049-kb54l"] Nov 03 14:24:56.515869 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:56.515833 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fa6a699-01c0-4236-8cf4-a9a6270ddf4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8359049-kb54l\" (UID: \"7fa6a699-01c0-4236-8cf4-a9a6270ddf4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8359049-kb54l" Nov 03 14:24:56.516057 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:56.515898 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7p6b\" (UniqueName: \"kubernetes.io/projected/7fa6a699-01c0-4236-8cf4-a9a6270ddf4e-kube-api-access-s7p6b\") pod \"maas-api-revoke-sub-f8359049-kb54l\" (UID: \"7fa6a699-01c0-4236-8cf4-a9a6270ddf4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8359049-kb54l" Nov 03 14:24:56.516057 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:56.516002 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fa6a699-01c0-4236-8cf4-a9a6270ddf4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8359049-kb54l\" (UID: \"7fa6a699-01c0-4236-8cf4-a9a6270ddf4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8359049-kb54l" Nov 03 14:24:56.523632 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:56.523609 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 14:24:56.617670 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:56.617639 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fa6a699-01c0-4236-8cf4-a9a6270ddf4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8359049-kb54l\" (UID: \"7fa6a699-01c0-4236-8cf4-a9a6270ddf4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8359049-kb54l" Nov 03 14:24:56.617893 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:56.617682 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7p6b\" (UniqueName: \"kubernetes.io/projected/7fa6a699-01c0-4236-8cf4-a9a6270ddf4e-kube-api-access-s7p6b\") pod \"maas-api-revoke-sub-f8359049-kb54l\" (UID: \"7fa6a699-01c0-4236-8cf4-a9a6270ddf4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8359049-kb54l" Nov 03 14:24:56.617893 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:56.617712 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fa6a699-01c0-4236-8cf4-a9a6270ddf4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8359049-kb54l\" (UID: \"7fa6a699-01c0-4236-8cf4-a9a6270ddf4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8359049-kb54l" Nov 03 14:24:56.618440 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:56.618419 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fa6a699-01c0-4236-8cf4-a9a6270ddf4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8359049-kb54l\" (UID: \"7fa6a699-01c0-4236-8cf4-a9a6270ddf4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8359049-kb54l" Nov 03 14:24:56.620181 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:56.620146 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fa6a699-01c0-4236-8cf4-a9a6270ddf4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8359049-kb54l\" (UID: \"7fa6a699-01c0-4236-8cf4-a9a6270ddf4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8359049-kb54l" Nov 03 14:24:56.695705 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:56.695637 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7p6b\" (UniqueName: \"kubernetes.io/projected/7fa6a699-01c0-4236-8cf4-a9a6270ddf4e-kube-api-access-s7p6b\") pod \"maas-api-revoke-sub-f8359049-kb54l\" (UID: \"7fa6a699-01c0-4236-8cf4-a9a6270ddf4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8359049-kb54l" Nov 03 14:24:56.740276 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:56.740242 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8359049-kb54l" Nov 03 14:24:57.114520 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:57.114427 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8359049-kb54l"] Nov 03 14:24:57.121115 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:24:57.121090 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fa6a699_01c0_4236_8cf4_a9a6270ddf4e.slice/crio-2db372f59bb52b2162e1e355b706667a0710570e0990d403794c9684bf94f392 WatchSource:0}: Error finding container 2db372f59bb52b2162e1e355b706667a0710570e0990d403794c9684bf94f392: Status 404 returned error can't find the container with id 2db372f59bb52b2162e1e355b706667a0710570e0990d403794c9684bf94f392 Nov 03 14:24:57.184216 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:57.184177 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b5daee-5fcb8897cf-kdkph"] Nov 03 14:24:57.292065 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:57.292025 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b5daee-797cb4f7f-ljgr2"] Nov 03 14:24:57.599732 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:57.599693 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8359049-kb54l" event={"ID":"7fa6a699-01c0-4236-8cf4-a9a6270ddf4e","Type":"ContainerStarted","Data":"7d06238fa5afcf9667e46f3717e58482c980bb978056d5526f733b88aa0d32d2"} Nov 03 14:24:57.600230 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:57.599758 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8359049-kb54l" event={"ID":"7fa6a699-01c0-4236-8cf4-a9a6270ddf4e","Type":"ContainerStarted","Data":"2db372f59bb52b2162e1e355b706667a0710570e0990d403794c9684bf94f392"} Nov 03 14:24:57.709342 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:57.709286 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8359049-kb54l" podStartSLOduration=1.709271268 podStartE2EDuration="1.709271268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:24:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:24:57.707191769 +0000 UTC m=+4779.717880017" watchObservedRunningTime="2026-11-03 14:24:57.709271268 +0000 UTC m=+4779.719959509" Nov 03 14:24:58.673304 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:58.673262 5979 generic.go:356] "Generic (PLEG): container finished" podID="7fa6a699-01c0-4236-8cf4-a9a6270ddf4e" containerID="7d06238fa5afcf9667e46f3717e58482c980bb978056d5526f733b88aa0d32d2" exitCode=0 Nov 03 14:24:58.673753 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:58.673444 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8359049-kb54l" event={"ID":"7fa6a699-01c0-4236-8cf4-a9a6270ddf4e","Type":"ContainerDied","Data":"7d06238fa5afcf9667e46f3717e58482c980bb978056d5526f733b88aa0d32d2"} Nov 03 14:24:59.838585 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:59.838556 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8359049-kb54l" Nov 03 14:24:59.865199 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:59.865165 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fa6a699-01c0-4236-8cf4-a9a6270ddf4e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fa6a699-01c0-4236-8cf4-a9a6270ddf4e-maas-api-xks-ca\") pod \"7fa6a699-01c0-4236-8cf4-a9a6270ddf4e\" (UID: \"7fa6a699-01c0-4236-8cf4-a9a6270ddf4e\") " Nov 03 14:24:59.865199 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:59.865201 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fa6a699-01c0-4236-8cf4-a9a6270ddf4e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fa6a699-01c0-4236-8cf4-a9a6270ddf4e-maas-api-openshift-ca\") pod \"7fa6a699-01c0-4236-8cf4-a9a6270ddf4e\" (UID: \"7fa6a699-01c0-4236-8cf4-a9a6270ddf4e\") " Nov 03 14:24:59.865425 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:59.865230 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fa6a699-01c0-4236-8cf4-a9a6270ddf4e-kube-api-access-s7p6b\" (UniqueName: \"kubernetes.io/projected/7fa6a699-01c0-4236-8cf4-a9a6270ddf4e-kube-api-access-s7p6b\") pod \"7fa6a699-01c0-4236-8cf4-a9a6270ddf4e\" (UID: \"7fa6a699-01c0-4236-8cf4-a9a6270ddf4e\") " Nov 03 14:24:59.865646 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:59.865614 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fa6a699-01c0-4236-8cf4-a9a6270ddf4e-maas-api-openshift-ca" pod "7fa6a699-01c0-4236-8cf4-a9a6270ddf4e" (UID: "7fa6a699-01c0-4236-8cf4-a9a6270ddf4e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:24:59.867311 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:59.867287 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fa6a699-01c0-4236-8cf4-a9a6270ddf4e-maas-api-xks-ca" pod "7fa6a699-01c0-4236-8cf4-a9a6270ddf4e" (UID: "7fa6a699-01c0-4236-8cf4-a9a6270ddf4e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:24:59.867539 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:59.867521 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fa6a699-01c0-4236-8cf4-a9a6270ddf4e-kube-api-access-s7p6b" pod "7fa6a699-01c0-4236-8cf4-a9a6270ddf4e" (UID: "7fa6a699-01c0-4236-8cf4-a9a6270ddf4e"). InnerVolumeSpecName "kube-api-access-s7p6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:24:59.966498 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:59.966405 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fa6a699-01c0-4236-8cf4-a9a6270ddf4e-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:24:59.966498 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:59.966438 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fa6a699-01c0-4236-8cf4-a9a6270ddf4e-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:24:59.966498 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:24:59.966449 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7p6b\" (UniqueName: \"kubernetes.io/projected/7fa6a699-01c0-4236-8cf4-a9a6270ddf4e-kube-api-access-s7p6b\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:25:00.693413 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:00.693369 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8359049-kb54l" event={"ID":"7fa6a699-01c0-4236-8cf4-a9a6270ddf4e","Type":"ContainerDied","Data":"2db372f59bb52b2162e1e355b706667a0710570e0990d403794c9684bf94f392"} Nov 03 14:25:00.693413 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:00.693413 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8359049-kb54l" Nov 03 14:25:00.693725 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:00.693434 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2db372f59bb52b2162e1e355b706667a0710570e0990d403794c9684bf94f392" Nov 03 14:25:00.795687 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:00.795652 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-42c707/distinct-w0-42c707-kserve-64949d6cb8-kq4w9"] Nov 03 14:25:02.184848 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:02.184809 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-b5daee-5fcb8897cf-kdkph" podUID="f0d0e2f4-b051-47a1-8118-88694a09dddc" containerName="payload-processing" containerID="cri-o://c8c75f62c1211c923954989848838a280c4d2947d703bd3adac5a3354778ae19" gracePeriod=25 Nov 03 14:25:02.292585 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:02.292544 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b5daee-797cb4f7f-ljgr2" podUID="97fe4000-c797-4c39-8872-38308ffdf700" containerName="payload-pre-processing" containerID="cri-o://8a802fe4ca0c79bf8310918e639ab746257b6612611e5d227be9e48278b826cd" gracePeriod=25 Nov 03 14:25:03.480426 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:03.480390 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b5daee-797cb4f7f-ljgr2" podUID="97fe4000-c797-4c39-8872-38308ffdf700" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:25:03.480853 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:03.480398 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-b5daee-5fcb8897cf-kdkph" podUID="f0d0e2f4-b051-47a1-8118-88694a09dddc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:25:07.234394 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:25:07.234353 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:25:12.928254 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:12.928210 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16decd81-wg8d8"] Nov 03 14:25:12.929014 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:12.928991 5979 cpu_manager.go:414] "containerMap: removing container" podUID="7fa6a699-01c0-4236-8cf4-a9a6270ddf4e" containerName="revoke-keys" Nov 03 14:25:12.929014 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:12.929013 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fa6a699-01c0-4236-8cf4-a9a6270ddf4e" containerName="revoke-keys" Nov 03 14:25:12.929237 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:12.929176 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fa6a699-01c0-4236-8cf4-a9a6270ddf4e" containerName="revoke-keys" Nov 03 14:25:12.933273 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:12.933253 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16decd81-wg8d8" Nov 03 14:25:12.982588 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:12.982560 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:25:13.028719 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:13.028676 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 14:25:13.031645 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:13.031617 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16decd81-wg8d8"] Nov 03 14:25:13.058158 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:13.058126 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a62f61a9-1d81-4bcc-b161-a6ce4d667fe3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16decd81-wg8d8\" (UID: \"a62f61a9-1d81-4bcc-b161-a6ce4d667fe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16decd81-wg8d8" Nov 03 14:25:13.058297 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:13.058269 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq7rm\" (UniqueName: \"kubernetes.io/projected/a62f61a9-1d81-4bcc-b161-a6ce4d667fe3-kube-api-access-gq7rm\") pod \"maas-api-revoke-sub-16decd81-wg8d8\" (UID: \"a62f61a9-1d81-4bcc-b161-a6ce4d667fe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16decd81-wg8d8" Nov 03 14:25:13.058442 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:13.058418 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a62f61a9-1d81-4bcc-b161-a6ce4d667fe3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16decd81-wg8d8\" (UID: \"a62f61a9-1d81-4bcc-b161-a6ce4d667fe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16decd81-wg8d8" Nov 03 14:25:13.160118 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:13.160082 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a62f61a9-1d81-4bcc-b161-a6ce4d667fe3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16decd81-wg8d8\" (UID: \"a62f61a9-1d81-4bcc-b161-a6ce4d667fe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16decd81-wg8d8" Nov 03 14:25:13.160304 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:13.160174 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a62f61a9-1d81-4bcc-b161-a6ce4d667fe3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16decd81-wg8d8\" (UID: \"a62f61a9-1d81-4bcc-b161-a6ce4d667fe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16decd81-wg8d8" Nov 03 14:25:13.160304 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:13.160282 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq7rm\" (UniqueName: \"kubernetes.io/projected/a62f61a9-1d81-4bcc-b161-a6ce4d667fe3-kube-api-access-gq7rm\") pod \"maas-api-revoke-sub-16decd81-wg8d8\" (UID: \"a62f61a9-1d81-4bcc-b161-a6ce4d667fe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16decd81-wg8d8" Nov 03 14:25:13.160868 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:13.160846 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a62f61a9-1d81-4bcc-b161-a6ce4d667fe3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16decd81-wg8d8\" (UID: \"a62f61a9-1d81-4bcc-b161-a6ce4d667fe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16decd81-wg8d8" Nov 03 14:25:13.162275 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:13.162257 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a62f61a9-1d81-4bcc-b161-a6ce4d667fe3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16decd81-wg8d8\" (UID: \"a62f61a9-1d81-4bcc-b161-a6ce4d667fe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16decd81-wg8d8" Nov 03 14:25:13.214187 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:13.214115 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq7rm\" (UniqueName: \"kubernetes.io/projected/a62f61a9-1d81-4bcc-b161-a6ce4d667fe3-kube-api-access-gq7rm\") pod \"maas-api-revoke-sub-16decd81-wg8d8\" (UID: \"a62f61a9-1d81-4bcc-b161-a6ce4d667fe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16decd81-wg8d8" Nov 03 14:25:13.246815 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:13.246786 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16decd81-wg8d8" Nov 03 14:25:13.480329 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:13.480242 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b5daee-797cb4f7f-ljgr2" podUID="97fe4000-c797-4c39-8872-38308ffdf700" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:25:13.480531 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:13.480242 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-b5daee-5fcb8897cf-kdkph" podUID="f0d0e2f4-b051-47a1-8118-88694a09dddc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:25:13.629379 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:13.629332 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16decd81-wg8d8"] Nov 03 14:25:13.634802 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:25:13.634768 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda62f61a9_1d81_4bcc_b161_a6ce4d667fe3.slice/crio-a62dbd29790bae64fd584608ccf9227e19b297a45a0a55478101146a7e9045a8 WatchSource:0}: Error finding container a62dbd29790bae64fd584608ccf9227e19b297a45a0a55478101146a7e9045a8: Status 404 returned error can't find the container with id a62dbd29790bae64fd584608ccf9227e19b297a45a0a55478101146a7e9045a8 Nov 03 14:25:13.780805 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:13.780721 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16decd81-wg8d8" event={"ID":"a62f61a9-1d81-4bcc-b161-a6ce4d667fe3","Type":"ContainerStarted","Data":"a62dbd29790bae64fd584608ccf9227e19b297a45a0a55478101146a7e9045a8"} Nov 03 14:25:14.510082 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:14.510048 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-42c707-69d5b98f4f-jwf86"] Nov 03 14:25:14.616984 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:14.616956 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-42c707-79c696bfc5-65rpj"] Nov 03 14:25:14.788266 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:14.788181 5979 generic.go:356] "Generic (PLEG): container finished" podID="a62f61a9-1d81-4bcc-b161-a6ce4d667fe3" containerID="94551948ae7a4e2e6b4ac64d76360784bd4b18dd6bf4db51bd441903374d71e2" exitCode=0 Nov 03 14:25:14.788433 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:14.788266 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16decd81-wg8d8" event={"ID":"a62f61a9-1d81-4bcc-b161-a6ce4d667fe3","Type":"ContainerDied","Data":"94551948ae7a4e2e6b4ac64d76360784bd4b18dd6bf4db51bd441903374d71e2"} Nov 03 14:25:15.827237 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:15.827182 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-42c707/distinct-w0-42c707-kserve-64949d6cb8-kq4w9" podUID="430d8f95-204c-4243-b771-a03e30303e77" containerName="main" containerID="cri-o://a0f2959ef1981d75e9f92d0ecec6408978d72894f804638fce4da5f87cbc5676" gracePeriod=45 Nov 03 14:25:16.039512 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:16.039461 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16decd81-wg8d8" Nov 03 14:25:16.131857 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:16.127932 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a62f61a9-1d81-4bcc-b161-a6ce4d667fe3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a62f61a9-1d81-4bcc-b161-a6ce4d667fe3-maas-api-openshift-ca\") pod \"a62f61a9-1d81-4bcc-b161-a6ce4d667fe3\" (UID: \"a62f61a9-1d81-4bcc-b161-a6ce4d667fe3\") " Nov 03 14:25:16.131857 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:16.128066 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a62f61a9-1d81-4bcc-b161-a6ce4d667fe3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a62f61a9-1d81-4bcc-b161-a6ce4d667fe3-maas-api-xks-ca\") pod \"a62f61a9-1d81-4bcc-b161-a6ce4d667fe3\" (UID: \"a62f61a9-1d81-4bcc-b161-a6ce4d667fe3\") " Nov 03 14:25:16.131857 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:16.128198 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a62f61a9-1d81-4bcc-b161-a6ce4d667fe3-kube-api-access-gq7rm\" (UniqueName: \"kubernetes.io/projected/a62f61a9-1d81-4bcc-b161-a6ce4d667fe3-kube-api-access-gq7rm\") pod \"a62f61a9-1d81-4bcc-b161-a6ce4d667fe3\" (UID: \"a62f61a9-1d81-4bcc-b161-a6ce4d667fe3\") " Nov 03 14:25:16.144781 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:16.144721 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a62f61a9-1d81-4bcc-b161-a6ce4d667fe3-maas-api-openshift-ca" pod "a62f61a9-1d81-4bcc-b161-a6ce4d667fe3" (UID: "a62f61a9-1d81-4bcc-b161-a6ce4d667fe3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:25:16.183521 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:16.182798 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a62f61a9-1d81-4bcc-b161-a6ce4d667fe3-kube-api-access-gq7rm" pod "a62f61a9-1d81-4bcc-b161-a6ce4d667fe3" (UID: "a62f61a9-1d81-4bcc-b161-a6ce4d667fe3"). InnerVolumeSpecName "kube-api-access-gq7rm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:25:16.185607 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:16.185552 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a62f61a9-1d81-4bcc-b161-a6ce4d667fe3-maas-api-xks-ca" pod "a62f61a9-1d81-4bcc-b161-a6ce4d667fe3" (UID: "a62f61a9-1d81-4bcc-b161-a6ce4d667fe3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:25:16.198647 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:16.198620 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-42c707/distinct-w0-42c707-kserve-64949d6cb8-kq4w9" Nov 03 14:25:16.229820 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:16.229789 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/430d8f95-204c-4243-b771-a03e30303e77-tls-certs\" (UniqueName: \"kubernetes.io/secret/430d8f95-204c-4243-b771-a03e30303e77-tls-certs\") pod \"430d8f95-204c-4243-b771-a03e30303e77\" (UID: \"430d8f95-204c-4243-b771-a03e30303e77\") " Nov 03 14:25:16.230005 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:16.229851 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/430d8f95-204c-4243-b771-a03e30303e77-dshm\" (UniqueName: \"kubernetes.io/empty-dir/430d8f95-204c-4243-b771-a03e30303e77-dshm\") pod \"430d8f95-204c-4243-b771-a03e30303e77\" (UID: \"430d8f95-204c-4243-b771-a03e30303e77\") " Nov 03 14:25:16.230005 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:16.229899 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/430d8f95-204c-4243-b771-a03e30303e77-kube-api-access-zchgp\" (UniqueName: \"kubernetes.io/projected/430d8f95-204c-4243-b771-a03e30303e77-kube-api-access-zchgp\") pod \"430d8f95-204c-4243-b771-a03e30303e77\" (UID: \"430d8f95-204c-4243-b771-a03e30303e77\") " Nov 03 14:25:16.230128 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:16.230022 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/430d8f95-204c-4243-b771-a03e30303e77-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/430d8f95-204c-4243-b771-a03e30303e77-model-cache\") pod \"430d8f95-204c-4243-b771-a03e30303e77\" (UID: \"430d8f95-204c-4243-b771-a03e30303e77\") " Nov 03 14:25:16.230183 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:16.230135 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/430d8f95-204c-4243-b771-a03e30303e77-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/430d8f95-204c-4243-b771-a03e30303e77-tmp-dir\") pod \"430d8f95-204c-4243-b771-a03e30303e77\" (UID: \"430d8f95-204c-4243-b771-a03e30303e77\") " Nov 03 14:25:16.230240 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:16.230219 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/430d8f95-204c-4243-b771-a03e30303e77-home\" (UniqueName: \"kubernetes.io/empty-dir/430d8f95-204c-4243-b771-a03e30303e77-home\") pod \"430d8f95-204c-4243-b771-a03e30303e77\" (UID: \"430d8f95-204c-4243-b771-a03e30303e77\") " Nov 03 14:25:16.230665 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:16.230646 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq7rm\" (UniqueName: \"kubernetes.io/projected/a62f61a9-1d81-4bcc-b161-a6ce4d667fe3-kube-api-access-gq7rm\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:25:16.230800 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:16.230667 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a62f61a9-1d81-4bcc-b161-a6ce4d667fe3-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:25:16.230800 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:16.230682 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a62f61a9-1d81-4bcc-b161-a6ce4d667fe3-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:25:16.230976 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:16.230922 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/430d8f95-204c-4243-b771-a03e30303e77-home" pod "430d8f95-204c-4243-b771-a03e30303e77" (UID: "430d8f95-204c-4243-b771-a03e30303e77"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:25:16.231826 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:16.231797 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/430d8f95-204c-4243-b771-a03e30303e77-model-cache" pod "430d8f95-204c-4243-b771-a03e30303e77" (UID: "430d8f95-204c-4243-b771-a03e30303e77"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:25:16.232412 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:16.232388 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/430d8f95-204c-4243-b771-a03e30303e77-tmp-dir" pod "430d8f95-204c-4243-b771-a03e30303e77" (UID: "430d8f95-204c-4243-b771-a03e30303e77"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:25:16.233658 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:16.233623 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/430d8f95-204c-4243-b771-a03e30303e77-tls-certs" pod "430d8f95-204c-4243-b771-a03e30303e77" (UID: "430d8f95-204c-4243-b771-a03e30303e77"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:25:16.235203 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:16.235177 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/430d8f95-204c-4243-b771-a03e30303e77-kube-api-access-zchgp" pod "430d8f95-204c-4243-b771-a03e30303e77" (UID: "430d8f95-204c-4243-b771-a03e30303e77"). InnerVolumeSpecName "kube-api-access-zchgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:25:16.236324 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:16.236295 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/430d8f95-204c-4243-b771-a03e30303e77-dshm" pod "430d8f95-204c-4243-b771-a03e30303e77" (UID: "430d8f95-204c-4243-b771-a03e30303e77"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:25:16.343207 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:16.340208 5979 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/430d8f95-204c-4243-b771-a03e30303e77-model-cache\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:25:16.343207 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:16.340250 5979 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/430d8f95-204c-4243-b771-a03e30303e77-tmp-dir\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:25:16.343207 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:16.340269 5979 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/430d8f95-204c-4243-b771-a03e30303e77-home\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:25:16.343207 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:16.340286 5979 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/430d8f95-204c-4243-b771-a03e30303e77-tls-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:25:16.344688 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:16.340298 5979 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/430d8f95-204c-4243-b771-a03e30303e77-dshm\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:25:16.344688 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:16.344654 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zchgp\" (UniqueName: \"kubernetes.io/projected/430d8f95-204c-4243-b771-a03e30303e77-kube-api-access-zchgp\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:25:16.803293 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:16.803266 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16decd81-wg8d8" Nov 03 14:25:16.803524 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:16.803300 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16decd81-wg8d8" event={"ID":"a62f61a9-1d81-4bcc-b161-a6ce4d667fe3","Type":"ContainerDied","Data":"a62dbd29790bae64fd584608ccf9227e19b297a45a0a55478101146a7e9045a8"} Nov 03 14:25:16.803524 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:16.803348 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a62dbd29790bae64fd584608ccf9227e19b297a45a0a55478101146a7e9045a8" Nov 03 14:25:16.804849 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:16.804825 5979 generic.go:356] "Generic (PLEG): container finished" podID="430d8f95-204c-4243-b771-a03e30303e77" containerID="a0f2959ef1981d75e9f92d0ecec6408978d72894f804638fce4da5f87cbc5676" exitCode=0 Nov 03 14:25:16.804980 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:16.804876 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-42c707/distinct-w0-42c707-kserve-64949d6cb8-kq4w9" Nov 03 14:25:16.804980 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:16.804907 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-42c707/distinct-w0-42c707-kserve-64949d6cb8-kq4w9" event={"ID":"430d8f95-204c-4243-b771-a03e30303e77","Type":"ContainerDied","Data":"a0f2959ef1981d75e9f92d0ecec6408978d72894f804638fce4da5f87cbc5676"} Nov 03 14:25:16.804980 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:16.804941 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-42c707/distinct-w0-42c707-kserve-64949d6cb8-kq4w9" event={"ID":"430d8f95-204c-4243-b771-a03e30303e77","Type":"ContainerDied","Data":"9575b1bb9e48389d621cf183c3c9bd0ff112c07023da9358421c151b7c881f4b"} Nov 03 14:25:16.804980 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:16.804958 5979 scope.go:122] "RemoveContainer" containerID="a0f2959ef1981d75e9f92d0ecec6408978d72894f804638fce4da5f87cbc5676" Nov 03 14:25:16.816154 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:16.816135 5979 scope.go:122] "RemoveContainer" containerID="a0f2959ef1981d75e9f92d0ecec6408978d72894f804638fce4da5f87cbc5676" Nov 03 14:25:16.816422 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:25:16.816402 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0f2959ef1981d75e9f92d0ecec6408978d72894f804638fce4da5f87cbc5676\": container with ID starting with a0f2959ef1981d75e9f92d0ecec6408978d72894f804638fce4da5f87cbc5676 not found: ID does not exist" containerID="a0f2959ef1981d75e9f92d0ecec6408978d72894f804638fce4da5f87cbc5676" Nov 03 14:25:16.816530 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:16.816433 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0f2959ef1981d75e9f92d0ecec6408978d72894f804638fce4da5f87cbc5676"} err="failed to get container status \"a0f2959ef1981d75e9f92d0ecec6408978d72894f804638fce4da5f87cbc5676\": rpc error: code = NotFound desc = could not find container \"a0f2959ef1981d75e9f92d0ecec6408978d72894f804638fce4da5f87cbc5676\": container with ID starting with a0f2959ef1981d75e9f92d0ecec6408978d72894f804638fce4da5f87cbc5676 not found: ID does not exist" Nov 03 14:25:16.963053 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:16.963013 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-42c707/distinct-w0-42c707-kserve-64949d6cb8-kq4w9"] Nov 03 14:25:16.968511 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:16.968483 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-42c707/distinct-w0-42c707-kserve-64949d6cb8-kq4w9"] Nov 03 14:25:18.116137 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:18.116103 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="430d8f95-204c-4243-b771-a03e30303e77" path="/var/lib/kubelet/pods/430d8f95-204c-4243-b771-a03e30303e77/volumes" Nov 03 14:25:19.510658 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:19.510606 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-42c707-69d5b98f4f-jwf86" podUID="3bc04196-6680-45dd-bb04-7a8a0f5aac29" containerName="payload-processing" containerID="cri-o://4970ac7ad4ab9d129cbff0e9733f8a60e4bda6174b30390c77b0e55a5290354c" gracePeriod=25 Nov 03 14:25:19.617603 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:19.617554 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-42c707-79c696bfc5-65rpj" podUID="a748a184-2a39-4997-bbd4-337399775ae4" containerName="payload-pre-processing" containerID="cri-o://ac1155c4426edf5418c844d00b403d3dad26089c2d744717994799338a5871c2" gracePeriod=25 Nov 03 14:25:21.313063 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:21.313027 5979 scope.go:122] "RemoveContainer" containerID="b65eb00a04749cce05c7b560ee88e7584bf04f040e5185da89fa8ce35294a721" Nov 03 14:25:21.325051 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:21.325024 5979 scope.go:122] "RemoveContainer" containerID="cd91282891faefe16536226f153f88d31daa75c15f4c140493065992678cc1b8" Nov 03 14:25:22.446598 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:22.446544 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b5daee-5fcb8897cf-kdkph" Nov 03 14:25:22.520508 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:22.520453 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0d0e2f4-b051-47a1-8118-88694a09dddc-kube-api-access-npxq6\" (UniqueName: \"kubernetes.io/projected/f0d0e2f4-b051-47a1-8118-88694a09dddc-kube-api-access-npxq6\") pod \"f0d0e2f4-b051-47a1-8118-88694a09dddc\" (UID: \"f0d0e2f4-b051-47a1-8118-88694a09dddc\") " Nov 03 14:25:22.520709 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:22.520656 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0d0e2f4-b051-47a1-8118-88694a09dddc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0d0e2f4-b051-47a1-8118-88694a09dddc-plugins-config-volume\") pod \"f0d0e2f4-b051-47a1-8118-88694a09dddc\" (UID: \"f0d0e2f4-b051-47a1-8118-88694a09dddc\") " Nov 03 14:25:22.521027 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:22.520992 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0d0e2f4-b051-47a1-8118-88694a09dddc-plugins-config-volume" pod "f0d0e2f4-b051-47a1-8118-88694a09dddc" (UID: "f0d0e2f4-b051-47a1-8118-88694a09dddc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:25:22.522583 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:22.522555 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0d0e2f4-b051-47a1-8118-88694a09dddc-kube-api-access-npxq6" pod "f0d0e2f4-b051-47a1-8118-88694a09dddc" (UID: "f0d0e2f4-b051-47a1-8118-88694a09dddc"). InnerVolumeSpecName "kube-api-access-npxq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:25:22.543418 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:22.543394 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b5daee-797cb4f7f-ljgr2" Nov 03 14:25:22.621456 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:22.621424 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97fe4000-c797-4c39-8872-38308ffdf700-kube-api-access-xzjn4\" (UniqueName: \"kubernetes.io/projected/97fe4000-c797-4c39-8872-38308ffdf700-kube-api-access-xzjn4\") pod \"97fe4000-c797-4c39-8872-38308ffdf700\" (UID: \"97fe4000-c797-4c39-8872-38308ffdf700\") " Nov 03 14:25:22.621699 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:22.621556 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97fe4000-c797-4c39-8872-38308ffdf700-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97fe4000-c797-4c39-8872-38308ffdf700-plugins-config-volume\") pod \"97fe4000-c797-4c39-8872-38308ffdf700\" (UID: \"97fe4000-c797-4c39-8872-38308ffdf700\") " Nov 03 14:25:22.621944 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:22.621926 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0d0e2f4-b051-47a1-8118-88694a09dddc-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:25:22.622063 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:22.621949 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npxq6\" (UniqueName: \"kubernetes.io/projected/f0d0e2f4-b051-47a1-8118-88694a09dddc-kube-api-access-npxq6\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:25:22.622063 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:22.621962 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97fe4000-c797-4c39-8872-38308ffdf700-plugins-config-volume" pod "97fe4000-c797-4c39-8872-38308ffdf700" (UID: "97fe4000-c797-4c39-8872-38308ffdf700"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:25:22.623451 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:22.623431 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97fe4000-c797-4c39-8872-38308ffdf700-kube-api-access-xzjn4" pod "97fe4000-c797-4c39-8872-38308ffdf700" (UID: "97fe4000-c797-4c39-8872-38308ffdf700"). InnerVolumeSpecName "kube-api-access-xzjn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:25:22.722785 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:22.722708 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzjn4\" (UniqueName: \"kubernetes.io/projected/97fe4000-c797-4c39-8872-38308ffdf700-kube-api-access-xzjn4\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:25:22.722785 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:22.722738 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97fe4000-c797-4c39-8872-38308ffdf700-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:25:22.846112 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:22.846075 5979 generic.go:356] "Generic (PLEG): container finished" podID="97fe4000-c797-4c39-8872-38308ffdf700" containerID="8a802fe4ca0c79bf8310918e639ab746257b6612611e5d227be9e48278b826cd" exitCode=0 Nov 03 14:25:22.846283 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:22.846121 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b5daee-797cb4f7f-ljgr2" Nov 03 14:25:22.846283 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:22.846160 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b5daee-797cb4f7f-ljgr2" event={"ID":"97fe4000-c797-4c39-8872-38308ffdf700","Type":"ContainerDied","Data":"8a802fe4ca0c79bf8310918e639ab746257b6612611e5d227be9e48278b826cd"} Nov 03 14:25:22.846283 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:22.846218 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b5daee-797cb4f7f-ljgr2" event={"ID":"97fe4000-c797-4c39-8872-38308ffdf700","Type":"ContainerDied","Data":"5b04ee55cec1c1152c553d8e751d732523002dbdad48d8ae55565dc939ac1c17"} Nov 03 14:25:22.846283 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:22.846246 5979 scope.go:122] "RemoveContainer" containerID="8a802fe4ca0c79bf8310918e639ab746257b6612611e5d227be9e48278b826cd" Nov 03 14:25:22.855869 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:22.853232 5979 generic.go:356] "Generic (PLEG): container finished" podID="f0d0e2f4-b051-47a1-8118-88694a09dddc" containerID="c8c75f62c1211c923954989848838a280c4d2947d703bd3adac5a3354778ae19" exitCode=0 Nov 03 14:25:22.855869 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:22.853323 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b5daee-5fcb8897cf-kdkph" event={"ID":"f0d0e2f4-b051-47a1-8118-88694a09dddc","Type":"ContainerDied","Data":"c8c75f62c1211c923954989848838a280c4d2947d703bd3adac5a3354778ae19"} Nov 03 14:25:22.855869 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:22.853413 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b5daee-5fcb8897cf-kdkph" event={"ID":"f0d0e2f4-b051-47a1-8118-88694a09dddc","Type":"ContainerDied","Data":"5861eb5d5de14af26d6593b07d6af04096792f4ac64fa697a2cb426dbdecbfb5"} Nov 03 14:25:22.855869 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:22.853530 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b5daee-5fcb8897cf-kdkph" Nov 03 14:25:22.868920 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:22.868900 5979 scope.go:122] "RemoveContainer" containerID="8a802fe4ca0c79bf8310918e639ab746257b6612611e5d227be9e48278b826cd" Nov 03 14:25:22.869217 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:25:22.869198 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a802fe4ca0c79bf8310918e639ab746257b6612611e5d227be9e48278b826cd\": container with ID starting with 8a802fe4ca0c79bf8310918e639ab746257b6612611e5d227be9e48278b826cd not found: ID does not exist" containerID="8a802fe4ca0c79bf8310918e639ab746257b6612611e5d227be9e48278b826cd" Nov 03 14:25:22.869304 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:22.869227 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a802fe4ca0c79bf8310918e639ab746257b6612611e5d227be9e48278b826cd"} err="failed to get container status \"8a802fe4ca0c79bf8310918e639ab746257b6612611e5d227be9e48278b826cd\": rpc error: code = NotFound desc = could not find container \"8a802fe4ca0c79bf8310918e639ab746257b6612611e5d227be9e48278b826cd\": container with ID starting with 8a802fe4ca0c79bf8310918e639ab746257b6612611e5d227be9e48278b826cd not found: ID does not exist" Nov 03 14:25:22.869304 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:22.869249 5979 scope.go:122] "RemoveContainer" containerID="c8c75f62c1211c923954989848838a280c4d2947d703bd3adac5a3354778ae19" Nov 03 14:25:22.877891 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:22.877873 5979 scope.go:122] "RemoveContainer" containerID="c8c75f62c1211c923954989848838a280c4d2947d703bd3adac5a3354778ae19" Nov 03 14:25:22.878157 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:25:22.878137 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8c75f62c1211c923954989848838a280c4d2947d703bd3adac5a3354778ae19\": container with ID starting with c8c75f62c1211c923954989848838a280c4d2947d703bd3adac5a3354778ae19 not found: ID does not exist" containerID="c8c75f62c1211c923954989848838a280c4d2947d703bd3adac5a3354778ae19" Nov 03 14:25:22.878216 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:22.878167 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8c75f62c1211c923954989848838a280c4d2947d703bd3adac5a3354778ae19"} err="failed to get container status \"c8c75f62c1211c923954989848838a280c4d2947d703bd3adac5a3354778ae19\": rpc error: code = NotFound desc = could not find container \"c8c75f62c1211c923954989848838a280c4d2947d703bd3adac5a3354778ae19\": container with ID starting with c8c75f62c1211c923954989848838a280c4d2947d703bd3adac5a3354778ae19 not found: ID does not exist" Nov 03 14:25:23.002006 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:23.001921 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b5daee-797cb4f7f-ljgr2"] Nov 03 14:25:23.004756 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:23.004727 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b5daee-797cb4f7f-ljgr2"] Nov 03 14:25:23.107276 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:23.107235 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b5daee-5fcb8897cf-kdkph"] Nov 03 14:25:23.110393 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:23.110363 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b5daee-5fcb8897cf-kdkph"] Nov 03 14:25:24.115854 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:24.115823 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97fe4000-c797-4c39-8872-38308ffdf700" path="/var/lib/kubelet/pods/97fe4000-c797-4c39-8872-38308ffdf700/volumes" Nov 03 14:25:24.116381 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:24.116298 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0d0e2f4-b051-47a1-8118-88694a09dddc" path="/var/lib/kubelet/pods/f0d0e2f4-b051-47a1-8118-88694a09dddc/volumes" Nov 03 14:25:28.236195 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:28.236156 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:25:29.049085 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:29.049035 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-42c707-69d5b98f4f-jwf86" podUID="3bc04196-6680-45dd-bb04-7a8a0f5aac29" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:25:29.049610 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:29.049576 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-42c707-79c696bfc5-65rpj" podUID="a748a184-2a39-4997-bbd4-337399775ae4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:25:37.259482 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:25:37.259416 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:25:39.032747 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:39.032709 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-42c707-69d5b98f4f-jwf86" podUID="3bc04196-6680-45dd-bb04-7a8a0f5aac29" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:25:39.032747 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:39.032728 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-42c707-79c696bfc5-65rpj" podUID="a748a184-2a39-4997-bbd4-337399775ae4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:25:39.785200 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:39.785171 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-42c707-69d5b98f4f-jwf86" Nov 03 14:25:39.830767 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:39.830735 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bc04196-6680-45dd-bb04-7a8a0f5aac29-kube-api-access-25jl2\" (UniqueName: \"kubernetes.io/projected/3bc04196-6680-45dd-bb04-7a8a0f5aac29-kube-api-access-25jl2\") pod \"3bc04196-6680-45dd-bb04-7a8a0f5aac29\" (UID: \"3bc04196-6680-45dd-bb04-7a8a0f5aac29\") " Nov 03 14:25:39.830972 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:39.830896 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bc04196-6680-45dd-bb04-7a8a0f5aac29-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bc04196-6680-45dd-bb04-7a8a0f5aac29-plugins-config-volume\") pod \"3bc04196-6680-45dd-bb04-7a8a0f5aac29\" (UID: \"3bc04196-6680-45dd-bb04-7a8a0f5aac29\") " Nov 03 14:25:39.831627 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:39.831590 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bc04196-6680-45dd-bb04-7a8a0f5aac29-plugins-config-volume" pod "3bc04196-6680-45dd-bb04-7a8a0f5aac29" (UID: "3bc04196-6680-45dd-bb04-7a8a0f5aac29"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:25:39.832954 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:39.832919 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bc04196-6680-45dd-bb04-7a8a0f5aac29-kube-api-access-25jl2" pod "3bc04196-6680-45dd-bb04-7a8a0f5aac29" (UID: "3bc04196-6680-45dd-bb04-7a8a0f5aac29"). InnerVolumeSpecName "kube-api-access-25jl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:25:39.875002 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:39.874978 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-42c707-79c696bfc5-65rpj" Nov 03 14:25:39.931864 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:39.931834 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a748a184-2a39-4997-bbd4-337399775ae4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a748a184-2a39-4997-bbd4-337399775ae4-plugins-config-volume\") pod \"a748a184-2a39-4997-bbd4-337399775ae4\" (UID: \"a748a184-2a39-4997-bbd4-337399775ae4\") " Nov 03 14:25:39.932053 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:39.932012 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a748a184-2a39-4997-bbd4-337399775ae4-kube-api-access-8n695\" (UniqueName: \"kubernetes.io/projected/a748a184-2a39-4997-bbd4-337399775ae4-kube-api-access-8n695\") pod \"a748a184-2a39-4997-bbd4-337399775ae4\" (UID: \"a748a184-2a39-4997-bbd4-337399775ae4\") " Nov 03 14:25:39.932292 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:39.932264 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a748a184-2a39-4997-bbd4-337399775ae4-plugins-config-volume" pod "a748a184-2a39-4997-bbd4-337399775ae4" (UID: "a748a184-2a39-4997-bbd4-337399775ae4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:25:39.932562 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:39.932539 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bc04196-6680-45dd-bb04-7a8a0f5aac29-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:25:39.932562 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:39.932562 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25jl2\" (UniqueName: \"kubernetes.io/projected/3bc04196-6680-45dd-bb04-7a8a0f5aac29-kube-api-access-25jl2\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:25:39.932781 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:39.932572 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a748a184-2a39-4997-bbd4-337399775ae4-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:25:39.934119 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:39.934097 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a748a184-2a39-4997-bbd4-337399775ae4-kube-api-access-8n695" pod "a748a184-2a39-4997-bbd4-337399775ae4" (UID: "a748a184-2a39-4997-bbd4-337399775ae4"). InnerVolumeSpecName "kube-api-access-8n695". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:25:39.959402 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:39.959371 5979 generic.go:356] "Generic (PLEG): container finished" podID="3bc04196-6680-45dd-bb04-7a8a0f5aac29" containerID="4970ac7ad4ab9d129cbff0e9733f8a60e4bda6174b30390c77b0e55a5290354c" exitCode=0 Nov 03 14:25:39.959553 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:39.959412 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-42c707-69d5b98f4f-jwf86" Nov 03 14:25:39.959553 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:39.959463 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-42c707-69d5b98f4f-jwf86" event={"ID":"3bc04196-6680-45dd-bb04-7a8a0f5aac29","Type":"ContainerDied","Data":"4970ac7ad4ab9d129cbff0e9733f8a60e4bda6174b30390c77b0e55a5290354c"} Nov 03 14:25:39.959553 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:39.959531 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-42c707-69d5b98f4f-jwf86" event={"ID":"3bc04196-6680-45dd-bb04-7a8a0f5aac29","Type":"ContainerDied","Data":"0c62f5cb8b9cb65c2d349d89c0afc0b43e99590dd627618bba88fffc35506a8f"} Nov 03 14:25:39.959730 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:39.959562 5979 scope.go:122] "RemoveContainer" containerID="4970ac7ad4ab9d129cbff0e9733f8a60e4bda6174b30390c77b0e55a5290354c" Nov 03 14:25:39.961085 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:39.961062 5979 generic.go:356] "Generic (PLEG): container finished" podID="a748a184-2a39-4997-bbd4-337399775ae4" containerID="ac1155c4426edf5418c844d00b403d3dad26089c2d744717994799338a5871c2" exitCode=0 Nov 03 14:25:39.961205 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:39.961098 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-42c707-79c696bfc5-65rpj" event={"ID":"a748a184-2a39-4997-bbd4-337399775ae4","Type":"ContainerDied","Data":"ac1155c4426edf5418c844d00b403d3dad26089c2d744717994799338a5871c2"} Nov 03 14:25:39.961205 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:39.961114 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-42c707-79c696bfc5-65rpj" Nov 03 14:25:39.961205 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:39.961137 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-42c707-79c696bfc5-65rpj" event={"ID":"a748a184-2a39-4997-bbd4-337399775ae4","Type":"ContainerDied","Data":"84b3889e99e3f88a20a6ce5613792feb1c58b364246078cb56fc45087117ecb0"} Nov 03 14:25:39.970973 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:39.970954 5979 scope.go:122] "RemoveContainer" containerID="4970ac7ad4ab9d129cbff0e9733f8a60e4bda6174b30390c77b0e55a5290354c" Nov 03 14:25:39.971255 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:25:39.971236 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4970ac7ad4ab9d129cbff0e9733f8a60e4bda6174b30390c77b0e55a5290354c\": container with ID starting with 4970ac7ad4ab9d129cbff0e9733f8a60e4bda6174b30390c77b0e55a5290354c not found: ID does not exist" containerID="4970ac7ad4ab9d129cbff0e9733f8a60e4bda6174b30390c77b0e55a5290354c" Nov 03 14:25:39.971349 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:39.971261 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4970ac7ad4ab9d129cbff0e9733f8a60e4bda6174b30390c77b0e55a5290354c"} err="failed to get container status \"4970ac7ad4ab9d129cbff0e9733f8a60e4bda6174b30390c77b0e55a5290354c\": rpc error: code = NotFound desc = could not find container \"4970ac7ad4ab9d129cbff0e9733f8a60e4bda6174b30390c77b0e55a5290354c\": container with ID starting with 4970ac7ad4ab9d129cbff0e9733f8a60e4bda6174b30390c77b0e55a5290354c not found: ID does not exist" Nov 03 14:25:39.971349 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:39.971276 5979 scope.go:122] "RemoveContainer" containerID="ac1155c4426edf5418c844d00b403d3dad26089c2d744717994799338a5871c2" Nov 03 14:25:39.979967 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:39.979951 5979 scope.go:122] "RemoveContainer" containerID="ac1155c4426edf5418c844d00b403d3dad26089c2d744717994799338a5871c2" Nov 03 14:25:39.980215 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:25:39.980194 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac1155c4426edf5418c844d00b403d3dad26089c2d744717994799338a5871c2\": container with ID starting with ac1155c4426edf5418c844d00b403d3dad26089c2d744717994799338a5871c2 not found: ID does not exist" containerID="ac1155c4426edf5418c844d00b403d3dad26089c2d744717994799338a5871c2" Nov 03 14:25:39.980281 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:39.980225 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac1155c4426edf5418c844d00b403d3dad26089c2d744717994799338a5871c2"} err="failed to get container status \"ac1155c4426edf5418c844d00b403d3dad26089c2d744717994799338a5871c2\": rpc error: code = NotFound desc = could not find container \"ac1155c4426edf5418c844d00b403d3dad26089c2d744717994799338a5871c2\": container with ID starting with ac1155c4426edf5418c844d00b403d3dad26089c2d744717994799338a5871c2 not found: ID does not exist" Nov 03 14:25:40.033948 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:40.033875 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8n695\" (UniqueName: \"kubernetes.io/projected/a748a184-2a39-4997-bbd4-337399775ae4-kube-api-access-8n695\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:25:40.116166 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:40.116129 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-42c707-79c696bfc5-65rpj"] Nov 03 14:25:40.116166 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:40.116172 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-42c707-79c696bfc5-65rpj"] Nov 03 14:25:40.217436 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:40.217399 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-42c707-69d5b98f4f-jwf86"] Nov 03 14:25:40.221349 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:40.221321 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-42c707-69d5b98f4f-jwf86"] Nov 03 14:25:41.045975 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:41.045941 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-037f380d-kfk4l"] Nov 03 14:25:41.092199 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:41.092163 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-037f380d-kfk4l"] Nov 03 14:25:42.116307 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:42.116273 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15b07cd0-6863-442d-8f1c-3015333f708d" path="/var/lib/kubelet/pods/15b07cd0-6863-442d-8f1c-3015333f708d/volumes" Nov 03 14:25:42.116725 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:42.116712 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bc04196-6680-45dd-bb04-7a8a0f5aac29" path="/var/lib/kubelet/pods/3bc04196-6680-45dd-bb04-7a8a0f5aac29/volumes" Nov 03 14:25:42.117107 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:42.117091 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a748a184-2a39-4997-bbd4-337399775ae4" path="/var/lib/kubelet/pods/a748a184-2a39-4997-bbd4-337399775ae4/volumes" Nov 03 14:25:51.753403 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:51.753364 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l"] Nov 03 14:25:51.754043 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:51.753892 5979 cpu_manager.go:414] "containerMap: removing container" podUID="a748a184-2a39-4997-bbd4-337399775ae4" containerName="payload-pre-processing" Nov 03 14:25:51.754043 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:51.753904 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a748a184-2a39-4997-bbd4-337399775ae4" containerName="payload-pre-processing" Nov 03 14:25:51.754043 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:51.753916 5979 cpu_manager.go:414] "containerMap: removing container" podUID="430d8f95-204c-4243-b771-a03e30303e77" containerName="main" Nov 03 14:25:51.754043 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:51.753921 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="430d8f95-204c-4243-b771-a03e30303e77" containerName="main" Nov 03 14:25:51.754043 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:51.753939 5979 cpu_manager.go:414] "containerMap: removing container" podUID="f0d0e2f4-b051-47a1-8118-88694a09dddc" containerName="payload-processing" Nov 03 14:25:51.754043 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:51.753943 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0d0e2f4-b051-47a1-8118-88694a09dddc" containerName="payload-processing" Nov 03 14:25:51.754043 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:51.753966 5979 cpu_manager.go:414] "containerMap: removing container" podUID="a62f61a9-1d81-4bcc-b161-a6ce4d667fe3" containerName="revoke-keys" Nov 03 14:25:51.754043 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:51.753971 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a62f61a9-1d81-4bcc-b161-a6ce4d667fe3" containerName="revoke-keys" Nov 03 14:25:51.754043 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:51.753982 5979 cpu_manager.go:414] "containerMap: removing container" podUID="97fe4000-c797-4c39-8872-38308ffdf700" containerName="payload-pre-processing" Nov 03 14:25:51.754043 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:51.753987 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97fe4000-c797-4c39-8872-38308ffdf700" containerName="payload-pre-processing" Nov 03 14:25:51.754043 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:51.753995 5979 cpu_manager.go:414] "containerMap: removing container" podUID="3bc04196-6680-45dd-bb04-7a8a0f5aac29" containerName="payload-processing" Nov 03 14:25:51.754043 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:51.753999 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bc04196-6680-45dd-bb04-7a8a0f5aac29" containerName="payload-processing" Nov 03 14:25:51.754700 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:51.754078 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="a62f61a9-1d81-4bcc-b161-a6ce4d667fe3" containerName="revoke-keys" Nov 03 14:25:51.754700 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:51.754088 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="97fe4000-c797-4c39-8872-38308ffdf700" containerName="payload-pre-processing" Nov 03 14:25:51.754700 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:51.754095 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bc04196-6680-45dd-bb04-7a8a0f5aac29" containerName="payload-processing" Nov 03 14:25:51.754700 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:51.754103 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="a748a184-2a39-4997-bbd4-337399775ae4" containerName="payload-pre-processing" Nov 03 14:25:51.754700 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:51.754108 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="430d8f95-204c-4243-b771-a03e30303e77" containerName="main" Nov 03 14:25:51.754700 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:51.754116 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0d0e2f4-b051-47a1-8118-88694a09dddc" containerName="payload-processing" Nov 03 14:25:51.759337 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:51.759317 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l" Nov 03 14:25:51.790846 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:51.790810 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-workload-certs\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l\" (UID: \"b81d253f-6cd7-4d97-a2ba-c9c22135df1d\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l" Nov 03 14:25:51.790846 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:51.790844 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-istio-data\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l\" (UID: \"b81d253f-6cd7-4d97-a2ba-c9c22135df1d\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l" Nov 03 14:25:51.791039 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:51.790873 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-istio-token\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l\" (UID: \"b81d253f-6cd7-4d97-a2ba-c9c22135df1d\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l" Nov 03 14:25:51.791039 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:51.790993 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-istio-podinfo\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l\" (UID: \"b81d253f-6cd7-4d97-a2ba-c9c22135df1d\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l" Nov 03 14:25:51.791039 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:51.791029 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-istio-envoy\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l\" (UID: \"b81d253f-6cd7-4d97-a2ba-c9c22135df1d\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l" Nov 03 14:25:51.791174 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:51.791106 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvxqb\" (UniqueName: \"kubernetes.io/projected/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-kube-api-access-dvxqb\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l\" (UID: \"b81d253f-6cd7-4d97-a2ba-c9c22135df1d\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l" Nov 03 14:25:51.791174 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:51.791153 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-workload-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l\" (UID: \"b81d253f-6cd7-4d97-a2ba-c9c22135df1d\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l" Nov 03 14:25:51.791258 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:51.791173 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-credential-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l\" (UID: \"b81d253f-6cd7-4d97-a2ba-c9c22135df1d\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l" Nov 03 14:25:51.791324 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:51.791301 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-istiod-ca-cert\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l\" (UID: \"b81d253f-6cd7-4d97-a2ba-c9c22135df1d\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l" Nov 03 14:25:51.808405 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:51.808382 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-42c707-openshift-default-dockercfg-7t7mv\"" Nov 03 14:25:51.859808 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:51.859770 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l"] Nov 03 14:25:51.892251 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:51.892213 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-istiod-ca-cert\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l\" (UID: \"b81d253f-6cd7-4d97-a2ba-c9c22135df1d\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l" Nov 03 14:25:51.892422 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:51.892305 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-workload-certs\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l\" (UID: \"b81d253f-6cd7-4d97-a2ba-c9c22135df1d\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l" Nov 03 14:25:51.892422 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:51.892332 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-istio-data\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l\" (UID: \"b81d253f-6cd7-4d97-a2ba-c9c22135df1d\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l" Nov 03 14:25:51.892422 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:51.892373 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-istio-token\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l\" (UID: \"b81d253f-6cd7-4d97-a2ba-c9c22135df1d\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l" Nov 03 14:25:51.892422 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:51.892408 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-istio-podinfo\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l\" (UID: \"b81d253f-6cd7-4d97-a2ba-c9c22135df1d\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l" Nov 03 14:25:51.892654 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:51.892450 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-istio-envoy\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l\" (UID: \"b81d253f-6cd7-4d97-a2ba-c9c22135df1d\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l" Nov 03 14:25:51.892654 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:51.892533 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvxqb\" (UniqueName: \"kubernetes.io/projected/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-kube-api-access-dvxqb\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l\" (UID: \"b81d253f-6cd7-4d97-a2ba-c9c22135df1d\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l" Nov 03 14:25:51.892654 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:51.892571 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-workload-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l\" (UID: \"b81d253f-6cd7-4d97-a2ba-c9c22135df1d\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l" Nov 03 14:25:51.892654 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:51.892595 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-credential-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l\" (UID: \"b81d253f-6cd7-4d97-a2ba-c9c22135df1d\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l" Nov 03 14:25:51.892939 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:51.892768 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-istio-data\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l\" (UID: \"b81d253f-6cd7-4d97-a2ba-c9c22135df1d\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l" Nov 03 14:25:51.893033 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:51.893010 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-credential-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l\" (UID: \"b81d253f-6cd7-4d97-a2ba-c9c22135df1d\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l" Nov 03 14:25:51.893105 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:51.893041 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-istiod-ca-cert\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l\" (UID: \"b81d253f-6cd7-4d97-a2ba-c9c22135df1d\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l" Nov 03 14:25:51.893215 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:51.893200 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-workload-certs\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l\" (UID: \"b81d253f-6cd7-4d97-a2ba-c9c22135df1d\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l" Nov 03 14:25:51.893315 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:51.893290 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-workload-socket\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l\" (UID: \"b81d253f-6cd7-4d97-a2ba-c9c22135df1d\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l" Nov 03 14:25:51.895144 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:51.895114 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-istio-envoy\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l\" (UID: \"b81d253f-6cd7-4d97-a2ba-c9c22135df1d\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l" Nov 03 14:25:51.895558 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:51.895535 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-istio-podinfo\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l\" (UID: \"b81d253f-6cd7-4d97-a2ba-c9c22135df1d\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l" Nov 03 14:25:51.946325 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:51.946287 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-istio-token\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l\" (UID: \"b81d253f-6cd7-4d97-a2ba-c9c22135df1d\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l" Nov 03 14:25:51.946548 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:51.946530 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvxqb\" (UniqueName: \"kubernetes.io/projected/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-kube-api-access-dvxqb\") pod \"e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l\" (UID: \"b81d253f-6cd7-4d97-a2ba-c9c22135df1d\") " pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l" Nov 03 14:25:52.073319 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:52.073233 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l" Nov 03 14:25:52.500028 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:52.499993 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l"] Nov 03 14:25:52.508274 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:25:52.508246 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb81d253f_6cd7_4d97_a2ba_c9c22135df1d.slice/crio-a56e1b555fc951b2eccc955a3cc0768a91562f5bf652b07b420cc1f83c7bbfdd WatchSource:0}: Error finding container a56e1b555fc951b2eccc955a3cc0768a91562f5bf652b07b420cc1f83c7bbfdd: Status 404 returned error can't find the container with id a56e1b555fc951b2eccc955a3cc0768a91562f5bf652b07b420cc1f83c7bbfdd Nov 03 14:25:52.510737 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:52.510703 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:25:52.510828 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:52.510769 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:25:52.510828 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:52.510797 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:25:53.084273 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:53.084224 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l" event={"ID":"b81d253f-6cd7-4d97-a2ba-c9c22135df1d","Type":"ContainerStarted","Data":"8cb2cf8a6463bebd0077d2d736994360b94d280023557ef9478239a13e223359"} Nov 03 14:25:53.084780 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:53.084287 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l" event={"ID":"b81d253f-6cd7-4d97-a2ba-c9c22135df1d","Type":"ContainerStarted","Data":"a56e1b555fc951b2eccc955a3cc0768a91562f5bf652b07b420cc1f83c7bbfdd"} Nov 03 14:25:53.197441 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:53.197373 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l" podStartSLOduration=2.197359991 podStartE2EDuration="2.197359991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:25:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:25:53.195022933 +0000 UTC m=+4835.205711175" watchObservedRunningTime="2026-11-03 14:25:53.197359991 +0000 UTC m=+4835.208048232" Nov 03 14:25:54.074422 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:54.074384 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l" Nov 03 14:25:54.076220 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:54.076188 5979 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.105:15021/healthz/ready\": dial tcp 10.134.1.105:15021: connect: connection refused" start-of-body= Nov 03 14:25:54.076369 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:54.076247 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l" podUID="b81d253f-6cd7-4d97-a2ba-c9c22135df1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.105:15021/healthz/ready\": dial tcp 10.134.1.105:15021: connect: connection refused" Nov 03 14:25:55.074048 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:55.074009 5979 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.105:15021/healthz/ready\": dial tcp 10.134.1.105:15021: connect: connection refused" start-of-body= Nov 03 14:25:55.074453 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:55.074070 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l" podUID="b81d253f-6cd7-4d97-a2ba-c9c22135df1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.105:15021/healthz/ready\": dial tcp 10.134.1.105:15021: connect: connection refused" Nov 03 14:25:55.603295 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:55.603259 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm"] Nov 03 14:25:55.603526 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:55.603461 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm" podUID="9650f53f-0ff8-49f0-ba84-1186f4753807" containerName="istio-proxy" containerID="cri-o://02f12789787e4198f74c4d0b0129c7475038d135f0ba2fdb4fcf607e1d730a6e" gracePeriod=30 Nov 03 14:25:56.074117 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:56.074065 5979 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.105:15021/healthz/ready\": dial tcp 10.134.1.105:15021: connect: connection refused" start-of-body= Nov 03 14:25:56.074669 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:56.074135 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l" podUID="b81d253f-6cd7-4d97-a2ba-c9c22135df1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.105:15021/healthz/ready\": dial tcp 10.134.1.105:15021: connect: connection refused" Nov 03 14:25:57.074189 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:57.074149 5979 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.105:15021/healthz/ready\": dial tcp 10.134.1.105:15021: connect: connection refused" start-of-body= Nov 03 14:25:57.074633 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:57.074208 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l" podUID="b81d253f-6cd7-4d97-a2ba-c9c22135df1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.105:15021/healthz/ready\": dial tcp 10.134.1.105:15021: connect: connection refused" Nov 03 14:25:58.074270 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:58.074236 5979 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.105:15021/healthz/ready\": dial tcp 10.134.1.105:15021: connect: connection refused" start-of-body= Nov 03 14:25:58.074830 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:58.074290 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l" podUID="b81d253f-6cd7-4d97-a2ba-c9c22135df1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.105:15021/healthz/ready\": dial tcp 10.134.1.105:15021: connect: connection refused" Nov 03 14:25:58.880558 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:58.880512 5979 patch_prober.go:29] interesting pod/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 14:25:58.880767 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:58.880581 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm" podUID="9650f53f-0ff8-49f0-ba84-1186f4753807" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 14:25:59.211521 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:59.211464 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l" Nov 03 14:25:59.211920 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:59.211760 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l" Nov 03 14:25:59.212521 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:25:59.212503 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l" Nov 03 14:26:00.861769 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:00.861741 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm" Nov 03 14:26:00.941427 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:00.941387 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9650f53f-0ff8-49f0-ba84-1186f4753807-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9650f53f-0ff8-49f0-ba84-1186f4753807-credential-socket\") pod \"9650f53f-0ff8-49f0-ba84-1186f4753807\" (UID: \"9650f53f-0ff8-49f0-ba84-1186f4753807\") " Nov 03 14:26:00.941427 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:00.941435 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9650f53f-0ff8-49f0-ba84-1186f4753807-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9650f53f-0ff8-49f0-ba84-1186f4753807-istiod-ca-cert\") pod \"9650f53f-0ff8-49f0-ba84-1186f4753807\" (UID: \"9650f53f-0ff8-49f0-ba84-1186f4753807\") " Nov 03 14:26:00.941753 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:00.941512 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9650f53f-0ff8-49f0-ba84-1186f4753807-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9650f53f-0ff8-49f0-ba84-1186f4753807-workload-certs\") pod \"9650f53f-0ff8-49f0-ba84-1186f4753807\" (UID: \"9650f53f-0ff8-49f0-ba84-1186f4753807\") " Nov 03 14:26:00.941753 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:00.941569 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9650f53f-0ff8-49f0-ba84-1186f4753807-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9650f53f-0ff8-49f0-ba84-1186f4753807-istio-data\") pod \"9650f53f-0ff8-49f0-ba84-1186f4753807\" (UID: \"9650f53f-0ff8-49f0-ba84-1186f4753807\") " Nov 03 14:26:00.941753 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:00.941609 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9650f53f-0ff8-49f0-ba84-1186f4753807-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9650f53f-0ff8-49f0-ba84-1186f4753807-workload-socket\") pod \"9650f53f-0ff8-49f0-ba84-1186f4753807\" (UID: \"9650f53f-0ff8-49f0-ba84-1186f4753807\") " Nov 03 14:26:00.941753 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:00.941648 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9650f53f-0ff8-49f0-ba84-1186f4753807-kube-api-access-z4q8p\" (UniqueName: \"kubernetes.io/projected/9650f53f-0ff8-49f0-ba84-1186f4753807-kube-api-access-z4q8p\") pod \"9650f53f-0ff8-49f0-ba84-1186f4753807\" (UID: \"9650f53f-0ff8-49f0-ba84-1186f4753807\") " Nov 03 14:26:00.941753 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:00.941683 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9650f53f-0ff8-49f0-ba84-1186f4753807-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9650f53f-0ff8-49f0-ba84-1186f4753807-istio-podinfo\") pod \"9650f53f-0ff8-49f0-ba84-1186f4753807\" (UID: \"9650f53f-0ff8-49f0-ba84-1186f4753807\") " Nov 03 14:26:00.942052 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:00.941775 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9650f53f-0ff8-49f0-ba84-1186f4753807-istio-token\" (UniqueName: \"kubernetes.io/projected/9650f53f-0ff8-49f0-ba84-1186f4753807-istio-token\") pod \"9650f53f-0ff8-49f0-ba84-1186f4753807\" (UID: \"9650f53f-0ff8-49f0-ba84-1186f4753807\") " Nov 03 14:26:00.942052 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:00.941758 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9650f53f-0ff8-49f0-ba84-1186f4753807-credential-socket" pod "9650f53f-0ff8-49f0-ba84-1186f4753807" (UID: "9650f53f-0ff8-49f0-ba84-1186f4753807"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:26:00.942052 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:00.941829 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9650f53f-0ff8-49f0-ba84-1186f4753807-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9650f53f-0ff8-49f0-ba84-1186f4753807-istio-envoy\") pod \"9650f53f-0ff8-49f0-ba84-1186f4753807\" (UID: \"9650f53f-0ff8-49f0-ba84-1186f4753807\") " Nov 03 14:26:00.942052 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:00.942000 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9650f53f-0ff8-49f0-ba84-1186f4753807-workload-certs" pod "9650f53f-0ff8-49f0-ba84-1186f4753807" (UID: "9650f53f-0ff8-49f0-ba84-1186f4753807"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:26:00.942052 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:00.942021 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9650f53f-0ff8-49f0-ba84-1186f4753807-istiod-ca-cert" pod "9650f53f-0ff8-49f0-ba84-1186f4753807" (UID: "9650f53f-0ff8-49f0-ba84-1186f4753807"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:26:00.942052 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:00.942037 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9650f53f-0ff8-49f0-ba84-1186f4753807-istio-data" pod "9650f53f-0ff8-49f0-ba84-1186f4753807" (UID: "9650f53f-0ff8-49f0-ba84-1186f4753807"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:26:00.942381 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:00.942205 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9650f53f-0ff8-49f0-ba84-1186f4753807-workload-socket" pod "9650f53f-0ff8-49f0-ba84-1186f4753807" (UID: "9650f53f-0ff8-49f0-ba84-1186f4753807"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:26:00.942440 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:00.942396 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9650f53f-0ff8-49f0-ba84-1186f4753807-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:00.942440 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:00.942414 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9650f53f-0ff8-49f0-ba84-1186f4753807-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:00.942440 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:00.942423 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9650f53f-0ff8-49f0-ba84-1186f4753807-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:00.942440 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:00.942433 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9650f53f-0ff8-49f0-ba84-1186f4753807-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:00.942440 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:00.942441 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9650f53f-0ff8-49f0-ba84-1186f4753807-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:00.944137 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:00.944107 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9650f53f-0ff8-49f0-ba84-1186f4753807-istio-podinfo" pod "9650f53f-0ff8-49f0-ba84-1186f4753807" (UID: "9650f53f-0ff8-49f0-ba84-1186f4753807"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:26:00.944253 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:00.944115 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9650f53f-0ff8-49f0-ba84-1186f4753807-istio-envoy" pod "9650f53f-0ff8-49f0-ba84-1186f4753807" (UID: "9650f53f-0ff8-49f0-ba84-1186f4753807"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:26:00.944388 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:00.944368 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9650f53f-0ff8-49f0-ba84-1186f4753807-istio-token" pod "9650f53f-0ff8-49f0-ba84-1186f4753807" (UID: "9650f53f-0ff8-49f0-ba84-1186f4753807"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:26:00.944529 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:00.944509 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9650f53f-0ff8-49f0-ba84-1186f4753807-kube-api-access-z4q8p" pod "9650f53f-0ff8-49f0-ba84-1186f4753807" (UID: "9650f53f-0ff8-49f0-ba84-1186f4753807"). InnerVolumeSpecName "kube-api-access-z4q8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:26:01.043420 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:01.043378 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4q8p\" (UniqueName: \"kubernetes.io/projected/9650f53f-0ff8-49f0-ba84-1186f4753807-kube-api-access-z4q8p\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:01.043420 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:01.043418 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9650f53f-0ff8-49f0-ba84-1186f4753807-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:01.043420 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:01.043428 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9650f53f-0ff8-49f0-ba84-1186f4753807-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:01.043769 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:01.043437 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9650f53f-0ff8-49f0-ba84-1186f4753807-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:01.138015 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:01.137971 5979 generic.go:356] "Generic (PLEG): container finished" podID="9650f53f-0ff8-49f0-ba84-1186f4753807" containerID="02f12789787e4198f74c4d0b0129c7475038d135f0ba2fdb4fcf607e1d730a6e" exitCode=0 Nov 03 14:26:01.138199 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:01.138091 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm" Nov 03 14:26:01.138199 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:01.138109 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm" event={"ID":"9650f53f-0ff8-49f0-ba84-1186f4753807","Type":"ContainerDied","Data":"02f12789787e4198f74c4d0b0129c7475038d135f0ba2fdb4fcf607e1d730a6e"} Nov 03 14:26:01.138199 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:01.138172 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm" event={"ID":"9650f53f-0ff8-49f0-ba84-1186f4753807","Type":"ContainerDied","Data":"214e3c8de1b69bb5931f0203341b17e64359684779b33b475f1e5433dc34e777"} Nov 03 14:26:01.138315 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:01.138209 5979 scope.go:122] "RemoveContainer" containerID="02f12789787e4198f74c4d0b0129c7475038d135f0ba2fdb4fcf607e1d730a6e" Nov 03 14:26:01.149262 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:01.149239 5979 scope.go:122] "RemoveContainer" containerID="51380ace269523c2098117a40084ef8dced7c5e9b09ab75ffbd43e81e869b668" Nov 03 14:26:01.159097 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:01.159074 5979 scope.go:122] "RemoveContainer" containerID="02f12789787e4198f74c4d0b0129c7475038d135f0ba2fdb4fcf607e1d730a6e" Nov 03 14:26:01.159379 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:26:01.159356 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02f12789787e4198f74c4d0b0129c7475038d135f0ba2fdb4fcf607e1d730a6e\": container with ID starting with 02f12789787e4198f74c4d0b0129c7475038d135f0ba2fdb4fcf607e1d730a6e not found: ID does not exist" containerID="02f12789787e4198f74c4d0b0129c7475038d135f0ba2fdb4fcf607e1d730a6e" Nov 03 14:26:01.159426 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:01.159393 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02f12789787e4198f74c4d0b0129c7475038d135f0ba2fdb4fcf607e1d730a6e"} err="failed to get container status \"02f12789787e4198f74c4d0b0129c7475038d135f0ba2fdb4fcf607e1d730a6e\": rpc error: code = NotFound desc = could not find container \"02f12789787e4198f74c4d0b0129c7475038d135f0ba2fdb4fcf607e1d730a6e\": container with ID starting with 02f12789787e4198f74c4d0b0129c7475038d135f0ba2fdb4fcf607e1d730a6e not found: ID does not exist" Nov 03 14:26:01.159426 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:01.159419 5979 scope.go:122] "RemoveContainer" containerID="51380ace269523c2098117a40084ef8dced7c5e9b09ab75ffbd43e81e869b668" Nov 03 14:26:01.159725 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:26:01.159707 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51380ace269523c2098117a40084ef8dced7c5e9b09ab75ffbd43e81e869b668\": container with ID starting with 51380ace269523c2098117a40084ef8dced7c5e9b09ab75ffbd43e81e869b668 not found: ID does not exist" containerID="51380ace269523c2098117a40084ef8dced7c5e9b09ab75ffbd43e81e869b668" Nov 03 14:26:01.159778 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:01.159734 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51380ace269523c2098117a40084ef8dced7c5e9b09ab75ffbd43e81e869b668"} err="failed to get container status \"51380ace269523c2098117a40084ef8dced7c5e9b09ab75ffbd43e81e869b668\": rpc error: code = NotFound desc = could not find container \"51380ace269523c2098117a40084ef8dced7c5e9b09ab75ffbd43e81e869b668\": container with ID starting with 51380ace269523c2098117a40084ef8dced7c5e9b09ab75ffbd43e81e869b668 not found: ID does not exist" Nov 03 14:26:01.295124 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:01.295030 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm"] Nov 03 14:26:01.299354 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:01.299322 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-b5daee-openshift-default-5c758f6489-lkvxm"] Nov 03 14:26:02.115920 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:02.115887 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9650f53f-0ff8-49f0-ba84-1186f4753807" path="/var/lib/kubelet/pods/9650f53f-0ff8-49f0-ba84-1186f4753807/volumes" Nov 03 14:26:07.234459 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:26:07.234416 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:26:09.870145 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:09.870109 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l"] Nov 03 14:26:09.870637 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:09.870371 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l" podUID="b81d253f-6cd7-4d97-a2ba-c9c22135df1d" containerName="istio-proxy" containerID="cri-o://8cb2cf8a6463bebd0077d2d736994360b94d280023557ef9478239a13e223359" gracePeriod=30 Nov 03 14:26:14.214219 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:14.214185 5979 patch_prober.go:29] interesting pod/e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Nov 03 14:26:14.214657 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:14.214252 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l" podUID="b81d253f-6cd7-4d97-a2ba-c9c22135df1d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 14:26:15.142743 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:15.142717 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l" Nov 03 14:26:15.211400 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:15.211360 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-workload-certs\") pod \"b81d253f-6cd7-4d97-a2ba-c9c22135df1d\" (UID: \"b81d253f-6cd7-4d97-a2ba-c9c22135df1d\") " Nov 03 14:26:15.211629 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:15.211419 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-istio-podinfo\") pod \"b81d253f-6cd7-4d97-a2ba-c9c22135df1d\" (UID: \"b81d253f-6cd7-4d97-a2ba-c9c22135df1d\") " Nov 03 14:26:15.211629 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:15.211452 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-istio-data\") pod \"b81d253f-6cd7-4d97-a2ba-c9c22135df1d\" (UID: \"b81d253f-6cd7-4d97-a2ba-c9c22135df1d\") " Nov 03 14:26:15.211629 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:15.211550 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-credential-socket\") pod \"b81d253f-6cd7-4d97-a2ba-c9c22135df1d\" (UID: \"b81d253f-6cd7-4d97-a2ba-c9c22135df1d\") " Nov 03 14:26:15.211815 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:15.211635 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-workload-socket\") pod \"b81d253f-6cd7-4d97-a2ba-c9c22135df1d\" (UID: \"b81d253f-6cd7-4d97-a2ba-c9c22135df1d\") " Nov 03 14:26:15.211815 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:15.211675 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-istio-token\" (UniqueName: \"kubernetes.io/projected/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-istio-token\") pod \"b81d253f-6cd7-4d97-a2ba-c9c22135df1d\" (UID: \"b81d253f-6cd7-4d97-a2ba-c9c22135df1d\") " Nov 03 14:26:15.211815 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:15.211704 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-workload-certs" pod "b81d253f-6cd7-4d97-a2ba-c9c22135df1d" (UID: "b81d253f-6cd7-4d97-a2ba-c9c22135df1d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:26:15.211815 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:15.211722 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-kube-api-access-dvxqb\" (UniqueName: \"kubernetes.io/projected/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-kube-api-access-dvxqb\") pod \"b81d253f-6cd7-4d97-a2ba-c9c22135df1d\" (UID: \"b81d253f-6cd7-4d97-a2ba-c9c22135df1d\") " Nov 03 14:26:15.211815 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:15.211783 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-istiod-ca-cert\") pod \"b81d253f-6cd7-4d97-a2ba-c9c22135df1d\" (UID: \"b81d253f-6cd7-4d97-a2ba-c9c22135df1d\") " Nov 03 14:26:15.212098 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:15.211887 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-credential-socket" pod "b81d253f-6cd7-4d97-a2ba-c9c22135df1d" (UID: "b81d253f-6cd7-4d97-a2ba-c9c22135df1d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:26:15.212098 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:15.211900 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-istio-data" pod "b81d253f-6cd7-4d97-a2ba-c9c22135df1d" (UID: "b81d253f-6cd7-4d97-a2ba-c9c22135df1d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:26:15.212098 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:15.212009 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-workload-socket" pod "b81d253f-6cd7-4d97-a2ba-c9c22135df1d" (UID: "b81d253f-6cd7-4d97-a2ba-c9c22135df1d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:26:15.212271 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:15.212135 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-istio-envoy\") pod \"b81d253f-6cd7-4d97-a2ba-c9c22135df1d\" (UID: \"b81d253f-6cd7-4d97-a2ba-c9c22135df1d\") " Nov 03 14:26:15.212271 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:15.212174 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-istiod-ca-cert" pod "b81d253f-6cd7-4d97-a2ba-c9c22135df1d" (UID: "b81d253f-6cd7-4d97-a2ba-c9c22135df1d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:26:15.212633 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:15.212600 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:15.212633 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:15.212625 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:15.212749 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:15.212638 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:15.212749 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:15.212651 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:15.212749 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:15.212664 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:15.213996 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:15.213965 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-kube-api-access-dvxqb" pod "b81d253f-6cd7-4d97-a2ba-c9c22135df1d" (UID: "b81d253f-6cd7-4d97-a2ba-c9c22135df1d"). InnerVolumeSpecName "kube-api-access-dvxqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:26:15.214116 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:15.214065 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-istio-envoy" pod "b81d253f-6cd7-4d97-a2ba-c9c22135df1d" (UID: "b81d253f-6cd7-4d97-a2ba-c9c22135df1d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:26:15.214242 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:15.214209 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-istio-token" pod "b81d253f-6cd7-4d97-a2ba-c9c22135df1d" (UID: "b81d253f-6cd7-4d97-a2ba-c9c22135df1d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:26:15.214242 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:15.214224 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-istio-podinfo" pod "b81d253f-6cd7-4d97-a2ba-c9c22135df1d" (UID: "b81d253f-6cd7-4d97-a2ba-c9c22135df1d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:26:15.226285 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:15.226261 5979 generic.go:356] "Generic (PLEG): container finished" podID="b81d253f-6cd7-4d97-a2ba-c9c22135df1d" containerID="8cb2cf8a6463bebd0077d2d736994360b94d280023557ef9478239a13e223359" exitCode=0 Nov 03 14:26:15.226412 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:15.226322 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l" Nov 03 14:26:15.226412 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:15.226348 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l" event={"ID":"b81d253f-6cd7-4d97-a2ba-c9c22135df1d","Type":"ContainerDied","Data":"8cb2cf8a6463bebd0077d2d736994360b94d280023557ef9478239a13e223359"} Nov 03 14:26:15.226412 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:15.226405 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l" event={"ID":"b81d253f-6cd7-4d97-a2ba-c9c22135df1d","Type":"ContainerDied","Data":"a56e1b555fc951b2eccc955a3cc0768a91562f5bf652b07b420cc1f83c7bbfdd"} Nov 03 14:26:15.226609 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:15.226432 5979 scope.go:122] "RemoveContainer" containerID="8cb2cf8a6463bebd0077d2d736994360b94d280023557ef9478239a13e223359" Nov 03 14:26:15.239958 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:15.239935 5979 scope.go:122] "RemoveContainer" containerID="8cb2cf8a6463bebd0077d2d736994360b94d280023557ef9478239a13e223359" Nov 03 14:26:15.240244 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:26:15.240223 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cb2cf8a6463bebd0077d2d736994360b94d280023557ef9478239a13e223359\": container with ID starting with 8cb2cf8a6463bebd0077d2d736994360b94d280023557ef9478239a13e223359 not found: ID does not exist" containerID="8cb2cf8a6463bebd0077d2d736994360b94d280023557ef9478239a13e223359" Nov 03 14:26:15.240311 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:15.240256 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cb2cf8a6463bebd0077d2d736994360b94d280023557ef9478239a13e223359"} err="failed to get container status \"8cb2cf8a6463bebd0077d2d736994360b94d280023557ef9478239a13e223359\": rpc error: code = NotFound desc = could not find container \"8cb2cf8a6463bebd0077d2d736994360b94d280023557ef9478239a13e223359\": container with ID starting with 8cb2cf8a6463bebd0077d2d736994360b94d280023557ef9478239a13e223359 not found: ID does not exist" Nov 03 14:26:15.314399 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:15.314366 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:15.314399 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:15.314395 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvxqb\" (UniqueName: \"kubernetes.io/projected/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-kube-api-access-dvxqb\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:15.314641 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:15.314408 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:15.314641 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:15.314416 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b81d253f-6cd7-4d97-a2ba-c9c22135df1d-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:15.419237 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:15.419144 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l"] Nov 03 14:26:15.425112 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:15.425080 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-42c707-openshift-default-dd85db49f-djm4l"] Nov 03 14:26:16.115681 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:16.115644 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b81d253f-6cd7-4d97-a2ba-c9c22135df1d" path="/var/lib/kubelet/pods/b81d253f-6cd7-4d97-a2ba-c9c22135df1d/volumes" Nov 03 14:26:17.619584 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:17.619550 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:26:37.234530 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:26:37.234488 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:26:42.802008 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:42.801971 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp"] Nov 03 14:26:42.802424 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:42.802173 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp" podUID="215d3ba5-9175-474d-9ad2-ff3cdb5ed411" containerName="istio-proxy" containerID="cri-o://1e0370196a1f27194e0d2dc6054f4e76bab990dbd4583ff9b2dc285c79ef4abe" gracePeriod=30 Nov 03 14:26:44.311639 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:44.311599 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 14:26:44.312071 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:44.311668 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp" podUID="215d3ba5-9175-474d-9ad2-ff3cdb5ed411" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 14:26:48.070174 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:48.070110 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp" Nov 03 14:26:48.208232 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:48.208201 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-kube-api-access-xttd7\" (UniqueName: \"kubernetes.io/projected/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-kube-api-access-xttd7\") pod \"215d3ba5-9175-474d-9ad2-ff3cdb5ed411\" (UID: \"215d3ba5-9175-474d-9ad2-ff3cdb5ed411\") " Nov 03 14:26:48.208425 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:48.208249 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-istiod-ca-cert\") pod \"215d3ba5-9175-474d-9ad2-ff3cdb5ed411\" (UID: \"215d3ba5-9175-474d-9ad2-ff3cdb5ed411\") " Nov 03 14:26:48.208425 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:48.208283 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-workload-socket\") pod \"215d3ba5-9175-474d-9ad2-ff3cdb5ed411\" (UID: \"215d3ba5-9175-474d-9ad2-ff3cdb5ed411\") " Nov 03 14:26:48.208425 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:48.208386 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-credential-socket\") pod \"215d3ba5-9175-474d-9ad2-ff3cdb5ed411\" (UID: \"215d3ba5-9175-474d-9ad2-ff3cdb5ed411\") " Nov 03 14:26:48.208629 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:48.208428 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-workload-certs\") pod \"215d3ba5-9175-474d-9ad2-ff3cdb5ed411\" (UID: \"215d3ba5-9175-474d-9ad2-ff3cdb5ed411\") " Nov 03 14:26:48.208629 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:48.208564 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-istio-data\") pod \"215d3ba5-9175-474d-9ad2-ff3cdb5ed411\" (UID: \"215d3ba5-9175-474d-9ad2-ff3cdb5ed411\") " Nov 03 14:26:48.208629 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:48.208569 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-workload-socket" pod "215d3ba5-9175-474d-9ad2-ff3cdb5ed411" (UID: "215d3ba5-9175-474d-9ad2-ff3cdb5ed411"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:26:48.208751 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:48.208672 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-istio-token\" (UniqueName: \"kubernetes.io/projected/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-istio-token\") pod \"215d3ba5-9175-474d-9ad2-ff3cdb5ed411\" (UID: \"215d3ba5-9175-474d-9ad2-ff3cdb5ed411\") " Nov 03 14:26:48.208751 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:48.208690 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-credential-socket" pod "215d3ba5-9175-474d-9ad2-ff3cdb5ed411" (UID: "215d3ba5-9175-474d-9ad2-ff3cdb5ed411"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:26:48.208751 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:48.208696 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-istiod-ca-cert" pod "215d3ba5-9175-474d-9ad2-ff3cdb5ed411" (UID: "215d3ba5-9175-474d-9ad2-ff3cdb5ed411"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:26:48.208751 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:48.208705 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-istio-podinfo\") pod \"215d3ba5-9175-474d-9ad2-ff3cdb5ed411\" (UID: \"215d3ba5-9175-474d-9ad2-ff3cdb5ed411\") " Nov 03 14:26:48.208972 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:48.208774 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-workload-certs" pod "215d3ba5-9175-474d-9ad2-ff3cdb5ed411" (UID: "215d3ba5-9175-474d-9ad2-ff3cdb5ed411"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:26:48.208972 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:48.208796 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-istio-envoy\") pod \"215d3ba5-9175-474d-9ad2-ff3cdb5ed411\" (UID: \"215d3ba5-9175-474d-9ad2-ff3cdb5ed411\") " Nov 03 14:26:48.208972 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:48.208801 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-istio-data" pod "215d3ba5-9175-474d-9ad2-ff3cdb5ed411" (UID: "215d3ba5-9175-474d-9ad2-ff3cdb5ed411"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:26:48.209294 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:48.209270 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:48.209294 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:48.209294 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:48.209556 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:48.209309 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:48.209556 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:48.209323 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:48.209556 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:48.209336 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:48.210800 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:48.210775 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-kube-api-access-xttd7" pod "215d3ba5-9175-474d-9ad2-ff3cdb5ed411" (UID: "215d3ba5-9175-474d-9ad2-ff3cdb5ed411"). InnerVolumeSpecName "kube-api-access-xttd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:26:48.210914 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:48.210810 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-istio-token" pod "215d3ba5-9175-474d-9ad2-ff3cdb5ed411" (UID: "215d3ba5-9175-474d-9ad2-ff3cdb5ed411"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:26:48.210914 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:48.210887 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-istio-podinfo" pod "215d3ba5-9175-474d-9ad2-ff3cdb5ed411" (UID: "215d3ba5-9175-474d-9ad2-ff3cdb5ed411"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:26:48.210994 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:48.210910 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-istio-envoy" pod "215d3ba5-9175-474d-9ad2-ff3cdb5ed411" (UID: "215d3ba5-9175-474d-9ad2-ff3cdb5ed411"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:26:48.310992 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:48.310956 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:48.310992 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:48.310988 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:48.310992 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:48.310998 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:48.311237 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:48.311006 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xttd7\" (UniqueName: \"kubernetes.io/projected/215d3ba5-9175-474d-9ad2-ff3cdb5ed411-kube-api-access-xttd7\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:26:48.435033 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:48.434999 5979 generic.go:356] "Generic (PLEG): container finished" podID="215d3ba5-9175-474d-9ad2-ff3cdb5ed411" containerID="1e0370196a1f27194e0d2dc6054f4e76bab990dbd4583ff9b2dc285c79ef4abe" exitCode=0 Nov 03 14:26:48.435259 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:48.435073 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp" Nov 03 14:26:48.435259 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:48.435181 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp" event={"ID":"215d3ba5-9175-474d-9ad2-ff3cdb5ed411","Type":"ContainerDied","Data":"1e0370196a1f27194e0d2dc6054f4e76bab990dbd4583ff9b2dc285c79ef4abe"} Nov 03 14:26:48.435259 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:48.435239 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp" event={"ID":"215d3ba5-9175-474d-9ad2-ff3cdb5ed411","Type":"ContainerDied","Data":"3a693f6a44a742dc1fcc7f6c0b1629196bfab1872285e1a66e9dbf8e431431f8"} Nov 03 14:26:48.435402 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:48.435265 5979 scope.go:122] "RemoveContainer" containerID="1e0370196a1f27194e0d2dc6054f4e76bab990dbd4583ff9b2dc285c79ef4abe" Nov 03 14:26:48.448440 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:48.448411 5979 scope.go:122] "RemoveContainer" containerID="1e0370196a1f27194e0d2dc6054f4e76bab990dbd4583ff9b2dc285c79ef4abe" Nov 03 14:26:48.448777 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:26:48.448750 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e0370196a1f27194e0d2dc6054f4e76bab990dbd4583ff9b2dc285c79ef4abe\": container with ID starting with 1e0370196a1f27194e0d2dc6054f4e76bab990dbd4583ff9b2dc285c79ef4abe not found: ID does not exist" containerID="1e0370196a1f27194e0d2dc6054f4e76bab990dbd4583ff9b2dc285c79ef4abe" Nov 03 14:26:48.448889 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:48.448788 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e0370196a1f27194e0d2dc6054f4e76bab990dbd4583ff9b2dc285c79ef4abe"} err="failed to get container status \"1e0370196a1f27194e0d2dc6054f4e76bab990dbd4583ff9b2dc285c79ef4abe\": rpc error: code = NotFound desc = could not find container \"1e0370196a1f27194e0d2dc6054f4e76bab990dbd4583ff9b2dc285c79ef4abe\": container with ID starting with 1e0370196a1f27194e0d2dc6054f4e76bab990dbd4583ff9b2dc285c79ef4abe not found: ID does not exist" Nov 03 14:26:48.608442 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:48.608408 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp"] Nov 03 14:26:48.614895 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:48.614871 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-nx2cp"] Nov 03 14:26:50.115064 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:50.115026 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="215d3ba5-9175-474d-9ad2-ff3cdb5ed411" path="/var/lib/kubelet/pods/215d3ba5-9175-474d-9ad2-ff3cdb5ed411/volumes" Nov 03 14:26:57.221749 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:26:57.221718 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:27:07.233792 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:27:07.233757 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:27:13.069599 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:13.069564 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ef29fb8-lfd8x"] Nov 03 14:27:13.114453 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:13.114410 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ef29fb8-lfd8x"] Nov 03 14:27:14.115080 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:14.115034 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73ac05ea-aa66-4656-8e64-c53dfa42eed9" path="/var/lib/kubelet/pods/73ac05ea-aa66-4656-8e64-c53dfa42eed9/volumes" Nov 03 14:27:21.413917 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:21.413883 5979 scope.go:122] "RemoveContainer" containerID="7a03dc6ed3aa868c481ebba0f8d3f98455f75cbd98a20ce34233115ffe89aa86" Nov 03 14:27:24.051507 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:24.051464 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4771b897-wrdwm"] Nov 03 14:27:24.097692 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:24.097656 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4771b897-wrdwm"] Nov 03 14:27:24.115787 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:24.115753 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12b107b5-7b5c-4db7-99d3-570163320ef3" path="/var/lib/kubelet/pods/12b107b5-7b5c-4db7-99d3-570163320ef3/volumes" Nov 03 14:27:25.324060 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:25.324026 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba40e004-z6twm"] Nov 03 14:27:25.324577 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:25.324561 5979 cpu_manager.go:414] "containerMap: removing container" podUID="9650f53f-0ff8-49f0-ba84-1186f4753807" containerName="istio-proxy" Nov 03 14:27:25.324636 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:25.324579 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9650f53f-0ff8-49f0-ba84-1186f4753807" containerName="istio-proxy" Nov 03 14:27:25.324636 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:25.324591 5979 cpu_manager.go:414] "containerMap: removing container" podUID="215d3ba5-9175-474d-9ad2-ff3cdb5ed411" containerName="istio-proxy" Nov 03 14:27:25.324636 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:25.324597 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="215d3ba5-9175-474d-9ad2-ff3cdb5ed411" containerName="istio-proxy" Nov 03 14:27:25.324636 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:25.324608 5979 cpu_manager.go:414] "containerMap: removing container" podUID="b81d253f-6cd7-4d97-a2ba-c9c22135df1d" containerName="istio-proxy" Nov 03 14:27:25.324636 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:25.324613 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b81d253f-6cd7-4d97-a2ba-c9c22135df1d" containerName="istio-proxy" Nov 03 14:27:25.324811 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:25.324718 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="9650f53f-0ff8-49f0-ba84-1186f4753807" containerName="istio-proxy" Nov 03 14:27:25.324811 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:25.324729 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="215d3ba5-9175-474d-9ad2-ff3cdb5ed411" containerName="istio-proxy" Nov 03 14:27:25.324811 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:25.324736 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="b81d253f-6cd7-4d97-a2ba-c9c22135df1d" containerName="istio-proxy" Nov 03 14:27:25.329913 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:25.329892 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba40e004-z6twm" Nov 03 14:27:25.380734 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:25.380704 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:27:25.425768 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:25.425738 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 14:27:25.429642 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:25.429615 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba40e004-z6twm"] Nov 03 14:27:25.455370 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:25.455336 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clh2j\" (UniqueName: \"kubernetes.io/projected/5a792dc9-3da9-45e8-ba2b-71885ba2505c-kube-api-access-clh2j\") pod \"maas-api-revoke-sub-ba40e004-z6twm\" (UID: \"5a792dc9-3da9-45e8-ba2b-71885ba2505c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba40e004-z6twm" Nov 03 14:27:25.455561 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:25.455498 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a792dc9-3da9-45e8-ba2b-71885ba2505c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba40e004-z6twm\" (UID: \"5a792dc9-3da9-45e8-ba2b-71885ba2505c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba40e004-z6twm" Nov 03 14:27:25.455656 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:25.455638 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a792dc9-3da9-45e8-ba2b-71885ba2505c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba40e004-z6twm\" (UID: \"5a792dc9-3da9-45e8-ba2b-71885ba2505c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba40e004-z6twm" Nov 03 14:27:25.556346 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:25.556303 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a792dc9-3da9-45e8-ba2b-71885ba2505c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba40e004-z6twm\" (UID: \"5a792dc9-3da9-45e8-ba2b-71885ba2505c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba40e004-z6twm" Nov 03 14:27:25.556567 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:25.556386 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clh2j\" (UniqueName: \"kubernetes.io/projected/5a792dc9-3da9-45e8-ba2b-71885ba2505c-kube-api-access-clh2j\") pod \"maas-api-revoke-sub-ba40e004-z6twm\" (UID: \"5a792dc9-3da9-45e8-ba2b-71885ba2505c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba40e004-z6twm" Nov 03 14:27:25.556567 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:25.556521 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a792dc9-3da9-45e8-ba2b-71885ba2505c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba40e004-z6twm\" (UID: \"5a792dc9-3da9-45e8-ba2b-71885ba2505c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba40e004-z6twm" Nov 03 14:27:25.557021 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:25.556998 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a792dc9-3da9-45e8-ba2b-71885ba2505c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba40e004-z6twm\" (UID: \"5a792dc9-3da9-45e8-ba2b-71885ba2505c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba40e004-z6twm" Nov 03 14:27:25.558812 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:25.558787 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a792dc9-3da9-45e8-ba2b-71885ba2505c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba40e004-z6twm\" (UID: \"5a792dc9-3da9-45e8-ba2b-71885ba2505c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba40e004-z6twm" Nov 03 14:27:25.610037 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:25.609965 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clh2j\" (UniqueName: \"kubernetes.io/projected/5a792dc9-3da9-45e8-ba2b-71885ba2505c-kube-api-access-clh2j\") pod \"maas-api-revoke-sub-ba40e004-z6twm\" (UID: \"5a792dc9-3da9-45e8-ba2b-71885ba2505c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba40e004-z6twm" Nov 03 14:27:25.643384 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:25.643356 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba40e004-z6twm" Nov 03 14:27:26.022957 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:26.022920 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba40e004-z6twm"] Nov 03 14:27:26.030265 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:27:26.030235 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a792dc9_3da9_45e8_ba2b_71885ba2505c.slice/crio-c239f428886d4f4e0bf8b3a1cfdc50f147171119e3ebe06d5046d1d452aad483 WatchSource:0}: Error finding container c239f428886d4f4e0bf8b3a1cfdc50f147171119e3ebe06d5046d1d452aad483: Status 404 returned error can't find the container with id c239f428886d4f4e0bf8b3a1cfdc50f147171119e3ebe06d5046d1d452aad483 Nov 03 14:27:26.669805 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:26.669755 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba40e004-z6twm" event={"ID":"5a792dc9-3da9-45e8-ba2b-71885ba2505c","Type":"ContainerStarted","Data":"316d21df51b20b9f613b2943c8413e69172e09a75c0518acc3745e1049dbed03"} Nov 03 14:27:26.670281 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:26.669813 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba40e004-z6twm" event={"ID":"5a792dc9-3da9-45e8-ba2b-71885ba2505c","Type":"ContainerStarted","Data":"c239f428886d4f4e0bf8b3a1cfdc50f147171119e3ebe06d5046d1d452aad483"} Nov 03 14:27:27.677453 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:27.677417 5979 generic.go:356] "Generic (PLEG): container finished" podID="5a792dc9-3da9-45e8-ba2b-71885ba2505c" containerID="316d21df51b20b9f613b2943c8413e69172e09a75c0518acc3745e1049dbed03" exitCode=0 Nov 03 14:27:27.677943 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:27.677513 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba40e004-z6twm" event={"ID":"5a792dc9-3da9-45e8-ba2b-71885ba2505c","Type":"ContainerDied","Data":"316d21df51b20b9f613b2943c8413e69172e09a75c0518acc3745e1049dbed03"} Nov 03 14:27:28.822947 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:28.822914 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba40e004-z6twm" Nov 03 14:27:28.895360 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:28.895320 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a792dc9-3da9-45e8-ba2b-71885ba2505c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a792dc9-3da9-45e8-ba2b-71885ba2505c-maas-api-xks-ca\") pod \"5a792dc9-3da9-45e8-ba2b-71885ba2505c\" (UID: \"5a792dc9-3da9-45e8-ba2b-71885ba2505c\") " Nov 03 14:27:28.895575 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:28.895500 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a792dc9-3da9-45e8-ba2b-71885ba2505c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a792dc9-3da9-45e8-ba2b-71885ba2505c-maas-api-openshift-ca\") pod \"5a792dc9-3da9-45e8-ba2b-71885ba2505c\" (UID: \"5a792dc9-3da9-45e8-ba2b-71885ba2505c\") " Nov 03 14:27:28.895666 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:28.895649 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a792dc9-3da9-45e8-ba2b-71885ba2505c-kube-api-access-clh2j\" (UniqueName: \"kubernetes.io/projected/5a792dc9-3da9-45e8-ba2b-71885ba2505c-kube-api-access-clh2j\") pod \"5a792dc9-3da9-45e8-ba2b-71885ba2505c\" (UID: \"5a792dc9-3da9-45e8-ba2b-71885ba2505c\") " Nov 03 14:27:28.895874 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:28.895847 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a792dc9-3da9-45e8-ba2b-71885ba2505c-maas-api-openshift-ca" pod "5a792dc9-3da9-45e8-ba2b-71885ba2505c" (UID: "5a792dc9-3da9-45e8-ba2b-71885ba2505c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:27:28.896087 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:28.896067 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a792dc9-3da9-45e8-ba2b-71885ba2505c-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:27:28.897622 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:28.897597 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a792dc9-3da9-45e8-ba2b-71885ba2505c-maas-api-xks-ca" pod "5a792dc9-3da9-45e8-ba2b-71885ba2505c" (UID: "5a792dc9-3da9-45e8-ba2b-71885ba2505c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:27:28.897714 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:28.897638 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a792dc9-3da9-45e8-ba2b-71885ba2505c-kube-api-access-clh2j" pod "5a792dc9-3da9-45e8-ba2b-71885ba2505c" (UID: "5a792dc9-3da9-45e8-ba2b-71885ba2505c"). InnerVolumeSpecName "kube-api-access-clh2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:27:28.997165 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:28.997079 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clh2j\" (UniqueName: \"kubernetes.io/projected/5a792dc9-3da9-45e8-ba2b-71885ba2505c-kube-api-access-clh2j\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:27:28.997165 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:28.997110 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a792dc9-3da9-45e8-ba2b-71885ba2505c-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:27:29.691998 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:29.691944 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba40e004-z6twm" Nov 03 14:27:29.691998 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:29.691974 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba40e004-z6twm" event={"ID":"5a792dc9-3da9-45e8-ba2b-71885ba2505c","Type":"ContainerDied","Data":"c239f428886d4f4e0bf8b3a1cfdc50f147171119e3ebe06d5046d1d452aad483"} Nov 03 14:27:29.692228 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:29.692037 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c239f428886d4f4e0bf8b3a1cfdc50f147171119e3ebe06d5046d1d452aad483" Nov 03 14:27:36.167459 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:36.167418 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-675bbdab-xk5xj"] Nov 03 14:27:36.167984 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:36.167969 5979 cpu_manager.go:414] "containerMap: removing container" podUID="9650f53f-0ff8-49f0-ba84-1186f4753807" containerName="istio-proxy" Nov 03 14:27:36.168035 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:36.167985 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9650f53f-0ff8-49f0-ba84-1186f4753807" containerName="istio-proxy" Nov 03 14:27:36.168035 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:36.168007 5979 cpu_manager.go:414] "containerMap: removing container" podUID="5a792dc9-3da9-45e8-ba2b-71885ba2505c" containerName="revoke-keys" Nov 03 14:27:36.168035 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:36.168012 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a792dc9-3da9-45e8-ba2b-71885ba2505c" containerName="revoke-keys" Nov 03 14:27:36.168139 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:36.168108 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a792dc9-3da9-45e8-ba2b-71885ba2505c" containerName="revoke-keys" Nov 03 14:27:36.168139 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:36.168121 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="9650f53f-0ff8-49f0-ba84-1186f4753807" containerName="istio-proxy" Nov 03 14:27:36.171701 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:36.171677 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-675bbdab-xk5xj" Nov 03 14:27:36.221590 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:36.221560 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:27:36.268082 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:36.268011 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 14:27:36.270643 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:36.270612 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-675bbdab-xk5xj"] Nov 03 14:27:36.278333 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:36.278300 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f552bb4d-99f9-4c4a-91ca-8ba563e522c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-675bbdab-xk5xj\" (UID: \"f552bb4d-99f9-4c4a-91ca-8ba563e522c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-675bbdab-xk5xj" Nov 03 14:27:36.278509 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:36.278484 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bmmg\" (UniqueName: \"kubernetes.io/projected/f552bb4d-99f9-4c4a-91ca-8ba563e522c4-kube-api-access-2bmmg\") pod \"maas-api-revoke-sub-675bbdab-xk5xj\" (UID: \"f552bb4d-99f9-4c4a-91ca-8ba563e522c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-675bbdab-xk5xj" Nov 03 14:27:36.278668 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:36.278650 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f552bb4d-99f9-4c4a-91ca-8ba563e522c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-675bbdab-xk5xj\" (UID: \"f552bb4d-99f9-4c4a-91ca-8ba563e522c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-675bbdab-xk5xj" Nov 03 14:27:36.379364 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:36.379319 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f552bb4d-99f9-4c4a-91ca-8ba563e522c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-675bbdab-xk5xj\" (UID: \"f552bb4d-99f9-4c4a-91ca-8ba563e522c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-675bbdab-xk5xj" Nov 03 14:27:36.379589 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:36.379426 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f552bb4d-99f9-4c4a-91ca-8ba563e522c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-675bbdab-xk5xj\" (UID: \"f552bb4d-99f9-4c4a-91ca-8ba563e522c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-675bbdab-xk5xj" Nov 03 14:27:36.379589 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:36.379572 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bmmg\" (UniqueName: \"kubernetes.io/projected/f552bb4d-99f9-4c4a-91ca-8ba563e522c4-kube-api-access-2bmmg\") pod \"maas-api-revoke-sub-675bbdab-xk5xj\" (UID: \"f552bb4d-99f9-4c4a-91ca-8ba563e522c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-675bbdab-xk5xj" Nov 03 14:27:36.380038 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:36.380016 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f552bb4d-99f9-4c4a-91ca-8ba563e522c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-675bbdab-xk5xj\" (UID: \"f552bb4d-99f9-4c4a-91ca-8ba563e522c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-675bbdab-xk5xj" Nov 03 14:27:36.381814 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:36.381791 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f552bb4d-99f9-4c4a-91ca-8ba563e522c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-675bbdab-xk5xj\" (UID: \"f552bb4d-99f9-4c4a-91ca-8ba563e522c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-675bbdab-xk5xj" Nov 03 14:27:36.435406 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:36.435320 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bmmg\" (UniqueName: \"kubernetes.io/projected/f552bb4d-99f9-4c4a-91ca-8ba563e522c4-kube-api-access-2bmmg\") pod \"maas-api-revoke-sub-675bbdab-xk5xj\" (UID: \"f552bb4d-99f9-4c4a-91ca-8ba563e522c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-675bbdab-xk5xj" Nov 03 14:27:36.485273 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:36.485235 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-675bbdab-xk5xj" Nov 03 14:27:36.857635 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:36.857549 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-675bbdab-xk5xj"] Nov 03 14:27:36.864386 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:27:36.864359 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf552bb4d_99f9_4c4a_91ca_8ba563e522c4.slice/crio-9ddc2e59ab37ca54ca7923c9ee95880c5acf9ec2288af160486aa2f5cc9b271d WatchSource:0}: Error finding container 9ddc2e59ab37ca54ca7923c9ee95880c5acf9ec2288af160486aa2f5cc9b271d: Status 404 returned error can't find the container with id 9ddc2e59ab37ca54ca7923c9ee95880c5acf9ec2288af160486aa2f5cc9b271d Nov 03 14:27:37.234729 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:27:37.234690 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:27:37.739422 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:37.739374 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-675bbdab-xk5xj" event={"ID":"f552bb4d-99f9-4c4a-91ca-8ba563e522c4","Type":"ContainerStarted","Data":"a64b215949e66fe31c04223ceda71908ef44b2571d34b348efc1582a35cc5715"} Nov 03 14:27:37.741512 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:37.739448 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-675bbdab-xk5xj" event={"ID":"f552bb4d-99f9-4c4a-91ca-8ba563e522c4","Type":"ContainerStarted","Data":"9ddc2e59ab37ca54ca7923c9ee95880c5acf9ec2288af160486aa2f5cc9b271d"} Nov 03 14:27:37.828864 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:27:37.828830 5979 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf552bb4d_99f9_4c4a_91ca_8ba563e522c4.slice/crio-conmon-a64b215949e66fe31c04223ceda71908ef44b2571d34b348efc1582a35cc5715.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf552bb4d_99f9_4c4a_91ca_8ba563e522c4.slice/crio-a64b215949e66fe31c04223ceda71908ef44b2571d34b348efc1582a35cc5715.scope\": RecentStats: unable to find data in memory cache]" Nov 03 14:27:37.883657 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:37.883600 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-675bbdab-xk5xj" podStartSLOduration=1.88358581 podStartE2EDuration="1.88358581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:27:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:27:37.847657817 +0000 UTC m=+4939.858346059" watchObservedRunningTime="2026-11-03 14:27:37.88358581 +0000 UTC m=+4939.894274051" Nov 03 14:27:37.917364 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:37.917324 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:27:38.747700 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:38.747664 5979 generic.go:356] "Generic (PLEG): container finished" podID="f552bb4d-99f9-4c4a-91ca-8ba563e522c4" containerID="a64b215949e66fe31c04223ceda71908ef44b2571d34b348efc1582a35cc5715" exitCode=0 Nov 03 14:27:38.748119 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:38.747750 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-675bbdab-xk5xj" event={"ID":"f552bb4d-99f9-4c4a-91ca-8ba563e522c4","Type":"ContainerDied","Data":"a64b215949e66fe31c04223ceda71908ef44b2571d34b348efc1582a35cc5715"} Nov 03 14:27:39.907720 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:39.907693 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-675bbdab-xk5xj" Nov 03 14:27:40.019814 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:40.019720 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f552bb4d-99f9-4c4a-91ca-8ba563e522c4-kube-api-access-2bmmg\" (UniqueName: \"kubernetes.io/projected/f552bb4d-99f9-4c4a-91ca-8ba563e522c4-kube-api-access-2bmmg\") pod \"f552bb4d-99f9-4c4a-91ca-8ba563e522c4\" (UID: \"f552bb4d-99f9-4c4a-91ca-8ba563e522c4\") " Nov 03 14:27:40.020076 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:40.019861 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f552bb4d-99f9-4c4a-91ca-8ba563e522c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f552bb4d-99f9-4c4a-91ca-8ba563e522c4-maas-api-openshift-ca\") pod \"f552bb4d-99f9-4c4a-91ca-8ba563e522c4\" (UID: \"f552bb4d-99f9-4c4a-91ca-8ba563e522c4\") " Nov 03 14:27:40.020076 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:40.020015 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f552bb4d-99f9-4c4a-91ca-8ba563e522c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f552bb4d-99f9-4c4a-91ca-8ba563e522c4-maas-api-xks-ca\") pod \"f552bb4d-99f9-4c4a-91ca-8ba563e522c4\" (UID: \"f552bb4d-99f9-4c4a-91ca-8ba563e522c4\") " Nov 03 14:27:40.020294 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:40.020269 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f552bb4d-99f9-4c4a-91ca-8ba563e522c4-maas-api-openshift-ca" pod "f552bb4d-99f9-4c4a-91ca-8ba563e522c4" (UID: "f552bb4d-99f9-4c4a-91ca-8ba563e522c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:27:40.022922 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:40.022892 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f552bb4d-99f9-4c4a-91ca-8ba563e522c4-kube-api-access-2bmmg" pod "f552bb4d-99f9-4c4a-91ca-8ba563e522c4" (UID: "f552bb4d-99f9-4c4a-91ca-8ba563e522c4"). InnerVolumeSpecName "kube-api-access-2bmmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:27:40.045524 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:40.024567 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f552bb4d-99f9-4c4a-91ca-8ba563e522c4-maas-api-xks-ca" pod "f552bb4d-99f9-4c4a-91ca-8ba563e522c4" (UID: "f552bb4d-99f9-4c4a-91ca-8ba563e522c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:27:40.124830 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:40.124795 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f552bb4d-99f9-4c4a-91ca-8ba563e522c4-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:27:40.124830 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:40.124833 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bmmg\" (UniqueName: \"kubernetes.io/projected/f552bb4d-99f9-4c4a-91ca-8ba563e522c4-kube-api-access-2bmmg\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:27:40.125057 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:40.124849 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f552bb4d-99f9-4c4a-91ca-8ba563e522c4-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:27:40.765108 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:40.765078 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-675bbdab-xk5xj" Nov 03 14:27:40.765294 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:40.765071 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-675bbdab-xk5xj" event={"ID":"f552bb4d-99f9-4c4a-91ca-8ba563e522c4","Type":"ContainerDied","Data":"9ddc2e59ab37ca54ca7923c9ee95880c5acf9ec2288af160486aa2f5cc9b271d"} Nov 03 14:27:40.765294 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:40.765210 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ddc2e59ab37ca54ca7923c9ee95880c5acf9ec2288af160486aa2f5cc9b271d" Nov 03 14:27:47.866320 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:47.866284 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0d285d0-hjshf"] Nov 03 14:27:47.866994 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:47.866973 5979 cpu_manager.go:414] "containerMap: removing container" podUID="f552bb4d-99f9-4c4a-91ca-8ba563e522c4" containerName="revoke-keys" Nov 03 14:27:47.867043 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:47.866996 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f552bb4d-99f9-4c4a-91ca-8ba563e522c4" containerName="revoke-keys" Nov 03 14:27:47.867114 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:47.867105 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="f552bb4d-99f9-4c4a-91ca-8ba563e522c4" containerName="revoke-keys" Nov 03 14:27:47.870810 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:47.870790 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0d285d0-hjshf" Nov 03 14:27:47.912294 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:47.912264 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2e3af4d-bd1b-450c-8306-048e2a720c93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0d285d0-hjshf\" (UID: \"f2e3af4d-bd1b-450c-8306-048e2a720c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0d285d0-hjshf" Nov 03 14:27:47.912415 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:47.912308 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2e3af4d-bd1b-450c-8306-048e2a720c93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0d285d0-hjshf\" (UID: \"f2e3af4d-bd1b-450c-8306-048e2a720c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0d285d0-hjshf" Nov 03 14:27:47.912415 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:47.912359 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t67jj\" (UniqueName: \"kubernetes.io/projected/f2e3af4d-bd1b-450c-8306-048e2a720c93-kube-api-access-t67jj\") pod \"maas-api-revoke-sub-a0d285d0-hjshf\" (UID: \"f2e3af4d-bd1b-450c-8306-048e2a720c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0d285d0-hjshf" Nov 03 14:27:47.919676 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:47.919654 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:27:47.966115 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:47.966083 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 14:27:47.968637 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:47.968615 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0d285d0-hjshf"] Nov 03 14:27:48.013135 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:48.013102 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2e3af4d-bd1b-450c-8306-048e2a720c93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0d285d0-hjshf\" (UID: \"f2e3af4d-bd1b-450c-8306-048e2a720c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0d285d0-hjshf" Nov 03 14:27:48.013321 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:48.013144 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2e3af4d-bd1b-450c-8306-048e2a720c93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0d285d0-hjshf\" (UID: \"f2e3af4d-bd1b-450c-8306-048e2a720c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0d285d0-hjshf" Nov 03 14:27:48.013395 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:48.013332 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t67jj\" (UniqueName: \"kubernetes.io/projected/f2e3af4d-bd1b-450c-8306-048e2a720c93-kube-api-access-t67jj\") pod \"maas-api-revoke-sub-a0d285d0-hjshf\" (UID: \"f2e3af4d-bd1b-450c-8306-048e2a720c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0d285d0-hjshf" Nov 03 14:27:48.013805 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:48.013785 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2e3af4d-bd1b-450c-8306-048e2a720c93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0d285d0-hjshf\" (UID: \"f2e3af4d-bd1b-450c-8306-048e2a720c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0d285d0-hjshf" Nov 03 14:27:48.015448 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:48.015427 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2e3af4d-bd1b-450c-8306-048e2a720c93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0d285d0-hjshf\" (UID: \"f2e3af4d-bd1b-450c-8306-048e2a720c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0d285d0-hjshf" Nov 03 14:27:48.068053 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:48.068025 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t67jj\" (UniqueName: \"kubernetes.io/projected/f2e3af4d-bd1b-450c-8306-048e2a720c93-kube-api-access-t67jj\") pod \"maas-api-revoke-sub-a0d285d0-hjshf\" (UID: \"f2e3af4d-bd1b-450c-8306-048e2a720c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0d285d0-hjshf" Nov 03 14:27:48.183633 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:48.183605 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0d285d0-hjshf" Nov 03 14:27:48.555605 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:48.555520 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0d285d0-hjshf"] Nov 03 14:27:48.562868 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:27:48.562830 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2e3af4d_bd1b_450c_8306_048e2a720c93.slice/crio-10ca1550826a9bd7d2cad7fed02b2625539ac2d19f5572ae11a695e5121f2032 WatchSource:0}: Error finding container 10ca1550826a9bd7d2cad7fed02b2625539ac2d19f5572ae11a695e5121f2032: Status 404 returned error can't find the container with id 10ca1550826a9bd7d2cad7fed02b2625539ac2d19f5572ae11a695e5121f2032 Nov 03 14:27:48.812074 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:48.811998 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0d285d0-hjshf" event={"ID":"f2e3af4d-bd1b-450c-8306-048e2a720c93","Type":"ContainerStarted","Data":"10ca1550826a9bd7d2cad7fed02b2625539ac2d19f5572ae11a695e5121f2032"} Nov 03 14:27:49.819710 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:49.819617 5979 generic.go:356] "Generic (PLEG): container finished" podID="f2e3af4d-bd1b-450c-8306-048e2a720c93" containerID="49c128dcd4ec425f013d2eb199536329988b0c6ffaecfb9dd52d562fedcc3d68" exitCode=0 Nov 03 14:27:49.820097 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:49.819699 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0d285d0-hjshf" event={"ID":"f2e3af4d-bd1b-450c-8306-048e2a720c93","Type":"ContainerDied","Data":"49c128dcd4ec425f013d2eb199536329988b0c6ffaecfb9dd52d562fedcc3d68"} Nov 03 14:27:50.964712 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:50.964687 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0d285d0-hjshf" Nov 03 14:27:51.066239 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:51.065442 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2e3af4d-bd1b-450c-8306-048e2a720c93-kube-api-access-t67jj\" (UniqueName: \"kubernetes.io/projected/f2e3af4d-bd1b-450c-8306-048e2a720c93-kube-api-access-t67jj\") pod \"f2e3af4d-bd1b-450c-8306-048e2a720c93\" (UID: \"f2e3af4d-bd1b-450c-8306-048e2a720c93\") " Nov 03 14:27:51.068399 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:51.068365 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2e3af4d-bd1b-450c-8306-048e2a720c93-kube-api-access-t67jj" pod "f2e3af4d-bd1b-450c-8306-048e2a720c93" (UID: "f2e3af4d-bd1b-450c-8306-048e2a720c93"). InnerVolumeSpecName "kube-api-access-t67jj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:27:51.166315 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:51.166278 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2e3af4d-bd1b-450c-8306-048e2a720c93-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2e3af4d-bd1b-450c-8306-048e2a720c93-maas-api-openshift-ca\") pod \"f2e3af4d-bd1b-450c-8306-048e2a720c93\" (UID: \"f2e3af4d-bd1b-450c-8306-048e2a720c93\") " Nov 03 14:27:51.166514 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:51.166347 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2e3af4d-bd1b-450c-8306-048e2a720c93-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2e3af4d-bd1b-450c-8306-048e2a720c93-maas-api-xks-ca\") pod \"f2e3af4d-bd1b-450c-8306-048e2a720c93\" (UID: \"f2e3af4d-bd1b-450c-8306-048e2a720c93\") " Nov 03 14:27:51.166722 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:51.166694 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2e3af4d-bd1b-450c-8306-048e2a720c93-maas-api-openshift-ca" pod "f2e3af4d-bd1b-450c-8306-048e2a720c93" (UID: "f2e3af4d-bd1b-450c-8306-048e2a720c93"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:27:51.166775 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:51.166762 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t67jj\" (UniqueName: \"kubernetes.io/projected/f2e3af4d-bd1b-450c-8306-048e2a720c93-kube-api-access-t67jj\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:27:51.168482 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:51.168447 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2e3af4d-bd1b-450c-8306-048e2a720c93-maas-api-xks-ca" pod "f2e3af4d-bd1b-450c-8306-048e2a720c93" (UID: "f2e3af4d-bd1b-450c-8306-048e2a720c93"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:27:51.268271 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:51.268234 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2e3af4d-bd1b-450c-8306-048e2a720c93-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:27:51.268271 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:51.268267 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2e3af4d-bd1b-450c-8306-048e2a720c93-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:27:51.832592 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:51.832553 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0d285d0-hjshf" Nov 03 14:27:51.832592 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:51.832583 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0d285d0-hjshf" event={"ID":"f2e3af4d-bd1b-450c-8306-048e2a720c93","Type":"ContainerDied","Data":"10ca1550826a9bd7d2cad7fed02b2625539ac2d19f5572ae11a695e5121f2032"} Nov 03 14:27:51.832828 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:51.832660 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10ca1550826a9bd7d2cad7fed02b2625539ac2d19f5572ae11a695e5121f2032" Nov 03 14:27:58.822251 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:58.822169 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dded4219-2tjcj"] Nov 03 14:27:58.822982 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:58.822964 5979 cpu_manager.go:414] "containerMap: removing container" podUID="f2e3af4d-bd1b-450c-8306-048e2a720c93" containerName="revoke-keys" Nov 03 14:27:58.823035 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:58.822984 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2e3af4d-bd1b-450c-8306-048e2a720c93" containerName="revoke-keys" Nov 03 14:27:58.823096 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:58.823087 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2e3af4d-bd1b-450c-8306-048e2a720c93" containerName="revoke-keys" Nov 03 14:27:58.826973 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:58.826954 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dded4219-2tjcj" Nov 03 14:27:58.876398 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:58.876362 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:27:58.922955 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:58.922924 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 14:27:58.925588 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:58.925563 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dded4219-2tjcj"] Nov 03 14:27:58.961021 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:58.960988 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba6aa3ab-9c2c-4bca-8748-b6b5f1abc34c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dded4219-2tjcj\" (UID: \"ba6aa3ab-9c2c-4bca-8748-b6b5f1abc34c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dded4219-2tjcj" Nov 03 14:27:58.961198 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:58.961144 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2g5q\" (UniqueName: \"kubernetes.io/projected/ba6aa3ab-9c2c-4bca-8748-b6b5f1abc34c-kube-api-access-q2g5q\") pod \"maas-api-revoke-sub-dded4219-2tjcj\" (UID: \"ba6aa3ab-9c2c-4bca-8748-b6b5f1abc34c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dded4219-2tjcj" Nov 03 14:27:58.961198 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:58.961189 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba6aa3ab-9c2c-4bca-8748-b6b5f1abc34c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dded4219-2tjcj\" (UID: \"ba6aa3ab-9c2c-4bca-8748-b6b5f1abc34c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dded4219-2tjcj" Nov 03 14:27:59.062895 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:59.062857 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba6aa3ab-9c2c-4bca-8748-b6b5f1abc34c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dded4219-2tjcj\" (UID: \"ba6aa3ab-9c2c-4bca-8748-b6b5f1abc34c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dded4219-2tjcj" Nov 03 14:27:59.063115 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:59.062954 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2g5q\" (UniqueName: \"kubernetes.io/projected/ba6aa3ab-9c2c-4bca-8748-b6b5f1abc34c-kube-api-access-q2g5q\") pod \"maas-api-revoke-sub-dded4219-2tjcj\" (UID: \"ba6aa3ab-9c2c-4bca-8748-b6b5f1abc34c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dded4219-2tjcj" Nov 03 14:27:59.063115 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:59.062984 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba6aa3ab-9c2c-4bca-8748-b6b5f1abc34c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dded4219-2tjcj\" (UID: \"ba6aa3ab-9c2c-4bca-8748-b6b5f1abc34c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dded4219-2tjcj" Nov 03 14:27:59.063623 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:59.063602 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba6aa3ab-9c2c-4bca-8748-b6b5f1abc34c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dded4219-2tjcj\" (UID: \"ba6aa3ab-9c2c-4bca-8748-b6b5f1abc34c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dded4219-2tjcj" Nov 03 14:27:59.065231 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:59.065206 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba6aa3ab-9c2c-4bca-8748-b6b5f1abc34c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dded4219-2tjcj\" (UID: \"ba6aa3ab-9c2c-4bca-8748-b6b5f1abc34c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dded4219-2tjcj" Nov 03 14:27:59.117228 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:59.117156 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2g5q\" (UniqueName: \"kubernetes.io/projected/ba6aa3ab-9c2c-4bca-8748-b6b5f1abc34c-kube-api-access-q2g5q\") pod \"maas-api-revoke-sub-dded4219-2tjcj\" (UID: \"ba6aa3ab-9c2c-4bca-8748-b6b5f1abc34c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dded4219-2tjcj" Nov 03 14:27:59.142140 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:59.142111 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dded4219-2tjcj" Nov 03 14:27:59.516744 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:59.516713 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dded4219-2tjcj"] Nov 03 14:27:59.526051 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:27:59.526023 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba6aa3ab_9c2c_4bca_8748_b6b5f1abc34c.slice/crio-26f73edaaa2f7f23158cef58b0729666f1aab104edece55774511f7ef50af608 WatchSource:0}: Error finding container 26f73edaaa2f7f23158cef58b0729666f1aab104edece55774511f7ef50af608: Status 404 returned error can't find the container with id 26f73edaaa2f7f23158cef58b0729666f1aab104edece55774511f7ef50af608 Nov 03 14:27:59.884497 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:59.884450 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dded4219-2tjcj" event={"ID":"ba6aa3ab-9c2c-4bca-8748-b6b5f1abc34c","Type":"ContainerStarted","Data":"843e6994b70b03b67e8af0f4a6687b7145e1d350231b40905ca73246196045da"} Nov 03 14:27:59.884927 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:59.884520 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dded4219-2tjcj" event={"ID":"ba6aa3ab-9c2c-4bca-8748-b6b5f1abc34c","Type":"ContainerStarted","Data":"26f73edaaa2f7f23158cef58b0729666f1aab104edece55774511f7ef50af608"} Nov 03 14:27:59.992902 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:27:59.992838 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dded4219-2tjcj" podStartSLOduration=1.992826392 podStartE2EDuration="1.992826392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:27:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:27:59.990254497 +0000 UTC m=+4962.000942738" watchObservedRunningTime="2026-11-03 14:27:59.992826392 +0000 UTC m=+4962.003514633" Nov 03 14:28:00.891992 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:00.891960 5979 generic.go:356] "Generic (PLEG): container finished" podID="ba6aa3ab-9c2c-4bca-8748-b6b5f1abc34c" containerID="843e6994b70b03b67e8af0f4a6687b7145e1d350231b40905ca73246196045da" exitCode=0 Nov 03 14:28:00.892389 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:00.892042 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dded4219-2tjcj" event={"ID":"ba6aa3ab-9c2c-4bca-8748-b6b5f1abc34c","Type":"ContainerDied","Data":"843e6994b70b03b67e8af0f4a6687b7145e1d350231b40905ca73246196045da"} Nov 03 14:28:02.079766 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:02.079743 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dded4219-2tjcj" Nov 03 14:28:02.198856 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:02.198781 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba6aa3ab-9c2c-4bca-8748-b6b5f1abc34c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba6aa3ab-9c2c-4bca-8748-b6b5f1abc34c-maas-api-openshift-ca\") pod \"ba6aa3ab-9c2c-4bca-8748-b6b5f1abc34c\" (UID: \"ba6aa3ab-9c2c-4bca-8748-b6b5f1abc34c\") " Nov 03 14:28:02.199043 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:02.198891 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba6aa3ab-9c2c-4bca-8748-b6b5f1abc34c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba6aa3ab-9c2c-4bca-8748-b6b5f1abc34c-maas-api-xks-ca\") pod \"ba6aa3ab-9c2c-4bca-8748-b6b5f1abc34c\" (UID: \"ba6aa3ab-9c2c-4bca-8748-b6b5f1abc34c\") " Nov 03 14:28:02.199043 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:02.198997 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba6aa3ab-9c2c-4bca-8748-b6b5f1abc34c-kube-api-access-q2g5q\" (UniqueName: \"kubernetes.io/projected/ba6aa3ab-9c2c-4bca-8748-b6b5f1abc34c-kube-api-access-q2g5q\") pod \"ba6aa3ab-9c2c-4bca-8748-b6b5f1abc34c\" (UID: \"ba6aa3ab-9c2c-4bca-8748-b6b5f1abc34c\") " Nov 03 14:28:02.199214 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:02.199183 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba6aa3ab-9c2c-4bca-8748-b6b5f1abc34c-maas-api-openshift-ca" pod "ba6aa3ab-9c2c-4bca-8748-b6b5f1abc34c" (UID: "ba6aa3ab-9c2c-4bca-8748-b6b5f1abc34c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:28:02.199605 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:02.199580 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba6aa3ab-9c2c-4bca-8748-b6b5f1abc34c-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:28:02.201187 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:02.201155 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba6aa3ab-9c2c-4bca-8748-b6b5f1abc34c-kube-api-access-q2g5q" pod "ba6aa3ab-9c2c-4bca-8748-b6b5f1abc34c" (UID: "ba6aa3ab-9c2c-4bca-8748-b6b5f1abc34c"). InnerVolumeSpecName "kube-api-access-q2g5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:28:02.201296 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:02.201163 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba6aa3ab-9c2c-4bca-8748-b6b5f1abc34c-maas-api-xks-ca" pod "ba6aa3ab-9c2c-4bca-8748-b6b5f1abc34c" (UID: "ba6aa3ab-9c2c-4bca-8748-b6b5f1abc34c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:28:02.301343 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:02.301298 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba6aa3ab-9c2c-4bca-8748-b6b5f1abc34c-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:28:02.301343 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:02.301340 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2g5q\" (UniqueName: \"kubernetes.io/projected/ba6aa3ab-9c2c-4bca-8748-b6b5f1abc34c-kube-api-access-q2g5q\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:28:02.905699 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:02.905672 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dded4219-2tjcj" Nov 03 14:28:02.905699 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:02.905680 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dded4219-2tjcj" event={"ID":"ba6aa3ab-9c2c-4bca-8748-b6b5f1abc34c","Type":"ContainerDied","Data":"26f73edaaa2f7f23158cef58b0729666f1aab104edece55774511f7ef50af608"} Nov 03 14:28:02.905920 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:02.905723 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26f73edaaa2f7f23158cef58b0729666f1aab104edece55774511f7ef50af608" Nov 03 14:28:05.798133 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:05.798099 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:28:07.233611 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:28:07.233574 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:28:12.347062 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:12.347026 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-60261a/sub-test-model-60261a-kserve-5cddbb6dd5-skpws"] Nov 03 14:28:21.442261 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:21.442225 5979 scope.go:122] "RemoveContainer" containerID="a01bab3feca0310a084d0c6ae0e7efde51057678e7717a05c6fb1a448e997fe6" Nov 03 14:28:21.451989 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:21.451968 5979 scope.go:122] "RemoveContainer" containerID="2c6e1257f013a9afc4ce63864437e1dc365a9ace095d28eaa206b0e5e2525243" Nov 03 14:28:27.367180 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:27.367138 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-60261a/sub-test-model-60261a-kserve-5cddbb6dd5-skpws" podUID="5d1bbe2d-c09b-4caa-86ec-a0a64465e419" containerName="main" containerID="cri-o://543360af1b7ca39f7373e4ad819fe6033ca5eb02fe8fae0fe163df2b6b8ff415" gracePeriod=45 Nov 03 14:28:27.628490 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:27.628411 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-60261a/sub-test-model-60261a-kserve-5cddbb6dd5-skpws" Nov 03 14:28:27.710004 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:27.709976 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d1bbe2d-c09b-4caa-86ec-a0a64465e419-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d1bbe2d-c09b-4caa-86ec-a0a64465e419-tmp-dir\") pod \"5d1bbe2d-c09b-4caa-86ec-a0a64465e419\" (UID: \"5d1bbe2d-c09b-4caa-86ec-a0a64465e419\") " Nov 03 14:28:27.710182 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:27.710014 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d1bbe2d-c09b-4caa-86ec-a0a64465e419-kube-api-access-mwx8s\" (UniqueName: \"kubernetes.io/projected/5d1bbe2d-c09b-4caa-86ec-a0a64465e419-kube-api-access-mwx8s\") pod \"5d1bbe2d-c09b-4caa-86ec-a0a64465e419\" (UID: \"5d1bbe2d-c09b-4caa-86ec-a0a64465e419\") " Nov 03 14:28:27.710182 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:27.710091 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d1bbe2d-c09b-4caa-86ec-a0a64465e419-home\" (UniqueName: \"kubernetes.io/empty-dir/5d1bbe2d-c09b-4caa-86ec-a0a64465e419-home\") pod \"5d1bbe2d-c09b-4caa-86ec-a0a64465e419\" (UID: \"5d1bbe2d-c09b-4caa-86ec-a0a64465e419\") " Nov 03 14:28:27.710182 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:27.710122 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d1bbe2d-c09b-4caa-86ec-a0a64465e419-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d1bbe2d-c09b-4caa-86ec-a0a64465e419-model-cache\") pod \"5d1bbe2d-c09b-4caa-86ec-a0a64465e419\" (UID: \"5d1bbe2d-c09b-4caa-86ec-a0a64465e419\") " Nov 03 14:28:27.710182 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:27.710141 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d1bbe2d-c09b-4caa-86ec-a0a64465e419-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d1bbe2d-c09b-4caa-86ec-a0a64465e419-dshm\") pod \"5d1bbe2d-c09b-4caa-86ec-a0a64465e419\" (UID: \"5d1bbe2d-c09b-4caa-86ec-a0a64465e419\") " Nov 03 14:28:27.710182 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:27.710174 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d1bbe2d-c09b-4caa-86ec-a0a64465e419-tls-certs\" (UniqueName: \"kubernetes.io/secret/5d1bbe2d-c09b-4caa-86ec-a0a64465e419-tls-certs\") pod \"5d1bbe2d-c09b-4caa-86ec-a0a64465e419\" (UID: \"5d1bbe2d-c09b-4caa-86ec-a0a64465e419\") " Nov 03 14:28:27.710452 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:27.710299 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d1bbe2d-c09b-4caa-86ec-a0a64465e419-tmp-dir" pod "5d1bbe2d-c09b-4caa-86ec-a0a64465e419" (UID: "5d1bbe2d-c09b-4caa-86ec-a0a64465e419"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:28:27.710452 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:27.710338 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d1bbe2d-c09b-4caa-86ec-a0a64465e419-home" pod "5d1bbe2d-c09b-4caa-86ec-a0a64465e419" (UID: "5d1bbe2d-c09b-4caa-86ec-a0a64465e419"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:28:27.710452 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:27.710360 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d1bbe2d-c09b-4caa-86ec-a0a64465e419-model-cache" pod "5d1bbe2d-c09b-4caa-86ec-a0a64465e419" (UID: "5d1bbe2d-c09b-4caa-86ec-a0a64465e419"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:28:27.710657 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:27.710617 5979 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d1bbe2d-c09b-4caa-86ec-a0a64465e419-home\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:28:27.710657 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:27.710636 5979 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d1bbe2d-c09b-4caa-86ec-a0a64465e419-model-cache\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:28:27.710657 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:27.710655 5979 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d1bbe2d-c09b-4caa-86ec-a0a64465e419-tmp-dir\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:28:27.712368 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:27.712345 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d1bbe2d-c09b-4caa-86ec-a0a64465e419-tls-certs" pod "5d1bbe2d-c09b-4caa-86ec-a0a64465e419" (UID: "5d1bbe2d-c09b-4caa-86ec-a0a64465e419"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:28:27.712502 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:27.712401 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d1bbe2d-c09b-4caa-86ec-a0a64465e419-kube-api-access-mwx8s" pod "5d1bbe2d-c09b-4caa-86ec-a0a64465e419" (UID: "5d1bbe2d-c09b-4caa-86ec-a0a64465e419"). InnerVolumeSpecName "kube-api-access-mwx8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:28:27.712502 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:27.712427 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d1bbe2d-c09b-4caa-86ec-a0a64465e419-dshm" pod "5d1bbe2d-c09b-4caa-86ec-a0a64465e419" (UID: "5d1bbe2d-c09b-4caa-86ec-a0a64465e419"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:28:27.812179 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:27.812142 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwx8s\" (UniqueName: \"kubernetes.io/projected/5d1bbe2d-c09b-4caa-86ec-a0a64465e419-kube-api-access-mwx8s\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:28:27.812179 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:27.812174 5979 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d1bbe2d-c09b-4caa-86ec-a0a64465e419-dshm\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:28:27.812179 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:27.812186 5979 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d1bbe2d-c09b-4caa-86ec-a0a64465e419-tls-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:28:28.062950 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:28.062915 5979 generic.go:356] "Generic (PLEG): container finished" podID="5d1bbe2d-c09b-4caa-86ec-a0a64465e419" containerID="543360af1b7ca39f7373e4ad819fe6033ca5eb02fe8fae0fe163df2b6b8ff415" exitCode=0 Nov 03 14:28:28.063137 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:28.062963 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-60261a/sub-test-model-60261a-kserve-5cddbb6dd5-skpws" Nov 03 14:28:28.063137 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:28.063002 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-60261a/sub-test-model-60261a-kserve-5cddbb6dd5-skpws" event={"ID":"5d1bbe2d-c09b-4caa-86ec-a0a64465e419","Type":"ContainerDied","Data":"543360af1b7ca39f7373e4ad819fe6033ca5eb02fe8fae0fe163df2b6b8ff415"} Nov 03 14:28:28.063137 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:28.063047 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-60261a/sub-test-model-60261a-kserve-5cddbb6dd5-skpws" event={"ID":"5d1bbe2d-c09b-4caa-86ec-a0a64465e419","Type":"ContainerDied","Data":"e2a9398a33a0562afcd15e7b89c8325e13bfeb3b0ebea2c3154228f04673f5fe"} Nov 03 14:28:28.063137 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:28.063064 5979 scope.go:122] "RemoveContainer" containerID="543360af1b7ca39f7373e4ad819fe6033ca5eb02fe8fae0fe163df2b6b8ff415" Nov 03 14:28:28.073449 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:28.073429 5979 scope.go:122] "RemoveContainer" containerID="543360af1b7ca39f7373e4ad819fe6033ca5eb02fe8fae0fe163df2b6b8ff415" Nov 03 14:28:28.073749 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:28:28.073727 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"543360af1b7ca39f7373e4ad819fe6033ca5eb02fe8fae0fe163df2b6b8ff415\": container with ID starting with 543360af1b7ca39f7373e4ad819fe6033ca5eb02fe8fae0fe163df2b6b8ff415 not found: ID does not exist" containerID="543360af1b7ca39f7373e4ad819fe6033ca5eb02fe8fae0fe163df2b6b8ff415" Nov 03 14:28:28.073792 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:28.073757 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"543360af1b7ca39f7373e4ad819fe6033ca5eb02fe8fae0fe163df2b6b8ff415"} err="failed to get container status \"543360af1b7ca39f7373e4ad819fe6033ca5eb02fe8fae0fe163df2b6b8ff415\": rpc error: code = NotFound desc = could not find container \"543360af1b7ca39f7373e4ad819fe6033ca5eb02fe8fae0fe163df2b6b8ff415\": container with ID starting with 543360af1b7ca39f7373e4ad819fe6033ca5eb02fe8fae0fe163df2b6b8ff415 not found: ID does not exist" Nov 03 14:28:28.230666 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:28.230630 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-60261a/sub-test-model-60261a-kserve-5cddbb6dd5-skpws"] Nov 03 14:28:28.277309 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:28.277272 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-60261a/sub-test-model-60261a-kserve-5cddbb6dd5-skpws"] Nov 03 14:28:30.114824 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:30.114782 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d1bbe2d-c09b-4caa-86ec-a0a64465e419" path="/var/lib/kubelet/pods/5d1bbe2d-c09b-4caa-86ec-a0a64465e419/volumes" Nov 03 14:28:36.871096 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:36.871060 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-efbdec-7c85dd48cc-8s447"] Nov 03 14:28:36.979435 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:36.979396 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-efbdec-84dbbbd855-n2fds"] Nov 03 14:28:37.234063 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:28:37.234027 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:28:39.320114 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:39.320077 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-60261a-77454f94c-ccbgm"] Nov 03 14:28:39.429827 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:39.429791 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-60261a-c86bf8c5f-pxlmt"] Nov 03 14:28:41.871593 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:41.871521 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-efbdec-7c85dd48cc-8s447" podUID="609e1b10-a933-48b3-b224-1476116d3e66" containerName="payload-processing" containerID="cri-o://adf9b3084e6d79ee34c230c3792704efbdd328feb6bf71ad7ca3b8f75d455a0f" gracePeriod=25 Nov 03 14:28:41.980698 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:41.980649 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-efbdec-84dbbbd855-n2fds" podUID="6e0e9df1-0191-446b-95d9-6e08b67f5ed2" containerName="payload-pre-processing" containerID="cri-o://febdaa01b20192744f094880b240e87a3ccd5a07cbe3562e0004f7c62a7ccd3b" gracePeriod=25 Nov 03 14:28:43.044237 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:43.044197 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-456f842c-rgb2l"] Nov 03 14:28:43.090733 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:43.090699 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-456f842c-rgb2l"] Nov 03 14:28:44.115652 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:44.115619 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bb75ba9-1a58-4a3a-8013-a85739105276" path="/var/lib/kubelet/pods/0bb75ba9-1a58-4a3a-8013-a85739105276/volumes" Nov 03 14:28:44.320695 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:44.320641 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-60261a-77454f94c-ccbgm" podUID="cdc023c5-a79b-49d0-9bed-ea1b749d4416" containerName="payload-processing" containerID="cri-o://e97c86d573fc01a8fd6e0cb8193420852302aa4fa64486f60a4eb35f39d17156" gracePeriod=25 Nov 03 14:28:44.430634 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:44.430573 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-60261a-c86bf8c5f-pxlmt" podUID="0e2ca1ff-46e0-4b67-a561-bebf0d231464" containerName="payload-pre-processing" containerID="cri-o://4a63747785a13b7b05bc1c66e6a2091613d8dc6142384b9a72dab5f003124ba9" gracePeriod=25 Nov 03 14:28:45.045105 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:45.045068 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d99263c-lsvgn"] Nov 03 14:28:45.091555 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:45.091514 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d99263c-lsvgn"] Nov 03 14:28:46.116207 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:46.116174 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d7217b4-7e79-4f5a-b05e-9eb0ad53e9da" path="/var/lib/kubelet/pods/0d7217b4-7e79-4f5a-b05e-9eb0ad53e9da/volumes" Nov 03 14:28:46.339953 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:46.339904 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-60261a-77454f94c-ccbgm" podUID="cdc023c5-a79b-49d0-9bed-ea1b749d4416" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:28:46.340128 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:46.339921 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-60261a-c86bf8c5f-pxlmt" podUID="0e2ca1ff-46e0-4b67-a561-bebf0d231464" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:28:46.776328 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:46.776287 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-efbdec-7c85dd48cc-8s447" podUID="609e1b10-a933-48b3-b224-1476116d3e66" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:28:46.776541 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:46.776345 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-efbdec-84dbbbd855-n2fds" podUID="6e0e9df1-0191-446b-95d9-6e08b67f5ed2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:28:47.963180 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:47.963138 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:28:49.180432 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:49.180394 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-60261a-76f8445748-w4cwv"] Nov 03 14:28:49.180850 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:49.180615 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-60261a-76f8445748-w4cwv" podUID="50674938-d53d-430b-a564-fa0bd1abed74" containerName="maas-api" containerID="cri-o://f337d4b4f7c390ef4d07325d4a4b22158b4fe6d1b2f359c6acf00b99d81b6b6a" gracePeriod=30 Nov 03 14:28:49.456020 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:49.455996 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-60261a-76f8445748-w4cwv" Nov 03 14:28:49.576027 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:49.575992 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50674938-d53d-430b-a564-fa0bd1abed74-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50674938-d53d-430b-a564-fa0bd1abed74-openshift-service-ca\") pod \"50674938-d53d-430b-a564-fa0bd1abed74\" (UID: \"50674938-d53d-430b-a564-fa0bd1abed74\") " Nov 03 14:28:49.576239 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:49.576083 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50674938-d53d-430b-a564-fa0bd1abed74-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50674938-d53d-430b-a564-fa0bd1abed74-maas-api-tls\") pod \"50674938-d53d-430b-a564-fa0bd1abed74\" (UID: \"50674938-d53d-430b-a564-fa0bd1abed74\") " Nov 03 14:28:49.576239 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:49.576151 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50674938-d53d-430b-a564-fa0bd1abed74-kube-api-access-7g6gg\" (UniqueName: \"kubernetes.io/projected/50674938-d53d-430b-a564-fa0bd1abed74-kube-api-access-7g6gg\") pod \"50674938-d53d-430b-a564-fa0bd1abed74\" (UID: \"50674938-d53d-430b-a564-fa0bd1abed74\") " Nov 03 14:28:49.576574 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:49.576242 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50674938-d53d-430b-a564-fa0bd1abed74-metrics-certs\" (UniqueName: \"kubernetes.io/secret/50674938-d53d-430b-a564-fa0bd1abed74-metrics-certs\") pod \"50674938-d53d-430b-a564-fa0bd1abed74\" (UID: \"50674938-d53d-430b-a564-fa0bd1abed74\") " Nov 03 14:28:49.577491 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:49.577443 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50674938-d53d-430b-a564-fa0bd1abed74-openshift-service-ca" pod "50674938-d53d-430b-a564-fa0bd1abed74" (UID: "50674938-d53d-430b-a564-fa0bd1abed74"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:28:49.578658 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:49.578624 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50674938-d53d-430b-a564-fa0bd1abed74-metrics-certs" pod "50674938-d53d-430b-a564-fa0bd1abed74" (UID: "50674938-d53d-430b-a564-fa0bd1abed74"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:28:49.579087 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:49.579054 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50674938-d53d-430b-a564-fa0bd1abed74-kube-api-access-7g6gg" pod "50674938-d53d-430b-a564-fa0bd1abed74" (UID: "50674938-d53d-430b-a564-fa0bd1abed74"). InnerVolumeSpecName "kube-api-access-7g6gg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:28:49.579216 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:49.579079 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50674938-d53d-430b-a564-fa0bd1abed74-maas-api-tls" pod "50674938-d53d-430b-a564-fa0bd1abed74" (UID: "50674938-d53d-430b-a564-fa0bd1abed74"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:28:49.677704 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:49.677667 5979 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50674938-d53d-430b-a564-fa0bd1abed74-metrics-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:28:49.677704 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:49.677697 5979 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50674938-d53d-430b-a564-fa0bd1abed74-openshift-service-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:28:49.677704 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:49.677708 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50674938-d53d-430b-a564-fa0bd1abed74-maas-api-tls\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:28:49.677952 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:49.677717 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7g6gg\" (UniqueName: \"kubernetes.io/projected/50674938-d53d-430b-a564-fa0bd1abed74-kube-api-access-7g6gg\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:28:50.204316 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:50.204281 5979 generic.go:356] "Generic (PLEG): container finished" podID="50674938-d53d-430b-a564-fa0bd1abed74" containerID="f337d4b4f7c390ef4d07325d4a4b22158b4fe6d1b2f359c6acf00b99d81b6b6a" exitCode=0 Nov 03 14:28:50.204794 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:50.204349 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-60261a-76f8445748-w4cwv" Nov 03 14:28:50.204937 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:50.204365 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-60261a-76f8445748-w4cwv" event={"ID":"50674938-d53d-430b-a564-fa0bd1abed74","Type":"ContainerDied","Data":"f337d4b4f7c390ef4d07325d4a4b22158b4fe6d1b2f359c6acf00b99d81b6b6a"} Nov 03 14:28:50.205086 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:50.204971 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-60261a-76f8445748-w4cwv" event={"ID":"50674938-d53d-430b-a564-fa0bd1abed74","Type":"ContainerDied","Data":"e3cc3696798825dd25cff30a69caf2a5f370b6c431288da754a9e803c0082fc3"} Nov 03 14:28:50.205086 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:50.205004 5979 scope.go:122] "RemoveContainer" containerID="f337d4b4f7c390ef4d07325d4a4b22158b4fe6d1b2f359c6acf00b99d81b6b6a" Nov 03 14:28:50.218341 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:50.218308 5979 scope.go:122] "RemoveContainer" containerID="f337d4b4f7c390ef4d07325d4a4b22158b4fe6d1b2f359c6acf00b99d81b6b6a" Nov 03 14:28:50.218668 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:28:50.218647 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f337d4b4f7c390ef4d07325d4a4b22158b4fe6d1b2f359c6acf00b99d81b6b6a\": container with ID starting with f337d4b4f7c390ef4d07325d4a4b22158b4fe6d1b2f359c6acf00b99d81b6b6a not found: ID does not exist" containerID="f337d4b4f7c390ef4d07325d4a4b22158b4fe6d1b2f359c6acf00b99d81b6b6a" Nov 03 14:28:50.218761 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:50.218677 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f337d4b4f7c390ef4d07325d4a4b22158b4fe6d1b2f359c6acf00b99d81b6b6a"} err="failed to get container status \"f337d4b4f7c390ef4d07325d4a4b22158b4fe6d1b2f359c6acf00b99d81b6b6a\": rpc error: code = NotFound desc = could not find container \"f337d4b4f7c390ef4d07325d4a4b22158b4fe6d1b2f359c6acf00b99d81b6b6a\": container with ID starting with f337d4b4f7c390ef4d07325d4a4b22158b4fe6d1b2f359c6acf00b99d81b6b6a not found: ID does not exist" Nov 03 14:28:50.358719 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:50.358686 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-60261a-76f8445748-w4cwv"] Nov 03 14:28:50.404938 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:50.404903 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-60261a-76f8445748-w4cwv"] Nov 03 14:28:52.115384 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:52.115346 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50674938-d53d-430b-a564-fa0bd1abed74" path="/var/lib/kubelet/pods/50674938-d53d-430b-a564-fa0bd1abed74/volumes" Nov 03 14:28:53.045093 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:53.045059 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87e0a1a4-kmz8f"] Nov 03 14:28:53.091763 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:53.091728 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87e0a1a4-kmz8f"] Nov 03 14:28:54.115706 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:54.115668 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab455ae0-3c29-4b6e-9caf-2c705e2a9d4b" path="/var/lib/kubelet/pods/ab455ae0-3c29-4b6e-9caf-2c705e2a9d4b/volumes" Nov 03 14:28:56.344879 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:56.344838 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-60261a-c86bf8c5f-pxlmt" podUID="0e2ca1ff-46e0-4b67-a561-bebf0d231464" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:28:56.345391 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:56.345304 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-60261a-77454f94c-ccbgm" podUID="cdc023c5-a79b-49d0-9bed-ea1b749d4416" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:28:56.776807 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:56.776767 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-efbdec-7c85dd48cc-8s447" podUID="609e1b10-a933-48b3-b224-1476116d3e66" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:28:56.776807 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:56.776791 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-efbdec-84dbbbd855-n2fds" podUID="6e0e9df1-0191-446b-95d9-6e08b67f5ed2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:28:58.043108 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:58.043073 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4683a8dc-jmf9m"] Nov 03 14:28:58.089543 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:58.089501 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4683a8dc-jmf9m"] Nov 03 14:28:58.115951 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:28:58.115912 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bf41e61-5fac-4e9a-9287-46bf62a5bcfa" path="/var/lib/kubelet/pods/9bf41e61-5fac-4e9a-9287-46bf62a5bcfa/volumes" Nov 03 14:29:01.457834 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:01.457796 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj"] Nov 03 14:29:01.458273 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:01.458018 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj" podUID="4723366a-b53b-40f7-a27a-d5cde461525d" containerName="istio-proxy" containerID="cri-o://78f537c5e25fcc7de464419ca6c3694c2f5921b25b7f21e362614923ccf286bc" gracePeriod=30 Nov 03 14:29:02.143882 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:02.143854 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-efbdec-7c85dd48cc-8s447" Nov 03 14:29:02.240864 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:02.240837 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-efbdec-84dbbbd855-n2fds" Nov 03 14:29:02.255376 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:02.255352 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/609e1b10-a933-48b3-b224-1476116d3e66-kube-api-access-bzsdr\" (UniqueName: \"kubernetes.io/projected/609e1b10-a933-48b3-b224-1476116d3e66-kube-api-access-bzsdr\") pod \"609e1b10-a933-48b3-b224-1476116d3e66\" (UID: \"609e1b10-a933-48b3-b224-1476116d3e66\") " Nov 03 14:29:02.255518 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:02.255503 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/609e1b10-a933-48b3-b224-1476116d3e66-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/609e1b10-a933-48b3-b224-1476116d3e66-plugins-config-volume\") pod \"609e1b10-a933-48b3-b224-1476116d3e66\" (UID: \"609e1b10-a933-48b3-b224-1476116d3e66\") " Nov 03 14:29:02.255847 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:02.255826 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/609e1b10-a933-48b3-b224-1476116d3e66-plugins-config-volume" pod "609e1b10-a933-48b3-b224-1476116d3e66" (UID: "609e1b10-a933-48b3-b224-1476116d3e66"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:29:02.257387 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:02.257366 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/609e1b10-a933-48b3-b224-1476116d3e66-kube-api-access-bzsdr" pod "609e1b10-a933-48b3-b224-1476116d3e66" (UID: "609e1b10-a933-48b3-b224-1476116d3e66"). InnerVolumeSpecName "kube-api-access-bzsdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:29:02.282675 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:02.282641 5979 generic.go:356] "Generic (PLEG): container finished" podID="6e0e9df1-0191-446b-95d9-6e08b67f5ed2" containerID="febdaa01b20192744f094880b240e87a3ccd5a07cbe3562e0004f7c62a7ccd3b" exitCode=0 Nov 03 14:29:02.282840 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:02.282691 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-efbdec-84dbbbd855-n2fds" Nov 03 14:29:02.282840 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:02.282732 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-efbdec-84dbbbd855-n2fds" event={"ID":"6e0e9df1-0191-446b-95d9-6e08b67f5ed2","Type":"ContainerDied","Data":"febdaa01b20192744f094880b240e87a3ccd5a07cbe3562e0004f7c62a7ccd3b"} Nov 03 14:29:02.282985 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:02.282860 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-efbdec-84dbbbd855-n2fds" event={"ID":"6e0e9df1-0191-446b-95d9-6e08b67f5ed2","Type":"ContainerDied","Data":"9861d275c9d6d2d14cd256e6a1064e15abe0f881b70d6d902038eb41695d87e0"} Nov 03 14:29:02.282985 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:02.282899 5979 scope.go:122] "RemoveContainer" containerID="febdaa01b20192744f094880b240e87a3ccd5a07cbe3562e0004f7c62a7ccd3b" Nov 03 14:29:02.284462 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:02.284431 5979 generic.go:356] "Generic (PLEG): container finished" podID="609e1b10-a933-48b3-b224-1476116d3e66" containerID="adf9b3084e6d79ee34c230c3792704efbdd328feb6bf71ad7ca3b8f75d455a0f" exitCode=0 Nov 03 14:29:02.284594 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:02.284503 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-efbdec-7c85dd48cc-8s447" Nov 03 14:29:02.284594 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:02.284517 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-efbdec-7c85dd48cc-8s447" event={"ID":"609e1b10-a933-48b3-b224-1476116d3e66","Type":"ContainerDied","Data":"adf9b3084e6d79ee34c230c3792704efbdd328feb6bf71ad7ca3b8f75d455a0f"} Nov 03 14:29:02.284594 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:02.284569 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-efbdec-7c85dd48cc-8s447" event={"ID":"609e1b10-a933-48b3-b224-1476116d3e66","Type":"ContainerDied","Data":"5887f8d4a903bce0280e63e2ea4e38c51c8427eb57fca79d031098f958e943ef"} Nov 03 14:29:02.293903 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:02.293868 5979 scope.go:122] "RemoveContainer" containerID="febdaa01b20192744f094880b240e87a3ccd5a07cbe3562e0004f7c62a7ccd3b" Nov 03 14:29:02.294207 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:29:02.294183 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"febdaa01b20192744f094880b240e87a3ccd5a07cbe3562e0004f7c62a7ccd3b\": container with ID starting with febdaa01b20192744f094880b240e87a3ccd5a07cbe3562e0004f7c62a7ccd3b not found: ID does not exist" containerID="febdaa01b20192744f094880b240e87a3ccd5a07cbe3562e0004f7c62a7ccd3b" Nov 03 14:29:02.294283 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:02.294218 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"febdaa01b20192744f094880b240e87a3ccd5a07cbe3562e0004f7c62a7ccd3b"} err="failed to get container status \"febdaa01b20192744f094880b240e87a3ccd5a07cbe3562e0004f7c62a7ccd3b\": rpc error: code = NotFound desc = could not find container \"febdaa01b20192744f094880b240e87a3ccd5a07cbe3562e0004f7c62a7ccd3b\": container with ID starting with febdaa01b20192744f094880b240e87a3ccd5a07cbe3562e0004f7c62a7ccd3b not found: ID does not exist" Nov 03 14:29:02.294283 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:02.294240 5979 scope.go:122] "RemoveContainer" containerID="adf9b3084e6d79ee34c230c3792704efbdd328feb6bf71ad7ca3b8f75d455a0f" Nov 03 14:29:02.303922 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:02.303900 5979 scope.go:122] "RemoveContainer" containerID="adf9b3084e6d79ee34c230c3792704efbdd328feb6bf71ad7ca3b8f75d455a0f" Nov 03 14:29:02.304219 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:29:02.304200 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adf9b3084e6d79ee34c230c3792704efbdd328feb6bf71ad7ca3b8f75d455a0f\": container with ID starting with adf9b3084e6d79ee34c230c3792704efbdd328feb6bf71ad7ca3b8f75d455a0f not found: ID does not exist" containerID="adf9b3084e6d79ee34c230c3792704efbdd328feb6bf71ad7ca3b8f75d455a0f" Nov 03 14:29:02.304290 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:02.304226 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adf9b3084e6d79ee34c230c3792704efbdd328feb6bf71ad7ca3b8f75d455a0f"} err="failed to get container status \"adf9b3084e6d79ee34c230c3792704efbdd328feb6bf71ad7ca3b8f75d455a0f\": rpc error: code = NotFound desc = could not find container \"adf9b3084e6d79ee34c230c3792704efbdd328feb6bf71ad7ca3b8f75d455a0f\": container with ID starting with adf9b3084e6d79ee34c230c3792704efbdd328feb6bf71ad7ca3b8f75d455a0f not found: ID does not exist" Nov 03 14:29:02.356384 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:02.356352 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e0e9df1-0191-446b-95d9-6e08b67f5ed2-kube-api-access-4gkjv\" (UniqueName: \"kubernetes.io/projected/6e0e9df1-0191-446b-95d9-6e08b67f5ed2-kube-api-access-4gkjv\") pod \"6e0e9df1-0191-446b-95d9-6e08b67f5ed2\" (UID: \"6e0e9df1-0191-446b-95d9-6e08b67f5ed2\") " Nov 03 14:29:02.356583 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:02.356446 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e0e9df1-0191-446b-95d9-6e08b67f5ed2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e0e9df1-0191-446b-95d9-6e08b67f5ed2-plugins-config-volume\") pod \"6e0e9df1-0191-446b-95d9-6e08b67f5ed2\" (UID: \"6e0e9df1-0191-446b-95d9-6e08b67f5ed2\") " Nov 03 14:29:02.356828 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:02.356805 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e0e9df1-0191-446b-95d9-6e08b67f5ed2-plugins-config-volume" pod "6e0e9df1-0191-446b-95d9-6e08b67f5ed2" (UID: "6e0e9df1-0191-446b-95d9-6e08b67f5ed2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:29:02.356900 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:02.356817 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzsdr\" (UniqueName: \"kubernetes.io/projected/609e1b10-a933-48b3-b224-1476116d3e66-kube-api-access-bzsdr\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:02.356900 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:02.356855 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/609e1b10-a933-48b3-b224-1476116d3e66-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:02.358390 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:02.358372 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e0e9df1-0191-446b-95d9-6e08b67f5ed2-kube-api-access-4gkjv" pod "6e0e9df1-0191-446b-95d9-6e08b67f5ed2" (UID: "6e0e9df1-0191-446b-95d9-6e08b67f5ed2"). InnerVolumeSpecName "kube-api-access-4gkjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:29:02.458148 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:02.458067 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gkjv\" (UniqueName: \"kubernetes.io/projected/6e0e9df1-0191-446b-95d9-6e08b67f5ed2-kube-api-access-4gkjv\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:02.458148 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:02.458095 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e0e9df1-0191-446b-95d9-6e08b67f5ed2-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:02.487994 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:02.487958 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-efbdec-7c85dd48cc-8s447"] Nov 03 14:29:02.492509 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:02.492485 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-efbdec-7c85dd48cc-8s447"] Nov 03 14:29:02.697120 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:02.697087 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-efbdec-84dbbbd855-n2fds"] Nov 03 14:29:02.700207 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:02.700157 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-efbdec-84dbbbd855-n2fds"] Nov 03 14:29:04.114455 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:04.114419 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="609e1b10-a933-48b3-b224-1476116d3e66" path="/var/lib/kubelet/pods/609e1b10-a933-48b3-b224-1476116d3e66/volumes" Nov 03 14:29:04.114944 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:04.114927 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e0e9df1-0191-446b-95d9-6e08b67f5ed2" path="/var/lib/kubelet/pods/6e0e9df1-0191-446b-95d9-6e08b67f5ed2/volumes" Nov 03 14:29:04.593239 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:04.593210 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-60261a-77454f94c-ccbgm" Nov 03 14:29:04.685387 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:04.685362 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdc023c5-a79b-49d0-9bed-ea1b749d4416-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdc023c5-a79b-49d0-9bed-ea1b749d4416-plugins-config-volume\") pod \"cdc023c5-a79b-49d0-9bed-ea1b749d4416\" (UID: \"cdc023c5-a79b-49d0-9bed-ea1b749d4416\") " Nov 03 14:29:04.685541 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:04.685521 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdc023c5-a79b-49d0-9bed-ea1b749d4416-kube-api-access-jxktq\" (UniqueName: \"kubernetes.io/projected/cdc023c5-a79b-49d0-9bed-ea1b749d4416-kube-api-access-jxktq\") pod \"cdc023c5-a79b-49d0-9bed-ea1b749d4416\" (UID: \"cdc023c5-a79b-49d0-9bed-ea1b749d4416\") " Nov 03 14:29:04.685755 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:04.685732 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdc023c5-a79b-49d0-9bed-ea1b749d4416-plugins-config-volume" pod "cdc023c5-a79b-49d0-9bed-ea1b749d4416" (UID: "cdc023c5-a79b-49d0-9bed-ea1b749d4416"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:29:04.686019 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:04.686003 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdc023c5-a79b-49d0-9bed-ea1b749d4416-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:04.687806 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:04.687741 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdc023c5-a79b-49d0-9bed-ea1b749d4416-kube-api-access-jxktq" pod "cdc023c5-a79b-49d0-9bed-ea1b749d4416" (UID: "cdc023c5-a79b-49d0-9bed-ea1b749d4416"). InnerVolumeSpecName "kube-api-access-jxktq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:29:04.704419 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:04.704398 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-60261a-c86bf8c5f-pxlmt" Nov 03 14:29:04.786887 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:04.786853 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e2ca1ff-46e0-4b67-a561-bebf0d231464-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e2ca1ff-46e0-4b67-a561-bebf0d231464-plugins-config-volume\") pod \"0e2ca1ff-46e0-4b67-a561-bebf0d231464\" (UID: \"0e2ca1ff-46e0-4b67-a561-bebf0d231464\") " Nov 03 14:29:04.787082 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:04.786968 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e2ca1ff-46e0-4b67-a561-bebf0d231464-kube-api-access-dv42p\" (UniqueName: \"kubernetes.io/projected/0e2ca1ff-46e0-4b67-a561-bebf0d231464-kube-api-access-dv42p\") pod \"0e2ca1ff-46e0-4b67-a561-bebf0d231464\" (UID: \"0e2ca1ff-46e0-4b67-a561-bebf0d231464\") " Nov 03 14:29:04.787288 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:04.787264 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e2ca1ff-46e0-4b67-a561-bebf0d231464-plugins-config-volume" pod "0e2ca1ff-46e0-4b67-a561-bebf0d231464" (UID: "0e2ca1ff-46e0-4b67-a561-bebf0d231464"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:29:04.787568 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:04.787542 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e2ca1ff-46e0-4b67-a561-bebf0d231464-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:04.787568 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:04.787564 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxktq\" (UniqueName: \"kubernetes.io/projected/cdc023c5-a79b-49d0-9bed-ea1b749d4416-kube-api-access-jxktq\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:04.789059 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:04.789036 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e2ca1ff-46e0-4b67-a561-bebf0d231464-kube-api-access-dv42p" pod "0e2ca1ff-46e0-4b67-a561-bebf0d231464" (UID: "0e2ca1ff-46e0-4b67-a561-bebf0d231464"). InnerVolumeSpecName "kube-api-access-dv42p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:29:04.888767 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:04.888726 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv42p\" (UniqueName: \"kubernetes.io/projected/0e2ca1ff-46e0-4b67-a561-bebf0d231464-kube-api-access-dv42p\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:05.305650 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:05.305617 5979 generic.go:356] "Generic (PLEG): container finished" podID="0e2ca1ff-46e0-4b67-a561-bebf0d231464" containerID="4a63747785a13b7b05bc1c66e6a2091613d8dc6142384b9a72dab5f003124ba9" exitCode=0 Nov 03 14:29:05.306143 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:05.305672 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-60261a-c86bf8c5f-pxlmt" Nov 03 14:29:05.306143 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:05.305696 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-60261a-c86bf8c5f-pxlmt" event={"ID":"0e2ca1ff-46e0-4b67-a561-bebf0d231464","Type":"ContainerDied","Data":"4a63747785a13b7b05bc1c66e6a2091613d8dc6142384b9a72dab5f003124ba9"} Nov 03 14:29:05.306143 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:05.305744 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-60261a-c86bf8c5f-pxlmt" event={"ID":"0e2ca1ff-46e0-4b67-a561-bebf0d231464","Type":"ContainerDied","Data":"c696a463be0e556ce53c685f65cc9bf7f451e92c3b22d2ce4c272149131bdcce"} Nov 03 14:29:05.306143 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:05.305766 5979 scope.go:122] "RemoveContainer" containerID="4a63747785a13b7b05bc1c66e6a2091613d8dc6142384b9a72dab5f003124ba9" Nov 03 14:29:05.307294 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:05.307273 5979 generic.go:356] "Generic (PLEG): container finished" podID="cdc023c5-a79b-49d0-9bed-ea1b749d4416" containerID="e97c86d573fc01a8fd6e0cb8193420852302aa4fa64486f60a4eb35f39d17156" exitCode=0 Nov 03 14:29:05.307394 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:05.307313 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-60261a-77454f94c-ccbgm" Nov 03 14:29:05.307394 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:05.307329 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-60261a-77454f94c-ccbgm" event={"ID":"cdc023c5-a79b-49d0-9bed-ea1b749d4416","Type":"ContainerDied","Data":"e97c86d573fc01a8fd6e0cb8193420852302aa4fa64486f60a4eb35f39d17156"} Nov 03 14:29:05.307560 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:05.307391 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-60261a-77454f94c-ccbgm" event={"ID":"cdc023c5-a79b-49d0-9bed-ea1b749d4416","Type":"ContainerDied","Data":"94058f424c06681052900201b008ae5844149e9bfc60972aa6d72ee23502db3b"} Nov 03 14:29:05.316070 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:05.316050 5979 scope.go:122] "RemoveContainer" containerID="4a63747785a13b7b05bc1c66e6a2091613d8dc6142384b9a72dab5f003124ba9" Nov 03 14:29:05.316389 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:29:05.316327 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a63747785a13b7b05bc1c66e6a2091613d8dc6142384b9a72dab5f003124ba9\": container with ID starting with 4a63747785a13b7b05bc1c66e6a2091613d8dc6142384b9a72dab5f003124ba9 not found: ID does not exist" containerID="4a63747785a13b7b05bc1c66e6a2091613d8dc6142384b9a72dab5f003124ba9" Nov 03 14:29:05.316389 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:05.316358 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a63747785a13b7b05bc1c66e6a2091613d8dc6142384b9a72dab5f003124ba9"} err="failed to get container status \"4a63747785a13b7b05bc1c66e6a2091613d8dc6142384b9a72dab5f003124ba9\": rpc error: code = NotFound desc = could not find container \"4a63747785a13b7b05bc1c66e6a2091613d8dc6142384b9a72dab5f003124ba9\": container with ID starting with 4a63747785a13b7b05bc1c66e6a2091613d8dc6142384b9a72dab5f003124ba9 not found: ID does not exist" Nov 03 14:29:05.316389 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:05.316375 5979 scope.go:122] "RemoveContainer" containerID="e97c86d573fc01a8fd6e0cb8193420852302aa4fa64486f60a4eb35f39d17156" Nov 03 14:29:05.325115 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:05.325098 5979 scope.go:122] "RemoveContainer" containerID="e97c86d573fc01a8fd6e0cb8193420852302aa4fa64486f60a4eb35f39d17156" Nov 03 14:29:05.325369 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:29:05.325351 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e97c86d573fc01a8fd6e0cb8193420852302aa4fa64486f60a4eb35f39d17156\": container with ID starting with e97c86d573fc01a8fd6e0cb8193420852302aa4fa64486f60a4eb35f39d17156 not found: ID does not exist" containerID="e97c86d573fc01a8fd6e0cb8193420852302aa4fa64486f60a4eb35f39d17156" Nov 03 14:29:05.325428 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:05.325376 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e97c86d573fc01a8fd6e0cb8193420852302aa4fa64486f60a4eb35f39d17156"} err="failed to get container status \"e97c86d573fc01a8fd6e0cb8193420852302aa4fa64486f60a4eb35f39d17156\": rpc error: code = NotFound desc = could not find container \"e97c86d573fc01a8fd6e0cb8193420852302aa4fa64486f60a4eb35f39d17156\": container with ID starting with e97c86d573fc01a8fd6e0cb8193420852302aa4fa64486f60a4eb35f39d17156 not found: ID does not exist" Nov 03 14:29:05.460999 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:05.460969 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-60261a-77454f94c-ccbgm"] Nov 03 14:29:05.507606 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:05.507568 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-60261a-77454f94c-ccbgm"] Nov 03 14:29:05.568060 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:05.567971 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-60261a-c86bf8c5f-pxlmt"] Nov 03 14:29:05.573837 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:05.573810 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-60261a-c86bf8c5f-pxlmt"] Nov 03 14:29:06.115309 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:06.115272 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e2ca1ff-46e0-4b67-a561-bebf0d231464" path="/var/lib/kubelet/pods/0e2ca1ff-46e0-4b67-a561-bebf0d231464/volumes" Nov 03 14:29:06.115741 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:06.115724 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdc023c5-a79b-49d0-9bed-ea1b749d4416" path="/var/lib/kubelet/pods/cdc023c5-a79b-49d0-9bed-ea1b749d4416/volumes" Nov 03 14:29:06.719831 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:06.719800 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj" Nov 03 14:29:06.812221 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:06.812128 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4723366a-b53b-40f7-a27a-d5cde461525d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4723366a-b53b-40f7-a27a-d5cde461525d-istio-data\") pod \"4723366a-b53b-40f7-a27a-d5cde461525d\" (UID: \"4723366a-b53b-40f7-a27a-d5cde461525d\") " Nov 03 14:29:06.812221 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:06.812203 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4723366a-b53b-40f7-a27a-d5cde461525d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4723366a-b53b-40f7-a27a-d5cde461525d-istiod-ca-cert\") pod \"4723366a-b53b-40f7-a27a-d5cde461525d\" (UID: \"4723366a-b53b-40f7-a27a-d5cde461525d\") " Nov 03 14:29:06.812514 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:06.812251 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4723366a-b53b-40f7-a27a-d5cde461525d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4723366a-b53b-40f7-a27a-d5cde461525d-workload-socket\") pod \"4723366a-b53b-40f7-a27a-d5cde461525d\" (UID: \"4723366a-b53b-40f7-a27a-d5cde461525d\") " Nov 03 14:29:06.812514 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:06.812293 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4723366a-b53b-40f7-a27a-d5cde461525d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4723366a-b53b-40f7-a27a-d5cde461525d-workload-certs\") pod \"4723366a-b53b-40f7-a27a-d5cde461525d\" (UID: \"4723366a-b53b-40f7-a27a-d5cde461525d\") " Nov 03 14:29:06.812514 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:06.812360 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4723366a-b53b-40f7-a27a-d5cde461525d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4723366a-b53b-40f7-a27a-d5cde461525d-istio-envoy\") pod \"4723366a-b53b-40f7-a27a-d5cde461525d\" (UID: \"4723366a-b53b-40f7-a27a-d5cde461525d\") " Nov 03 14:29:06.812514 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:06.812389 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4723366a-b53b-40f7-a27a-d5cde461525d-kube-api-access-266k5\" (UniqueName: \"kubernetes.io/projected/4723366a-b53b-40f7-a27a-d5cde461525d-kube-api-access-266k5\") pod \"4723366a-b53b-40f7-a27a-d5cde461525d\" (UID: \"4723366a-b53b-40f7-a27a-d5cde461525d\") " Nov 03 14:29:06.812514 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:06.812417 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4723366a-b53b-40f7-a27a-d5cde461525d-istio-data" pod "4723366a-b53b-40f7-a27a-d5cde461525d" (UID: "4723366a-b53b-40f7-a27a-d5cde461525d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:29:06.812514 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:06.812451 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4723366a-b53b-40f7-a27a-d5cde461525d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4723366a-b53b-40f7-a27a-d5cde461525d-credential-socket\") pod \"4723366a-b53b-40f7-a27a-d5cde461525d\" (UID: \"4723366a-b53b-40f7-a27a-d5cde461525d\") " Nov 03 14:29:06.812863 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:06.812592 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4723366a-b53b-40f7-a27a-d5cde461525d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4723366a-b53b-40f7-a27a-d5cde461525d-istio-podinfo\") pod \"4723366a-b53b-40f7-a27a-d5cde461525d\" (UID: \"4723366a-b53b-40f7-a27a-d5cde461525d\") " Nov 03 14:29:06.812863 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:06.812640 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4723366a-b53b-40f7-a27a-d5cde461525d-workload-socket" pod "4723366a-b53b-40f7-a27a-d5cde461525d" (UID: "4723366a-b53b-40f7-a27a-d5cde461525d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:29:06.812863 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:06.812656 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4723366a-b53b-40f7-a27a-d5cde461525d-istio-token\" (UniqueName: \"kubernetes.io/projected/4723366a-b53b-40f7-a27a-d5cde461525d-istio-token\") pod \"4723366a-b53b-40f7-a27a-d5cde461525d\" (UID: \"4723366a-b53b-40f7-a27a-d5cde461525d\") " Nov 03 14:29:06.812863 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:06.812674 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4723366a-b53b-40f7-a27a-d5cde461525d-workload-certs" pod "4723366a-b53b-40f7-a27a-d5cde461525d" (UID: "4723366a-b53b-40f7-a27a-d5cde461525d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:29:06.812863 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:06.812771 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4723366a-b53b-40f7-a27a-d5cde461525d-istiod-ca-cert" pod "4723366a-b53b-40f7-a27a-d5cde461525d" (UID: "4723366a-b53b-40f7-a27a-d5cde461525d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:29:06.812863 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:06.812779 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4723366a-b53b-40f7-a27a-d5cde461525d-credential-socket" pod "4723366a-b53b-40f7-a27a-d5cde461525d" (UID: "4723366a-b53b-40f7-a27a-d5cde461525d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:29:06.813179 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:06.813104 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4723366a-b53b-40f7-a27a-d5cde461525d-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:06.813179 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:06.813124 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4723366a-b53b-40f7-a27a-d5cde461525d-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:06.813179 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:06.813141 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4723366a-b53b-40f7-a27a-d5cde461525d-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:06.813179 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:06.813156 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4723366a-b53b-40f7-a27a-d5cde461525d-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:06.813179 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:06.813170 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4723366a-b53b-40f7-a27a-d5cde461525d-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:06.814724 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:06.814691 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4723366a-b53b-40f7-a27a-d5cde461525d-istio-envoy" pod "4723366a-b53b-40f7-a27a-d5cde461525d" (UID: "4723366a-b53b-40f7-a27a-d5cde461525d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:29:06.814842 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:06.814795 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4723366a-b53b-40f7-a27a-d5cde461525d-kube-api-access-266k5" pod "4723366a-b53b-40f7-a27a-d5cde461525d" (UID: "4723366a-b53b-40f7-a27a-d5cde461525d"). InnerVolumeSpecName "kube-api-access-266k5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:29:06.815069 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:06.815050 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4723366a-b53b-40f7-a27a-d5cde461525d-istio-token" pod "4723366a-b53b-40f7-a27a-d5cde461525d" (UID: "4723366a-b53b-40f7-a27a-d5cde461525d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:29:06.815147 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:06.815050 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4723366a-b53b-40f7-a27a-d5cde461525d-istio-podinfo" pod "4723366a-b53b-40f7-a27a-d5cde461525d" (UID: "4723366a-b53b-40f7-a27a-d5cde461525d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:29:06.914174 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:06.914126 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4723366a-b53b-40f7-a27a-d5cde461525d-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:06.914174 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:06.914162 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4723366a-b53b-40f7-a27a-d5cde461525d-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:06.914174 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:06.914178 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-266k5\" (UniqueName: \"kubernetes.io/projected/4723366a-b53b-40f7-a27a-d5cde461525d-kube-api-access-266k5\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:06.914536 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:06.914192 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4723366a-b53b-40f7-a27a-d5cde461525d-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:07.262094 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:29:07.262045 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:29:07.323899 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:07.323859 5979 generic.go:356] "Generic (PLEG): container finished" podID="4723366a-b53b-40f7-a27a-d5cde461525d" containerID="78f537c5e25fcc7de464419ca6c3694c2f5921b25b7f21e362614923ccf286bc" exitCode=0 Nov 03 14:29:07.324096 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:07.323918 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj" Nov 03 14:29:07.324096 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:07.323955 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj" event={"ID":"4723366a-b53b-40f7-a27a-d5cde461525d","Type":"ContainerDied","Data":"78f537c5e25fcc7de464419ca6c3694c2f5921b25b7f21e362614923ccf286bc"} Nov 03 14:29:07.324096 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:07.324032 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj" event={"ID":"4723366a-b53b-40f7-a27a-d5cde461525d","Type":"ContainerDied","Data":"58188e0fdcedc4b434a10c721fa559c81542c0c09a7cfb25b5eff520ecb0ed9d"} Nov 03 14:29:07.324096 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:07.324069 5979 scope.go:122] "RemoveContainer" containerID="78f537c5e25fcc7de464419ca6c3694c2f5921b25b7f21e362614923ccf286bc" Nov 03 14:29:07.335058 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:07.335041 5979 scope.go:122] "RemoveContainer" containerID="bbccee5e13015c2853354a6cbc64fccec9b6655e2cd856a2b03a7bfb19300918" Nov 03 14:29:07.346110 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:07.346088 5979 scope.go:122] "RemoveContainer" containerID="78f537c5e25fcc7de464419ca6c3694c2f5921b25b7f21e362614923ccf286bc" Nov 03 14:29:07.346374 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:29:07.346350 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78f537c5e25fcc7de464419ca6c3694c2f5921b25b7f21e362614923ccf286bc\": container with ID starting with 78f537c5e25fcc7de464419ca6c3694c2f5921b25b7f21e362614923ccf286bc not found: ID does not exist" containerID="78f537c5e25fcc7de464419ca6c3694c2f5921b25b7f21e362614923ccf286bc" Nov 03 14:29:07.346453 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:07.346386 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78f537c5e25fcc7de464419ca6c3694c2f5921b25b7f21e362614923ccf286bc"} err="failed to get container status \"78f537c5e25fcc7de464419ca6c3694c2f5921b25b7f21e362614923ccf286bc\": rpc error: code = NotFound desc = could not find container \"78f537c5e25fcc7de464419ca6c3694c2f5921b25b7f21e362614923ccf286bc\": container with ID starting with 78f537c5e25fcc7de464419ca6c3694c2f5921b25b7f21e362614923ccf286bc not found: ID does not exist" Nov 03 14:29:07.346453 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:07.346410 5979 scope.go:122] "RemoveContainer" containerID="bbccee5e13015c2853354a6cbc64fccec9b6655e2cd856a2b03a7bfb19300918" Nov 03 14:29:07.346694 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:29:07.346677 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbccee5e13015c2853354a6cbc64fccec9b6655e2cd856a2b03a7bfb19300918\": container with ID starting with bbccee5e13015c2853354a6cbc64fccec9b6655e2cd856a2b03a7bfb19300918 not found: ID does not exist" containerID="bbccee5e13015c2853354a6cbc64fccec9b6655e2cd856a2b03a7bfb19300918" Nov 03 14:29:07.346768 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:07.346703 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbccee5e13015c2853354a6cbc64fccec9b6655e2cd856a2b03a7bfb19300918"} err="failed to get container status \"bbccee5e13015c2853354a6cbc64fccec9b6655e2cd856a2b03a7bfb19300918\": rpc error: code = NotFound desc = could not find container \"bbccee5e13015c2853354a6cbc64fccec9b6655e2cd856a2b03a7bfb19300918\": container with ID starting with bbccee5e13015c2853354a6cbc64fccec9b6655e2cd856a2b03a7bfb19300918 not found: ID does not exist" Nov 03 14:29:07.477996 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:07.477966 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj"] Nov 03 14:29:07.480780 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:07.480727 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-60261a-openshift-default-784665dc57-vq4rj"] Nov 03 14:29:08.115719 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:08.115690 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4723366a-b53b-40f7-a27a-d5cde461525d" path="/var/lib/kubelet/pods/4723366a-b53b-40f7-a27a-d5cde461525d/volumes" Nov 03 14:29:11.148012 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:11.147971 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:29:12.445006 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:12.444970 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43512dd5-7vqnz"] Nov 03 14:29:12.445454 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:12.445441 5979 cpu_manager.go:414] "containerMap: removing container" podUID="50674938-d53d-430b-a564-fa0bd1abed74" containerName="maas-api" Nov 03 14:29:12.445527 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:12.445455 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50674938-d53d-430b-a564-fa0bd1abed74" containerName="maas-api" Nov 03 14:29:12.445527 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:12.445489 5979 cpu_manager.go:414] "containerMap: removing container" podUID="5d1bbe2d-c09b-4caa-86ec-a0a64465e419" containerName="main" Nov 03 14:29:12.445527 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:12.445496 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d1bbe2d-c09b-4caa-86ec-a0a64465e419" containerName="main" Nov 03 14:29:12.445527 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:12.445521 5979 cpu_manager.go:414] "containerMap: removing container" podUID="4723366a-b53b-40f7-a27a-d5cde461525d" containerName="istio-proxy" Nov 03 14:29:12.445527 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:12.445526 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4723366a-b53b-40f7-a27a-d5cde461525d" containerName="istio-proxy" Nov 03 14:29:12.445714 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:12.445537 5979 cpu_manager.go:414] "containerMap: removing container" podUID="609e1b10-a933-48b3-b224-1476116d3e66" containerName="payload-processing" Nov 03 14:29:12.445714 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:12.445542 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="609e1b10-a933-48b3-b224-1476116d3e66" containerName="payload-processing" Nov 03 14:29:12.445714 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:12.445552 5979 cpu_manager.go:414] "containerMap: removing container" podUID="ba6aa3ab-9c2c-4bca-8748-b6b5f1abc34c" containerName="revoke-keys" Nov 03 14:29:12.445714 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:12.445556 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba6aa3ab-9c2c-4bca-8748-b6b5f1abc34c" containerName="revoke-keys" Nov 03 14:29:12.445714 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:12.445563 5979 cpu_manager.go:414] "containerMap: removing container" podUID="0e2ca1ff-46e0-4b67-a561-bebf0d231464" containerName="payload-pre-processing" Nov 03 14:29:12.445714 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:12.445567 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e2ca1ff-46e0-4b67-a561-bebf0d231464" containerName="payload-pre-processing" Nov 03 14:29:12.445714 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:12.445578 5979 cpu_manager.go:414] "containerMap: removing container" podUID="4723366a-b53b-40f7-a27a-d5cde461525d" containerName="istio-proxy" Nov 03 14:29:12.445714 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:12.445583 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4723366a-b53b-40f7-a27a-d5cde461525d" containerName="istio-proxy" Nov 03 14:29:12.445714 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:12.445593 5979 cpu_manager.go:414] "containerMap: removing container" podUID="6e0e9df1-0191-446b-95d9-6e08b67f5ed2" containerName="payload-pre-processing" Nov 03 14:29:12.445714 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:12.445598 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e0e9df1-0191-446b-95d9-6e08b67f5ed2" containerName="payload-pre-processing" Nov 03 14:29:12.445714 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:12.445607 5979 cpu_manager.go:414] "containerMap: removing container" podUID="cdc023c5-a79b-49d0-9bed-ea1b749d4416" containerName="payload-processing" Nov 03 14:29:12.445714 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:12.445612 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdc023c5-a79b-49d0-9bed-ea1b749d4416" containerName="payload-processing" Nov 03 14:29:12.445714 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:12.445683 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba6aa3ab-9c2c-4bca-8748-b6b5f1abc34c" containerName="revoke-keys" Nov 03 14:29:12.445714 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:12.445694 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e2ca1ff-46e0-4b67-a561-bebf0d231464" containerName="payload-pre-processing" Nov 03 14:29:12.445714 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:12.445701 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="4723366a-b53b-40f7-a27a-d5cde461525d" containerName="istio-proxy" Nov 03 14:29:12.445714 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:12.445709 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e0e9df1-0191-446b-95d9-6e08b67f5ed2" containerName="payload-pre-processing" Nov 03 14:29:12.445714 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:12.445716 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdc023c5-a79b-49d0-9bed-ea1b749d4416" containerName="payload-processing" Nov 03 14:29:12.446238 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:12.445723 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="50674938-d53d-430b-a564-fa0bd1abed74" containerName="maas-api" Nov 03 14:29:12.446238 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:12.445730 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d1bbe2d-c09b-4caa-86ec-a0a64465e419" containerName="main" Nov 03 14:29:12.446238 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:12.445740 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="609e1b10-a933-48b3-b224-1476116d3e66" containerName="payload-processing" Nov 03 14:29:12.450494 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:12.450462 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43512dd5-7vqnz" Nov 03 14:29:12.472050 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:12.472018 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e029367c-8b59-49eb-ad8f-60f024f5a5c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43512dd5-7vqnz\" (UID: \"e029367c-8b59-49eb-ad8f-60f024f5a5c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43512dd5-7vqnz" Nov 03 14:29:12.472199 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:12.472070 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9zcc\" (UniqueName: \"kubernetes.io/projected/e029367c-8b59-49eb-ad8f-60f024f5a5c7-kube-api-access-q9zcc\") pod \"maas-api-revoke-sub-43512dd5-7vqnz\" (UID: \"e029367c-8b59-49eb-ad8f-60f024f5a5c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43512dd5-7vqnz" Nov 03 14:29:12.472246 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:12.472200 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e029367c-8b59-49eb-ad8f-60f024f5a5c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43512dd5-7vqnz\" (UID: \"e029367c-8b59-49eb-ad8f-60f024f5a5c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43512dd5-7vqnz" Nov 03 14:29:12.499663 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:12.499625 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:29:12.546131 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:12.546096 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 14:29:12.548351 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:12.548327 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43512dd5-7vqnz"] Nov 03 14:29:12.573486 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:12.573447 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e029367c-8b59-49eb-ad8f-60f024f5a5c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43512dd5-7vqnz\" (UID: \"e029367c-8b59-49eb-ad8f-60f024f5a5c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43512dd5-7vqnz" Nov 03 14:29:12.573672 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:12.573506 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9zcc\" (UniqueName: \"kubernetes.io/projected/e029367c-8b59-49eb-ad8f-60f024f5a5c7-kube-api-access-q9zcc\") pod \"maas-api-revoke-sub-43512dd5-7vqnz\" (UID: \"e029367c-8b59-49eb-ad8f-60f024f5a5c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43512dd5-7vqnz" Nov 03 14:29:12.573672 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:12.573549 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e029367c-8b59-49eb-ad8f-60f024f5a5c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43512dd5-7vqnz\" (UID: \"e029367c-8b59-49eb-ad8f-60f024f5a5c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43512dd5-7vqnz" Nov 03 14:29:12.574173 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:12.574151 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e029367c-8b59-49eb-ad8f-60f024f5a5c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43512dd5-7vqnz\" (UID: \"e029367c-8b59-49eb-ad8f-60f024f5a5c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43512dd5-7vqnz" Nov 03 14:29:12.575781 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:12.575761 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e029367c-8b59-49eb-ad8f-60f024f5a5c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43512dd5-7vqnz\" (UID: \"e029367c-8b59-49eb-ad8f-60f024f5a5c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43512dd5-7vqnz" Nov 03 14:29:12.627461 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:12.627423 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9zcc\" (UniqueName: \"kubernetes.io/projected/e029367c-8b59-49eb-ad8f-60f024f5a5c7-kube-api-access-q9zcc\") pod \"maas-api-revoke-sub-43512dd5-7vqnz\" (UID: \"e029367c-8b59-49eb-ad8f-60f024f5a5c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43512dd5-7vqnz" Nov 03 14:29:12.763504 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:12.763401 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43512dd5-7vqnz" Nov 03 14:29:13.134634 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:13.134607 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43512dd5-7vqnz"] Nov 03 14:29:13.143030 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:29:13.142998 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode029367c_8b59_49eb_ad8f_60f024f5a5c7.slice/crio-ac2413df06b491eefb9b5459dcae1924f295eefe0dc1b21f4c9a84f5903d8323 WatchSource:0}: Error finding container ac2413df06b491eefb9b5459dcae1924f295eefe0dc1b21f4c9a84f5903d8323: Status 404 returned error can't find the container with id ac2413df06b491eefb9b5459dcae1924f295eefe0dc1b21f4c9a84f5903d8323 Nov 03 14:29:13.364399 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:13.364362 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43512dd5-7vqnz" event={"ID":"e029367c-8b59-49eb-ad8f-60f024f5a5c7","Type":"ContainerStarted","Data":"ac2413df06b491eefb9b5459dcae1924f295eefe0dc1b21f4c9a84f5903d8323"} Nov 03 14:29:14.371974 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:14.371934 5979 generic.go:356] "Generic (PLEG): container finished" podID="e029367c-8b59-49eb-ad8f-60f024f5a5c7" containerID="e16a312f87b7590b0f3d867d227ce5ee18364773c8cd27dbe6eeb5cc4d7a0f8e" exitCode=0 Nov 03 14:29:14.372363 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:14.372019 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43512dd5-7vqnz" event={"ID":"e029367c-8b59-49eb-ad8f-60f024f5a5c7","Type":"ContainerDied","Data":"e16a312f87b7590b0f3d867d227ce5ee18364773c8cd27dbe6eeb5cc4d7a0f8e"} Nov 03 14:29:15.521543 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:15.521520 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43512dd5-7vqnz" Nov 03 14:29:15.601343 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:15.601312 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e029367c-8b59-49eb-ad8f-60f024f5a5c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e029367c-8b59-49eb-ad8f-60f024f5a5c7-maas-api-openshift-ca\") pod \"e029367c-8b59-49eb-ad8f-60f024f5a5c7\" (UID: \"e029367c-8b59-49eb-ad8f-60f024f5a5c7\") " Nov 03 14:29:15.601555 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:15.601394 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e029367c-8b59-49eb-ad8f-60f024f5a5c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e029367c-8b59-49eb-ad8f-60f024f5a5c7-maas-api-xks-ca\") pod \"e029367c-8b59-49eb-ad8f-60f024f5a5c7\" (UID: \"e029367c-8b59-49eb-ad8f-60f024f5a5c7\") " Nov 03 14:29:15.601555 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:15.601421 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e029367c-8b59-49eb-ad8f-60f024f5a5c7-kube-api-access-q9zcc\" (UniqueName: \"kubernetes.io/projected/e029367c-8b59-49eb-ad8f-60f024f5a5c7-kube-api-access-q9zcc\") pod \"e029367c-8b59-49eb-ad8f-60f024f5a5c7\" (UID: \"e029367c-8b59-49eb-ad8f-60f024f5a5c7\") " Nov 03 14:29:15.601792 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:15.601769 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e029367c-8b59-49eb-ad8f-60f024f5a5c7-maas-api-openshift-ca" pod "e029367c-8b59-49eb-ad8f-60f024f5a5c7" (UID: "e029367c-8b59-49eb-ad8f-60f024f5a5c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:29:15.603582 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:15.603555 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e029367c-8b59-49eb-ad8f-60f024f5a5c7-maas-api-xks-ca" pod "e029367c-8b59-49eb-ad8f-60f024f5a5c7" (UID: "e029367c-8b59-49eb-ad8f-60f024f5a5c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:29:15.603700 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:15.603610 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e029367c-8b59-49eb-ad8f-60f024f5a5c7-kube-api-access-q9zcc" pod "e029367c-8b59-49eb-ad8f-60f024f5a5c7" (UID: "e029367c-8b59-49eb-ad8f-60f024f5a5c7"). InnerVolumeSpecName "kube-api-access-q9zcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:29:15.702714 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:15.702679 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e029367c-8b59-49eb-ad8f-60f024f5a5c7-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:15.702714 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:15.702709 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e029367c-8b59-49eb-ad8f-60f024f5a5c7-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:15.702714 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:15.702720 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9zcc\" (UniqueName: \"kubernetes.io/projected/e029367c-8b59-49eb-ad8f-60f024f5a5c7-kube-api-access-q9zcc\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:16.385834 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:16.385806 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43512dd5-7vqnz" Nov 03 14:29:16.385834 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:16.385822 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43512dd5-7vqnz" event={"ID":"e029367c-8b59-49eb-ad8f-60f024f5a5c7","Type":"ContainerDied","Data":"ac2413df06b491eefb9b5459dcae1924f295eefe0dc1b21f4c9a84f5903d8323"} Nov 03 14:29:16.386035 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:16.385871 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac2413df06b491eefb9b5459dcae1924f295eefe0dc1b21f4c9a84f5903d8323" Nov 03 14:29:18.324552 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:18.324511 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv"] Nov 03 14:29:18.324951 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:18.324730 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv" podUID="e73bb972-7cdd-4a6a-aae0-4345d0630989" containerName="istio-proxy" containerID="cri-o://1a5c1ff335b0329062bdb928f4584e35d2e0842f6f869d033edff69eabc799d8" gracePeriod=30 Nov 03 14:29:19.523663 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:19.523626 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4834eea6-rxz6c"] Nov 03 14:29:19.524112 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:19.524097 5979 cpu_manager.go:414] "containerMap: removing container" podUID="e029367c-8b59-49eb-ad8f-60f024f5a5c7" containerName="revoke-keys" Nov 03 14:29:19.524156 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:19.524114 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e029367c-8b59-49eb-ad8f-60f024f5a5c7" containerName="revoke-keys" Nov 03 14:29:19.524247 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:19.524236 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="e029367c-8b59-49eb-ad8f-60f024f5a5c7" containerName="revoke-keys" Nov 03 14:29:19.524287 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:19.524255 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="4723366a-b53b-40f7-a27a-d5cde461525d" containerName="istio-proxy" Nov 03 14:29:19.532145 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:19.532116 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4834eea6-rxz6c" Nov 03 14:29:19.542958 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:19.542930 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh7vf\" (UniqueName: \"kubernetes.io/projected/b97ade37-319a-4885-ba04-9d943641ac1d-kube-api-access-zh7vf\") pod \"maas-api-revoke-sub-4834eea6-rxz6c\" (UID: \"b97ade37-319a-4885-ba04-9d943641ac1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4834eea6-rxz6c" Nov 03 14:29:19.543076 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:19.542980 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b97ade37-319a-4885-ba04-9d943641ac1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4834eea6-rxz6c\" (UID: \"b97ade37-319a-4885-ba04-9d943641ac1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4834eea6-rxz6c" Nov 03 14:29:19.543125 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:19.543083 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b97ade37-319a-4885-ba04-9d943641ac1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4834eea6-rxz6c\" (UID: \"b97ade37-319a-4885-ba04-9d943641ac1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4834eea6-rxz6c" Nov 03 14:29:19.584257 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:19.584226 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4834eea6-rxz6c"] Nov 03 14:29:19.601454 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:19.601426 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:29:19.628261 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:19.628228 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 14:29:19.644147 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:19.644113 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh7vf\" (UniqueName: \"kubernetes.io/projected/b97ade37-319a-4885-ba04-9d943641ac1d-kube-api-access-zh7vf\") pod \"maas-api-revoke-sub-4834eea6-rxz6c\" (UID: \"b97ade37-319a-4885-ba04-9d943641ac1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4834eea6-rxz6c" Nov 03 14:29:19.644312 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:19.644179 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b97ade37-319a-4885-ba04-9d943641ac1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4834eea6-rxz6c\" (UID: \"b97ade37-319a-4885-ba04-9d943641ac1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4834eea6-rxz6c" Nov 03 14:29:19.644312 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:19.644230 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b97ade37-319a-4885-ba04-9d943641ac1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4834eea6-rxz6c\" (UID: \"b97ade37-319a-4885-ba04-9d943641ac1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4834eea6-rxz6c" Nov 03 14:29:19.644982 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:19.644948 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b97ade37-319a-4885-ba04-9d943641ac1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4834eea6-rxz6c\" (UID: \"b97ade37-319a-4885-ba04-9d943641ac1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4834eea6-rxz6c" Nov 03 14:29:19.646619 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:19.646599 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b97ade37-319a-4885-ba04-9d943641ac1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4834eea6-rxz6c\" (UID: \"b97ade37-319a-4885-ba04-9d943641ac1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4834eea6-rxz6c" Nov 03 14:29:19.698671 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:19.698642 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh7vf\" (UniqueName: \"kubernetes.io/projected/b97ade37-319a-4885-ba04-9d943641ac1d-kube-api-access-zh7vf\") pod \"maas-api-revoke-sub-4834eea6-rxz6c\" (UID: \"b97ade37-319a-4885-ba04-9d943641ac1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4834eea6-rxz6c" Nov 03 14:29:19.849195 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:19.849095 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4834eea6-rxz6c" Nov 03 14:29:20.219933 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:20.219892 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4834eea6-rxz6c"] Nov 03 14:29:20.232920 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:29:20.229411 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb97ade37_319a_4885_ba04_9d943641ac1d.slice/crio-154b75c9c75c204e40781c713dfb92232684544cbe870cdf1628be0d2c06bd6e WatchSource:0}: Error finding container 154b75c9c75c204e40781c713dfb92232684544cbe870cdf1628be0d2c06bd6e: Status 404 returned error can't find the container with id 154b75c9c75c204e40781c713dfb92232684544cbe870cdf1628be0d2c06bd6e Nov 03 14:29:20.410534 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:20.410501 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4834eea6-rxz6c" event={"ID":"b97ade37-319a-4885-ba04-9d943641ac1d","Type":"ContainerStarted","Data":"154b75c9c75c204e40781c713dfb92232684544cbe870cdf1628be0d2c06bd6e"} Nov 03 14:29:21.417901 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:21.417866 5979 generic.go:356] "Generic (PLEG): container finished" podID="b97ade37-319a-4885-ba04-9d943641ac1d" containerID="d4cbea2b2e3d98a7690384dda63efadd775aa70d4fc34146614de73a26e3deaa" exitCode=0 Nov 03 14:29:21.418271 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:21.417948 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4834eea6-rxz6c" event={"ID":"b97ade37-319a-4885-ba04-9d943641ac1d","Type":"ContainerDied","Data":"d4cbea2b2e3d98a7690384dda63efadd775aa70d4fc34146614de73a26e3deaa"} Nov 03 14:29:22.566048 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:22.566023 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4834eea6-rxz6c" Nov 03 14:29:22.679399 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:22.679308 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b97ade37-319a-4885-ba04-9d943641ac1d-kube-api-access-zh7vf\" (UniqueName: \"kubernetes.io/projected/b97ade37-319a-4885-ba04-9d943641ac1d-kube-api-access-zh7vf\") pod \"b97ade37-319a-4885-ba04-9d943641ac1d\" (UID: \"b97ade37-319a-4885-ba04-9d943641ac1d\") " Nov 03 14:29:22.679399 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:22.679364 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b97ade37-319a-4885-ba04-9d943641ac1d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b97ade37-319a-4885-ba04-9d943641ac1d-maas-api-openshift-ca\") pod \"b97ade37-319a-4885-ba04-9d943641ac1d\" (UID: \"b97ade37-319a-4885-ba04-9d943641ac1d\") " Nov 03 14:29:22.679646 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:22.679592 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b97ade37-319a-4885-ba04-9d943641ac1d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b97ade37-319a-4885-ba04-9d943641ac1d-maas-api-xks-ca\") pod \"b97ade37-319a-4885-ba04-9d943641ac1d\" (UID: \"b97ade37-319a-4885-ba04-9d943641ac1d\") " Nov 03 14:29:22.680014 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:22.679861 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b97ade37-319a-4885-ba04-9d943641ac1d-maas-api-openshift-ca" pod "b97ade37-319a-4885-ba04-9d943641ac1d" (UID: "b97ade37-319a-4885-ba04-9d943641ac1d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:29:22.680014 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:22.680000 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b97ade37-319a-4885-ba04-9d943641ac1d-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:22.681821 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:22.681787 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b97ade37-319a-4885-ba04-9d943641ac1d-maas-api-xks-ca" pod "b97ade37-319a-4885-ba04-9d943641ac1d" (UID: "b97ade37-319a-4885-ba04-9d943641ac1d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:29:22.682011 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:22.681993 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b97ade37-319a-4885-ba04-9d943641ac1d-kube-api-access-zh7vf" pod "b97ade37-319a-4885-ba04-9d943641ac1d" (UID: "b97ade37-319a-4885-ba04-9d943641ac1d"). InnerVolumeSpecName "kube-api-access-zh7vf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:29:22.781761 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:22.781719 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh7vf\" (UniqueName: \"kubernetes.io/projected/b97ade37-319a-4885-ba04-9d943641ac1d-kube-api-access-zh7vf\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:22.781761 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:22.781757 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b97ade37-319a-4885-ba04-9d943641ac1d-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:23.211540 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:23.211452 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-420450a7-jp4nb"] Nov 03 14:29:23.212159 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:23.212139 5979 cpu_manager.go:414] "containerMap: removing container" podUID="b97ade37-319a-4885-ba04-9d943641ac1d" containerName="revoke-keys" Nov 03 14:29:23.212272 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:23.212161 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b97ade37-319a-4885-ba04-9d943641ac1d" containerName="revoke-keys" Nov 03 14:29:23.212272 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:23.212262 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="b97ade37-319a-4885-ba04-9d943641ac1d" containerName="revoke-keys" Nov 03 14:29:23.215331 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:23.215304 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-420450a7-jp4nb" Nov 03 14:29:23.271854 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:23.271814 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-420450a7-jp4nb"] Nov 03 14:29:23.287198 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:23.287166 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcc26\" (UniqueName: \"kubernetes.io/projected/062dae17-169d-4881-95ba-dd1d307af352-kube-api-access-xcc26\") pod \"maas-api-revoke-sub-420450a7-jp4nb\" (UID: \"062dae17-169d-4881-95ba-dd1d307af352\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-420450a7-jp4nb" Nov 03 14:29:23.287377 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:23.287207 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/062dae17-169d-4881-95ba-dd1d307af352-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-420450a7-jp4nb\" (UID: \"062dae17-169d-4881-95ba-dd1d307af352\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-420450a7-jp4nb" Nov 03 14:29:23.287377 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:23.287316 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/062dae17-169d-4881-95ba-dd1d307af352-maas-api-xks-ca\") pod \"maas-api-revoke-sub-420450a7-jp4nb\" (UID: \"062dae17-169d-4881-95ba-dd1d307af352\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-420450a7-jp4nb" Nov 03 14:29:23.388745 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:23.388705 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcc26\" (UniqueName: \"kubernetes.io/projected/062dae17-169d-4881-95ba-dd1d307af352-kube-api-access-xcc26\") pod \"maas-api-revoke-sub-420450a7-jp4nb\" (UID: \"062dae17-169d-4881-95ba-dd1d307af352\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-420450a7-jp4nb" Nov 03 14:29:23.388903 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:23.388770 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/062dae17-169d-4881-95ba-dd1d307af352-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-420450a7-jp4nb\" (UID: \"062dae17-169d-4881-95ba-dd1d307af352\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-420450a7-jp4nb" Nov 03 14:29:23.388903 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:23.388817 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/062dae17-169d-4881-95ba-dd1d307af352-maas-api-xks-ca\") pod \"maas-api-revoke-sub-420450a7-jp4nb\" (UID: \"062dae17-169d-4881-95ba-dd1d307af352\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-420450a7-jp4nb" Nov 03 14:29:23.389461 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:23.389436 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/062dae17-169d-4881-95ba-dd1d307af352-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-420450a7-jp4nb\" (UID: \"062dae17-169d-4881-95ba-dd1d307af352\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-420450a7-jp4nb" Nov 03 14:29:23.391239 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:23.391219 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/062dae17-169d-4881-95ba-dd1d307af352-maas-api-xks-ca\") pod \"maas-api-revoke-sub-420450a7-jp4nb\" (UID: \"062dae17-169d-4881-95ba-dd1d307af352\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-420450a7-jp4nb" Nov 03 14:29:23.433105 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:23.433079 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4834eea6-rxz6c" Nov 03 14:29:23.433280 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:23.433117 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4834eea6-rxz6c" event={"ID":"b97ade37-319a-4885-ba04-9d943641ac1d","Type":"ContainerDied","Data":"154b75c9c75c204e40781c713dfb92232684544cbe870cdf1628be0d2c06bd6e"} Nov 03 14:29:23.433280 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:23.433172 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="154b75c9c75c204e40781c713dfb92232684544cbe870cdf1628be0d2c06bd6e" Nov 03 14:29:23.442398 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:23.442371 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcc26\" (UniqueName: \"kubernetes.io/projected/062dae17-169d-4881-95ba-dd1d307af352-kube-api-access-xcc26\") pod \"maas-api-revoke-sub-420450a7-jp4nb\" (UID: \"062dae17-169d-4881-95ba-dd1d307af352\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-420450a7-jp4nb" Nov 03 14:29:23.529889 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:23.529855 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-420450a7-jp4nb" Nov 03 14:29:23.600883 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:23.600848 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv" Nov 03 14:29:23.693003 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:23.692971 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e73bb972-7cdd-4a6a-aae0-4345d0630989-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e73bb972-7cdd-4a6a-aae0-4345d0630989-workload-certs\") pod \"e73bb972-7cdd-4a6a-aae0-4345d0630989\" (UID: \"e73bb972-7cdd-4a6a-aae0-4345d0630989\") " Nov 03 14:29:23.693176 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:23.693050 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e73bb972-7cdd-4a6a-aae0-4345d0630989-kube-api-access-p6vkv\" (UniqueName: \"kubernetes.io/projected/e73bb972-7cdd-4a6a-aae0-4345d0630989-kube-api-access-p6vkv\") pod \"e73bb972-7cdd-4a6a-aae0-4345d0630989\" (UID: \"e73bb972-7cdd-4a6a-aae0-4345d0630989\") " Nov 03 14:29:23.693176 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:23.693071 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e73bb972-7cdd-4a6a-aae0-4345d0630989-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e73bb972-7cdd-4a6a-aae0-4345d0630989-istiod-ca-cert\") pod \"e73bb972-7cdd-4a6a-aae0-4345d0630989\" (UID: \"e73bb972-7cdd-4a6a-aae0-4345d0630989\") " Nov 03 14:29:23.693176 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:23.693135 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e73bb972-7cdd-4a6a-aae0-4345d0630989-istio-token\" (UniqueName: \"kubernetes.io/projected/e73bb972-7cdd-4a6a-aae0-4345d0630989-istio-token\") pod \"e73bb972-7cdd-4a6a-aae0-4345d0630989\" (UID: \"e73bb972-7cdd-4a6a-aae0-4345d0630989\") " Nov 03 14:29:23.693176 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:23.693155 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e73bb972-7cdd-4a6a-aae0-4345d0630989-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e73bb972-7cdd-4a6a-aae0-4345d0630989-istio-envoy\") pod \"e73bb972-7cdd-4a6a-aae0-4345d0630989\" (UID: \"e73bb972-7cdd-4a6a-aae0-4345d0630989\") " Nov 03 14:29:23.693446 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:23.693230 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e73bb972-7cdd-4a6a-aae0-4345d0630989-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e73bb972-7cdd-4a6a-aae0-4345d0630989-workload-socket\") pod \"e73bb972-7cdd-4a6a-aae0-4345d0630989\" (UID: \"e73bb972-7cdd-4a6a-aae0-4345d0630989\") " Nov 03 14:29:23.693446 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:23.693240 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e73bb972-7cdd-4a6a-aae0-4345d0630989-workload-certs" pod "e73bb972-7cdd-4a6a-aae0-4345d0630989" (UID: "e73bb972-7cdd-4a6a-aae0-4345d0630989"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:29:23.693446 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:23.693274 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e73bb972-7cdd-4a6a-aae0-4345d0630989-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e73bb972-7cdd-4a6a-aae0-4345d0630989-istio-data\") pod \"e73bb972-7cdd-4a6a-aae0-4345d0630989\" (UID: \"e73bb972-7cdd-4a6a-aae0-4345d0630989\") " Nov 03 14:29:23.693446 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:23.693301 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e73bb972-7cdd-4a6a-aae0-4345d0630989-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e73bb972-7cdd-4a6a-aae0-4345d0630989-istio-podinfo\") pod \"e73bb972-7cdd-4a6a-aae0-4345d0630989\" (UID: \"e73bb972-7cdd-4a6a-aae0-4345d0630989\") " Nov 03 14:29:23.693446 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:23.693328 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e73bb972-7cdd-4a6a-aae0-4345d0630989-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e73bb972-7cdd-4a6a-aae0-4345d0630989-credential-socket\") pod \"e73bb972-7cdd-4a6a-aae0-4345d0630989\" (UID: \"e73bb972-7cdd-4a6a-aae0-4345d0630989\") " Nov 03 14:29:23.693753 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:23.693554 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e73bb972-7cdd-4a6a-aae0-4345d0630989-istio-data" pod "e73bb972-7cdd-4a6a-aae0-4345d0630989" (UID: "e73bb972-7cdd-4a6a-aae0-4345d0630989"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:29:23.693753 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:23.693583 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e73bb972-7cdd-4a6a-aae0-4345d0630989-istiod-ca-cert" pod "e73bb972-7cdd-4a6a-aae0-4345d0630989" (UID: "e73bb972-7cdd-4a6a-aae0-4345d0630989"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:29:23.693753 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:23.693714 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e73bb972-7cdd-4a6a-aae0-4345d0630989-workload-socket" pod "e73bb972-7cdd-4a6a-aae0-4345d0630989" (UID: "e73bb972-7cdd-4a6a-aae0-4345d0630989"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:29:23.693926 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:23.693791 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e73bb972-7cdd-4a6a-aae0-4345d0630989-credential-socket" pod "e73bb972-7cdd-4a6a-aae0-4345d0630989" (UID: "e73bb972-7cdd-4a6a-aae0-4345d0630989"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:29:23.693985 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:23.693964 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e73bb972-7cdd-4a6a-aae0-4345d0630989-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:23.693985 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:23.693983 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e73bb972-7cdd-4a6a-aae0-4345d0630989-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:23.694103 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:23.693992 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e73bb972-7cdd-4a6a-aae0-4345d0630989-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:23.694103 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:23.694000 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e73bb972-7cdd-4a6a-aae0-4345d0630989-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:23.694103 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:23.694010 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e73bb972-7cdd-4a6a-aae0-4345d0630989-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:23.695823 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:23.695785 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e73bb972-7cdd-4a6a-aae0-4345d0630989-istio-envoy" pod "e73bb972-7cdd-4a6a-aae0-4345d0630989" (UID: "e73bb972-7cdd-4a6a-aae0-4345d0630989"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:29:23.697085 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:23.697058 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e73bb972-7cdd-4a6a-aae0-4345d0630989-istio-token" pod "e73bb972-7cdd-4a6a-aae0-4345d0630989" (UID: "e73bb972-7cdd-4a6a-aae0-4345d0630989"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:29:23.697203 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:23.697086 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e73bb972-7cdd-4a6a-aae0-4345d0630989-istio-podinfo" pod "e73bb972-7cdd-4a6a-aae0-4345d0630989" (UID: "e73bb972-7cdd-4a6a-aae0-4345d0630989"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:29:23.697203 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:23.697107 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e73bb972-7cdd-4a6a-aae0-4345d0630989-kube-api-access-p6vkv" pod "e73bb972-7cdd-4a6a-aae0-4345d0630989" (UID: "e73bb972-7cdd-4a6a-aae0-4345d0630989"). InnerVolumeSpecName "kube-api-access-p6vkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:29:23.754366 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:23.754281 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3c5dbc/distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd"] Nov 03 14:29:23.776207 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:23.771703 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3c5dbc/premium-w4-3c5dbc-kserve-577d8db86-mc72z"] Nov 03 14:29:23.795099 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:23.795065 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e73bb972-7cdd-4a6a-aae0-4345d0630989-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:23.795099 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:23.795095 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e73bb972-7cdd-4a6a-aae0-4345d0630989-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:23.795099 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:23.795106 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e73bb972-7cdd-4a6a-aae0-4345d0630989-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:23.795099 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:23.795115 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6vkv\" (UniqueName: \"kubernetes.io/projected/e73bb972-7cdd-4a6a-aae0-4345d0630989-kube-api-access-p6vkv\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:23.935821 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:23.935785 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-420450a7-jp4nb"] Nov 03 14:29:23.943215 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:29:23.943187 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod062dae17_169d_4881_95ba_dd1d307af352.slice/crio-29df9b9060bd9573f120ccf1f9317abae2975fa1149fbc24dd14e575e089b952 WatchSource:0}: Error finding container 29df9b9060bd9573f120ccf1f9317abae2975fa1149fbc24dd14e575e089b952: Status 404 returned error can't find the container with id 29df9b9060bd9573f120ccf1f9317abae2975fa1149fbc24dd14e575e089b952 Nov 03 14:29:24.441263 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:24.441218 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-420450a7-jp4nb" event={"ID":"062dae17-169d-4881-95ba-dd1d307af352","Type":"ContainerStarted","Data":"1fa37934ff204b2bfcda7431e40b2f793c44579c54b68473d0ea7d6fb71630a0"} Nov 03 14:29:24.441558 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:24.441293 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-420450a7-jp4nb" event={"ID":"062dae17-169d-4881-95ba-dd1d307af352","Type":"ContainerStarted","Data":"29df9b9060bd9573f120ccf1f9317abae2975fa1149fbc24dd14e575e089b952"} Nov 03 14:29:24.442963 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:24.442938 5979 generic.go:356] "Generic (PLEG): container finished" podID="e73bb972-7cdd-4a6a-aae0-4345d0630989" containerID="1a5c1ff335b0329062bdb928f4584e35d2e0842f6f869d033edff69eabc799d8" exitCode=0 Nov 03 14:29:24.443072 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:24.442988 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv" Nov 03 14:29:24.443072 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:24.443013 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv" event={"ID":"e73bb972-7cdd-4a6a-aae0-4345d0630989","Type":"ContainerDied","Data":"1a5c1ff335b0329062bdb928f4584e35d2e0842f6f869d033edff69eabc799d8"} Nov 03 14:29:24.443072 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:24.443057 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv" event={"ID":"e73bb972-7cdd-4a6a-aae0-4345d0630989","Type":"ContainerDied","Data":"55c0615996ff2de819668b93628c591919491fd5d6e37ae5b77334af1723ed25"} Nov 03 14:29:24.443187 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:24.443078 5979 scope.go:122] "RemoveContainer" containerID="1a5c1ff335b0329062bdb928f4584e35d2e0842f6f869d033edff69eabc799d8" Nov 03 14:29:24.452979 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:24.452957 5979 scope.go:122] "RemoveContainer" containerID="34bffb888a3da97cb7456853f1a0f47fa4c8a1f0126f74a9075af4538585d433" Nov 03 14:29:24.461832 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:24.461806 5979 scope.go:122] "RemoveContainer" containerID="1a5c1ff335b0329062bdb928f4584e35d2e0842f6f869d033edff69eabc799d8" Nov 03 14:29:24.462130 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:29:24.462110 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a5c1ff335b0329062bdb928f4584e35d2e0842f6f869d033edff69eabc799d8\": container with ID starting with 1a5c1ff335b0329062bdb928f4584e35d2e0842f6f869d033edff69eabc799d8 not found: ID does not exist" containerID="1a5c1ff335b0329062bdb928f4584e35d2e0842f6f869d033edff69eabc799d8" Nov 03 14:29:24.462197 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:24.462141 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a5c1ff335b0329062bdb928f4584e35d2e0842f6f869d033edff69eabc799d8"} err="failed to get container status \"1a5c1ff335b0329062bdb928f4584e35d2e0842f6f869d033edff69eabc799d8\": rpc error: code = NotFound desc = could not find container \"1a5c1ff335b0329062bdb928f4584e35d2e0842f6f869d033edff69eabc799d8\": container with ID starting with 1a5c1ff335b0329062bdb928f4584e35d2e0842f6f869d033edff69eabc799d8 not found: ID does not exist" Nov 03 14:29:24.462197 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:24.462160 5979 scope.go:122] "RemoveContainer" containerID="34bffb888a3da97cb7456853f1a0f47fa4c8a1f0126f74a9075af4538585d433" Nov 03 14:29:24.462429 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:29:24.462411 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34bffb888a3da97cb7456853f1a0f47fa4c8a1f0126f74a9075af4538585d433\": container with ID starting with 34bffb888a3da97cb7456853f1a0f47fa4c8a1f0126f74a9075af4538585d433 not found: ID does not exist" containerID="34bffb888a3da97cb7456853f1a0f47fa4c8a1f0126f74a9075af4538585d433" Nov 03 14:29:24.462528 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:24.462440 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34bffb888a3da97cb7456853f1a0f47fa4c8a1f0126f74a9075af4538585d433"} err="failed to get container status \"34bffb888a3da97cb7456853f1a0f47fa4c8a1f0126f74a9075af4538585d433\": rpc error: code = NotFound desc = could not find container \"34bffb888a3da97cb7456853f1a0f47fa4c8a1f0126f74a9075af4538585d433\": container with ID starting with 34bffb888a3da97cb7456853f1a0f47fa4c8a1f0126f74a9075af4538585d433 not found: ID does not exist" Nov 03 14:29:24.549623 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:24.549564 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-420450a7-jp4nb" podStartSLOduration=1.5495469480000001 podStartE2EDuration="1.549546948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:29:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:29:24.549462381 +0000 UTC m=+5046.560150626" watchObservedRunningTime="2026-11-03 14:29:24.549546948 +0000 UTC m=+5046.560235190" Nov 03 14:29:24.658207 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:24.658162 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv"] Nov 03 14:29:24.661114 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:24.661090 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-efbdec-openshift-default-594cb76467-fjcpv"] Nov 03 14:29:25.451248 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:25.451210 5979 generic.go:356] "Generic (PLEG): container finished" podID="062dae17-169d-4881-95ba-dd1d307af352" containerID="1fa37934ff204b2bfcda7431e40b2f793c44579c54b68473d0ea7d6fb71630a0" exitCode=0 Nov 03 14:29:25.451444 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:25.451274 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-420450a7-jp4nb" event={"ID":"062dae17-169d-4881-95ba-dd1d307af352","Type":"ContainerDied","Data":"1fa37934ff204b2bfcda7431e40b2f793c44579c54b68473d0ea7d6fb71630a0"} Nov 03 14:29:26.115015 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:26.114984 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e73bb972-7cdd-4a6a-aae0-4345d0630989" path="/var/lib/kubelet/pods/e73bb972-7cdd-4a6a-aae0-4345d0630989/volumes" Nov 03 14:29:26.589094 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:26.589070 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-420450a7-jp4nb" Nov 03 14:29:26.625660 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:26.625631 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/062dae17-169d-4881-95ba-dd1d307af352-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/062dae17-169d-4881-95ba-dd1d307af352-maas-api-openshift-ca\") pod \"062dae17-169d-4881-95ba-dd1d307af352\" (UID: \"062dae17-169d-4881-95ba-dd1d307af352\") " Nov 03 14:29:26.625794 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:26.625691 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/062dae17-169d-4881-95ba-dd1d307af352-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/062dae17-169d-4881-95ba-dd1d307af352-maas-api-xks-ca\") pod \"062dae17-169d-4881-95ba-dd1d307af352\" (UID: \"062dae17-169d-4881-95ba-dd1d307af352\") " Nov 03 14:29:26.625849 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:26.625837 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/062dae17-169d-4881-95ba-dd1d307af352-kube-api-access-xcc26\" (UniqueName: \"kubernetes.io/projected/062dae17-169d-4881-95ba-dd1d307af352-kube-api-access-xcc26\") pod \"062dae17-169d-4881-95ba-dd1d307af352\" (UID: \"062dae17-169d-4881-95ba-dd1d307af352\") " Nov 03 14:29:26.626070 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:26.626043 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/062dae17-169d-4881-95ba-dd1d307af352-maas-api-openshift-ca" pod "062dae17-169d-4881-95ba-dd1d307af352" (UID: "062dae17-169d-4881-95ba-dd1d307af352"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:29:26.626255 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:26.626226 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/062dae17-169d-4881-95ba-dd1d307af352-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:26.627814 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:26.627791 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/062dae17-169d-4881-95ba-dd1d307af352-kube-api-access-xcc26" pod "062dae17-169d-4881-95ba-dd1d307af352" (UID: "062dae17-169d-4881-95ba-dd1d307af352"). InnerVolumeSpecName "kube-api-access-xcc26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:29:26.627814 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:26.627800 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/062dae17-169d-4881-95ba-dd1d307af352-maas-api-xks-ca" pod "062dae17-169d-4881-95ba-dd1d307af352" (UID: "062dae17-169d-4881-95ba-dd1d307af352"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:29:26.727097 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:26.727014 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcc26\" (UniqueName: \"kubernetes.io/projected/062dae17-169d-4881-95ba-dd1d307af352-kube-api-access-xcc26\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:26.727097 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:26.727044 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/062dae17-169d-4881-95ba-dd1d307af352-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:27.465074 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:27.465037 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-420450a7-jp4nb" event={"ID":"062dae17-169d-4881-95ba-dd1d307af352","Type":"ContainerDied","Data":"29df9b9060bd9573f120ccf1f9317abae2975fa1149fbc24dd14e575e089b952"} Nov 03 14:29:27.465074 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:27.465067 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-420450a7-jp4nb" Nov 03 14:29:27.465615 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:27.465088 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29df9b9060bd9573f120ccf1f9317abae2975fa1149fbc24dd14e575e089b952" Nov 03 14:29:32.340711 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:32.340673 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bba0b6c-rxsmw"] Nov 03 14:29:32.341140 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:32.341127 5979 cpu_manager.go:414] "containerMap: removing container" podUID="e73bb972-7cdd-4a6a-aae0-4345d0630989" containerName="istio-proxy" Nov 03 14:29:32.341189 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:32.341140 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e73bb972-7cdd-4a6a-aae0-4345d0630989" containerName="istio-proxy" Nov 03 14:29:32.341189 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:32.341165 5979 cpu_manager.go:414] "containerMap: removing container" podUID="062dae17-169d-4881-95ba-dd1d307af352" containerName="revoke-keys" Nov 03 14:29:32.341189 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:32.341170 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="062dae17-169d-4881-95ba-dd1d307af352" containerName="revoke-keys" Nov 03 14:29:32.341189 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:32.341188 5979 cpu_manager.go:414] "containerMap: removing container" podUID="e73bb972-7cdd-4a6a-aae0-4345d0630989" containerName="istio-proxy" Nov 03 14:29:32.341325 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:32.341193 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e73bb972-7cdd-4a6a-aae0-4345d0630989" containerName="istio-proxy" Nov 03 14:29:32.341325 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:32.341265 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="e73bb972-7cdd-4a6a-aae0-4345d0630989" containerName="istio-proxy" Nov 03 14:29:32.341325 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:32.341276 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="062dae17-169d-4881-95ba-dd1d307af352" containerName="revoke-keys" Nov 03 14:29:32.344644 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:32.344619 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bba0b6c-rxsmw" Nov 03 14:29:32.440125 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:32.440088 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:29:32.440379 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:32.440364 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 14:29:32.443046 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:32.443021 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bba0b6c-rxsmw"] Nov 03 14:29:32.487268 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:32.487235 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6134692-b548-43a9-b097-6bce1f7ed9f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2bba0b6c-rxsmw\" (UID: \"e6134692-b548-43a9-b097-6bce1f7ed9f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bba0b6c-rxsmw" Nov 03 14:29:32.487430 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:32.487384 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vg8h\" (UniqueName: \"kubernetes.io/projected/e6134692-b548-43a9-b097-6bce1f7ed9f9-kube-api-access-5vg8h\") pod \"maas-api-revoke-sub-2bba0b6c-rxsmw\" (UID: \"e6134692-b548-43a9-b097-6bce1f7ed9f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bba0b6c-rxsmw" Nov 03 14:29:32.487430 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:32.487411 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6134692-b548-43a9-b097-6bce1f7ed9f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2bba0b6c-rxsmw\" (UID: \"e6134692-b548-43a9-b097-6bce1f7ed9f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bba0b6c-rxsmw" Nov 03 14:29:32.588610 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:32.588570 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6134692-b548-43a9-b097-6bce1f7ed9f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2bba0b6c-rxsmw\" (UID: \"e6134692-b548-43a9-b097-6bce1f7ed9f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bba0b6c-rxsmw" Nov 03 14:29:32.588837 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:32.588714 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vg8h\" (UniqueName: \"kubernetes.io/projected/e6134692-b548-43a9-b097-6bce1f7ed9f9-kube-api-access-5vg8h\") pod \"maas-api-revoke-sub-2bba0b6c-rxsmw\" (UID: \"e6134692-b548-43a9-b097-6bce1f7ed9f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bba0b6c-rxsmw" Nov 03 14:29:32.588837 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:32.588750 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6134692-b548-43a9-b097-6bce1f7ed9f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2bba0b6c-rxsmw\" (UID: \"e6134692-b548-43a9-b097-6bce1f7ed9f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bba0b6c-rxsmw" Nov 03 14:29:32.589382 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:32.589354 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6134692-b548-43a9-b097-6bce1f7ed9f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2bba0b6c-rxsmw\" (UID: \"e6134692-b548-43a9-b097-6bce1f7ed9f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bba0b6c-rxsmw" Nov 03 14:29:32.591035 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:32.590986 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6134692-b548-43a9-b097-6bce1f7ed9f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2bba0b6c-rxsmw\" (UID: \"e6134692-b548-43a9-b097-6bce1f7ed9f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bba0b6c-rxsmw" Nov 03 14:29:32.642104 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:32.642069 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vg8h\" (UniqueName: \"kubernetes.io/projected/e6134692-b548-43a9-b097-6bce1f7ed9f9-kube-api-access-5vg8h\") pod \"maas-api-revoke-sub-2bba0b6c-rxsmw\" (UID: \"e6134692-b548-43a9-b097-6bce1f7ed9f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bba0b6c-rxsmw" Nov 03 14:29:32.657603 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:32.657572 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bba0b6c-rxsmw" Nov 03 14:29:33.036355 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:33.036321 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bba0b6c-rxsmw"] Nov 03 14:29:33.043423 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:29:33.043394 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6134692_b548_43a9_b097_6bce1f7ed9f9.slice/crio-ecc773bda50520e0e768e1bd7577f89742de619f2cbff5c21f1a6834dc5cadb2 WatchSource:0}: Error finding container ecc773bda50520e0e768e1bd7577f89742de619f2cbff5c21f1a6834dc5cadb2: Status 404 returned error can't find the container with id ecc773bda50520e0e768e1bd7577f89742de619f2cbff5c21f1a6834dc5cadb2 Nov 03 14:29:33.498818 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:33.498778 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bba0b6c-rxsmw" event={"ID":"e6134692-b548-43a9-b097-6bce1f7ed9f9","Type":"ContainerStarted","Data":"3bfe311038d0eaecbc8ce66e3024b3833c4ee147728d54d8e68f7e57c9a392bf"} Nov 03 14:29:33.499199 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:33.498836 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bba0b6c-rxsmw" event={"ID":"e6134692-b548-43a9-b097-6bce1f7ed9f9","Type":"ContainerStarted","Data":"ecc773bda50520e0e768e1bd7577f89742de619f2cbff5c21f1a6834dc5cadb2"} Nov 03 14:29:33.652383 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:33.652335 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bba0b6c-rxsmw" podStartSLOduration=1.652323026 podStartE2EDuration="1.652323026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:29:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:29:33.632696459 +0000 UTC m=+5055.643384699" watchObservedRunningTime="2026-11-03 14:29:33.652323026 +0000 UTC m=+5055.663011268" Nov 03 14:29:34.506341 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:34.506307 5979 generic.go:356] "Generic (PLEG): container finished" podID="e6134692-b548-43a9-b097-6bce1f7ed9f9" containerID="3bfe311038d0eaecbc8ce66e3024b3833c4ee147728d54d8e68f7e57c9a392bf" exitCode=0 Nov 03 14:29:34.506782 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:34.506386 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bba0b6c-rxsmw" event={"ID":"e6134692-b548-43a9-b097-6bce1f7ed9f9","Type":"ContainerDied","Data":"3bfe311038d0eaecbc8ce66e3024b3833c4ee147728d54d8e68f7e57c9a392bf"} Nov 03 14:29:34.910512 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:34.910454 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69d6d864-lqm5g"] Nov 03 14:29:34.911039 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:34.911024 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="e73bb972-7cdd-4a6a-aae0-4345d0630989" containerName="istio-proxy" Nov 03 14:29:34.914281 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:34.914257 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d6d864-lqm5g" Nov 03 14:29:34.971723 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:34.971688 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69d6d864-lqm5g"] Nov 03 14:29:35.025724 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:35.025687 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8d3d4f9-8d57-4ff2-99fa-57ffeedfea04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69d6d864-lqm5g\" (UID: \"a8d3d4f9-8d57-4ff2-99fa-57ffeedfea04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d6d864-lqm5g" Nov 03 14:29:35.025919 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:35.025816 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8d3d4f9-8d57-4ff2-99fa-57ffeedfea04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69d6d864-lqm5g\" (UID: \"a8d3d4f9-8d57-4ff2-99fa-57ffeedfea04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d6d864-lqm5g" Nov 03 14:29:35.025919 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:35.025863 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krdfg\" (UniqueName: \"kubernetes.io/projected/a8d3d4f9-8d57-4ff2-99fa-57ffeedfea04-kube-api-access-krdfg\") pod \"maas-api-revoke-sub-69d6d864-lqm5g\" (UID: \"a8d3d4f9-8d57-4ff2-99fa-57ffeedfea04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d6d864-lqm5g" Nov 03 14:29:35.126852 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:35.126807 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8d3d4f9-8d57-4ff2-99fa-57ffeedfea04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69d6d864-lqm5g\" (UID: \"a8d3d4f9-8d57-4ff2-99fa-57ffeedfea04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d6d864-lqm5g" Nov 03 14:29:35.127063 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:35.126878 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krdfg\" (UniqueName: \"kubernetes.io/projected/a8d3d4f9-8d57-4ff2-99fa-57ffeedfea04-kube-api-access-krdfg\") pod \"maas-api-revoke-sub-69d6d864-lqm5g\" (UID: \"a8d3d4f9-8d57-4ff2-99fa-57ffeedfea04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d6d864-lqm5g" Nov 03 14:29:35.127063 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:35.126955 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8d3d4f9-8d57-4ff2-99fa-57ffeedfea04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69d6d864-lqm5g\" (UID: \"a8d3d4f9-8d57-4ff2-99fa-57ffeedfea04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d6d864-lqm5g" Nov 03 14:29:35.127628 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:35.127608 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8d3d4f9-8d57-4ff2-99fa-57ffeedfea04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69d6d864-lqm5g\" (UID: \"a8d3d4f9-8d57-4ff2-99fa-57ffeedfea04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d6d864-lqm5g" Nov 03 14:29:35.129162 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:35.129140 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8d3d4f9-8d57-4ff2-99fa-57ffeedfea04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69d6d864-lqm5g\" (UID: \"a8d3d4f9-8d57-4ff2-99fa-57ffeedfea04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d6d864-lqm5g" Nov 03 14:29:35.180533 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:35.180421 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krdfg\" (UniqueName: \"kubernetes.io/projected/a8d3d4f9-8d57-4ff2-99fa-57ffeedfea04-kube-api-access-krdfg\") pod \"maas-api-revoke-sub-69d6d864-lqm5g\" (UID: \"a8d3d4f9-8d57-4ff2-99fa-57ffeedfea04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d6d864-lqm5g" Nov 03 14:29:35.227172 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:35.227131 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d6d864-lqm5g" Nov 03 14:29:35.599606 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:35.599567 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69d6d864-lqm5g"] Nov 03 14:29:35.609637 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:29:35.609595 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8d3d4f9_8d57_4ff2_99fa_57ffeedfea04.slice/crio-f5827a41acdbe67b3cfb3c88b7beb26fe1ec8840a29e1c8017cf6e7f59f4d8fb WatchSource:0}: Error finding container f5827a41acdbe67b3cfb3c88b7beb26fe1ec8840a29e1c8017cf6e7f59f4d8fb: Status 404 returned error can't find the container with id f5827a41acdbe67b3cfb3c88b7beb26fe1ec8840a29e1c8017cf6e7f59f4d8fb Nov 03 14:29:35.674850 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:35.674821 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bba0b6c-rxsmw" Nov 03 14:29:35.733566 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:35.733528 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6134692-b548-43a9-b097-6bce1f7ed9f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6134692-b548-43a9-b097-6bce1f7ed9f9-maas-api-xks-ca\") pod \"e6134692-b548-43a9-b097-6bce1f7ed9f9\" (UID: \"e6134692-b548-43a9-b097-6bce1f7ed9f9\") " Nov 03 14:29:35.733758 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:35.733646 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6134692-b548-43a9-b097-6bce1f7ed9f9-kube-api-access-5vg8h\" (UniqueName: \"kubernetes.io/projected/e6134692-b548-43a9-b097-6bce1f7ed9f9-kube-api-access-5vg8h\") pod \"e6134692-b548-43a9-b097-6bce1f7ed9f9\" (UID: \"e6134692-b548-43a9-b097-6bce1f7ed9f9\") " Nov 03 14:29:35.733758 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:35.733732 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6134692-b548-43a9-b097-6bce1f7ed9f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6134692-b548-43a9-b097-6bce1f7ed9f9-maas-api-openshift-ca\") pod \"e6134692-b548-43a9-b097-6bce1f7ed9f9\" (UID: \"e6134692-b548-43a9-b097-6bce1f7ed9f9\") " Nov 03 14:29:35.734097 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:35.734072 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6134692-b548-43a9-b097-6bce1f7ed9f9-maas-api-openshift-ca" pod "e6134692-b548-43a9-b097-6bce1f7ed9f9" (UID: "e6134692-b548-43a9-b097-6bce1f7ed9f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:29:35.735666 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:35.735645 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6134692-b548-43a9-b097-6bce1f7ed9f9-maas-api-xks-ca" pod "e6134692-b548-43a9-b097-6bce1f7ed9f9" (UID: "e6134692-b548-43a9-b097-6bce1f7ed9f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:29:35.735777 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:35.735717 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6134692-b548-43a9-b097-6bce1f7ed9f9-kube-api-access-5vg8h" pod "e6134692-b548-43a9-b097-6bce1f7ed9f9" (UID: "e6134692-b548-43a9-b097-6bce1f7ed9f9"). InnerVolumeSpecName "kube-api-access-5vg8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:29:35.835542 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:35.835435 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vg8h\" (UniqueName: \"kubernetes.io/projected/e6134692-b548-43a9-b097-6bce1f7ed9f9-kube-api-access-5vg8h\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:35.835542 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:35.835463 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6134692-b548-43a9-b097-6bce1f7ed9f9-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:35.835542 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:35.835499 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6134692-b548-43a9-b097-6bce1f7ed9f9-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:36.520846 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:36.520811 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d6d864-lqm5g" event={"ID":"a8d3d4f9-8d57-4ff2-99fa-57ffeedfea04","Type":"ContainerStarted","Data":"7486f3f6396e27c5c863fb860ca99043da5eb539411b09e5ca3abbfb00acea5b"} Nov 03 14:29:36.520973 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:36.520882 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d6d864-lqm5g" event={"ID":"a8d3d4f9-8d57-4ff2-99fa-57ffeedfea04","Type":"ContainerStarted","Data":"f5827a41acdbe67b3cfb3c88b7beb26fe1ec8840a29e1c8017cf6e7f59f4d8fb"} Nov 03 14:29:36.522123 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:36.522102 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bba0b6c-rxsmw" event={"ID":"e6134692-b548-43a9-b097-6bce1f7ed9f9","Type":"ContainerDied","Data":"ecc773bda50520e0e768e1bd7577f89742de619f2cbff5c21f1a6834dc5cadb2"} Nov 03 14:29:36.522123 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:36.522118 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bba0b6c-rxsmw" Nov 03 14:29:36.522264 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:36.522139 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ecc773bda50520e0e768e1bd7577f89742de619f2cbff5c21f1a6834dc5cadb2" Nov 03 14:29:37.233569 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:29:37.233528 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:29:37.529315 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:37.529224 5979 generic.go:356] "Generic (PLEG): container finished" podID="a8d3d4f9-8d57-4ff2-99fa-57ffeedfea04" containerID="7486f3f6396e27c5c863fb860ca99043da5eb539411b09e5ca3abbfb00acea5b" exitCode=0 Nov 03 14:29:37.529526 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:37.529310 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d6d864-lqm5g" event={"ID":"a8d3d4f9-8d57-4ff2-99fa-57ffeedfea04","Type":"ContainerDied","Data":"7486f3f6396e27c5c863fb860ca99043da5eb539411b09e5ca3abbfb00acea5b"} Nov 03 14:29:37.691576 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:37.691546 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d6d864-lqm5g" Nov 03 14:29:37.758719 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:37.758691 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8d3d4f9-8d57-4ff2-99fa-57ffeedfea04-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8d3d4f9-8d57-4ff2-99fa-57ffeedfea04-maas-api-xks-ca\") pod \"a8d3d4f9-8d57-4ff2-99fa-57ffeedfea04\" (UID: \"a8d3d4f9-8d57-4ff2-99fa-57ffeedfea04\") " Nov 03 14:29:37.758926 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:37.758745 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8d3d4f9-8d57-4ff2-99fa-57ffeedfea04-kube-api-access-krdfg\" (UniqueName: \"kubernetes.io/projected/a8d3d4f9-8d57-4ff2-99fa-57ffeedfea04-kube-api-access-krdfg\") pod \"a8d3d4f9-8d57-4ff2-99fa-57ffeedfea04\" (UID: \"a8d3d4f9-8d57-4ff2-99fa-57ffeedfea04\") " Nov 03 14:29:37.758926 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:37.758793 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8d3d4f9-8d57-4ff2-99fa-57ffeedfea04-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8d3d4f9-8d57-4ff2-99fa-57ffeedfea04-maas-api-openshift-ca\") pod \"a8d3d4f9-8d57-4ff2-99fa-57ffeedfea04\" (UID: \"a8d3d4f9-8d57-4ff2-99fa-57ffeedfea04\") " Nov 03 14:29:37.759238 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:37.759217 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8d3d4f9-8d57-4ff2-99fa-57ffeedfea04-maas-api-openshift-ca" pod "a8d3d4f9-8d57-4ff2-99fa-57ffeedfea04" (UID: "a8d3d4f9-8d57-4ff2-99fa-57ffeedfea04"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:29:37.760793 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:37.760763 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8d3d4f9-8d57-4ff2-99fa-57ffeedfea04-maas-api-xks-ca" pod "a8d3d4f9-8d57-4ff2-99fa-57ffeedfea04" (UID: "a8d3d4f9-8d57-4ff2-99fa-57ffeedfea04"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:29:37.760897 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:37.760876 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8d3d4f9-8d57-4ff2-99fa-57ffeedfea04-kube-api-access-krdfg" pod "a8d3d4f9-8d57-4ff2-99fa-57ffeedfea04" (UID: "a8d3d4f9-8d57-4ff2-99fa-57ffeedfea04"). InnerVolumeSpecName "kube-api-access-krdfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:29:37.859994 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:37.859909 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8d3d4f9-8d57-4ff2-99fa-57ffeedfea04-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:37.859994 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:37.859939 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krdfg\" (UniqueName: \"kubernetes.io/projected/a8d3d4f9-8d57-4ff2-99fa-57ffeedfea04-kube-api-access-krdfg\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:37.859994 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:37.859957 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8d3d4f9-8d57-4ff2-99fa-57ffeedfea04-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:38.536752 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:38.536714 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d6d864-lqm5g" event={"ID":"a8d3d4f9-8d57-4ff2-99fa-57ffeedfea04","Type":"ContainerDied","Data":"f5827a41acdbe67b3cfb3c88b7beb26fe1ec8840a29e1c8017cf6e7f59f4d8fb"} Nov 03 14:29:38.537159 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:38.536764 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69d6d864-lqm5g" Nov 03 14:29:38.537159 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:38.536801 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5827a41acdbe67b3cfb3c88b7beb26fe1ec8840a29e1c8017cf6e7f59f4d8fb" Nov 03 14:29:38.779536 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:38.779489 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd" podUID="a4b4df54-0aac-419d-8bb2-3258267bc088" containerName="main" containerID="cri-o://7e0b9b76152fbf6f744744032118cefd3ac91dc36b339256ad93539ede57dd44" gracePeriod=45 Nov 03 14:29:38.796876 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:38.796841 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-3c5dbc/premium-w4-3c5dbc-kserve-577d8db86-mc72z" podUID="64d52ec5-fb72-4616-8d20-8b90043af828" containerName="main" containerID="cri-o://161695f2a055d41eddba2c4f40a2b949bfe9883c11fa05945c528a97f0a75828" gracePeriod=45 Nov 03 14:29:39.074094 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:39.074068 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd" Nov 03 14:29:39.076848 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:39.076829 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3c5dbc/premium-w4-3c5dbc-kserve-577d8db86-mc72z" Nov 03 14:29:39.173062 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:39.173022 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64d52ec5-fb72-4616-8d20-8b90043af828-kube-api-access-n6t2v\" (UniqueName: \"kubernetes.io/projected/64d52ec5-fb72-4616-8d20-8b90043af828-kube-api-access-n6t2v\") pod \"64d52ec5-fb72-4616-8d20-8b90043af828\" (UID: \"64d52ec5-fb72-4616-8d20-8b90043af828\") " Nov 03 14:29:39.173287 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:39.173083 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4b4df54-0aac-419d-8bb2-3258267bc088-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4b4df54-0aac-419d-8bb2-3258267bc088-tmp-dir\") pod \"a4b4df54-0aac-419d-8bb2-3258267bc088\" (UID: \"a4b4df54-0aac-419d-8bb2-3258267bc088\") " Nov 03 14:29:39.173287 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:39.173154 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4b4df54-0aac-419d-8bb2-3258267bc088-tls-certs\" (UniqueName: \"kubernetes.io/secret/a4b4df54-0aac-419d-8bb2-3258267bc088-tls-certs\") pod \"a4b4df54-0aac-419d-8bb2-3258267bc088\" (UID: \"a4b4df54-0aac-419d-8bb2-3258267bc088\") " Nov 03 14:29:39.173287 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:39.173173 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64d52ec5-fb72-4616-8d20-8b90043af828-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64d52ec5-fb72-4616-8d20-8b90043af828-model-cache\") pod \"64d52ec5-fb72-4616-8d20-8b90043af828\" (UID: \"64d52ec5-fb72-4616-8d20-8b90043af828\") " Nov 03 14:29:39.173287 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:39.173232 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64d52ec5-fb72-4616-8d20-8b90043af828-home\" (UniqueName: \"kubernetes.io/empty-dir/64d52ec5-fb72-4616-8d20-8b90043af828-home\") pod \"64d52ec5-fb72-4616-8d20-8b90043af828\" (UID: \"64d52ec5-fb72-4616-8d20-8b90043af828\") " Nov 03 14:29:39.173597 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:39.173290 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64d52ec5-fb72-4616-8d20-8b90043af828-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64d52ec5-fb72-4616-8d20-8b90043af828-tmp-dir\") pod \"64d52ec5-fb72-4616-8d20-8b90043af828\" (UID: \"64d52ec5-fb72-4616-8d20-8b90043af828\") " Nov 03 14:29:39.173597 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:39.173316 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64d52ec5-fb72-4616-8d20-8b90043af828-dshm\" (UniqueName: \"kubernetes.io/empty-dir/64d52ec5-fb72-4616-8d20-8b90043af828-dshm\") pod \"64d52ec5-fb72-4616-8d20-8b90043af828\" (UID: \"64d52ec5-fb72-4616-8d20-8b90043af828\") " Nov 03 14:29:39.173597 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:39.173338 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4b4df54-0aac-419d-8bb2-3258267bc088-tmp-dir" pod "a4b4df54-0aac-419d-8bb2-3258267bc088" (UID: "a4b4df54-0aac-419d-8bb2-3258267bc088"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:29:39.173597 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:39.173350 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64d52ec5-fb72-4616-8d20-8b90043af828-tls-certs\" (UniqueName: \"kubernetes.io/secret/64d52ec5-fb72-4616-8d20-8b90043af828-tls-certs\") pod \"64d52ec5-fb72-4616-8d20-8b90043af828\" (UID: \"64d52ec5-fb72-4616-8d20-8b90043af828\") " Nov 03 14:29:39.173597 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:39.173423 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4b4df54-0aac-419d-8bb2-3258267bc088-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4b4df54-0aac-419d-8bb2-3258267bc088-model-cache\") pod \"a4b4df54-0aac-419d-8bb2-3258267bc088\" (UID: \"a4b4df54-0aac-419d-8bb2-3258267bc088\") " Nov 03 14:29:39.173597 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:39.173535 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4b4df54-0aac-419d-8bb2-3258267bc088-home\" (UniqueName: \"kubernetes.io/empty-dir/a4b4df54-0aac-419d-8bb2-3258267bc088-home\") pod \"a4b4df54-0aac-419d-8bb2-3258267bc088\" (UID: \"a4b4df54-0aac-419d-8bb2-3258267bc088\") " Nov 03 14:29:39.173597 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:39.173544 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64d52ec5-fb72-4616-8d20-8b90043af828-model-cache" pod "64d52ec5-fb72-4616-8d20-8b90043af828" (UID: "64d52ec5-fb72-4616-8d20-8b90043af828"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:29:39.173597 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:39.173568 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4b4df54-0aac-419d-8bb2-3258267bc088-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4b4df54-0aac-419d-8bb2-3258267bc088-dshm\") pod \"a4b4df54-0aac-419d-8bb2-3258267bc088\" (UID: \"a4b4df54-0aac-419d-8bb2-3258267bc088\") " Nov 03 14:29:39.174046 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:39.173559 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64d52ec5-fb72-4616-8d20-8b90043af828-home" pod "64d52ec5-fb72-4616-8d20-8b90043af828" (UID: "64d52ec5-fb72-4616-8d20-8b90043af828"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:29:39.174046 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:39.173637 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4b4df54-0aac-419d-8bb2-3258267bc088-kube-api-access-87shq\" (UniqueName: \"kubernetes.io/projected/a4b4df54-0aac-419d-8bb2-3258267bc088-kube-api-access-87shq\") pod \"a4b4df54-0aac-419d-8bb2-3258267bc088\" (UID: \"a4b4df54-0aac-419d-8bb2-3258267bc088\") " Nov 03 14:29:39.174046 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:39.173615 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64d52ec5-fb72-4616-8d20-8b90043af828-tmp-dir" pod "64d52ec5-fb72-4616-8d20-8b90043af828" (UID: "64d52ec5-fb72-4616-8d20-8b90043af828"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:29:39.174046 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:39.173774 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4b4df54-0aac-419d-8bb2-3258267bc088-model-cache" pod "a4b4df54-0aac-419d-8bb2-3258267bc088" (UID: "a4b4df54-0aac-419d-8bb2-3258267bc088"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:29:39.174046 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:39.173857 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4b4df54-0aac-419d-8bb2-3258267bc088-home" pod "a4b4df54-0aac-419d-8bb2-3258267bc088" (UID: "a4b4df54-0aac-419d-8bb2-3258267bc088"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:29:39.174325 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:39.174134 5979 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64d52ec5-fb72-4616-8d20-8b90043af828-model-cache\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:39.174325 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:39.174155 5979 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64d52ec5-fb72-4616-8d20-8b90043af828-home\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:39.174325 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:39.174196 5979 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64d52ec5-fb72-4616-8d20-8b90043af828-tmp-dir\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:39.174325 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:39.174211 5979 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4b4df54-0aac-419d-8bb2-3258267bc088-model-cache\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:39.174325 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:39.174225 5979 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4b4df54-0aac-419d-8bb2-3258267bc088-home\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:39.174325 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:39.174239 5979 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4b4df54-0aac-419d-8bb2-3258267bc088-tmp-dir\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:39.176246 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:39.176218 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64d52ec5-fb72-4616-8d20-8b90043af828-dshm" pod "64d52ec5-fb72-4616-8d20-8b90043af828" (UID: "64d52ec5-fb72-4616-8d20-8b90043af828"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:29:39.176365 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:39.176257 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4b4df54-0aac-419d-8bb2-3258267bc088-kube-api-access-87shq" pod "a4b4df54-0aac-419d-8bb2-3258267bc088" (UID: "a4b4df54-0aac-419d-8bb2-3258267bc088"). InnerVolumeSpecName "kube-api-access-87shq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:29:39.176365 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:39.176276 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4b4df54-0aac-419d-8bb2-3258267bc088-tls-certs" pod "a4b4df54-0aac-419d-8bb2-3258267bc088" (UID: "a4b4df54-0aac-419d-8bb2-3258267bc088"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:29:39.176365 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:39.176301 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64d52ec5-fb72-4616-8d20-8b90043af828-tls-certs" pod "64d52ec5-fb72-4616-8d20-8b90043af828" (UID: "64d52ec5-fb72-4616-8d20-8b90043af828"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:29:39.176365 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:39.176314 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4b4df54-0aac-419d-8bb2-3258267bc088-dshm" pod "a4b4df54-0aac-419d-8bb2-3258267bc088" (UID: "a4b4df54-0aac-419d-8bb2-3258267bc088"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:29:39.176546 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:39.176390 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64d52ec5-fb72-4616-8d20-8b90043af828-kube-api-access-n6t2v" pod "64d52ec5-fb72-4616-8d20-8b90043af828" (UID: "64d52ec5-fb72-4616-8d20-8b90043af828"). InnerVolumeSpecName "kube-api-access-n6t2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:29:39.276068 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:39.276001 5979 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4b4df54-0aac-419d-8bb2-3258267bc088-tls-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:39.276068 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:39.276052 5979 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64d52ec5-fb72-4616-8d20-8b90043af828-dshm\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:39.276313 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:39.276077 5979 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64d52ec5-fb72-4616-8d20-8b90043af828-tls-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:39.276313 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:39.276089 5979 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4b4df54-0aac-419d-8bb2-3258267bc088-dshm\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:39.276313 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:39.276103 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87shq\" (UniqueName: \"kubernetes.io/projected/a4b4df54-0aac-419d-8bb2-3258267bc088-kube-api-access-87shq\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:39.276313 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:39.276124 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6t2v\" (UniqueName: \"kubernetes.io/projected/64d52ec5-fb72-4616-8d20-8b90043af828-kube-api-access-n6t2v\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:39.543863 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:39.543827 5979 generic.go:356] "Generic (PLEG): container finished" podID="64d52ec5-fb72-4616-8d20-8b90043af828" containerID="161695f2a055d41eddba2c4f40a2b949bfe9883c11fa05945c528a97f0a75828" exitCode=0 Nov 03 14:29:39.544298 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:39.543869 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3c5dbc/premium-w4-3c5dbc-kserve-577d8db86-mc72z" Nov 03 14:29:39.544298 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:39.543907 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3c5dbc/premium-w4-3c5dbc-kserve-577d8db86-mc72z" event={"ID":"64d52ec5-fb72-4616-8d20-8b90043af828","Type":"ContainerDied","Data":"161695f2a055d41eddba2c4f40a2b949bfe9883c11fa05945c528a97f0a75828"} Nov 03 14:29:39.544298 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:39.543961 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3c5dbc/premium-w4-3c5dbc-kserve-577d8db86-mc72z" event={"ID":"64d52ec5-fb72-4616-8d20-8b90043af828","Type":"ContainerDied","Data":"5dd1a6c1f8a24f8ac6b784bc52834e02ef996fcfa4117ebf76ab8d20456fc836"} Nov 03 14:29:39.544298 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:39.543986 5979 scope.go:122] "RemoveContainer" containerID="161695f2a055d41eddba2c4f40a2b949bfe9883c11fa05945c528a97f0a75828" Nov 03 14:29:39.545342 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:39.545316 5979 generic.go:356] "Generic (PLEG): container finished" podID="a4b4df54-0aac-419d-8bb2-3258267bc088" containerID="7e0b9b76152fbf6f744744032118cefd3ac91dc36b339256ad93539ede57dd44" exitCode=0 Nov 03 14:29:39.545342 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:39.545346 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd" event={"ID":"a4b4df54-0aac-419d-8bb2-3258267bc088","Type":"ContainerDied","Data":"7e0b9b76152fbf6f744744032118cefd3ac91dc36b339256ad93539ede57dd44"} Nov 03 14:29:39.545522 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:39.545370 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd" Nov 03 14:29:39.545522 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:39.545375 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3c5dbc/distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd" event={"ID":"a4b4df54-0aac-419d-8bb2-3258267bc088","Type":"ContainerDied","Data":"e2b4e6437bbf2f3ae387c04570c3ba824a4abc75f30cd9b28bade1a5caca5d14"} Nov 03 14:29:39.554671 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:39.554653 5979 scope.go:122] "RemoveContainer" containerID="161695f2a055d41eddba2c4f40a2b949bfe9883c11fa05945c528a97f0a75828" Nov 03 14:29:39.554962 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:29:39.554931 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"161695f2a055d41eddba2c4f40a2b949bfe9883c11fa05945c528a97f0a75828\": container with ID starting with 161695f2a055d41eddba2c4f40a2b949bfe9883c11fa05945c528a97f0a75828 not found: ID does not exist" containerID="161695f2a055d41eddba2c4f40a2b949bfe9883c11fa05945c528a97f0a75828" Nov 03 14:29:39.555042 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:39.554975 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"161695f2a055d41eddba2c4f40a2b949bfe9883c11fa05945c528a97f0a75828"} err="failed to get container status \"161695f2a055d41eddba2c4f40a2b949bfe9883c11fa05945c528a97f0a75828\": rpc error: code = NotFound desc = could not find container \"161695f2a055d41eddba2c4f40a2b949bfe9883c11fa05945c528a97f0a75828\": container with ID starting with 161695f2a055d41eddba2c4f40a2b949bfe9883c11fa05945c528a97f0a75828 not found: ID does not exist" Nov 03 14:29:39.555042 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:39.554996 5979 scope.go:122] "RemoveContainer" containerID="7e0b9b76152fbf6f744744032118cefd3ac91dc36b339256ad93539ede57dd44" Nov 03 14:29:39.563158 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:39.563140 5979 scope.go:122] "RemoveContainer" containerID="7e0b9b76152fbf6f744744032118cefd3ac91dc36b339256ad93539ede57dd44" Nov 03 14:29:39.563428 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:29:39.563410 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e0b9b76152fbf6f744744032118cefd3ac91dc36b339256ad93539ede57dd44\": container with ID starting with 7e0b9b76152fbf6f744744032118cefd3ac91dc36b339256ad93539ede57dd44 not found: ID does not exist" containerID="7e0b9b76152fbf6f744744032118cefd3ac91dc36b339256ad93539ede57dd44" Nov 03 14:29:39.563491 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:39.563435 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e0b9b76152fbf6f744744032118cefd3ac91dc36b339256ad93539ede57dd44"} err="failed to get container status \"7e0b9b76152fbf6f744744032118cefd3ac91dc36b339256ad93539ede57dd44\": rpc error: code = NotFound desc = could not find container \"7e0b9b76152fbf6f744744032118cefd3ac91dc36b339256ad93539ede57dd44\": container with ID starting with 7e0b9b76152fbf6f744744032118cefd3ac91dc36b339256ad93539ede57dd44 not found: ID does not exist" Nov 03 14:29:39.698059 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:39.698020 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3c5dbc/premium-w4-3c5dbc-kserve-577d8db86-mc72z"] Nov 03 14:29:39.701803 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:39.701776 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-3c5dbc/premium-w4-3c5dbc-kserve-577d8db86-mc72z"] Nov 03 14:29:39.803019 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:39.802950 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3c5dbc/distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd"] Nov 03 14:29:39.807430 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:39.807405 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-3c5dbc/distinct-2-w4-3c5dbc-kserve-74bf74fc9d-bfhzd"] Nov 03 14:29:39.965238 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:39.965195 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-367381-55bcdb6c57-6kxd4"] Nov 03 14:29:39.965506 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:39.965456 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-367381-55bcdb6c57-6kxd4" podUID="80a05ef5-573e-4b3a-aaaf-6c91d6cba523" containerName="maas-api" containerID="cri-o://d8bcca5fb8b1879744bfebdaea6f6157f5967f83b66df8718ecbd647e5b44bb2" gracePeriod=30 Nov 03 14:29:40.114583 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:40.114548 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64d52ec5-fb72-4616-8d20-8b90043af828" path="/var/lib/kubelet/pods/64d52ec5-fb72-4616-8d20-8b90043af828/volumes" Nov 03 14:29:40.115058 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:40.115039 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4b4df54-0aac-419d-8bb2-3258267bc088" path="/var/lib/kubelet/pods/a4b4df54-0aac-419d-8bb2-3258267bc088/volumes" Nov 03 14:29:40.227906 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:40.227878 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-367381-55bcdb6c57-6kxd4" Nov 03 14:29:40.394662 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:40.394629 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80a05ef5-573e-4b3a-aaaf-6c91d6cba523-metrics-certs\" (UniqueName: \"kubernetes.io/secret/80a05ef5-573e-4b3a-aaaf-6c91d6cba523-metrics-certs\") pod \"80a05ef5-573e-4b3a-aaaf-6c91d6cba523\" (UID: \"80a05ef5-573e-4b3a-aaaf-6c91d6cba523\") " Nov 03 14:29:40.394841 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:40.394722 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80a05ef5-573e-4b3a-aaaf-6c91d6cba523-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80a05ef5-573e-4b3a-aaaf-6c91d6cba523-maas-api-tls\") pod \"80a05ef5-573e-4b3a-aaaf-6c91d6cba523\" (UID: \"80a05ef5-573e-4b3a-aaaf-6c91d6cba523\") " Nov 03 14:29:40.394841 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:40.394757 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80a05ef5-573e-4b3a-aaaf-6c91d6cba523-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80a05ef5-573e-4b3a-aaaf-6c91d6cba523-openshift-service-ca\") pod \"80a05ef5-573e-4b3a-aaaf-6c91d6cba523\" (UID: \"80a05ef5-573e-4b3a-aaaf-6c91d6cba523\") " Nov 03 14:29:40.394841 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:40.394798 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80a05ef5-573e-4b3a-aaaf-6c91d6cba523-kube-api-access-2kjmx\" (UniqueName: \"kubernetes.io/projected/80a05ef5-573e-4b3a-aaaf-6c91d6cba523-kube-api-access-2kjmx\") pod \"80a05ef5-573e-4b3a-aaaf-6c91d6cba523\" (UID: \"80a05ef5-573e-4b3a-aaaf-6c91d6cba523\") " Nov 03 14:29:40.395209 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:40.395179 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80a05ef5-573e-4b3a-aaaf-6c91d6cba523-openshift-service-ca" pod "80a05ef5-573e-4b3a-aaaf-6c91d6cba523" (UID: "80a05ef5-573e-4b3a-aaaf-6c91d6cba523"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:29:40.396875 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:40.396851 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80a05ef5-573e-4b3a-aaaf-6c91d6cba523-maas-api-tls" pod "80a05ef5-573e-4b3a-aaaf-6c91d6cba523" (UID: "80a05ef5-573e-4b3a-aaaf-6c91d6cba523"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:29:40.396974 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:40.396951 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80a05ef5-573e-4b3a-aaaf-6c91d6cba523-kube-api-access-2kjmx" pod "80a05ef5-573e-4b3a-aaaf-6c91d6cba523" (UID: "80a05ef5-573e-4b3a-aaaf-6c91d6cba523"). InnerVolumeSpecName "kube-api-access-2kjmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:29:40.397121 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:40.397102 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80a05ef5-573e-4b3a-aaaf-6c91d6cba523-metrics-certs" pod "80a05ef5-573e-4b3a-aaaf-6c91d6cba523" (UID: "80a05ef5-573e-4b3a-aaaf-6c91d6cba523"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:29:40.502697 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:40.502659 5979 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80a05ef5-573e-4b3a-aaaf-6c91d6cba523-openshift-service-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:40.502697 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:40.502695 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kjmx\" (UniqueName: \"kubernetes.io/projected/80a05ef5-573e-4b3a-aaaf-6c91d6cba523-kube-api-access-2kjmx\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:40.502974 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:40.502713 5979 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80a05ef5-573e-4b3a-aaaf-6c91d6cba523-metrics-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:40.502974 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:40.502729 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80a05ef5-573e-4b3a-aaaf-6c91d6cba523-maas-api-tls\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:40.552279 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:40.552244 5979 generic.go:356] "Generic (PLEG): container finished" podID="80a05ef5-573e-4b3a-aaaf-6c91d6cba523" containerID="d8bcca5fb8b1879744bfebdaea6f6157f5967f83b66df8718ecbd647e5b44bb2" exitCode=0 Nov 03 14:29:40.552745 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:40.552289 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-367381-55bcdb6c57-6kxd4" Nov 03 14:29:40.552745 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:40.552334 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-367381-55bcdb6c57-6kxd4" event={"ID":"80a05ef5-573e-4b3a-aaaf-6c91d6cba523","Type":"ContainerDied","Data":"d8bcca5fb8b1879744bfebdaea6f6157f5967f83b66df8718ecbd647e5b44bb2"} Nov 03 14:29:40.552745 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:40.552387 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-367381-55bcdb6c57-6kxd4" event={"ID":"80a05ef5-573e-4b3a-aaaf-6c91d6cba523","Type":"ContainerDied","Data":"f307cc8d760340800ac1121e250f8ef74718bc6ce4a7d06ba458447e1fbec878"} Nov 03 14:29:40.552745 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:40.552415 5979 scope.go:122] "RemoveContainer" containerID="d8bcca5fb8b1879744bfebdaea6f6157f5967f83b66df8718ecbd647e5b44bb2" Nov 03 14:29:40.562005 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:40.561985 5979 scope.go:122] "RemoveContainer" containerID="d8bcca5fb8b1879744bfebdaea6f6157f5967f83b66df8718ecbd647e5b44bb2" Nov 03 14:29:40.562250 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:29:40.562234 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8bcca5fb8b1879744bfebdaea6f6157f5967f83b66df8718ecbd647e5b44bb2\": container with ID starting with d8bcca5fb8b1879744bfebdaea6f6157f5967f83b66df8718ecbd647e5b44bb2 not found: ID does not exist" containerID="d8bcca5fb8b1879744bfebdaea6f6157f5967f83b66df8718ecbd647e5b44bb2" Nov 03 14:29:40.562302 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:40.562257 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8bcca5fb8b1879744bfebdaea6f6157f5967f83b66df8718ecbd647e5b44bb2"} err="failed to get container status \"d8bcca5fb8b1879744bfebdaea6f6157f5967f83b66df8718ecbd647e5b44bb2\": rpc error: code = NotFound desc = could not find container \"d8bcca5fb8b1879744bfebdaea6f6157f5967f83b66df8718ecbd647e5b44bb2\": container with ID starting with d8bcca5fb8b1879744bfebdaea6f6157f5967f83b66df8718ecbd647e5b44bb2 not found: ID does not exist" Nov 03 14:29:40.706772 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:40.706689 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-367381-55bcdb6c57-6kxd4"] Nov 03 14:29:40.711310 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:40.711275 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-367381-55bcdb6c57-6kxd4"] Nov 03 14:29:42.038284 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:42.038251 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:29:42.114256 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:42.114221 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80a05ef5-573e-4b3a-aaaf-6c91d6cba523" path="/var/lib/kubelet/pods/80a05ef5-573e-4b3a-aaaf-6c91d6cba523/volumes" Nov 03 14:29:45.859734 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:45.859698 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a8bd0ab-d7tlh"] Nov 03 14:29:45.860165 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:45.860152 5979 cpu_manager.go:414] "containerMap: removing container" podUID="80a05ef5-573e-4b3a-aaaf-6c91d6cba523" containerName="maas-api" Nov 03 14:29:45.860219 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:45.860166 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80a05ef5-573e-4b3a-aaaf-6c91d6cba523" containerName="maas-api" Nov 03 14:29:45.860219 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:45.860175 5979 cpu_manager.go:414] "containerMap: removing container" podUID="a8d3d4f9-8d57-4ff2-99fa-57ffeedfea04" containerName="revoke-keys" Nov 03 14:29:45.860219 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:45.860180 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8d3d4f9-8d57-4ff2-99fa-57ffeedfea04" containerName="revoke-keys" Nov 03 14:29:45.860219 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:45.860197 5979 cpu_manager.go:414] "containerMap: removing container" podUID="a4b4df54-0aac-419d-8bb2-3258267bc088" containerName="main" Nov 03 14:29:45.860219 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:45.860201 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4b4df54-0aac-419d-8bb2-3258267bc088" containerName="main" Nov 03 14:29:45.860219 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:45.860214 5979 cpu_manager.go:414] "containerMap: removing container" podUID="e6134692-b548-43a9-b097-6bce1f7ed9f9" containerName="revoke-keys" Nov 03 14:29:45.860219 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:45.860218 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6134692-b548-43a9-b097-6bce1f7ed9f9" containerName="revoke-keys" Nov 03 14:29:45.860441 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:45.860225 5979 cpu_manager.go:414] "containerMap: removing container" podUID="64d52ec5-fb72-4616-8d20-8b90043af828" containerName="main" Nov 03 14:29:45.860441 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:45.860230 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64d52ec5-fb72-4616-8d20-8b90043af828" containerName="main" Nov 03 14:29:45.860441 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:45.860291 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="80a05ef5-573e-4b3a-aaaf-6c91d6cba523" containerName="maas-api" Nov 03 14:29:45.860441 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:45.860299 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8d3d4f9-8d57-4ff2-99fa-57ffeedfea04" containerName="revoke-keys" Nov 03 14:29:45.860441 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:45.860307 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4b4df54-0aac-419d-8bb2-3258267bc088" containerName="main" Nov 03 14:29:45.860441 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:45.860313 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6134692-b548-43a9-b097-6bce1f7ed9f9" containerName="revoke-keys" Nov 03 14:29:45.860441 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:45.860319 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="64d52ec5-fb72-4616-8d20-8b90043af828" containerName="main" Nov 03 14:29:45.865197 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:45.865175 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a8bd0ab-d7tlh" Nov 03 14:29:45.914567 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:45.914529 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:29:45.914567 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:45.914535 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:29:45.957343 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:45.957308 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfzrv\" (UniqueName: \"kubernetes.io/projected/09d30900-2c56-479b-96d7-b5da683a4356-kube-api-access-dfzrv\") pod \"maas-api-revoke-sub-1a8bd0ab-d7tlh\" (UID: \"09d30900-2c56-479b-96d7-b5da683a4356\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a8bd0ab-d7tlh" Nov 03 14:29:45.957560 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:45.957398 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09d30900-2c56-479b-96d7-b5da683a4356-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a8bd0ab-d7tlh\" (UID: \"09d30900-2c56-479b-96d7-b5da683a4356\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a8bd0ab-d7tlh" Nov 03 14:29:45.957612 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:45.957575 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09d30900-2c56-479b-96d7-b5da683a4356-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a8bd0ab-d7tlh\" (UID: \"09d30900-2c56-479b-96d7-b5da683a4356\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a8bd0ab-d7tlh" Nov 03 14:29:45.960611 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:45.960589 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:29:45.960611 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:45.960589 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 14:29:45.963214 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:45.963190 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a8bd0ab-d7tlh"] Nov 03 14:29:46.058693 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:46.058651 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09d30900-2c56-479b-96d7-b5da683a4356-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a8bd0ab-d7tlh\" (UID: \"09d30900-2c56-479b-96d7-b5da683a4356\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a8bd0ab-d7tlh" Nov 03 14:29:46.058890 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:46.058712 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfzrv\" (UniqueName: \"kubernetes.io/projected/09d30900-2c56-479b-96d7-b5da683a4356-kube-api-access-dfzrv\") pod \"maas-api-revoke-sub-1a8bd0ab-d7tlh\" (UID: \"09d30900-2c56-479b-96d7-b5da683a4356\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a8bd0ab-d7tlh" Nov 03 14:29:46.058890 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:46.058791 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09d30900-2c56-479b-96d7-b5da683a4356-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a8bd0ab-d7tlh\" (UID: \"09d30900-2c56-479b-96d7-b5da683a4356\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a8bd0ab-d7tlh" Nov 03 14:29:46.059431 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:46.059407 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09d30900-2c56-479b-96d7-b5da683a4356-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a8bd0ab-d7tlh\" (UID: \"09d30900-2c56-479b-96d7-b5da683a4356\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a8bd0ab-d7tlh" Nov 03 14:29:46.061075 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:46.061052 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09d30900-2c56-479b-96d7-b5da683a4356-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a8bd0ab-d7tlh\" (UID: \"09d30900-2c56-479b-96d7-b5da683a4356\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a8bd0ab-d7tlh" Nov 03 14:29:46.113182 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:46.113112 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfzrv\" (UniqueName: \"kubernetes.io/projected/09d30900-2c56-479b-96d7-b5da683a4356-kube-api-access-dfzrv\") pod \"maas-api-revoke-sub-1a8bd0ab-d7tlh\" (UID: \"09d30900-2c56-479b-96d7-b5da683a4356\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a8bd0ab-d7tlh" Nov 03 14:29:46.178781 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:46.178740 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a8bd0ab-d7tlh" Nov 03 14:29:46.551371 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:46.551333 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a8bd0ab-d7tlh"] Nov 03 14:29:46.558079 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:29:46.558045 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09d30900_2c56_479b_96d7_b5da683a4356.slice/crio-aba93708aa7b2e3d8b7590bc663a41fea05a0eb233ae333ba56cf66ecf5efe88 WatchSource:0}: Error finding container aba93708aa7b2e3d8b7590bc663a41fea05a0eb233ae333ba56cf66ecf5efe88: Status 404 returned error can't find the container with id aba93708aa7b2e3d8b7590bc663a41fea05a0eb233ae333ba56cf66ecf5efe88 Nov 03 14:29:46.593432 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:46.593401 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a8bd0ab-d7tlh" event={"ID":"09d30900-2c56-479b-96d7-b5da683a4356","Type":"ContainerStarted","Data":"aba93708aa7b2e3d8b7590bc663a41fea05a0eb233ae333ba56cf66ecf5efe88"} Nov 03 14:29:47.602791 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:47.602755 5979 generic.go:356] "Generic (PLEG): container finished" podID="09d30900-2c56-479b-96d7-b5da683a4356" containerID="58a47de9fc5ea66812ade3ab8bf57b07b36b89e7f464c1fbbeb88ba77092f029" exitCode=0 Nov 03 14:29:47.603187 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:47.602803 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a8bd0ab-d7tlh" event={"ID":"09d30900-2c56-479b-96d7-b5da683a4356","Type":"ContainerDied","Data":"58a47de9fc5ea66812ade3ab8bf57b07b36b89e7f464c1fbbeb88ba77092f029"} Nov 03 14:29:48.754393 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:48.754366 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a8bd0ab-d7tlh" Nov 03 14:29:48.787359 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:48.787333 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09d30900-2c56-479b-96d7-b5da683a4356-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09d30900-2c56-479b-96d7-b5da683a4356-maas-api-openshift-ca\") pod \"09d30900-2c56-479b-96d7-b5da683a4356\" (UID: \"09d30900-2c56-479b-96d7-b5da683a4356\") " Nov 03 14:29:48.787562 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:48.787443 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09d30900-2c56-479b-96d7-b5da683a4356-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09d30900-2c56-479b-96d7-b5da683a4356-maas-api-xks-ca\") pod \"09d30900-2c56-479b-96d7-b5da683a4356\" (UID: \"09d30900-2c56-479b-96d7-b5da683a4356\") " Nov 03 14:29:48.787648 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:48.787599 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09d30900-2c56-479b-96d7-b5da683a4356-kube-api-access-dfzrv\" (UniqueName: \"kubernetes.io/projected/09d30900-2c56-479b-96d7-b5da683a4356-kube-api-access-dfzrv\") pod \"09d30900-2c56-479b-96d7-b5da683a4356\" (UID: \"09d30900-2c56-479b-96d7-b5da683a4356\") " Nov 03 14:29:48.787787 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:48.787762 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09d30900-2c56-479b-96d7-b5da683a4356-maas-api-openshift-ca" pod "09d30900-2c56-479b-96d7-b5da683a4356" (UID: "09d30900-2c56-479b-96d7-b5da683a4356"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:29:48.787933 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:48.787911 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09d30900-2c56-479b-96d7-b5da683a4356-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:48.789621 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:48.789590 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09d30900-2c56-479b-96d7-b5da683a4356-maas-api-xks-ca" pod "09d30900-2c56-479b-96d7-b5da683a4356" (UID: "09d30900-2c56-479b-96d7-b5da683a4356"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:29:48.789752 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:48.789720 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09d30900-2c56-479b-96d7-b5da683a4356-kube-api-access-dfzrv" pod "09d30900-2c56-479b-96d7-b5da683a4356" (UID: "09d30900-2c56-479b-96d7-b5da683a4356"). InnerVolumeSpecName "kube-api-access-dfzrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:29:48.889088 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:48.889024 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09d30900-2c56-479b-96d7-b5da683a4356-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:48.889088 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:48.889063 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfzrv\" (UniqueName: \"kubernetes.io/projected/09d30900-2c56-479b-96d7-b5da683a4356-kube-api-access-dfzrv\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:49.624422 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:49.624388 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a8bd0ab-d7tlh" Nov 03 14:29:49.624656 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:49.624421 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a8bd0ab-d7tlh" event={"ID":"09d30900-2c56-479b-96d7-b5da683a4356","Type":"ContainerDied","Data":"aba93708aa7b2e3d8b7590bc663a41fea05a0eb233ae333ba56cf66ecf5efe88"} Nov 03 14:29:49.624656 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:49.624483 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aba93708aa7b2e3d8b7590bc663a41fea05a0eb233ae333ba56cf66ecf5efe88" Nov 03 14:29:49.718311 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:49.718280 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld"] Nov 03 14:29:49.718523 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:49.718494 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld" podUID="f76845b8-e625-4425-ad28-74612d33985c" containerName="istio-proxy" containerID="cri-o://f2d4e8845a5e7f053b9899cc345a5e90e1d95ca83cbf5184e38dcf144c002eb7" gracePeriod=30 Nov 03 14:29:50.870921 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:50.870887 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-578862a6-5fdq5"] Nov 03 14:29:50.871352 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:50.871345 5979 cpu_manager.go:414] "containerMap: removing container" podUID="09d30900-2c56-479b-96d7-b5da683a4356" containerName="revoke-keys" Nov 03 14:29:50.871416 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:50.871356 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09d30900-2c56-479b-96d7-b5da683a4356" containerName="revoke-keys" Nov 03 14:29:50.871453 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:50.871437 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="09d30900-2c56-479b-96d7-b5da683a4356" containerName="revoke-keys" Nov 03 14:29:50.874976 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:50.874949 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-578862a6-5fdq5" Nov 03 14:29:50.909992 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:50.909963 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e1bdabe-7240-4fcb-b1d8-a95827a342cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-578862a6-5fdq5\" (UID: \"4e1bdabe-7240-4fcb-b1d8-a95827a342cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-578862a6-5fdq5" Nov 03 14:29:50.910141 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:50.910125 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gdvl\" (UniqueName: \"kubernetes.io/projected/4e1bdabe-7240-4fcb-b1d8-a95827a342cd-kube-api-access-6gdvl\") pod \"maas-api-revoke-sub-578862a6-5fdq5\" (UID: \"4e1bdabe-7240-4fcb-b1d8-a95827a342cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-578862a6-5fdq5" Nov 03 14:29:50.910198 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:50.910160 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e1bdabe-7240-4fcb-b1d8-a95827a342cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-578862a6-5fdq5\" (UID: \"4e1bdabe-7240-4fcb-b1d8-a95827a342cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-578862a6-5fdq5" Nov 03 14:29:50.924649 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:50.924624 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:29:50.924649 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:50.924629 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:29:50.924856 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:50.924845 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:29:50.925358 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:50.925339 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 14:29:50.973467 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:50.973435 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-578862a6-5fdq5"] Nov 03 14:29:51.010889 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:51.010852 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gdvl\" (UniqueName: \"kubernetes.io/projected/4e1bdabe-7240-4fcb-b1d8-a95827a342cd-kube-api-access-6gdvl\") pod \"maas-api-revoke-sub-578862a6-5fdq5\" (UID: \"4e1bdabe-7240-4fcb-b1d8-a95827a342cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-578862a6-5fdq5" Nov 03 14:29:51.010889 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:51.010891 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e1bdabe-7240-4fcb-b1d8-a95827a342cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-578862a6-5fdq5\" (UID: \"4e1bdabe-7240-4fcb-b1d8-a95827a342cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-578862a6-5fdq5" Nov 03 14:29:51.011084 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:51.010925 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e1bdabe-7240-4fcb-b1d8-a95827a342cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-578862a6-5fdq5\" (UID: \"4e1bdabe-7240-4fcb-b1d8-a95827a342cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-578862a6-5fdq5" Nov 03 14:29:51.011558 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:51.011534 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e1bdabe-7240-4fcb-b1d8-a95827a342cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-578862a6-5fdq5\" (UID: \"4e1bdabe-7240-4fcb-b1d8-a95827a342cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-578862a6-5fdq5" Nov 03 14:29:51.013334 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:51.013311 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e1bdabe-7240-4fcb-b1d8-a95827a342cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-578862a6-5fdq5\" (UID: \"4e1bdabe-7240-4fcb-b1d8-a95827a342cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-578862a6-5fdq5" Nov 03 14:29:51.064383 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:51.064353 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gdvl\" (UniqueName: \"kubernetes.io/projected/4e1bdabe-7240-4fcb-b1d8-a95827a342cd-kube-api-access-6gdvl\") pod \"maas-api-revoke-sub-578862a6-5fdq5\" (UID: \"4e1bdabe-7240-4fcb-b1d8-a95827a342cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-578862a6-5fdq5" Nov 03 14:29:51.188986 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:51.188948 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-578862a6-5fdq5" Nov 03 14:29:51.562256 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:51.562177 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-578862a6-5fdq5"] Nov 03 14:29:51.570465 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:29:51.570428 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e1bdabe_7240_4fcb_b1d8_a95827a342cd.slice/crio-1d0ba00f93b91b545d3bffbf925602ba5b6689aaa4e4cae39f6deb4e7f5e3a35 WatchSource:0}: Error finding container 1d0ba00f93b91b545d3bffbf925602ba5b6689aaa4e4cae39f6deb4e7f5e3a35: Status 404 returned error can't find the container with id 1d0ba00f93b91b545d3bffbf925602ba5b6689aaa4e4cae39f6deb4e7f5e3a35 Nov 03 14:29:51.637147 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:51.637113 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-578862a6-5fdq5" event={"ID":"4e1bdabe-7240-4fcb-b1d8-a95827a342cd","Type":"ContainerStarted","Data":"1d0ba00f93b91b545d3bffbf925602ba5b6689aaa4e4cae39f6deb4e7f5e3a35"} Nov 03 14:29:52.644067 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:52.644026 5979 generic.go:356] "Generic (PLEG): container finished" podID="4e1bdabe-7240-4fcb-b1d8-a95827a342cd" containerID="6e77c5d70e66e509f48ef38c56a18862088221cd413b20a3ded830f39f04c78d" exitCode=0 Nov 03 14:29:52.644455 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:52.644116 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-578862a6-5fdq5" event={"ID":"4e1bdabe-7240-4fcb-b1d8-a95827a342cd","Type":"ContainerDied","Data":"6e77c5d70e66e509f48ef38c56a18862088221cd413b20a3ded830f39f04c78d"} Nov 03 14:29:53.791718 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:53.791692 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-578862a6-5fdq5" Nov 03 14:29:53.841710 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:53.841677 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e1bdabe-7240-4fcb-b1d8-a95827a342cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e1bdabe-7240-4fcb-b1d8-a95827a342cd-maas-api-openshift-ca\") pod \"4e1bdabe-7240-4fcb-b1d8-a95827a342cd\" (UID: \"4e1bdabe-7240-4fcb-b1d8-a95827a342cd\") " Nov 03 14:29:53.841880 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:53.841766 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e1bdabe-7240-4fcb-b1d8-a95827a342cd-kube-api-access-6gdvl\" (UniqueName: \"kubernetes.io/projected/4e1bdabe-7240-4fcb-b1d8-a95827a342cd-kube-api-access-6gdvl\") pod \"4e1bdabe-7240-4fcb-b1d8-a95827a342cd\" (UID: \"4e1bdabe-7240-4fcb-b1d8-a95827a342cd\") " Nov 03 14:29:53.841921 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:53.841886 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e1bdabe-7240-4fcb-b1d8-a95827a342cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e1bdabe-7240-4fcb-b1d8-a95827a342cd-maas-api-xks-ca\") pod \"4e1bdabe-7240-4fcb-b1d8-a95827a342cd\" (UID: \"4e1bdabe-7240-4fcb-b1d8-a95827a342cd\") " Nov 03 14:29:53.842098 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:53.842076 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e1bdabe-7240-4fcb-b1d8-a95827a342cd-maas-api-openshift-ca" pod "4e1bdabe-7240-4fcb-b1d8-a95827a342cd" (UID: "4e1bdabe-7240-4fcb-b1d8-a95827a342cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:29:53.842311 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:53.842286 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e1bdabe-7240-4fcb-b1d8-a95827a342cd-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:53.843844 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:53.843820 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e1bdabe-7240-4fcb-b1d8-a95827a342cd-kube-api-access-6gdvl" pod "4e1bdabe-7240-4fcb-b1d8-a95827a342cd" (UID: "4e1bdabe-7240-4fcb-b1d8-a95827a342cd"). InnerVolumeSpecName "kube-api-access-6gdvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:29:53.843966 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:53.843835 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e1bdabe-7240-4fcb-b1d8-a95827a342cd-maas-api-xks-ca" pod "4e1bdabe-7240-4fcb-b1d8-a95827a342cd" (UID: "4e1bdabe-7240-4fcb-b1d8-a95827a342cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:29:53.944117 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:53.944040 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gdvl\" (UniqueName: \"kubernetes.io/projected/4e1bdabe-7240-4fcb-b1d8-a95827a342cd-kube-api-access-6gdvl\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:53.944117 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:53.944081 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e1bdabe-7240-4fcb-b1d8-a95827a342cd-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:54.659010 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:54.658979 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-578862a6-5fdq5" Nov 03 14:29:54.659191 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:54.659013 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-578862a6-5fdq5" event={"ID":"4e1bdabe-7240-4fcb-b1d8-a95827a342cd","Type":"ContainerDied","Data":"1d0ba00f93b91b545d3bffbf925602ba5b6689aaa4e4cae39f6deb4e7f5e3a35"} Nov 03 14:29:54.659191 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:54.659062 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d0ba00f93b91b545d3bffbf925602ba5b6689aaa4e4cae39f6deb4e7f5e3a35" Nov 03 14:29:54.981088 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:54.981063 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld" Nov 03 14:29:55.054526 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:55.054490 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f76845b8-e625-4425-ad28-74612d33985c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f76845b8-e625-4425-ad28-74612d33985c-credential-socket\") pod \"f76845b8-e625-4425-ad28-74612d33985c\" (UID: \"f76845b8-e625-4425-ad28-74612d33985c\") " Nov 03 14:29:55.054690 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:55.054537 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f76845b8-e625-4425-ad28-74612d33985c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f76845b8-e625-4425-ad28-74612d33985c-istio-data\") pod \"f76845b8-e625-4425-ad28-74612d33985c\" (UID: \"f76845b8-e625-4425-ad28-74612d33985c\") " Nov 03 14:29:55.054690 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:55.054611 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f76845b8-e625-4425-ad28-74612d33985c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f76845b8-e625-4425-ad28-74612d33985c-workload-socket\") pod \"f76845b8-e625-4425-ad28-74612d33985c\" (UID: \"f76845b8-e625-4425-ad28-74612d33985c\") " Nov 03 14:29:55.054690 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:55.054639 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f76845b8-e625-4425-ad28-74612d33985c-istio-token\" (UniqueName: \"kubernetes.io/projected/f76845b8-e625-4425-ad28-74612d33985c-istio-token\") pod \"f76845b8-e625-4425-ad28-74612d33985c\" (UID: \"f76845b8-e625-4425-ad28-74612d33985c\") " Nov 03 14:29:55.054690 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:55.054666 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f76845b8-e625-4425-ad28-74612d33985c-kube-api-access-9cwfx\" (UniqueName: \"kubernetes.io/projected/f76845b8-e625-4425-ad28-74612d33985c-kube-api-access-9cwfx\") pod \"f76845b8-e625-4425-ad28-74612d33985c\" (UID: \"f76845b8-e625-4425-ad28-74612d33985c\") " Nov 03 14:29:55.054910 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:55.054694 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f76845b8-e625-4425-ad28-74612d33985c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f76845b8-e625-4425-ad28-74612d33985c-workload-certs\") pod \"f76845b8-e625-4425-ad28-74612d33985c\" (UID: \"f76845b8-e625-4425-ad28-74612d33985c\") " Nov 03 14:29:55.054910 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:55.054733 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f76845b8-e625-4425-ad28-74612d33985c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f76845b8-e625-4425-ad28-74612d33985c-istiod-ca-cert\") pod \"f76845b8-e625-4425-ad28-74612d33985c\" (UID: \"f76845b8-e625-4425-ad28-74612d33985c\") " Nov 03 14:29:55.054910 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:55.054808 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f76845b8-e625-4425-ad28-74612d33985c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f76845b8-e625-4425-ad28-74612d33985c-istio-envoy\") pod \"f76845b8-e625-4425-ad28-74612d33985c\" (UID: \"f76845b8-e625-4425-ad28-74612d33985c\") " Nov 03 14:29:55.054910 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:55.054850 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f76845b8-e625-4425-ad28-74612d33985c-credential-socket" pod "f76845b8-e625-4425-ad28-74612d33985c" (UID: "f76845b8-e625-4425-ad28-74612d33985c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:29:55.054910 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:55.054861 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f76845b8-e625-4425-ad28-74612d33985c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f76845b8-e625-4425-ad28-74612d33985c-istio-podinfo\") pod \"f76845b8-e625-4425-ad28-74612d33985c\" (UID: \"f76845b8-e625-4425-ad28-74612d33985c\") " Nov 03 14:29:55.055184 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:55.054942 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f76845b8-e625-4425-ad28-74612d33985c-workload-socket" pod "f76845b8-e625-4425-ad28-74612d33985c" (UID: "f76845b8-e625-4425-ad28-74612d33985c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:29:55.055184 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:55.055005 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f76845b8-e625-4425-ad28-74612d33985c-istio-data" pod "f76845b8-e625-4425-ad28-74612d33985c" (UID: "f76845b8-e625-4425-ad28-74612d33985c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:29:55.055184 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:55.055064 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f76845b8-e625-4425-ad28-74612d33985c-workload-certs" pod "f76845b8-e625-4425-ad28-74612d33985c" (UID: "f76845b8-e625-4425-ad28-74612d33985c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:29:55.055184 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:55.055154 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f76845b8-e625-4425-ad28-74612d33985c-istiod-ca-cert" pod "f76845b8-e625-4425-ad28-74612d33985c" (UID: "f76845b8-e625-4425-ad28-74612d33985c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:29:55.055417 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:55.055348 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f76845b8-e625-4425-ad28-74612d33985c-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:55.055417 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:55.055369 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f76845b8-e625-4425-ad28-74612d33985c-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:55.055417 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:55.055383 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f76845b8-e625-4425-ad28-74612d33985c-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:55.055417 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:55.055396 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f76845b8-e625-4425-ad28-74612d33985c-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:55.055417 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:55.055410 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f76845b8-e625-4425-ad28-74612d33985c-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:55.056972 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:55.056944 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f76845b8-e625-4425-ad28-74612d33985c-istio-token" pod "f76845b8-e625-4425-ad28-74612d33985c" (UID: "f76845b8-e625-4425-ad28-74612d33985c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:29:55.057101 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:55.056989 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f76845b8-e625-4425-ad28-74612d33985c-kube-api-access-9cwfx" pod "f76845b8-e625-4425-ad28-74612d33985c" (UID: "f76845b8-e625-4425-ad28-74612d33985c"). InnerVolumeSpecName "kube-api-access-9cwfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:29:55.057101 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:55.057037 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f76845b8-e625-4425-ad28-74612d33985c-istio-podinfo" pod "f76845b8-e625-4425-ad28-74612d33985c" (UID: "f76845b8-e625-4425-ad28-74612d33985c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:29:55.057101 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:55.057059 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f76845b8-e625-4425-ad28-74612d33985c-istio-envoy" pod "f76845b8-e625-4425-ad28-74612d33985c" (UID: "f76845b8-e625-4425-ad28-74612d33985c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:29:55.161851 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:55.157164 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f76845b8-e625-4425-ad28-74612d33985c-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:55.161851 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:55.157233 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f76845b8-e625-4425-ad28-74612d33985c-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:55.161851 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:55.157253 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f76845b8-e625-4425-ad28-74612d33985c-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:55.161851 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:55.157267 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cwfx\" (UniqueName: \"kubernetes.io/projected/f76845b8-e625-4425-ad28-74612d33985c-kube-api-access-9cwfx\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:29:55.667711 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:55.667678 5979 generic.go:356] "Generic (PLEG): container finished" podID="f76845b8-e625-4425-ad28-74612d33985c" containerID="f2d4e8845a5e7f053b9899cc345a5e90e1d95ca83cbf5184e38dcf144c002eb7" exitCode=0 Nov 03 14:29:55.667992 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:55.667749 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld" Nov 03 14:29:55.667992 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:55.667766 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld" event={"ID":"f76845b8-e625-4425-ad28-74612d33985c","Type":"ContainerDied","Data":"f2d4e8845a5e7f053b9899cc345a5e90e1d95ca83cbf5184e38dcf144c002eb7"} Nov 03 14:29:55.667992 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:55.667855 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld" event={"ID":"f76845b8-e625-4425-ad28-74612d33985c","Type":"ContainerDied","Data":"a5d5d299592355bcf8c4c61f54b30c30349c5cf61b1bee75a61bc67a85b45300"} Nov 03 14:29:55.667992 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:55.667883 5979 scope.go:122] "RemoveContainer" containerID="f2d4e8845a5e7f053b9899cc345a5e90e1d95ca83cbf5184e38dcf144c002eb7" Nov 03 14:29:55.678028 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:55.678004 5979 scope.go:122] "RemoveContainer" containerID="a7cf676f1fa259a460515dd1a4b7a3c48b4d7a44850101fcad73a470b97e9673" Nov 03 14:29:55.687342 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:55.687316 5979 scope.go:122] "RemoveContainer" containerID="f2d4e8845a5e7f053b9899cc345a5e90e1d95ca83cbf5184e38dcf144c002eb7" Nov 03 14:29:55.687666 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:29:55.687640 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2d4e8845a5e7f053b9899cc345a5e90e1d95ca83cbf5184e38dcf144c002eb7\": container with ID starting with f2d4e8845a5e7f053b9899cc345a5e90e1d95ca83cbf5184e38dcf144c002eb7 not found: ID does not exist" containerID="f2d4e8845a5e7f053b9899cc345a5e90e1d95ca83cbf5184e38dcf144c002eb7" Nov 03 14:29:55.687769 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:55.687673 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2d4e8845a5e7f053b9899cc345a5e90e1d95ca83cbf5184e38dcf144c002eb7"} err="failed to get container status \"f2d4e8845a5e7f053b9899cc345a5e90e1d95ca83cbf5184e38dcf144c002eb7\": rpc error: code = NotFound desc = could not find container \"f2d4e8845a5e7f053b9899cc345a5e90e1d95ca83cbf5184e38dcf144c002eb7\": container with ID starting with f2d4e8845a5e7f053b9899cc345a5e90e1d95ca83cbf5184e38dcf144c002eb7 not found: ID does not exist" Nov 03 14:29:55.687769 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:55.687693 5979 scope.go:122] "RemoveContainer" containerID="a7cf676f1fa259a460515dd1a4b7a3c48b4d7a44850101fcad73a470b97e9673" Nov 03 14:29:55.687982 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:29:55.687964 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7cf676f1fa259a460515dd1a4b7a3c48b4d7a44850101fcad73a470b97e9673\": container with ID starting with a7cf676f1fa259a460515dd1a4b7a3c48b4d7a44850101fcad73a470b97e9673 not found: ID does not exist" containerID="a7cf676f1fa259a460515dd1a4b7a3c48b4d7a44850101fcad73a470b97e9673" Nov 03 14:29:55.688029 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:55.687988 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7cf676f1fa259a460515dd1a4b7a3c48b4d7a44850101fcad73a470b97e9673"} err="failed to get container status \"a7cf676f1fa259a460515dd1a4b7a3c48b4d7a44850101fcad73a470b97e9673\": rpc error: code = NotFound desc = could not find container \"a7cf676f1fa259a460515dd1a4b7a3c48b4d7a44850101fcad73a470b97e9673\": container with ID starting with a7cf676f1fa259a460515dd1a4b7a3c48b4d7a44850101fcad73a470b97e9673 not found: ID does not exist" Nov 03 14:29:55.867908 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:55.867864 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld"] Nov 03 14:29:55.914337 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:55.914298 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-367381-openshift-default-fcbc6879-mz8ld"] Nov 03 14:29:56.115499 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:29:56.115416 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f76845b8-e625-4425-ad28-74612d33985c" path="/var/lib/kubelet/pods/f76845b8-e625-4425-ad28-74612d33985c/volumes" Nov 03 14:30:00.046992 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.046954 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8359049-kb54l"] Nov 03 14:30:00.093207 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.093167 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8359049-kb54l"] Nov 03 14:30:00.114301 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.114263 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fa6a699-01c0-4236-8cf4-a9a6270ddf4e" path="/var/lib/kubelet/pods/7fa6a699-01c0-4236-8cf4-a9a6270ddf4e/volumes" Nov 03 14:30:00.163818 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.163780 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895270-7csjj"] Nov 03 14:30:00.164401 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.164381 5979 cpu_manager.go:414] "containerMap: removing container" podUID="f76845b8-e625-4425-ad28-74612d33985c" containerName="istio-proxy" Nov 03 14:30:00.164505 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.164403 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f76845b8-e625-4425-ad28-74612d33985c" containerName="istio-proxy" Nov 03 14:30:00.164505 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.164421 5979 cpu_manager.go:414] "containerMap: removing container" podUID="4e1bdabe-7240-4fcb-b1d8-a95827a342cd" containerName="revoke-keys" Nov 03 14:30:00.164505 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.164427 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e1bdabe-7240-4fcb-b1d8-a95827a342cd" containerName="revoke-keys" Nov 03 14:30:00.164505 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.164457 5979 cpu_manager.go:414] "containerMap: removing container" podUID="f76845b8-e625-4425-ad28-74612d33985c" containerName="istio-proxy" Nov 03 14:30:00.164505 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.164462 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f76845b8-e625-4425-ad28-74612d33985c" containerName="istio-proxy" Nov 03 14:30:00.164723 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.164571 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="f76845b8-e625-4425-ad28-74612d33985c" containerName="istio-proxy" Nov 03 14:30:00.164723 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.164583 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e1bdabe-7240-4fcb-b1d8-a95827a342cd" containerName="revoke-keys" Nov 03 14:30:00.169974 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.169953 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895270-7csjj" Nov 03 14:30:00.170090 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.170023 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895270-69dbd"] Nov 03 14:30:00.170702 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.170683 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="f76845b8-e625-4425-ad28-74612d33985c" containerName="istio-proxy" Nov 03 14:30:00.174549 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.174528 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895270-2qcwp"] Nov 03 14:30:00.174683 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.174661 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895270-69dbd" Nov 03 14:30:00.178691 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.178601 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895270-4vv76"] Nov 03 14:30:00.178775 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.178717 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895270-2qcwp" Nov 03 14:30:00.182833 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.182812 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895270-4vv76" Nov 03 14:30:00.193799 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.193777 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895270-6trc6"] Nov 03 14:30:00.197740 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.197722 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895270-6trc6" Nov 03 14:30:00.218896 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.218848 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 14:30:00.219012 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.218998 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:30:00.219105 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.219089 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:30:00.224823 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.224799 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895270-7csjj"] Nov 03 14:30:00.231884 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.231864 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895270-69dbd"] Nov 03 14:30:00.267228 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.267200 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895270-4vv76"] Nov 03 14:30:00.269945 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.269916 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895270-6trc6"] Nov 03 14:30:00.288396 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.288366 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895270-2qcwp"] Nov 03 14:30:00.329499 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.329393 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phj7c\" (UniqueName: \"kubernetes.io/projected/8cf08c4a-4b04-4a73-a2dd-39bd58066be9-kube-api-access-phj7c\") pod \"maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895270-6trc6\" (UID: \"8cf08c4a-4b04-4a73-a2dd-39bd58066be9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895270-6trc6" Nov 03 14:30:00.329499 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.329438 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t2g7\" (UniqueName: \"kubernetes.io/projected/a0b8e1f3-1744-4fe0-8f2d-04b3ec7a13ad-kube-api-access-2t2g7\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895270-7csjj\" (UID: \"a0b8e1f3-1744-4fe0-8f2d-04b3ec7a13ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895270-7csjj" Nov 03 14:30:00.329499 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.329463 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b8e1f3-1744-4fe0-8f2d-04b3ec7a13ad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895270-7csjj\" (UID: \"a0b8e1f3-1744-4fe0-8f2d-04b3ec7a13ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895270-7csjj" Nov 03 14:30:00.329727 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.329543 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwc6s\" (UniqueName: \"kubernetes.io/projected/041bbdde-47d2-46eb-a5d4-bae94c32990f-kube-api-access-pwc6s\") pod \"maas-api-key-cleanup-29895270-4vv76\" (UID: \"041bbdde-47d2-46eb-a5d4-bae94c32990f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895270-4vv76" Nov 03 14:30:00.329727 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.329574 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cf08c4a-4b04-4a73-a2dd-39bd58066be9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895270-6trc6\" (UID: \"8cf08c4a-4b04-4a73-a2dd-39bd58066be9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895270-6trc6" Nov 03 14:30:00.329727 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.329623 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68fcc652-c24d-4b11-84a0-8e424f2e130c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895270-2qcwp\" (UID: \"68fcc652-c24d-4b11-84a0-8e424f2e130c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895270-2qcwp" Nov 03 14:30:00.329727 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.329705 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwh6g\" (UniqueName: \"kubernetes.io/projected/68fcc652-c24d-4b11-84a0-8e424f2e130c-kube-api-access-mwh6g\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895270-2qcwp\" (UID: \"68fcc652-c24d-4b11-84a0-8e424f2e130c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895270-2qcwp" Nov 03 14:30:00.329871 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.329749 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d4wm\" (UniqueName: \"kubernetes.io/projected/417d374a-e02d-4721-a1df-e2b912877928-kube-api-access-8d4wm\") pod \"maas-api-key-cleanup-e2e-worker-w1-34b13c-29895270-69dbd\" (UID: \"417d374a-e02d-4721-a1df-e2b912877928\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895270-69dbd" Nov 03 14:30:00.329871 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.329786 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/041bbdde-47d2-46eb-a5d4-bae94c32990f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895270-4vv76\" (UID: \"041bbdde-47d2-46eb-a5d4-bae94c32990f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895270-4vv76" Nov 03 14:30:00.329871 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.329832 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/417d374a-e02d-4721-a1df-e2b912877928-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-34b13c-29895270-69dbd\" (UID: \"417d374a-e02d-4721-a1df-e2b912877928\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895270-69dbd" Nov 03 14:30:00.430462 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.430426 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68fcc652-c24d-4b11-84a0-8e424f2e130c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895270-2qcwp\" (UID: \"68fcc652-c24d-4b11-84a0-8e424f2e130c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895270-2qcwp" Nov 03 14:30:00.430681 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.430494 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwh6g\" (UniqueName: \"kubernetes.io/projected/68fcc652-c24d-4b11-84a0-8e424f2e130c-kube-api-access-mwh6g\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895270-2qcwp\" (UID: \"68fcc652-c24d-4b11-84a0-8e424f2e130c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895270-2qcwp" Nov 03 14:30:00.430681 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.430549 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d4wm\" (UniqueName: \"kubernetes.io/projected/417d374a-e02d-4721-a1df-e2b912877928-kube-api-access-8d4wm\") pod \"maas-api-key-cleanup-e2e-worker-w1-34b13c-29895270-69dbd\" (UID: \"417d374a-e02d-4721-a1df-e2b912877928\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895270-69dbd" Nov 03 14:30:00.430681 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.430579 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/041bbdde-47d2-46eb-a5d4-bae94c32990f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895270-4vv76\" (UID: \"041bbdde-47d2-46eb-a5d4-bae94c32990f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895270-4vv76" Nov 03 14:30:00.430681 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.430645 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/417d374a-e02d-4721-a1df-e2b912877928-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-34b13c-29895270-69dbd\" (UID: \"417d374a-e02d-4721-a1df-e2b912877928\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895270-69dbd" Nov 03 14:30:00.430923 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.430712 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phj7c\" (UniqueName: \"kubernetes.io/projected/8cf08c4a-4b04-4a73-a2dd-39bd58066be9-kube-api-access-phj7c\") pod \"maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895270-6trc6\" (UID: \"8cf08c4a-4b04-4a73-a2dd-39bd58066be9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895270-6trc6" Nov 03 14:30:00.430923 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.430794 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t2g7\" (UniqueName: \"kubernetes.io/projected/a0b8e1f3-1744-4fe0-8f2d-04b3ec7a13ad-kube-api-access-2t2g7\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895270-7csjj\" (UID: \"a0b8e1f3-1744-4fe0-8f2d-04b3ec7a13ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895270-7csjj" Nov 03 14:30:00.430923 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.430831 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b8e1f3-1744-4fe0-8f2d-04b3ec7a13ad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895270-7csjj\" (UID: \"a0b8e1f3-1744-4fe0-8f2d-04b3ec7a13ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895270-7csjj" Nov 03 14:30:00.430923 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.430856 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwc6s\" (UniqueName: \"kubernetes.io/projected/041bbdde-47d2-46eb-a5d4-bae94c32990f-kube-api-access-pwc6s\") pod \"maas-api-key-cleanup-29895270-4vv76\" (UID: \"041bbdde-47d2-46eb-a5d4-bae94c32990f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895270-4vv76" Nov 03 14:30:00.430923 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.430881 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cf08c4a-4b04-4a73-a2dd-39bd58066be9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895270-6trc6\" (UID: \"8cf08c4a-4b04-4a73-a2dd-39bd58066be9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895270-6trc6" Nov 03 14:30:00.431294 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.431273 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68fcc652-c24d-4b11-84a0-8e424f2e130c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895270-2qcwp\" (UID: \"68fcc652-c24d-4b11-84a0-8e424f2e130c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895270-2qcwp" Nov 03 14:30:00.431420 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.431399 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/417d374a-e02d-4721-a1df-e2b912877928-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-34b13c-29895270-69dbd\" (UID: \"417d374a-e02d-4721-a1df-e2b912877928\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895270-69dbd" Nov 03 14:30:00.431505 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.431398 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/041bbdde-47d2-46eb-a5d4-bae94c32990f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895270-4vv76\" (UID: \"041bbdde-47d2-46eb-a5d4-bae94c32990f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895270-4vv76" Nov 03 14:30:00.431652 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.431628 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b8e1f3-1744-4fe0-8f2d-04b3ec7a13ad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895270-7csjj\" (UID: \"a0b8e1f3-1744-4fe0-8f2d-04b3ec7a13ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895270-7csjj" Nov 03 14:30:00.431691 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.431637 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cf08c4a-4b04-4a73-a2dd-39bd58066be9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895270-6trc6\" (UID: \"8cf08c4a-4b04-4a73-a2dd-39bd58066be9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895270-6trc6" Nov 03 14:30:00.485353 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.485312 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d4wm\" (UniqueName: \"kubernetes.io/projected/417d374a-e02d-4721-a1df-e2b912877928-kube-api-access-8d4wm\") pod \"maas-api-key-cleanup-e2e-worker-w1-34b13c-29895270-69dbd\" (UID: \"417d374a-e02d-4721-a1df-e2b912877928\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895270-69dbd" Nov 03 14:30:00.485633 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.485610 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwh6g\" (UniqueName: \"kubernetes.io/projected/68fcc652-c24d-4b11-84a0-8e424f2e130c-kube-api-access-mwh6g\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895270-2qcwp\" (UID: \"68fcc652-c24d-4b11-84a0-8e424f2e130c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895270-2qcwp" Nov 03 14:30:00.485745 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.485696 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t2g7\" (UniqueName: \"kubernetes.io/projected/a0b8e1f3-1744-4fe0-8f2d-04b3ec7a13ad-kube-api-access-2t2g7\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895270-7csjj\" (UID: \"a0b8e1f3-1744-4fe0-8f2d-04b3ec7a13ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895270-7csjj" Nov 03 14:30:00.485745 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.485714 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phj7c\" (UniqueName: \"kubernetes.io/projected/8cf08c4a-4b04-4a73-a2dd-39bd58066be9-kube-api-access-phj7c\") pod \"maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895270-6trc6\" (UID: \"8cf08c4a-4b04-4a73-a2dd-39bd58066be9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895270-6trc6" Nov 03 14:30:00.485855 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.485838 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwc6s\" (UniqueName: \"kubernetes.io/projected/041bbdde-47d2-46eb-a5d4-bae94c32990f-kube-api-access-pwc6s\") pod \"maas-api-key-cleanup-29895270-4vv76\" (UID: \"041bbdde-47d2-46eb-a5d4-bae94c32990f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895270-4vv76" Nov 03 14:30:00.487158 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.487142 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895270-7csjj" Nov 03 14:30:00.495463 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.495442 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895270-69dbd" Nov 03 14:30:00.502900 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.502877 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895270-2qcwp" Nov 03 14:30:00.510992 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.510780 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895270-4vv76" Nov 03 14:30:00.518009 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.517985 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895270-6trc6" Nov 03 14:30:00.876794 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.876708 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895270-69dbd"] Nov 03 14:30:00.884125 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:30:00.884088 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod417d374a_e02d_4721_a1df_e2b912877928.slice/crio-ccc3490245896533bdf620d45f374386531f80052cf618caf1a0553868efd807 WatchSource:0}: Error finding container ccc3490245896533bdf620d45f374386531f80052cf618caf1a0553868efd807: Status 404 returned error can't find the container with id ccc3490245896533bdf620d45f374386531f80052cf618caf1a0553868efd807 Nov 03 14:30:00.930246 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.930215 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895270-7csjj"] Nov 03 14:30:00.937696 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:30:00.937663 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0b8e1f3_1744_4fe0_8f2d_04b3ec7a13ad.slice/crio-9cc598a8c5bb1c972c1fae270b5dfccf189a41f1df3062b0290ef24ba34f4107 WatchSource:0}: Error finding container 9cc598a8c5bb1c972c1fae270b5dfccf189a41f1df3062b0290ef24ba34f4107: Status 404 returned error can't find the container with id 9cc598a8c5bb1c972c1fae270b5dfccf189a41f1df3062b0290ef24ba34f4107 Nov 03 14:30:00.988984 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.988947 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895270-2qcwp"] Nov 03 14:30:00.991006 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.990979 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895270-6trc6"] Nov 03 14:30:00.993354 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:00.993331 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895270-4vv76"] Nov 03 14:30:00.993680 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:30:00.993654 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68fcc652_c24d_4b11_84a0_8e424f2e130c.slice/crio-bac321ad96cb3f61fed51945e9fa668e99c002a007dade8041d917342b724fa3 WatchSource:0}: Error finding container bac321ad96cb3f61fed51945e9fa668e99c002a007dade8041d917342b724fa3: Status 404 returned error can't find the container with id bac321ad96cb3f61fed51945e9fa668e99c002a007dade8041d917342b724fa3 Nov 03 14:30:00.997423 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:30:00.997354 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod041bbdde_47d2_46eb_a5d4_bae94c32990f.slice/crio-3385378cf6b77b8ca4172174967cf4faacb543b85ba0af9fd73e3adfa9e32e32 WatchSource:0}: Error finding container 3385378cf6b77b8ca4172174967cf4faacb543b85ba0af9fd73e3adfa9e32e32: Status 404 returned error can't find the container with id 3385378cf6b77b8ca4172174967cf4faacb543b85ba0af9fd73e3adfa9e32e32 Nov 03 14:30:00.998840 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:30:00.998815 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cf08c4a_4b04_4a73_a2dd_39bd58066be9.slice/crio-f1a53efa0ea499e54effd9bcbcd7d7780e604b4f7c8f4c72bcaa72799fd083af WatchSource:0}: Error finding container f1a53efa0ea499e54effd9bcbcd7d7780e604b4f7c8f4c72bcaa72799fd083af: Status 404 returned error can't find the container with id f1a53efa0ea499e54effd9bcbcd7d7780e604b4f7c8f4c72bcaa72799fd083af Nov 03 14:30:01.711380 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:01.711278 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895270-69dbd" event={"ID":"417d374a-e02d-4721-a1df-e2b912877928","Type":"ContainerStarted","Data":"093949fffacfd25c1950374a2e88eddf3555624d2f278dd6817dc23d33403fd2"} Nov 03 14:30:01.711380 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:01.711347 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895270-69dbd" event={"ID":"417d374a-e02d-4721-a1df-e2b912877928","Type":"ContainerStarted","Data":"ccc3490245896533bdf620d45f374386531f80052cf618caf1a0553868efd807"} Nov 03 14:30:01.713020 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:01.712990 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895270-2qcwp" event={"ID":"68fcc652-c24d-4b11-84a0-8e424f2e130c","Type":"ContainerStarted","Data":"48e6bc6f24b557938b5df23974fe38147efed914c069a56f57adef4c2aea7401"} Nov 03 14:30:01.713155 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:01.713036 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895270-2qcwp" event={"ID":"68fcc652-c24d-4b11-84a0-8e424f2e130c","Type":"ContainerStarted","Data":"bac321ad96cb3f61fed51945e9fa668e99c002a007dade8041d917342b724fa3"} Nov 03 14:30:01.714649 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:01.714623 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895270-4vv76" event={"ID":"041bbdde-47d2-46eb-a5d4-bae94c32990f","Type":"ContainerStarted","Data":"21badc749416e7b25f49ac7328422e524d9b2812e4cb66cbedb2250af34b5e2d"} Nov 03 14:30:01.714771 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:01.714671 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895270-4vv76" event={"ID":"041bbdde-47d2-46eb-a5d4-bae94c32990f","Type":"ContainerStarted","Data":"3385378cf6b77b8ca4172174967cf4faacb543b85ba0af9fd73e3adfa9e32e32"} Nov 03 14:30:01.716045 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:01.716025 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895270-7csjj" event={"ID":"a0b8e1f3-1744-4fe0-8f2d-04b3ec7a13ad","Type":"ContainerStarted","Data":"47898655976a7f7f7e2ba713d7314054d92afb7d7eabb6326330ed0c7c7d9e89"} Nov 03 14:30:01.716138 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:01.716063 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895270-7csjj" event={"ID":"a0b8e1f3-1744-4fe0-8f2d-04b3ec7a13ad","Type":"ContainerStarted","Data":"9cc598a8c5bb1c972c1fae270b5dfccf189a41f1df3062b0290ef24ba34f4107"} Nov 03 14:30:01.717393 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:01.717372 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895270-6trc6" event={"ID":"8cf08c4a-4b04-4a73-a2dd-39bd58066be9","Type":"ContainerStarted","Data":"1ba481db5904b02b33f4ed1d241ef8820495517e34d0442522541b770f596cfc"} Nov 03 14:30:01.717518 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:01.717410 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895270-6trc6" event={"ID":"8cf08c4a-4b04-4a73-a2dd-39bd58066be9","Type":"ContainerStarted","Data":"f1a53efa0ea499e54effd9bcbcd7d7780e604b4f7c8f4c72bcaa72799fd083af"} Nov 03 14:30:01.820269 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:01.820212 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895270-69dbd" podStartSLOduration=1.8201971540000002 podStartE2EDuration="1.820197154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:30:01.818938072 +0000 UTC m=+5083.829626315" watchObservedRunningTime="2026-11-03 14:30:01.820197154 +0000 UTC m=+5083.830885390" Nov 03 14:30:01.930081 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:01.930017 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895270-4vv76" podStartSLOduration=1.930004367 podStartE2EDuration="1.930004367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:30:01.928819832 +0000 UTC m=+5083.939508076" watchObservedRunningTime="2026-11-03 14:30:01.930004367 +0000 UTC m=+5083.940692609" Nov 03 14:30:02.038499 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:02.038379 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895270-7csjj" podStartSLOduration=2.038366088 podStartE2EDuration="2.038366088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:30:02.035588193 +0000 UTC m=+5084.046276434" watchObservedRunningTime="2026-11-03 14:30:02.038366088 +0000 UTC m=+5084.049054339" Nov 03 14:30:02.144508 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:02.144312 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895270-2qcwp" podStartSLOduration=2.144297484 podStartE2EDuration="2.144297484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:30:02.142134039 +0000 UTC m=+5084.152822281" watchObservedRunningTime="2026-11-03 14:30:02.144297484 +0000 UTC m=+5084.154985726" Nov 03 14:30:02.248346 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:02.248288 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895270-6trc6" podStartSLOduration=2.248270436 podStartE2EDuration="2.248270436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:30:02.248163785 +0000 UTC m=+5084.258852027" watchObservedRunningTime="2026-11-03 14:30:02.248270436 +0000 UTC m=+5084.258958678" Nov 03 14:30:02.668204 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:02.668171 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895270-6trc6"] Nov 03 14:30:02.728336 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:02.728298 5979 generic.go:356] "Generic (PLEG): container finished" podID="041bbdde-47d2-46eb-a5d4-bae94c32990f" containerID="21badc749416e7b25f49ac7328422e524d9b2812e4cb66cbedb2250af34b5e2d" exitCode=60 Nov 03 14:30:02.728858 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:02.728377 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895270-4vv76" event={"ID":"041bbdde-47d2-46eb-a5d4-bae94c32990f","Type":"ContainerDied","Data":"21badc749416e7b25f49ac7328422e524d9b2812e4cb66cbedb2250af34b5e2d"} Nov 03 14:30:02.728858 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:02.728667 5979 scope.go:122] "RemoveContainer" containerID="21badc749416e7b25f49ac7328422e524d9b2812e4cb66cbedb2250af34b5e2d" Nov 03 14:30:02.730233 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:02.730076 5979 generic.go:356] "Generic (PLEG): container finished" podID="a0b8e1f3-1744-4fe0-8f2d-04b3ec7a13ad" containerID="47898655976a7f7f7e2ba713d7314054d92afb7d7eabb6326330ed0c7c7d9e89" exitCode=60 Nov 03 14:30:02.730233 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:02.730156 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895270-7csjj" event={"ID":"a0b8e1f3-1744-4fe0-8f2d-04b3ec7a13ad","Type":"ContainerDied","Data":"47898655976a7f7f7e2ba713d7314054d92afb7d7eabb6326330ed0c7c7d9e89"} Nov 03 14:30:02.730394 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:02.730373 5979 scope.go:122] "RemoveContainer" containerID="47898655976a7f7f7e2ba713d7314054d92afb7d7eabb6326330ed0c7c7d9e89" Nov 03 14:30:02.731669 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:02.731645 5979 generic.go:356] "Generic (PLEG): container finished" podID="8cf08c4a-4b04-4a73-a2dd-39bd58066be9" containerID="1ba481db5904b02b33f4ed1d241ef8820495517e34d0442522541b770f596cfc" exitCode=60 Nov 03 14:30:02.731733 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:02.731712 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895270-6trc6" event={"ID":"8cf08c4a-4b04-4a73-a2dd-39bd58066be9","Type":"ContainerDied","Data":"1ba481db5904b02b33f4ed1d241ef8820495517e34d0442522541b770f596cfc"} Nov 03 14:30:02.731829 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:02.731810 5979 scope.go:122] "RemoveContainer" containerID="1ba481db5904b02b33f4ed1d241ef8820495517e34d0442522541b770f596cfc" Nov 03 14:30:02.733446 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:02.733421 5979 generic.go:356] "Generic (PLEG): container finished" podID="417d374a-e02d-4721-a1df-e2b912877928" containerID="093949fffacfd25c1950374a2e88eddf3555624d2f278dd6817dc23d33403fd2" exitCode=60 Nov 03 14:30:02.733545 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:02.733503 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895270-69dbd" event={"ID":"417d374a-e02d-4721-a1df-e2b912877928","Type":"ContainerDied","Data":"093949fffacfd25c1950374a2e88eddf3555624d2f278dd6817dc23d33403fd2"} Nov 03 14:30:02.733669 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:02.733651 5979 scope.go:122] "RemoveContainer" containerID="093949fffacfd25c1950374a2e88eddf3555624d2f278dd6817dc23d33403fd2" Nov 03 14:30:02.735159 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:02.735138 5979 generic.go:356] "Generic (PLEG): container finished" podID="68fcc652-c24d-4b11-84a0-8e424f2e130c" containerID="48e6bc6f24b557938b5df23974fe38147efed914c069a56f57adef4c2aea7401" exitCode=60 Nov 03 14:30:02.735255 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:02.735206 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895270-2qcwp" event={"ID":"68fcc652-c24d-4b11-84a0-8e424f2e130c","Type":"ContainerDied","Data":"48e6bc6f24b557938b5df23974fe38147efed914c069a56f57adef4c2aea7401"} Nov 03 14:30:02.735359 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:02.735347 5979 scope.go:122] "RemoveContainer" containerID="48e6bc6f24b557938b5df23974fe38147efed914c069a56f57adef4c2aea7401" Nov 03 14:30:03.748128 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:03.748093 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895270-4vv76" event={"ID":"041bbdde-47d2-46eb-a5d4-bae94c32990f","Type":"ContainerStarted","Data":"053416e69c0c6fc50394b2a661216656e59e7504b1c9ae1c8adb704807dbbb66"} Nov 03 14:30:03.750648 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:03.750618 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895270-7csjj" event={"ID":"a0b8e1f3-1744-4fe0-8f2d-04b3ec7a13ad","Type":"ContainerStarted","Data":"1542f831b2e0541c5ba5fdf18cd7426614cf3d496970bddf0d88ca282316a324"} Nov 03 14:30:03.753766 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:03.753740 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895270-6trc6" event={"ID":"8cf08c4a-4b04-4a73-a2dd-39bd58066be9","Type":"ContainerStarted","Data":"ee7609015fa22b59df360701c13e59fd2b6b9441bc707ef0dfa7964b36115c54"} Nov 03 14:30:03.753884 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:03.753765 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895270-6trc6" podUID="8cf08c4a-4b04-4a73-a2dd-39bd58066be9" containerName="cleanup" containerID="cri-o://ee7609015fa22b59df360701c13e59fd2b6b9441bc707ef0dfa7964b36115c54" gracePeriod=30 Nov 03 14:30:03.756431 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:03.756401 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895270-69dbd" event={"ID":"417d374a-e02d-4721-a1df-e2b912877928","Type":"ContainerStarted","Data":"5a4aa16352189cc0b9626fd3a0663c27080862d0f1f62f3ef231a861e1745e25"} Nov 03 14:30:03.758360 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:03.758338 5979 generic.go:356] "Generic (PLEG): container finished" podID="68fcc652-c24d-4b11-84a0-8e424f2e130c" containerID="f3fb5be71f134f4111f96da355c3d91a2527bb909ba69b7bc7e8e2e7d0dcd585" exitCode=60 Nov 03 14:30:03.758484 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:03.758367 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895270-2qcwp" event={"ID":"68fcc652-c24d-4b11-84a0-8e424f2e130c","Type":"ContainerDied","Data":"f3fb5be71f134f4111f96da355c3d91a2527bb909ba69b7bc7e8e2e7d0dcd585"} Nov 03 14:30:03.758484 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:03.758403 5979 scope.go:122] "RemoveContainer" containerID="48e6bc6f24b557938b5df23974fe38147efed914c069a56f57adef4c2aea7401" Nov 03 14:30:03.758721 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:03.758705 5979 scope.go:122] "RemoveContainer" containerID="f3fb5be71f134f4111f96da355c3d91a2527bb909ba69b7bc7e8e2e7d0dcd585" Nov 03 14:30:03.758950 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:30:03.758928 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895270-2qcwp_odh-ai-gateway-infra(68fcc652-c24d-4b11-84a0-8e424f2e130c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895270-2qcwp" podUID="68fcc652-c24d-4b11-84a0-8e424f2e130c" Nov 03 14:30:04.766787 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:04.766746 5979 generic.go:356] "Generic (PLEG): container finished" podID="041bbdde-47d2-46eb-a5d4-bae94c32990f" containerID="053416e69c0c6fc50394b2a661216656e59e7504b1c9ae1c8adb704807dbbb66" exitCode=60 Nov 03 14:30:04.767259 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:04.766824 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895270-4vv76" event={"ID":"041bbdde-47d2-46eb-a5d4-bae94c32990f","Type":"ContainerDied","Data":"053416e69c0c6fc50394b2a661216656e59e7504b1c9ae1c8adb704807dbbb66"} Nov 03 14:30:04.767259 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:04.766886 5979 scope.go:122] "RemoveContainer" containerID="21badc749416e7b25f49ac7328422e524d9b2812e4cb66cbedb2250af34b5e2d" Nov 03 14:30:04.767259 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:04.767201 5979 scope.go:122] "RemoveContainer" containerID="053416e69c0c6fc50394b2a661216656e59e7504b1c9ae1c8adb704807dbbb66" Nov 03 14:30:04.767568 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:30:04.767539 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895270-4vv76_odh-ai-gateway-infra(041bbdde-47d2-46eb-a5d4-bae94c32990f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895270-4vv76" podUID="041bbdde-47d2-46eb-a5d4-bae94c32990f" Nov 03 14:30:04.768964 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:04.768943 5979 generic.go:356] "Generic (PLEG): container finished" podID="a0b8e1f3-1744-4fe0-8f2d-04b3ec7a13ad" containerID="1542f831b2e0541c5ba5fdf18cd7426614cf3d496970bddf0d88ca282316a324" exitCode=60 Nov 03 14:30:04.769046 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:04.769002 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895270-7csjj" event={"ID":"a0b8e1f3-1744-4fe0-8f2d-04b3ec7a13ad","Type":"ContainerDied","Data":"1542f831b2e0541c5ba5fdf18cd7426614cf3d496970bddf0d88ca282316a324"} Nov 03 14:30:04.769281 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:04.769267 5979 scope.go:122] "RemoveContainer" containerID="1542f831b2e0541c5ba5fdf18cd7426614cf3d496970bddf0d88ca282316a324" Nov 03 14:30:04.769546 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:30:04.769529 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895270-7csjj_odh-ai-gateway-infra(a0b8e1f3-1744-4fe0-8f2d-04b3ec7a13ad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895270-7csjj" podUID="a0b8e1f3-1744-4fe0-8f2d-04b3ec7a13ad" Nov 03 14:30:04.770805 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:04.770781 5979 generic.go:356] "Generic (PLEG): container finished" podID="417d374a-e02d-4721-a1df-e2b912877928" containerID="5a4aa16352189cc0b9626fd3a0663c27080862d0f1f62f3ef231a861e1745e25" exitCode=60 Nov 03 14:30:04.770897 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:04.770807 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895270-69dbd" event={"ID":"417d374a-e02d-4721-a1df-e2b912877928","Type":"ContainerDied","Data":"5a4aa16352189cc0b9626fd3a0663c27080862d0f1f62f3ef231a861e1745e25"} Nov 03 14:30:04.771033 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:04.771018 5979 scope.go:122] "RemoveContainer" containerID="5a4aa16352189cc0b9626fd3a0663c27080862d0f1f62f3ef231a861e1745e25" Nov 03 14:30:04.771264 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:30:04.771246 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-34b13c-29895270-69dbd_odh-ai-gateway-infra(417d374a-e02d-4721-a1df-e2b912877928)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895270-69dbd" podUID="417d374a-e02d-4721-a1df-e2b912877928" Nov 03 14:30:04.772642 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:04.772624 5979 scope.go:122] "RemoveContainer" containerID="f3fb5be71f134f4111f96da355c3d91a2527bb909ba69b7bc7e8e2e7d0dcd585" Nov 03 14:30:04.772831 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:30:04.772811 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895270-2qcwp_odh-ai-gateway-infra(68fcc652-c24d-4b11-84a0-8e424f2e130c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895270-2qcwp" podUID="68fcc652-c24d-4b11-84a0-8e424f2e130c" Nov 03 14:30:04.780557 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:04.780537 5979 scope.go:122] "RemoveContainer" containerID="47898655976a7f7f7e2ba713d7314054d92afb7d7eabb6326330ed0c7c7d9e89" Nov 03 14:30:04.793157 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:04.793135 5979 scope.go:122] "RemoveContainer" containerID="093949fffacfd25c1950374a2e88eddf3555624d2f278dd6817dc23d33403fd2" Nov 03 14:30:05.418379 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:05.418344 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5"] Nov 03 14:30:05.423418 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:05.423396 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5" Nov 03 14:30:05.472606 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:05.472576 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-3c5dbc-openshift-default-dockercfg-hfbq4\"" Nov 03 14:30:05.483808 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:05.483777 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5"] Nov 03 14:30:05.488713 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:05.488686 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-credential-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5\" (UID: \"3b7f6f99-6953-4354-8c33-b4b1765aa4d9\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5" Nov 03 14:30:05.488865 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:05.488794 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-istio-token\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5\" (UID: \"3b7f6f99-6953-4354-8c33-b4b1765aa4d9\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5" Nov 03 14:30:05.488865 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:05.488822 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-istio-data\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5\" (UID: \"3b7f6f99-6953-4354-8c33-b4b1765aa4d9\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5" Nov 03 14:30:05.489010 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:05.488881 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-istio-podinfo\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5\" (UID: \"3b7f6f99-6953-4354-8c33-b4b1765aa4d9\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5" Nov 03 14:30:05.489010 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:05.488905 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-workload-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5\" (UID: \"3b7f6f99-6953-4354-8c33-b4b1765aa4d9\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5" Nov 03 14:30:05.489010 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:05.488953 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-workload-certs\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5\" (UID: \"3b7f6f99-6953-4354-8c33-b4b1765aa4d9\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5" Nov 03 14:30:05.489010 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:05.488977 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-istiod-ca-cert\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5\" (UID: \"3b7f6f99-6953-4354-8c33-b4b1765aa4d9\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5" Nov 03 14:30:05.489010 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:05.489010 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svxs5\" (UniqueName: \"kubernetes.io/projected/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-kube-api-access-svxs5\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5\" (UID: \"3b7f6f99-6953-4354-8c33-b4b1765aa4d9\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5" Nov 03 14:30:05.489254 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:05.489053 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-istio-envoy\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5\" (UID: \"3b7f6f99-6953-4354-8c33-b4b1765aa4d9\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5" Nov 03 14:30:05.590046 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:05.590012 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-workload-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5\" (UID: \"3b7f6f99-6953-4354-8c33-b4b1765aa4d9\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5" Nov 03 14:30:05.590224 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:05.590078 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-workload-certs\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5\" (UID: \"3b7f6f99-6953-4354-8c33-b4b1765aa4d9\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5" Nov 03 14:30:05.590224 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:05.590117 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-istiod-ca-cert\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5\" (UID: \"3b7f6f99-6953-4354-8c33-b4b1765aa4d9\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5" Nov 03 14:30:05.590224 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:05.590169 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svxs5\" (UniqueName: \"kubernetes.io/projected/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-kube-api-access-svxs5\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5\" (UID: \"3b7f6f99-6953-4354-8c33-b4b1765aa4d9\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5" Nov 03 14:30:05.590412 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:05.590223 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-istio-envoy\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5\" (UID: \"3b7f6f99-6953-4354-8c33-b4b1765aa4d9\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5" Nov 03 14:30:05.590412 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:05.590261 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-credential-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5\" (UID: \"3b7f6f99-6953-4354-8c33-b4b1765aa4d9\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5" Nov 03 14:30:05.590412 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:05.590344 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-istio-token\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5\" (UID: \"3b7f6f99-6953-4354-8c33-b4b1765aa4d9\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5" Nov 03 14:30:05.590412 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:05.590375 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-istio-data\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5\" (UID: \"3b7f6f99-6953-4354-8c33-b4b1765aa4d9\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5" Nov 03 14:30:05.590412 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:05.590409 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-istio-podinfo\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5\" (UID: \"3b7f6f99-6953-4354-8c33-b4b1765aa4d9\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5" Nov 03 14:30:05.590771 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:05.590508 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-workload-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5\" (UID: \"3b7f6f99-6953-4354-8c33-b4b1765aa4d9\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5" Nov 03 14:30:05.590771 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:05.590561 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-workload-certs\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5\" (UID: \"3b7f6f99-6953-4354-8c33-b4b1765aa4d9\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5" Nov 03 14:30:05.590881 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:05.590817 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-istio-data\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5\" (UID: \"3b7f6f99-6953-4354-8c33-b4b1765aa4d9\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5" Nov 03 14:30:05.590881 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:05.590847 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-credential-socket\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5\" (UID: \"3b7f6f99-6953-4354-8c33-b4b1765aa4d9\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5" Nov 03 14:30:05.590961 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:05.590886 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-istiod-ca-cert\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5\" (UID: \"3b7f6f99-6953-4354-8c33-b4b1765aa4d9\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5" Nov 03 14:30:05.592561 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:05.592540 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-istio-envoy\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5\" (UID: \"3b7f6f99-6953-4354-8c33-b4b1765aa4d9\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5" Nov 03 14:30:05.592779 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:05.592763 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-istio-podinfo\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5\" (UID: \"3b7f6f99-6953-4354-8c33-b4b1765aa4d9\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5" Nov 03 14:30:05.644383 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:05.644347 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-istio-token\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5\" (UID: \"3b7f6f99-6953-4354-8c33-b4b1765aa4d9\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5" Nov 03 14:30:05.644810 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:05.644783 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svxs5\" (UniqueName: \"kubernetes.io/projected/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-kube-api-access-svxs5\") pod \"e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5\" (UID: \"3b7f6f99-6953-4354-8c33-b4b1765aa4d9\") " pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5" Nov 03 14:30:05.736464 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:05.736386 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5" Nov 03 14:30:05.781667 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:05.781634 5979 scope.go:122] "RemoveContainer" containerID="1542f831b2e0541c5ba5fdf18cd7426614cf3d496970bddf0d88ca282316a324" Nov 03 14:30:05.782099 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:30:05.781949 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895270-7csjj_odh-ai-gateway-infra(a0b8e1f3-1744-4fe0-8f2d-04b3ec7a13ad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895270-7csjj" podUID="a0b8e1f3-1744-4fe0-8f2d-04b3ec7a13ad" Nov 03 14:30:05.783374 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:05.783351 5979 scope.go:122] "RemoveContainer" containerID="5a4aa16352189cc0b9626fd3a0663c27080862d0f1f62f3ef231a861e1745e25" Nov 03 14:30:05.783627 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:30:05.783606 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-34b13c-29895270-69dbd_odh-ai-gateway-infra(417d374a-e02d-4721-a1df-e2b912877928)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895270-69dbd" podUID="417d374a-e02d-4721-a1df-e2b912877928" Nov 03 14:30:05.785107 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:05.785092 5979 scope.go:122] "RemoveContainer" containerID="053416e69c0c6fc50394b2a661216656e59e7504b1c9ae1c8adb704807dbbb66" Nov 03 14:30:05.785266 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:30:05.785251 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895270-4vv76_odh-ai-gateway-infra(041bbdde-47d2-46eb-a5d4-bae94c32990f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895270-4vv76" podUID="041bbdde-47d2-46eb-a5d4-bae94c32990f" Nov 03 14:30:06.163159 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:06.163127 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5"] Nov 03 14:30:06.171123 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:30:06.171095 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b7f6f99_6953_4354_8c33_b4b1765aa4d9.slice/crio-3032ab3745e3dfc57426675ce7ac64d88ee6aed0120307535d3fb31a14666bb7 WatchSource:0}: Error finding container 3032ab3745e3dfc57426675ce7ac64d88ee6aed0120307535d3fb31a14666bb7: Status 404 returned error can't find the container with id 3032ab3745e3dfc57426675ce7ac64d88ee6aed0120307535d3fb31a14666bb7 Nov 03 14:30:06.173384 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:06.173354 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:30:06.173510 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:06.173418 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:30:06.173510 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:06.173444 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:30:06.792452 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:06.792411 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5" event={"ID":"3b7f6f99-6953-4354-8c33-b4b1765aa4d9","Type":"ContainerStarted","Data":"fc3cf0a74ba1f3b9c4bcb54cf13c49bf128346038645cdc9e5f09d5c3825204b"} Nov 03 14:30:06.792846 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:06.792490 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5" event={"ID":"3b7f6f99-6953-4354-8c33-b4b1765aa4d9","Type":"ContainerStarted","Data":"3032ab3745e3dfc57426675ce7ac64d88ee6aed0120307535d3fb31a14666bb7"} Nov 03 14:30:06.905047 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:06.904998 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5" podStartSLOduration=1.904985938 podStartE2EDuration="1.904985938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:30:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:30:06.904124267 +0000 UTC m=+5088.914812509" watchObservedRunningTime="2026-11-03 14:30:06.904985938 +0000 UTC m=+5088.915674158" Nov 03 14:30:07.233538 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:30:07.233498 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:30:07.736669 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:07.736636 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5" Nov 03 14:30:07.738344 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:07.738317 5979 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.122:15021/healthz/ready\": dial tcp 10.134.1.122:15021: connect: connection refused" start-of-body= Nov 03 14:30:07.738481 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:07.738363 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5" podUID="3b7f6f99-6953-4354-8c33-b4b1765aa4d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.122:15021/healthz/ready\": dial tcp 10.134.1.122:15021: connect: connection refused" Nov 03 14:30:08.736969 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:08.736929 5979 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.122:15021/healthz/ready\": dial tcp 10.134.1.122:15021: connect: connection refused" start-of-body= Nov 03 14:30:08.737341 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:08.736992 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5" podUID="3b7f6f99-6953-4354-8c33-b4b1765aa4d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.122:15021/healthz/ready\": dial tcp 10.134.1.122:15021: connect: connection refused" Nov 03 14:30:09.114537 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:09.114424 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-34b13c-6bb577bd6f-r6tz2"] Nov 03 14:30:09.737363 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:09.737324 5979 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.122:15021/healthz/ready\": dial tcp 10.134.1.122:15021: connect: connection refused" start-of-body= Nov 03 14:30:09.737810 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:09.737384 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5" podUID="3b7f6f99-6953-4354-8c33-b4b1765aa4d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.122:15021/healthz/ready\": dial tcp 10.134.1.122:15021: connect: connection refused" Nov 03 14:30:10.736876 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:10.736832 5979 patch_prober.go:29] interesting pod/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.122:15021/healthz/ready\": dial tcp 10.134.1.122:15021: connect: connection refused" start-of-body= Nov 03 14:30:10.737042 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:10.736900 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5" podUID="3b7f6f99-6953-4354-8c33-b4b1765aa4d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.122:15021/healthz/ready\": dial tcp 10.134.1.122:15021: connect: connection refused" Nov 03 14:30:11.536170 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:11.536125 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5"] Nov 03 14:30:11.536680 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:11.536359 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5" podUID="3b7f6f99-6953-4354-8c33-b4b1765aa4d9" containerName="istio-proxy" containerID="cri-o://fc3cf0a74ba1f3b9c4bcb54cf13c49bf128346038645cdc9e5f09d5c3825204b" gracePeriod=30 Nov 03 14:30:14.115335 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:14.115297 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-34b13c-6bb577bd6f-r6tz2" podUID="420ec7c1-8dae-4500-9aec-5ed3d05433ff" containerName="payload-processing" containerID="cri-o://ddd30a3395444916b242909258c11f2bf4f37b87ac28222f509f8f0cbef63bd1" gracePeriod=25 Nov 03 14:30:14.367170 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:14.367105 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-34b13c-6bb577bd6f-r6tz2" Nov 03 14:30:14.517778 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:14.517741 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/420ec7c1-8dae-4500-9aec-5ed3d05433ff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/420ec7c1-8dae-4500-9aec-5ed3d05433ff-plugins-config-volume\") pod \"420ec7c1-8dae-4500-9aec-5ed3d05433ff\" (UID: \"420ec7c1-8dae-4500-9aec-5ed3d05433ff\") " Nov 03 14:30:14.517981 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:14.517921 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/420ec7c1-8dae-4500-9aec-5ed3d05433ff-kube-api-access-k76np\" (UniqueName: \"kubernetes.io/projected/420ec7c1-8dae-4500-9aec-5ed3d05433ff-kube-api-access-k76np\") pod \"420ec7c1-8dae-4500-9aec-5ed3d05433ff\" (UID: \"420ec7c1-8dae-4500-9aec-5ed3d05433ff\") " Nov 03 14:30:14.518193 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:14.518168 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/420ec7c1-8dae-4500-9aec-5ed3d05433ff-plugins-config-volume" pod "420ec7c1-8dae-4500-9aec-5ed3d05433ff" (UID: "420ec7c1-8dae-4500-9aec-5ed3d05433ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:30:14.520049 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:14.520017 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/420ec7c1-8dae-4500-9aec-5ed3d05433ff-kube-api-access-k76np" pod "420ec7c1-8dae-4500-9aec-5ed3d05433ff" (UID: "420ec7c1-8dae-4500-9aec-5ed3d05433ff"). InnerVolumeSpecName "kube-api-access-k76np". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:30:14.618908 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:14.618829 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/420ec7c1-8dae-4500-9aec-5ed3d05433ff-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:14.618908 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:14.618856 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k76np\" (UniqueName: \"kubernetes.io/projected/420ec7c1-8dae-4500-9aec-5ed3d05433ff-kube-api-access-k76np\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:14.841319 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:14.841281 5979 generic.go:356] "Generic (PLEG): container finished" podID="420ec7c1-8dae-4500-9aec-5ed3d05433ff" containerID="ddd30a3395444916b242909258c11f2bf4f37b87ac28222f509f8f0cbef63bd1" exitCode=0 Nov 03 14:30:14.841537 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:14.841326 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-34b13c-6bb577bd6f-r6tz2" Nov 03 14:30:14.841537 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:14.841362 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-34b13c-6bb577bd6f-r6tz2" event={"ID":"420ec7c1-8dae-4500-9aec-5ed3d05433ff","Type":"ContainerDied","Data":"ddd30a3395444916b242909258c11f2bf4f37b87ac28222f509f8f0cbef63bd1"} Nov 03 14:30:14.841537 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:14.841412 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-34b13c-6bb577bd6f-r6tz2" event={"ID":"420ec7c1-8dae-4500-9aec-5ed3d05433ff","Type":"ContainerDied","Data":"3db19283fb94cbc52c22aa38f9f555b8d109e9d8f882dedb894b87950fb4d7b5"} Nov 03 14:30:14.841537 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:14.841434 5979 scope.go:122] "RemoveContainer" containerID="ddd30a3395444916b242909258c11f2bf4f37b87ac28222f509f8f0cbef63bd1" Nov 03 14:30:14.851320 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:14.851302 5979 scope.go:122] "RemoveContainer" containerID="ddd30a3395444916b242909258c11f2bf4f37b87ac28222f509f8f0cbef63bd1" Nov 03 14:30:14.851606 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:30:14.851590 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddd30a3395444916b242909258c11f2bf4f37b87ac28222f509f8f0cbef63bd1\": container with ID starting with ddd30a3395444916b242909258c11f2bf4f37b87ac28222f509f8f0cbef63bd1 not found: ID does not exist" containerID="ddd30a3395444916b242909258c11f2bf4f37b87ac28222f509f8f0cbef63bd1" Nov 03 14:30:14.851668 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:14.851617 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddd30a3395444916b242909258c11f2bf4f37b87ac28222f509f8f0cbef63bd1"} err="failed to get container status \"ddd30a3395444916b242909258c11f2bf4f37b87ac28222f509f8f0cbef63bd1\": rpc error: code = NotFound desc = could not find container \"ddd30a3395444916b242909258c11f2bf4f37b87ac28222f509f8f0cbef63bd1\": container with ID starting with ddd30a3395444916b242909258c11f2bf4f37b87ac28222f509f8f0cbef63bd1 not found: ID does not exist" Nov 03 14:30:14.994911 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:14.994874 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-34b13c-6bb577bd6f-r6tz2"] Nov 03 14:30:15.047519 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:15.043720 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-34b13c-6bb577bd6f-r6tz2"] Nov 03 14:30:15.108145 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:15.108108 5979 scope.go:122] "RemoveContainer" containerID="f3fb5be71f134f4111f96da355c3d91a2527bb909ba69b7bc7e8e2e7d0dcd585" Nov 03 14:30:15.108348 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:15.108310 5979 scope.go:122] "RemoveContainer" containerID="053416e69c0c6fc50394b2a661216656e59e7504b1c9ae1c8adb704807dbbb66" Nov 03 14:30:15.850873 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:15.850828 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895270-4vv76" event={"ID":"041bbdde-47d2-46eb-a5d4-bae94c32990f","Type":"ContainerStarted","Data":"478481ff0a852ee02edc28102b622e144feaba17ce09a116a087e605af7cf9d1"} Nov 03 14:30:15.854330 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:15.854302 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895270-2qcwp" event={"ID":"68fcc652-c24d-4b11-84a0-8e424f2e130c","Type":"ContainerStarted","Data":"42ae374a72b592d0ad847fb3dcdf15c3ed4061008efa158ac97852dda4eb5e49"} Nov 03 14:30:16.108741 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:16.108655 5979 scope.go:122] "RemoveContainer" containerID="1542f831b2e0541c5ba5fdf18cd7426614cf3d496970bddf0d88ca282316a324" Nov 03 14:30:16.115031 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:16.115001 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="420ec7c1-8dae-4500-9aec-5ed3d05433ff" path="/var/lib/kubelet/pods/420ec7c1-8dae-4500-9aec-5ed3d05433ff/volumes" Nov 03 14:30:16.862645 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:16.862615 5979 generic.go:356] "Generic (PLEG): container finished" podID="041bbdde-47d2-46eb-a5d4-bae94c32990f" containerID="478481ff0a852ee02edc28102b622e144feaba17ce09a116a087e605af7cf9d1" exitCode=60 Nov 03 14:30:16.863115 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:16.862683 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895270-4vv76" event={"ID":"041bbdde-47d2-46eb-a5d4-bae94c32990f","Type":"ContainerDied","Data":"478481ff0a852ee02edc28102b622e144feaba17ce09a116a087e605af7cf9d1"} Nov 03 14:30:16.863115 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:16.862736 5979 scope.go:122] "RemoveContainer" containerID="053416e69c0c6fc50394b2a661216656e59e7504b1c9ae1c8adb704807dbbb66" Nov 03 14:30:16.863251 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:16.863117 5979 scope.go:122] "RemoveContainer" containerID="478481ff0a852ee02edc28102b622e144feaba17ce09a116a087e605af7cf9d1" Nov 03 14:30:16.863414 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:30:16.863392 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895270-4vv76_odh-ai-gateway-infra(041bbdde-47d2-46eb-a5d4-bae94c32990f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895270-4vv76" podUID="041bbdde-47d2-46eb-a5d4-bae94c32990f" Nov 03 14:30:16.864758 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:16.864733 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895270-7csjj" event={"ID":"a0b8e1f3-1744-4fe0-8f2d-04b3ec7a13ad","Type":"ContainerStarted","Data":"c7e7982171c27bdca318069de709fee946ed9b6b1699b9518a21688acba134b0"} Nov 03 14:30:16.866565 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:16.866543 5979 generic.go:356] "Generic (PLEG): container finished" podID="68fcc652-c24d-4b11-84a0-8e424f2e130c" containerID="42ae374a72b592d0ad847fb3dcdf15c3ed4061008efa158ac97852dda4eb5e49" exitCode=60 Nov 03 14:30:16.866731 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:16.866610 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895270-2qcwp" event={"ID":"68fcc652-c24d-4b11-84a0-8e424f2e130c","Type":"ContainerDied","Data":"42ae374a72b592d0ad847fb3dcdf15c3ed4061008efa158ac97852dda4eb5e49"} Nov 03 14:30:16.866890 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:16.866874 5979 scope.go:122] "RemoveContainer" containerID="42ae374a72b592d0ad847fb3dcdf15c3ed4061008efa158ac97852dda4eb5e49" Nov 03 14:30:16.867097 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:30:16.867081 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895270-2qcwp_odh-ai-gateway-infra(68fcc652-c24d-4b11-84a0-8e424f2e130c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895270-2qcwp" podUID="68fcc652-c24d-4b11-84a0-8e424f2e130c" Nov 03 14:30:16.875451 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:16.875432 5979 scope.go:122] "RemoveContainer" containerID="f3fb5be71f134f4111f96da355c3d91a2527bb909ba69b7bc7e8e2e7d0dcd585" Nov 03 14:30:16.949693 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:16.949657 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:30:16.967686 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:16.967660 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895270-4vv76"] Nov 03 14:30:17.050009 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:17.049981 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16decd81-wg8d8"] Nov 03 14:30:17.052554 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:17.052530 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16decd81-wg8d8"] Nov 03 14:30:17.090544 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:30:17.090510 5979 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0b8e1f3_1744_4fe0_8f2d_04b3ec7a13ad.slice/crio-conmon-c7e7982171c27bdca318069de709fee946ed9b6b1699b9518a21688acba134b0.scope\": RecentStats: unable to find data in memory cache]" Nov 03 14:30:17.090705 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:17.090541 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895270-2qcwp"] Nov 03 14:30:17.108050 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:17.108016 5979 scope.go:122] "RemoveContainer" containerID="5a4aa16352189cc0b9626fd3a0663c27080862d0f1f62f3ef231a861e1745e25" Nov 03 14:30:17.543515 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:17.532794 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5" Nov 03 14:30:17.657768 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:17.657738 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-istio-podinfo\") pod \"3b7f6f99-6953-4354-8c33-b4b1765aa4d9\" (UID: \"3b7f6f99-6953-4354-8c33-b4b1765aa4d9\") " Nov 03 14:30:17.657939 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:17.657787 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-istio-data\") pod \"3b7f6f99-6953-4354-8c33-b4b1765aa4d9\" (UID: \"3b7f6f99-6953-4354-8c33-b4b1765aa4d9\") " Nov 03 14:30:17.657939 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:17.657826 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-workload-certs\") pod \"3b7f6f99-6953-4354-8c33-b4b1765aa4d9\" (UID: \"3b7f6f99-6953-4354-8c33-b4b1765aa4d9\") " Nov 03 14:30:17.657939 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:17.657904 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-workload-socket\") pod \"3b7f6f99-6953-4354-8c33-b4b1765aa4d9\" (UID: \"3b7f6f99-6953-4354-8c33-b4b1765aa4d9\") " Nov 03 14:30:17.663019 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:17.657958 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-istiod-ca-cert\") pod \"3b7f6f99-6953-4354-8c33-b4b1765aa4d9\" (UID: \"3b7f6f99-6953-4354-8c33-b4b1765aa4d9\") " Nov 03 14:30:17.663019 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:17.658053 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-credential-socket\") pod \"3b7f6f99-6953-4354-8c33-b4b1765aa4d9\" (UID: \"3b7f6f99-6953-4354-8c33-b4b1765aa4d9\") " Nov 03 14:30:17.663019 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:17.658111 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-istio-token\" (UniqueName: \"kubernetes.io/projected/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-istio-token\") pod \"3b7f6f99-6953-4354-8c33-b4b1765aa4d9\" (UID: \"3b7f6f99-6953-4354-8c33-b4b1765aa4d9\") " Nov 03 14:30:17.663019 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:17.658134 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-istio-envoy\") pod \"3b7f6f99-6953-4354-8c33-b4b1765aa4d9\" (UID: \"3b7f6f99-6953-4354-8c33-b4b1765aa4d9\") " Nov 03 14:30:17.663019 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:17.658170 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-kube-api-access-svxs5\" (UniqueName: \"kubernetes.io/projected/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-kube-api-access-svxs5\") pod \"3b7f6f99-6953-4354-8c33-b4b1765aa4d9\" (UID: \"3b7f6f99-6953-4354-8c33-b4b1765aa4d9\") " Nov 03 14:30:17.663019 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:17.661062 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-credential-socket" pod "3b7f6f99-6953-4354-8c33-b4b1765aa4d9" (UID: "3b7f6f99-6953-4354-8c33-b4b1765aa4d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:30:17.663019 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:17.661077 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-workload-socket" pod "3b7f6f99-6953-4354-8c33-b4b1765aa4d9" (UID: "3b7f6f99-6953-4354-8c33-b4b1765aa4d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:30:17.663019 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:17.661349 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-istiod-ca-cert" pod "3b7f6f99-6953-4354-8c33-b4b1765aa4d9" (UID: "3b7f6f99-6953-4354-8c33-b4b1765aa4d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:30:17.663019 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:17.661559 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-istio-data" pod "3b7f6f99-6953-4354-8c33-b4b1765aa4d9" (UID: "3b7f6f99-6953-4354-8c33-b4b1765aa4d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:30:17.663019 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:17.661717 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-workload-certs" pod "3b7f6f99-6953-4354-8c33-b4b1765aa4d9" (UID: "3b7f6f99-6953-4354-8c33-b4b1765aa4d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:30:17.663574 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:17.663508 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-kube-api-access-svxs5" pod "3b7f6f99-6953-4354-8c33-b4b1765aa4d9" (UID: "3b7f6f99-6953-4354-8c33-b4b1765aa4d9"). InnerVolumeSpecName "kube-api-access-svxs5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:30:17.664219 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:17.664124 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-istio-token" pod "3b7f6f99-6953-4354-8c33-b4b1765aa4d9" (UID: "3b7f6f99-6953-4354-8c33-b4b1765aa4d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:30:17.664219 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:17.664191 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-istio-podinfo" pod "3b7f6f99-6953-4354-8c33-b4b1765aa4d9" (UID: "3b7f6f99-6953-4354-8c33-b4b1765aa4d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:30:17.665813 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:17.665792 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-istio-envoy" pod "3b7f6f99-6953-4354-8c33-b4b1765aa4d9" (UID: "3b7f6f99-6953-4354-8c33-b4b1765aa4d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:30:17.759948 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:17.759865 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:17.759948 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:17.759896 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:17.759948 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:17.759908 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:17.759948 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:17.759921 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:17.759948 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:17.759933 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:17.759948 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:17.759947 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:17.760282 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:17.759968 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:17.760282 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:17.759985 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:17.760282 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:17.759997 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svxs5\" (UniqueName: \"kubernetes.io/projected/3b7f6f99-6953-4354-8c33-b4b1765aa4d9-kube-api-access-svxs5\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:17.874466 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:17.874430 5979 generic.go:356] "Generic (PLEG): container finished" podID="a0b8e1f3-1744-4fe0-8f2d-04b3ec7a13ad" containerID="c7e7982171c27bdca318069de709fee946ed9b6b1699b9518a21688acba134b0" exitCode=60 Nov 03 14:30:17.874994 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:17.874502 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895270-7csjj" event={"ID":"a0b8e1f3-1744-4fe0-8f2d-04b3ec7a13ad","Type":"ContainerDied","Data":"c7e7982171c27bdca318069de709fee946ed9b6b1699b9518a21688acba134b0"} Nov 03 14:30:17.874994 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:17.874558 5979 scope.go:122] "RemoveContainer" containerID="1542f831b2e0541c5ba5fdf18cd7426614cf3d496970bddf0d88ca282316a324" Nov 03 14:30:17.874994 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:17.874937 5979 scope.go:122] "RemoveContainer" containerID="c7e7982171c27bdca318069de709fee946ed9b6b1699b9518a21688acba134b0" Nov 03 14:30:17.875216 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:30:17.875192 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895270-7csjj_odh-ai-gateway-infra(a0b8e1f3-1744-4fe0-8f2d-04b3ec7a13ad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895270-7csjj" podUID="a0b8e1f3-1744-4fe0-8f2d-04b3ec7a13ad" Nov 03 14:30:17.875948 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:17.875926 5979 generic.go:356] "Generic (PLEG): container finished" podID="3b7f6f99-6953-4354-8c33-b4b1765aa4d9" containerID="fc3cf0a74ba1f3b9c4bcb54cf13c49bf128346038645cdc9e5f09d5c3825204b" exitCode=0 Nov 03 14:30:17.876058 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:17.875984 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5" Nov 03 14:30:17.876058 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:17.876006 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5" event={"ID":"3b7f6f99-6953-4354-8c33-b4b1765aa4d9","Type":"ContainerDied","Data":"fc3cf0a74ba1f3b9c4bcb54cf13c49bf128346038645cdc9e5f09d5c3825204b"} Nov 03 14:30:17.876058 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:17.876053 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5" event={"ID":"3b7f6f99-6953-4354-8c33-b4b1765aa4d9","Type":"ContainerDied","Data":"3032ab3745e3dfc57426675ce7ac64d88ee6aed0120307535d3fb31a14666bb7"} Nov 03 14:30:17.877822 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:17.877802 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895270-69dbd" event={"ID":"417d374a-e02d-4721-a1df-e2b912877928","Type":"ContainerStarted","Data":"26beb71454079646a5f70660f671d1ce42f6082756d290b84e7f6f7fe6f4d034"} Nov 03 14:30:17.889869 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:17.889842 5979 scope.go:122] "RemoveContainer" containerID="fc3cf0a74ba1f3b9c4bcb54cf13c49bf128346038645cdc9e5f09d5c3825204b" Nov 03 14:30:17.900721 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:17.900694 5979 scope.go:122] "RemoveContainer" containerID="fc3cf0a74ba1f3b9c4bcb54cf13c49bf128346038645cdc9e5f09d5c3825204b" Nov 03 14:30:17.901073 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:30:17.901050 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc3cf0a74ba1f3b9c4bcb54cf13c49bf128346038645cdc9e5f09d5c3825204b\": container with ID starting with fc3cf0a74ba1f3b9c4bcb54cf13c49bf128346038645cdc9e5f09d5c3825204b not found: ID does not exist" containerID="fc3cf0a74ba1f3b9c4bcb54cf13c49bf128346038645cdc9e5f09d5c3825204b" Nov 03 14:30:17.901162 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:17.901089 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc3cf0a74ba1f3b9c4bcb54cf13c49bf128346038645cdc9e5f09d5c3825204b"} err="failed to get container status \"fc3cf0a74ba1f3b9c4bcb54cf13c49bf128346038645cdc9e5f09d5c3825204b\": rpc error: code = NotFound desc = could not find container \"fc3cf0a74ba1f3b9c4bcb54cf13c49bf128346038645cdc9e5f09d5c3825204b\": container with ID starting with fc3cf0a74ba1f3b9c4bcb54cf13c49bf128346038645cdc9e5f09d5c3825204b not found: ID does not exist" Nov 03 14:30:18.030295 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:18.030230 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895270-4vv76" Nov 03 14:30:18.063352 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:18.063323 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895270-2qcwp" Nov 03 14:30:18.114829 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:18.114791 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a62f61a9-1d81-4bcc-b161-a6ce4d667fe3" path="/var/lib/kubelet/pods/a62f61a9-1d81-4bcc-b161-a6ce4d667fe3/volumes" Nov 03 14:30:18.163873 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:18.163845 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68fcc652-c24d-4b11-84a0-8e424f2e130c-kube-api-access-mwh6g\" (UniqueName: \"kubernetes.io/projected/68fcc652-c24d-4b11-84a0-8e424f2e130c-kube-api-access-mwh6g\") pod \"68fcc652-c24d-4b11-84a0-8e424f2e130c\" (UID: \"68fcc652-c24d-4b11-84a0-8e424f2e130c\") " Nov 03 14:30:18.164021 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:18.163990 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/041bbdde-47d2-46eb-a5d4-bae94c32990f-kube-api-access-pwc6s\" (UniqueName: \"kubernetes.io/projected/041bbdde-47d2-46eb-a5d4-bae94c32990f-kube-api-access-pwc6s\") pod \"041bbdde-47d2-46eb-a5d4-bae94c32990f\" (UID: \"041bbdde-47d2-46eb-a5d4-bae94c32990f\") " Nov 03 14:30:18.164069 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:18.164055 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/041bbdde-47d2-46eb-a5d4-bae94c32990f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/041bbdde-47d2-46eb-a5d4-bae94c32990f-maas-api-service-ca\") pod \"041bbdde-47d2-46eb-a5d4-bae94c32990f\" (UID: \"041bbdde-47d2-46eb-a5d4-bae94c32990f\") " Nov 03 14:30:18.164173 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:18.164156 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68fcc652-c24d-4b11-84a0-8e424f2e130c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68fcc652-c24d-4b11-84a0-8e424f2e130c-maas-api-service-ca\") pod \"68fcc652-c24d-4b11-84a0-8e424f2e130c\" (UID: \"68fcc652-c24d-4b11-84a0-8e424f2e130c\") " Nov 03 14:30:18.164453 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:18.164424 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/041bbdde-47d2-46eb-a5d4-bae94c32990f-maas-api-service-ca" pod "041bbdde-47d2-46eb-a5d4-bae94c32990f" (UID: "041bbdde-47d2-46eb-a5d4-bae94c32990f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:30:18.164561 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:18.164506 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68fcc652-c24d-4b11-84a0-8e424f2e130c-maas-api-service-ca" pod "68fcc652-c24d-4b11-84a0-8e424f2e130c" (UID: "68fcc652-c24d-4b11-84a0-8e424f2e130c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:30:18.165832 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:18.165809 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68fcc652-c24d-4b11-84a0-8e424f2e130c-kube-api-access-mwh6g" pod "68fcc652-c24d-4b11-84a0-8e424f2e130c" (UID: "68fcc652-c24d-4b11-84a0-8e424f2e130c"). InnerVolumeSpecName "kube-api-access-mwh6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:30:18.165924 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:18.165894 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/041bbdde-47d2-46eb-a5d4-bae94c32990f-kube-api-access-pwc6s" pod "041bbdde-47d2-46eb-a5d4-bae94c32990f" (UID: "041bbdde-47d2-46eb-a5d4-bae94c32990f"). InnerVolumeSpecName "kube-api-access-pwc6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:30:18.229532 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:18.229492 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895270-7csjj"] Nov 03 14:30:18.266530 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:18.266493 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68fcc652-c24d-4b11-84a0-8e424f2e130c-maas-api-service-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:18.266530 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:18.266530 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwh6g\" (UniqueName: \"kubernetes.io/projected/68fcc652-c24d-4b11-84a0-8e424f2e130c-kube-api-access-mwh6g\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:18.266814 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:18.266554 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwc6s\" (UniqueName: \"kubernetes.io/projected/041bbdde-47d2-46eb-a5d4-bae94c32990f-kube-api-access-pwc6s\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:18.266814 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:18.266572 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/041bbdde-47d2-46eb-a5d4-bae94c32990f-maas-api-service-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:18.332314 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:18.332224 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5"] Nov 03 14:30:18.337636 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:18.337605 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3c5dbc-openshift-default-9bcf9f669-6pdm5"] Nov 03 14:30:18.888983 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:18.888955 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895270-4vv76" Nov 03 14:30:18.889442 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:18.888993 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895270-4vv76" event={"ID":"041bbdde-47d2-46eb-a5d4-bae94c32990f","Type":"ContainerDied","Data":"3385378cf6b77b8ca4172174967cf4faacb543b85ba0af9fd73e3adfa9e32e32"} Nov 03 14:30:18.889442 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:18.889076 5979 scope.go:122] "RemoveContainer" containerID="478481ff0a852ee02edc28102b622e144feaba17ce09a116a087e605af7cf9d1" Nov 03 14:30:18.894779 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:18.894753 5979 generic.go:356] "Generic (PLEG): container finished" podID="417d374a-e02d-4721-a1df-e2b912877928" containerID="26beb71454079646a5f70660f671d1ce42f6082756d290b84e7f6f7fe6f4d034" exitCode=60 Nov 03 14:30:18.894903 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:18.894820 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895270-69dbd" event={"ID":"417d374a-e02d-4721-a1df-e2b912877928","Type":"ContainerDied","Data":"26beb71454079646a5f70660f671d1ce42f6082756d290b84e7f6f7fe6f4d034"} Nov 03 14:30:18.895244 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:18.895224 5979 scope.go:122] "RemoveContainer" containerID="26beb71454079646a5f70660f671d1ce42f6082756d290b84e7f6f7fe6f4d034" Nov 03 14:30:18.895607 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:30:18.895529 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-34b13c-29895270-69dbd_odh-ai-gateway-infra(417d374a-e02d-4721-a1df-e2b912877928)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895270-69dbd" podUID="417d374a-e02d-4721-a1df-e2b912877928" Nov 03 14:30:18.900783 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:18.900767 5979 scope.go:122] "RemoveContainer" containerID="5a4aa16352189cc0b9626fd3a0663c27080862d0f1f62f3ef231a861e1745e25" Nov 03 14:30:18.901112 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:18.901089 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895270-2qcwp" event={"ID":"68fcc652-c24d-4b11-84a0-8e424f2e130c","Type":"ContainerDied","Data":"bac321ad96cb3f61fed51945e9fa668e99c002a007dade8041d917342b724fa3"} Nov 03 14:30:18.901209 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:18.901158 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895270-2qcwp" Nov 03 14:30:18.914803 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:18.914776 5979 scope.go:122] "RemoveContainer" containerID="42ae374a72b592d0ad847fb3dcdf15c3ed4061008efa158ac97852dda4eb5e49" Nov 03 14:30:19.044762 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:19.044730 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895270-7csjj" Nov 03 14:30:19.115267 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:19.115232 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895270-69dbd"] Nov 03 14:30:19.152081 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:19.152002 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895270-2qcwp"] Nov 03 14:30:19.156494 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:19.156445 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-d341af-29895270-2qcwp"] Nov 03 14:30:19.177100 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:19.177072 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0b8e1f3-1744-4fe0-8f2d-04b3ec7a13ad-kube-api-access-2t2g7\" (UniqueName: \"kubernetes.io/projected/a0b8e1f3-1744-4fe0-8f2d-04b3ec7a13ad-kube-api-access-2t2g7\") pod \"a0b8e1f3-1744-4fe0-8f2d-04b3ec7a13ad\" (UID: \"a0b8e1f3-1744-4fe0-8f2d-04b3ec7a13ad\") " Nov 03 14:30:19.177256 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:19.177132 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0b8e1f3-1744-4fe0-8f2d-04b3ec7a13ad-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b8e1f3-1744-4fe0-8f2d-04b3ec7a13ad-maas-api-service-ca\") pod \"a0b8e1f3-1744-4fe0-8f2d-04b3ec7a13ad\" (UID: \"a0b8e1f3-1744-4fe0-8f2d-04b3ec7a13ad\") " Nov 03 14:30:19.177542 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:19.177460 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0b8e1f3-1744-4fe0-8f2d-04b3ec7a13ad-maas-api-service-ca" pod "a0b8e1f3-1744-4fe0-8f2d-04b3ec7a13ad" (UID: "a0b8e1f3-1744-4fe0-8f2d-04b3ec7a13ad"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:30:19.179220 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:19.179193 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0b8e1f3-1744-4fe0-8f2d-04b3ec7a13ad-kube-api-access-2t2g7" pod "a0b8e1f3-1744-4fe0-8f2d-04b3ec7a13ad" (UID: "a0b8e1f3-1744-4fe0-8f2d-04b3ec7a13ad"). InnerVolumeSpecName "kube-api-access-2t2g7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:30:19.279192 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:19.278920 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2t2g7\" (UniqueName: \"kubernetes.io/projected/a0b8e1f3-1744-4fe0-8f2d-04b3ec7a13ad-kube-api-access-2t2g7\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:19.279192 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:19.278956 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0b8e1f3-1744-4fe0-8f2d-04b3ec7a13ad-maas-api-service-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:19.371804 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:19.371766 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895270-4vv76"] Nov 03 14:30:19.382115 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:19.382071 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895270-4vv76"] Nov 03 14:30:19.909043 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:19.909005 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895270-7csjj" Nov 03 14:30:19.909043 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:19.909035 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895270-7csjj" event={"ID":"a0b8e1f3-1744-4fe0-8f2d-04b3ec7a13ad","Type":"ContainerDied","Data":"9cc598a8c5bb1c972c1fae270b5dfccf189a41f1df3062b0290ef24ba34f4107"} Nov 03 14:30:19.909615 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:19.909105 5979 scope.go:122] "RemoveContainer" containerID="c7e7982171c27bdca318069de709fee946ed9b6b1699b9518a21688acba134b0" Nov 03 14:30:20.058156 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:20.058125 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895270-69dbd" Nov 03 14:30:20.085922 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:20.085883 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/417d374a-e02d-4721-a1df-e2b912877928-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/417d374a-e02d-4721-a1df-e2b912877928-maas-api-service-ca\") pod \"417d374a-e02d-4721-a1df-e2b912877928\" (UID: \"417d374a-e02d-4721-a1df-e2b912877928\") " Nov 03 14:30:20.086105 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:20.086011 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/417d374a-e02d-4721-a1df-e2b912877928-kube-api-access-8d4wm\" (UniqueName: \"kubernetes.io/projected/417d374a-e02d-4721-a1df-e2b912877928-kube-api-access-8d4wm\") pod \"417d374a-e02d-4721-a1df-e2b912877928\" (UID: \"417d374a-e02d-4721-a1df-e2b912877928\") " Nov 03 14:30:20.086254 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:20.086231 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/417d374a-e02d-4721-a1df-e2b912877928-maas-api-service-ca" pod "417d374a-e02d-4721-a1df-e2b912877928" (UID: "417d374a-e02d-4721-a1df-e2b912877928"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:30:20.086333 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:20.086320 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/417d374a-e02d-4721-a1df-e2b912877928-maas-api-service-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:20.088225 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:20.088197 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/417d374a-e02d-4721-a1df-e2b912877928-kube-api-access-8d4wm" pod "417d374a-e02d-4721-a1df-e2b912877928" (UID: "417d374a-e02d-4721-a1df-e2b912877928"). InnerVolumeSpecName "kube-api-access-8d4wm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:30:20.115686 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:20.115651 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="041bbdde-47d2-46eb-a5d4-bae94c32990f" path="/var/lib/kubelet/pods/041bbdde-47d2-46eb-a5d4-bae94c32990f/volumes" Nov 03 14:30:20.116192 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:20.116177 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b7f6f99-6953-4354-8c33-b4b1765aa4d9" path="/var/lib/kubelet/pods/3b7f6f99-6953-4354-8c33-b4b1765aa4d9/volumes" Nov 03 14:30:20.116553 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:20.116538 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68fcc652-c24d-4b11-84a0-8e424f2e130c" path="/var/lib/kubelet/pods/68fcc652-c24d-4b11-84a0-8e424f2e130c/volumes" Nov 03 14:30:20.171536 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:20.171447 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895270-7csjj"] Nov 03 14:30:20.175011 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:20.174983 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-2bceb1-29895270-7csjj"] Nov 03 14:30:20.186846 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:20.186816 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d4wm\" (UniqueName: \"kubernetes.io/projected/417d374a-e02d-4721-a1df-e2b912877928-kube-api-access-8d4wm\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:20.919878 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:20.919847 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895270-69dbd" Nov 03 14:30:20.920383 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:20.919848 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895270-69dbd" event={"ID":"417d374a-e02d-4721-a1df-e2b912877928","Type":"ContainerDied","Data":"ccc3490245896533bdf620d45f374386531f80052cf618caf1a0553868efd807"} Nov 03 14:30:20.920383 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:20.920009 5979 scope.go:122] "RemoveContainer" containerID="26beb71454079646a5f70660f671d1ce42f6082756d290b84e7f6f7fe6f4d034" Nov 03 14:30:21.073003 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:21.072972 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895270-69dbd"] Nov 03 14:30:21.119580 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:21.119547 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-34b13c-29895270-69dbd"] Nov 03 14:30:21.516591 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:21.516550 5979 scope.go:122] "RemoveContainer" containerID="8f65d5867c2cfe608b77a613457f667af16a99f3d984522d5180781de70e65e8" Nov 03 14:30:21.525850 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:21.525827 5979 scope.go:122] "RemoveContainer" containerID="9f93770d223351f867177c52585cafd9b5e342493f0429489a6ba7ef8ea6144a" Nov 03 14:30:21.534450 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:21.534421 5979 scope.go:122] "RemoveContainer" containerID="2bf8a76175d2954784636ee7c41c6e80dce1e34cbd5972432c0c1e74b9f23009" Nov 03 14:30:21.542906 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:21.542887 5979 scope.go:122] "RemoveContainer" containerID="cd92d9fcbd6f18d0b2912cb9d9eba675dea71acead1a8455846476eb444850b7" Nov 03 14:30:22.114392 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:22.114354 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="417d374a-e02d-4721-a1df-e2b912877928" path="/var/lib/kubelet/pods/417d374a-e02d-4721-a1df-e2b912877928/volumes" Nov 03 14:30:22.114826 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:22.114811 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0b8e1f3-1744-4fe0-8f2d-04b3ec7a13ad" path="/var/lib/kubelet/pods/a0b8e1f3-1744-4fe0-8f2d-04b3ec7a13ad/volumes" Nov 03 14:30:33.792376 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:30:33.792339 5979 manager.go:1119] Failed to create existing container: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68fcc652_c24d_4b11_84a0_8e424f2e130c.slice/crio-bac321ad96cb3f61fed51945e9fa668e99c002a007dade8041d917342b724fa3: Error finding container bac321ad96cb3f61fed51945e9fa668e99c002a007dade8041d917342b724fa3: Status 404 returned error can't find the container with id bac321ad96cb3f61fed51945e9fa668e99c002a007dade8041d917342b724fa3 Nov 03 14:30:33.793003 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:30:33.792982 5979 manager.go:1119] Failed to create existing container: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod041bbdde_47d2_46eb_a5d4_bae94c32990f.slice/crio-3385378cf6b77b8ca4172174967cf4faacb543b85ba0af9fd73e3adfa9e32e32: Error finding container 3385378cf6b77b8ca4172174967cf4faacb543b85ba0af9fd73e3adfa9e32e32: Status 404 returned error can't find the container with id 3385378cf6b77b8ca4172174967cf4faacb543b85ba0af9fd73e3adfa9e32e32 Nov 03 14:30:33.822821 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:30:33.822769 5979 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0b8e1f3_1744_4fe0_8f2d_04b3ec7a13ad.slice/crio-9cc598a8c5bb1c972c1fae270b5dfccf189a41f1df3062b0290ef24ba34f4107\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cf08c4a_4b04_4a73_a2dd_39bd58066be9.slice/crio-ee7609015fa22b59df360701c13e59fd2b6b9441bc707ef0dfa7964b36115c54.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0b8e1f3_1744_4fe0_8f2d_04b3ec7a13ad.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod417d374a_e02d_4721_a1df_e2b912877928.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68fcc652_c24d_4b11_84a0_8e424f2e130c.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cf08c4a_4b04_4a73_a2dd_39bd58066be9.slice/crio-conmon-ee7609015fa22b59df360701c13e59fd2b6b9441bc707ef0dfa7964b36115c54.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod417d374a_e02d_4721_a1df_e2b912877928.slice/crio-ccc3490245896533bdf620d45f374386531f80052cf618caf1a0553868efd807\": RecentStats: unable to find data in memory cache]" Nov 03 14:30:33.973966 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:33.957747 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895270-6trc6" Nov 03 14:30:33.995090 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:33.995054 5979 generic.go:356] "Generic (PLEG): container finished" podID="8cf08c4a-4b04-4a73-a2dd-39bd58066be9" containerID="ee7609015fa22b59df360701c13e59fd2b6b9441bc707ef0dfa7964b36115c54" exitCode=137 Nov 03 14:30:33.995262 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:33.995102 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895270-6trc6" Nov 03 14:30:33.995262 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:33.995135 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895270-6trc6" event={"ID":"8cf08c4a-4b04-4a73-a2dd-39bd58066be9","Type":"ContainerDied","Data":"ee7609015fa22b59df360701c13e59fd2b6b9441bc707ef0dfa7964b36115c54"} Nov 03 14:30:33.995262 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:33.995182 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895270-6trc6" event={"ID":"8cf08c4a-4b04-4a73-a2dd-39bd58066be9","Type":"ContainerDied","Data":"f1a53efa0ea499e54effd9bcbcd7d7780e604b4f7c8f4c72bcaa72799fd083af"} Nov 03 14:30:33.995262 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:33.995203 5979 scope.go:122] "RemoveContainer" containerID="ee7609015fa22b59df360701c13e59fd2b6b9441bc707ef0dfa7964b36115c54" Nov 03 14:30:34.004515 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.004494 5979 scope.go:122] "RemoveContainer" containerID="1ba481db5904b02b33f4ed1d241ef8820495517e34d0442522541b770f596cfc" Nov 03 14:30:34.012745 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.012728 5979 scope.go:122] "RemoveContainer" containerID="ee7609015fa22b59df360701c13e59fd2b6b9441bc707ef0dfa7964b36115c54" Nov 03 14:30:34.013013 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:30:34.012994 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee7609015fa22b59df360701c13e59fd2b6b9441bc707ef0dfa7964b36115c54\": container with ID starting with ee7609015fa22b59df360701c13e59fd2b6b9441bc707ef0dfa7964b36115c54 not found: ID does not exist" containerID="ee7609015fa22b59df360701c13e59fd2b6b9441bc707ef0dfa7964b36115c54" Nov 03 14:30:34.013091 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.013022 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee7609015fa22b59df360701c13e59fd2b6b9441bc707ef0dfa7964b36115c54"} err="failed to get container status \"ee7609015fa22b59df360701c13e59fd2b6b9441bc707ef0dfa7964b36115c54\": rpc error: code = NotFound desc = could not find container \"ee7609015fa22b59df360701c13e59fd2b6b9441bc707ef0dfa7964b36115c54\": container with ID starting with ee7609015fa22b59df360701c13e59fd2b6b9441bc707ef0dfa7964b36115c54 not found: ID does not exist" Nov 03 14:30:34.013091 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.013038 5979 scope.go:122] "RemoveContainer" containerID="1ba481db5904b02b33f4ed1d241ef8820495517e34d0442522541b770f596cfc" Nov 03 14:30:34.013268 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:30:34.013250 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ba481db5904b02b33f4ed1d241ef8820495517e34d0442522541b770f596cfc\": container with ID starting with 1ba481db5904b02b33f4ed1d241ef8820495517e34d0442522541b770f596cfc not found: ID does not exist" containerID="1ba481db5904b02b33f4ed1d241ef8820495517e34d0442522541b770f596cfc" Nov 03 14:30:34.013315 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.013272 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ba481db5904b02b33f4ed1d241ef8820495517e34d0442522541b770f596cfc"} err="failed to get container status \"1ba481db5904b02b33f4ed1d241ef8820495517e34d0442522541b770f596cfc\": rpc error: code = NotFound desc = could not find container \"1ba481db5904b02b33f4ed1d241ef8820495517e34d0442522541b770f596cfc\": container with ID starting with 1ba481db5904b02b33f4ed1d241ef8820495517e34d0442522541b770f596cfc not found: ID does not exist" Nov 03 14:30:34.031177 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.031151 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cf08c4a-4b04-4a73-a2dd-39bd58066be9-kube-api-access-phj7c\" (UniqueName: \"kubernetes.io/projected/8cf08c4a-4b04-4a73-a2dd-39bd58066be9-kube-api-access-phj7c\") pod \"8cf08c4a-4b04-4a73-a2dd-39bd58066be9\" (UID: \"8cf08c4a-4b04-4a73-a2dd-39bd58066be9\") " Nov 03 14:30:34.031282 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.031257 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cf08c4a-4b04-4a73-a2dd-39bd58066be9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cf08c4a-4b04-4a73-a2dd-39bd58066be9-maas-api-service-ca\") pod \"8cf08c4a-4b04-4a73-a2dd-39bd58066be9\" (UID: \"8cf08c4a-4b04-4a73-a2dd-39bd58066be9\") " Nov 03 14:30:34.031674 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.031650 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cf08c4a-4b04-4a73-a2dd-39bd58066be9-maas-api-service-ca" pod "8cf08c4a-4b04-4a73-a2dd-39bd58066be9" (UID: "8cf08c4a-4b04-4a73-a2dd-39bd58066be9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:30:34.033111 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.033091 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cf08c4a-4b04-4a73-a2dd-39bd58066be9-kube-api-access-phj7c" pod "8cf08c4a-4b04-4a73-a2dd-39bd58066be9" (UID: "8cf08c4a-4b04-4a73-a2dd-39bd58066be9"). InnerVolumeSpecName "kube-api-access-phj7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:30:34.132760 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.132723 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phj7c\" (UniqueName: \"kubernetes.io/projected/8cf08c4a-4b04-4a73-a2dd-39bd58066be9-kube-api-access-phj7c\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:34.132760 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.132758 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cf08c4a-4b04-4a73-a2dd-39bd58066be9-maas-api-service-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:30:34.202719 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.202686 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8"] Nov 03 14:30:34.203222 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.203184 5979 cpu_manager.go:414] "containerMap: removing container" podUID="68fcc652-c24d-4b11-84a0-8e424f2e130c" containerName="cleanup" Nov 03 14:30:34.203222 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.203204 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68fcc652-c24d-4b11-84a0-8e424f2e130c" containerName="cleanup" Nov 03 14:30:34.203222 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.203227 5979 cpu_manager.go:414] "containerMap: removing container" podUID="417d374a-e02d-4721-a1df-e2b912877928" containerName="cleanup" Nov 03 14:30:34.204718 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.203235 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="417d374a-e02d-4721-a1df-e2b912877928" containerName="cleanup" Nov 03 14:30:34.204718 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.203247 5979 cpu_manager.go:414] "containerMap: removing container" podUID="a0b8e1f3-1744-4fe0-8f2d-04b3ec7a13ad" containerName="cleanup" Nov 03 14:30:34.204718 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.203267 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0b8e1f3-1744-4fe0-8f2d-04b3ec7a13ad" containerName="cleanup" Nov 03 14:30:34.204718 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.203280 5979 cpu_manager.go:414] "containerMap: removing container" podUID="3b7f6f99-6953-4354-8c33-b4b1765aa4d9" containerName="istio-proxy" Nov 03 14:30:34.204718 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.203285 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b7f6f99-6953-4354-8c33-b4b1765aa4d9" containerName="istio-proxy" Nov 03 14:30:34.204718 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.203296 5979 cpu_manager.go:414] "containerMap: removing container" podUID="a0b8e1f3-1744-4fe0-8f2d-04b3ec7a13ad" containerName="cleanup" Nov 03 14:30:34.204718 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.203301 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0b8e1f3-1744-4fe0-8f2d-04b3ec7a13ad" containerName="cleanup" Nov 03 14:30:34.204718 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.203308 5979 cpu_manager.go:414] "containerMap: removing container" podUID="417d374a-e02d-4721-a1df-e2b912877928" containerName="cleanup" Nov 03 14:30:34.204718 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.203312 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="417d374a-e02d-4721-a1df-e2b912877928" containerName="cleanup" Nov 03 14:30:34.204718 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.203323 5979 cpu_manager.go:414] "containerMap: removing container" podUID="68fcc652-c24d-4b11-84a0-8e424f2e130c" containerName="cleanup" Nov 03 14:30:34.204718 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.203328 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68fcc652-c24d-4b11-84a0-8e424f2e130c" containerName="cleanup" Nov 03 14:30:34.204718 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.203336 5979 cpu_manager.go:414] "containerMap: removing container" podUID="420ec7c1-8dae-4500-9aec-5ed3d05433ff" containerName="payload-processing" Nov 03 14:30:34.204718 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.203342 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="420ec7c1-8dae-4500-9aec-5ed3d05433ff" containerName="payload-processing" Nov 03 14:30:34.204718 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.203351 5979 cpu_manager.go:414] "containerMap: removing container" podUID="041bbdde-47d2-46eb-a5d4-bae94c32990f" containerName="cleanup" Nov 03 14:30:34.204718 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.203355 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="041bbdde-47d2-46eb-a5d4-bae94c32990f" containerName="cleanup" Nov 03 14:30:34.204718 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.203366 5979 cpu_manager.go:414] "containerMap: removing container" podUID="8cf08c4a-4b04-4a73-a2dd-39bd58066be9" containerName="cleanup" Nov 03 14:30:34.204718 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.203372 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cf08c4a-4b04-4a73-a2dd-39bd58066be9" containerName="cleanup" Nov 03 14:30:34.204718 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.203388 5979 cpu_manager.go:414] "containerMap: removing container" podUID="041bbdde-47d2-46eb-a5d4-bae94c32990f" containerName="cleanup" Nov 03 14:30:34.204718 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.203393 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="041bbdde-47d2-46eb-a5d4-bae94c32990f" containerName="cleanup" Nov 03 14:30:34.204718 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.203404 5979 cpu_manager.go:414] "containerMap: removing container" podUID="041bbdde-47d2-46eb-a5d4-bae94c32990f" containerName="cleanup" Nov 03 14:30:34.204718 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.203411 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="041bbdde-47d2-46eb-a5d4-bae94c32990f" containerName="cleanup" Nov 03 14:30:34.204718 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.203550 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="041bbdde-47d2-46eb-a5d4-bae94c32990f" containerName="cleanup" Nov 03 14:30:34.204718 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.203563 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0b8e1f3-1744-4fe0-8f2d-04b3ec7a13ad" containerName="cleanup" Nov 03 14:30:34.204718 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.203570 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cf08c4a-4b04-4a73-a2dd-39bd58066be9" containerName="cleanup" Nov 03 14:30:34.204718 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.203577 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="041bbdde-47d2-46eb-a5d4-bae94c32990f" containerName="cleanup" Nov 03 14:30:34.204718 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.203582 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="68fcc652-c24d-4b11-84a0-8e424f2e130c" containerName="cleanup" Nov 03 14:30:34.204718 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.203592 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="417d374a-e02d-4721-a1df-e2b912877928" containerName="cleanup" Nov 03 14:30:34.204718 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.203603 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="68fcc652-c24d-4b11-84a0-8e424f2e130c" containerName="cleanup" Nov 03 14:30:34.204718 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.203614 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="417d374a-e02d-4721-a1df-e2b912877928" containerName="cleanup" Nov 03 14:30:34.204718 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.203620 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b7f6f99-6953-4354-8c33-b4b1765aa4d9" containerName="istio-proxy" Nov 03 14:30:34.204718 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.203626 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0b8e1f3-1744-4fe0-8f2d-04b3ec7a13ad" containerName="cleanup" Nov 03 14:30:34.204718 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.203632 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="417d374a-e02d-4721-a1df-e2b912877928" containerName="cleanup" Nov 03 14:30:34.204718 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.203638 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="68fcc652-c24d-4b11-84a0-8e424f2e130c" containerName="cleanup" Nov 03 14:30:34.204718 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.203645 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="420ec7c1-8dae-4500-9aec-5ed3d05433ff" containerName="payload-processing" Nov 03 14:30:34.208431 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.208413 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8" Nov 03 14:30:34.257187 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.257156 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-34b13c-openshift-default-dockercfg-kf4m8\"" Nov 03 14:30:34.265033 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.264962 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8"] Nov 03 14:30:34.335074 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.335037 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmh8d\" (UniqueName: \"kubernetes.io/projected/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-kube-api-access-zmh8d\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8\" (UID: \"5bf7a9e5-acfd-4d55-9355-daf95a0797a1\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8" Nov 03 14:30:34.335279 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.335094 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-credential-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8\" (UID: \"5bf7a9e5-acfd-4d55-9355-daf95a0797a1\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8" Nov 03 14:30:34.335279 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.335236 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-workload-certs\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8\" (UID: \"5bf7a9e5-acfd-4d55-9355-daf95a0797a1\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8" Nov 03 14:30:34.335408 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.335288 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-workload-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8\" (UID: \"5bf7a9e5-acfd-4d55-9355-daf95a0797a1\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8" Nov 03 14:30:34.335408 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.335359 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-istio-data\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8\" (UID: \"5bf7a9e5-acfd-4d55-9355-daf95a0797a1\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8" Nov 03 14:30:34.335581 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.335437 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-istio-envoy\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8\" (UID: \"5bf7a9e5-acfd-4d55-9355-daf95a0797a1\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8" Nov 03 14:30:34.335581 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.335463 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-istio-token\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8\" (UID: \"5bf7a9e5-acfd-4d55-9355-daf95a0797a1\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8" Nov 03 14:30:34.335699 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.335604 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-istio-podinfo\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8\" (UID: \"5bf7a9e5-acfd-4d55-9355-daf95a0797a1\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8" Nov 03 14:30:34.335699 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.335687 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-istiod-ca-cert\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8\" (UID: \"5bf7a9e5-acfd-4d55-9355-daf95a0797a1\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8" Nov 03 14:30:34.419881 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.419846 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895270-6trc6"] Nov 03 14:30:34.423797 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.423768 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-3c5dbc-29895270-6trc6"] Nov 03 14:30:34.436947 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.436916 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmh8d\" (UniqueName: \"kubernetes.io/projected/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-kube-api-access-zmh8d\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8\" (UID: \"5bf7a9e5-acfd-4d55-9355-daf95a0797a1\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8" Nov 03 14:30:34.437105 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.436967 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-credential-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8\" (UID: \"5bf7a9e5-acfd-4d55-9355-daf95a0797a1\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8" Nov 03 14:30:34.437105 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.437017 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-workload-certs\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8\" (UID: \"5bf7a9e5-acfd-4d55-9355-daf95a0797a1\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8" Nov 03 14:30:34.437105 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.437070 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-workload-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8\" (UID: \"5bf7a9e5-acfd-4d55-9355-daf95a0797a1\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8" Nov 03 14:30:34.437105 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.437095 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-istio-data\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8\" (UID: \"5bf7a9e5-acfd-4d55-9355-daf95a0797a1\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8" Nov 03 14:30:34.437343 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.437152 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-istio-envoy\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8\" (UID: \"5bf7a9e5-acfd-4d55-9355-daf95a0797a1\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8" Nov 03 14:30:34.437343 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.437178 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-istio-token\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8\" (UID: \"5bf7a9e5-acfd-4d55-9355-daf95a0797a1\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8" Nov 03 14:30:34.437343 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.437241 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-istio-podinfo\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8\" (UID: \"5bf7a9e5-acfd-4d55-9355-daf95a0797a1\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8" Nov 03 14:30:34.437343 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.437305 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-istiod-ca-cert\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8\" (UID: \"5bf7a9e5-acfd-4d55-9355-daf95a0797a1\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8" Nov 03 14:30:34.437596 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.437493 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-credential-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8\" (UID: \"5bf7a9e5-acfd-4d55-9355-daf95a0797a1\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8" Nov 03 14:30:34.437596 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.437532 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-workload-certs\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8\" (UID: \"5bf7a9e5-acfd-4d55-9355-daf95a0797a1\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8" Nov 03 14:30:34.437596 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.437568 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-workload-socket\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8\" (UID: \"5bf7a9e5-acfd-4d55-9355-daf95a0797a1\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8" Nov 03 14:30:34.437800 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.437744 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-istio-data\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8\" (UID: \"5bf7a9e5-acfd-4d55-9355-daf95a0797a1\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8" Nov 03 14:30:34.438084 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.438062 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-istiod-ca-cert\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8\" (UID: \"5bf7a9e5-acfd-4d55-9355-daf95a0797a1\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8" Nov 03 14:30:34.439427 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.439406 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-istio-envoy\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8\" (UID: \"5bf7a9e5-acfd-4d55-9355-daf95a0797a1\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8" Nov 03 14:30:34.439656 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.439639 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-istio-podinfo\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8\" (UID: \"5bf7a9e5-acfd-4d55-9355-daf95a0797a1\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8" Nov 03 14:30:34.490638 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.490609 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-istio-token\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8\" (UID: \"5bf7a9e5-acfd-4d55-9355-daf95a0797a1\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8" Nov 03 14:30:34.490796 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.490708 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmh8d\" (UniqueName: \"kubernetes.io/projected/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-kube-api-access-zmh8d\") pod \"e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8\" (UID: \"5bf7a9e5-acfd-4d55-9355-daf95a0797a1\") " pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8" Nov 03 14:30:34.521168 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.521090 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8" Nov 03 14:30:34.944217 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.944179 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8"] Nov 03 14:30:34.951091 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:30:34.951058 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bf7a9e5_acfd_4d55_9355_daf95a0797a1.slice/crio-b2cd9061928efb0fda02f5b90dc6b2bbcb318686c3c6802eca138b3daa43a474 WatchSource:0}: Error finding container b2cd9061928efb0fda02f5b90dc6b2bbcb318686c3c6802eca138b3daa43a474: Status 404 returned error can't find the container with id b2cd9061928efb0fda02f5b90dc6b2bbcb318686c3c6802eca138b3daa43a474 Nov 03 14:30:34.953601 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.953564 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:30:34.953705 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.953685 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:30:34.953747 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:34.953735 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:30:35.003003 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:35.002971 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8" event={"ID":"5bf7a9e5-acfd-4d55-9355-daf95a0797a1","Type":"ContainerStarted","Data":"b2cd9061928efb0fda02f5b90dc6b2bbcb318686c3c6802eca138b3daa43a474"} Nov 03 14:30:36.009449 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:36.009413 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8" event={"ID":"5bf7a9e5-acfd-4d55-9355-daf95a0797a1","Type":"ContainerStarted","Data":"47727653a277761a7a8acfbea396499719d6fb00dbebcde8930767eb9c687888"} Nov 03 14:30:36.114564 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:36.114533 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cf08c4a-4b04-4a73-a2dd-39bd58066be9" path="/var/lib/kubelet/pods/8cf08c4a-4b04-4a73-a2dd-39bd58066be9/volumes" Nov 03 14:30:36.175017 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:36.174963 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8" podStartSLOduration=2.174948741 podStartE2EDuration="2.174948741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:30:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:30:36.168946753 +0000 UTC m=+5118.179635024" watchObservedRunningTime="2026-11-03 14:30:36.174948741 +0000 UTC m=+5118.185636982" Nov 03 14:30:36.521716 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:36.521679 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8" Nov 03 14:30:36.523123 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:36.523096 5979 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.123:15021/healthz/ready\": dial tcp 10.134.1.123:15021: connect: connection refused" start-of-body= Nov 03 14:30:36.523257 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:36.523148 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8" podUID="5bf7a9e5-acfd-4d55-9355-daf95a0797a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.123:15021/healthz/ready\": dial tcp 10.134.1.123:15021: connect: connection refused" Nov 03 14:30:37.233676 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:30:37.233638 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:30:37.522435 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:37.522341 5979 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.123:15021/healthz/ready\": dial tcp 10.134.1.123:15021: connect: connection refused" start-of-body= Nov 03 14:30:37.522435 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:37.522398 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8" podUID="5bf7a9e5-acfd-4d55-9355-daf95a0797a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.123:15021/healthz/ready\": dial tcp 10.134.1.123:15021: connect: connection refused" Nov 03 14:30:38.522278 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:38.522246 5979 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.123:15021/healthz/ready\": dial tcp 10.134.1.123:15021: connect: connection refused" start-of-body= Nov 03 14:30:38.522734 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:38.522299 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8" podUID="5bf7a9e5-acfd-4d55-9355-daf95a0797a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.123:15021/healthz/ready\": dial tcp 10.134.1.123:15021: connect: connection refused" Nov 03 14:30:39.521676 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:39.521640 5979 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.123:15021/healthz/ready\": dial tcp 10.134.1.123:15021: connect: connection refused" start-of-body= Nov 03 14:30:39.521852 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:39.521693 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8" podUID="5bf7a9e5-acfd-4d55-9355-daf95a0797a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.123:15021/healthz/ready\": dial tcp 10.134.1.123:15021: connect: connection refused" Nov 03 14:30:40.521973 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:40.521938 5979 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.123:15021/healthz/ready\": dial tcp 10.134.1.123:15021: connect: connection refused" start-of-body= Nov 03 14:30:40.522386 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:40.521993 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8" podUID="5bf7a9e5-acfd-4d55-9355-daf95a0797a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.123:15021/healthz/ready\": dial tcp 10.134.1.123:15021: connect: connection refused" Nov 03 14:30:41.522562 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:41.522522 5979 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.123:15021/healthz/ready\": dial tcp 10.134.1.123:15021: connect: connection refused" start-of-body= Nov 03 14:30:41.523021 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:41.522588 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8" podUID="5bf7a9e5-acfd-4d55-9355-daf95a0797a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.123:15021/healthz/ready\": dial tcp 10.134.1.123:15021: connect: connection refused" Nov 03 14:30:41.949384 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:41.949335 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-2bceb1-6f4bbf98ff-k6vzq"] Nov 03 14:30:42.604972 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:42.604924 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8" Nov 03 14:30:42.605692 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:42.605667 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8" Nov 03 14:30:42.606933 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:42.606910 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8" Nov 03 14:30:46.950213 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:46.950164 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2bceb1-6f4bbf98ff-k6vzq" podUID="92497d85-07e4-4286-aabc-6ad4036733e4" containerName="payload-processing" containerID="cri-o://83e5c9dfc47b7ae4cf91360c3bc28373737b434015b5ad435f80a721b81804fe" gracePeriod=25 Nov 03 14:30:50.478676 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:50.478632 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:30:55.342770 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:55.342729 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8"] Nov 03 14:30:55.343234 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:55.342926 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8" podUID="5bf7a9e5-acfd-4d55-9355-daf95a0797a1" containerName="istio-proxy" containerID="cri-o://47727653a277761a7a8acfbea396499719d6fb00dbebcde8930767eb9c687888" gracePeriod=30 Nov 03 14:30:55.707530 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:55.707495 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2bceb1-6f4bbf98ff-k6vzq" podUID="92497d85-07e4-4286-aabc-6ad4036733e4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:30:57.608069 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:57.608027 5979 patch_prober.go:29] interesting pod/e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 14:30:57.608448 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:30:57.608117 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8" podUID="5bf7a9e5-acfd-4d55-9355-daf95a0797a1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 14:31:00.596385 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:00.596350 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8" Nov 03 14:31:00.634533 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:00.634501 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-istiod-ca-cert\") pod \"5bf7a9e5-acfd-4d55-9355-daf95a0797a1\" (UID: \"5bf7a9e5-acfd-4d55-9355-daf95a0797a1\") " Nov 03 14:31:00.634723 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:00.634603 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-istio-data\") pod \"5bf7a9e5-acfd-4d55-9355-daf95a0797a1\" (UID: \"5bf7a9e5-acfd-4d55-9355-daf95a0797a1\") " Nov 03 14:31:00.634723 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:00.634632 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-kube-api-access-zmh8d\" (UniqueName: \"kubernetes.io/projected/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-kube-api-access-zmh8d\") pod \"5bf7a9e5-acfd-4d55-9355-daf95a0797a1\" (UID: \"5bf7a9e5-acfd-4d55-9355-daf95a0797a1\") " Nov 03 14:31:00.634723 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:00.634678 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-istio-envoy\") pod \"5bf7a9e5-acfd-4d55-9355-daf95a0797a1\" (UID: \"5bf7a9e5-acfd-4d55-9355-daf95a0797a1\") " Nov 03 14:31:00.634723 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:00.634707 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-workload-certs\") pod \"5bf7a9e5-acfd-4d55-9355-daf95a0797a1\" (UID: \"5bf7a9e5-acfd-4d55-9355-daf95a0797a1\") " Nov 03 14:31:00.634878 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:00.634738 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-credential-socket\") pod \"5bf7a9e5-acfd-4d55-9355-daf95a0797a1\" (UID: \"5bf7a9e5-acfd-4d55-9355-daf95a0797a1\") " Nov 03 14:31:00.634878 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:00.634768 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-istio-token\" (UniqueName: \"kubernetes.io/projected/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-istio-token\") pod \"5bf7a9e5-acfd-4d55-9355-daf95a0797a1\" (UID: \"5bf7a9e5-acfd-4d55-9355-daf95a0797a1\") " Nov 03 14:31:00.634878 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:00.634804 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-workload-socket\") pod \"5bf7a9e5-acfd-4d55-9355-daf95a0797a1\" (UID: \"5bf7a9e5-acfd-4d55-9355-daf95a0797a1\") " Nov 03 14:31:00.634878 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:00.634853 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-istio-podinfo\") pod \"5bf7a9e5-acfd-4d55-9355-daf95a0797a1\" (UID: \"5bf7a9e5-acfd-4d55-9355-daf95a0797a1\") " Nov 03 14:31:00.635024 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:00.634878 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-istio-data" pod "5bf7a9e5-acfd-4d55-9355-daf95a0797a1" (UID: "5bf7a9e5-acfd-4d55-9355-daf95a0797a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:31:00.635024 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:00.634878 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-istiod-ca-cert" pod "5bf7a9e5-acfd-4d55-9355-daf95a0797a1" (UID: "5bf7a9e5-acfd-4d55-9355-daf95a0797a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:31:00.635137 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:00.635035 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-credential-socket" pod "5bf7a9e5-acfd-4d55-9355-daf95a0797a1" (UID: "5bf7a9e5-acfd-4d55-9355-daf95a0797a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:31:00.635198 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:00.635163 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:00.635198 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:00.635180 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:00.635198 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:00.635194 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:00.635381 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:00.635298 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-workload-certs" pod "5bf7a9e5-acfd-4d55-9355-daf95a0797a1" (UID: "5bf7a9e5-acfd-4d55-9355-daf95a0797a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:31:00.635457 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:00.635421 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-workload-socket" pod "5bf7a9e5-acfd-4d55-9355-daf95a0797a1" (UID: "5bf7a9e5-acfd-4d55-9355-daf95a0797a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:31:00.637281 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:00.637257 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-kube-api-access-zmh8d" pod "5bf7a9e5-acfd-4d55-9355-daf95a0797a1" (UID: "5bf7a9e5-acfd-4d55-9355-daf95a0797a1"). InnerVolumeSpecName "kube-api-access-zmh8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:31:00.637411 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:00.637370 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-istio-podinfo" pod "5bf7a9e5-acfd-4d55-9355-daf95a0797a1" (UID: "5bf7a9e5-acfd-4d55-9355-daf95a0797a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:31:00.637496 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:00.637419 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-istio-token" pod "5bf7a9e5-acfd-4d55-9355-daf95a0797a1" (UID: "5bf7a9e5-acfd-4d55-9355-daf95a0797a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:31:00.637618 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:00.637595 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-istio-envoy" pod "5bf7a9e5-acfd-4d55-9355-daf95a0797a1" (UID: "5bf7a9e5-acfd-4d55-9355-daf95a0797a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:31:00.736798 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:00.736760 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:00.736798 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:00.736788 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:00.736798 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:00.736799 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmh8d\" (UniqueName: \"kubernetes.io/projected/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-kube-api-access-zmh8d\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:00.736798 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:00.736809 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:00.737086 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:00.736818 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:00.737086 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:00.736826 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bf7a9e5-acfd-4d55-9355-daf95a0797a1-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:01.151569 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:01.151538 5979 generic.go:356] "Generic (PLEG): container finished" podID="5bf7a9e5-acfd-4d55-9355-daf95a0797a1" containerID="47727653a277761a7a8acfbea396499719d6fb00dbebcde8930767eb9c687888" exitCode=0 Nov 03 14:31:01.151792 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:01.151587 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8" Nov 03 14:31:01.151792 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:01.151632 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8" event={"ID":"5bf7a9e5-acfd-4d55-9355-daf95a0797a1","Type":"ContainerDied","Data":"47727653a277761a7a8acfbea396499719d6fb00dbebcde8930767eb9c687888"} Nov 03 14:31:01.151792 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:01.151692 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8" event={"ID":"5bf7a9e5-acfd-4d55-9355-daf95a0797a1","Type":"ContainerDied","Data":"b2cd9061928efb0fda02f5b90dc6b2bbcb318686c3c6802eca138b3daa43a474"} Nov 03 14:31:01.151792 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:01.151720 5979 scope.go:122] "RemoveContainer" containerID="47727653a277761a7a8acfbea396499719d6fb00dbebcde8930767eb9c687888" Nov 03 14:31:01.161791 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:01.161775 5979 scope.go:122] "RemoveContainer" containerID="47727653a277761a7a8acfbea396499719d6fb00dbebcde8930767eb9c687888" Nov 03 14:31:01.162072 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:31:01.162050 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47727653a277761a7a8acfbea396499719d6fb00dbebcde8930767eb9c687888\": container with ID starting with 47727653a277761a7a8acfbea396499719d6fb00dbebcde8930767eb9c687888 not found: ID does not exist" containerID="47727653a277761a7a8acfbea396499719d6fb00dbebcde8930767eb9c687888" Nov 03 14:31:01.162119 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:01.162081 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47727653a277761a7a8acfbea396499719d6fb00dbebcde8930767eb9c687888"} err="failed to get container status \"47727653a277761a7a8acfbea396499719d6fb00dbebcde8930767eb9c687888\": rpc error: code = NotFound desc = could not find container \"47727653a277761a7a8acfbea396499719d6fb00dbebcde8930767eb9c687888\": container with ID starting with 47727653a277761a7a8acfbea396499719d6fb00dbebcde8930767eb9c687888 not found: ID does not exist" Nov 03 14:31:01.305826 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:01.305786 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8"] Nov 03 14:31:01.352235 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:01.352200 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-34b13c-openshift-default-876454cf9-scxp8"] Nov 03 14:31:02.115999 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:02.115951 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bf7a9e5-acfd-4d55-9355-daf95a0797a1" path="/var/lib/kubelet/pods/5bf7a9e5-acfd-4d55-9355-daf95a0797a1/volumes" Nov 03 14:31:05.707698 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:05.707652 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2bceb1-6f4bbf98ff-k6vzq" podUID="92497d85-07e4-4286-aabc-6ad4036733e4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:31:07.191284 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:07.191254 5979 generic.go:356] "Generic (PLEG): container finished" podID="92497d85-07e4-4286-aabc-6ad4036733e4" containerID="83e5c9dfc47b7ae4cf91360c3bc28373737b434015b5ad435f80a721b81804fe" exitCode=0 Nov 03 14:31:07.191723 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:07.191335 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2bceb1-6f4bbf98ff-k6vzq" event={"ID":"92497d85-07e4-4286-aabc-6ad4036733e4","Type":"ContainerDied","Data":"83e5c9dfc47b7ae4cf91360c3bc28373737b434015b5ad435f80a721b81804fe"} Nov 03 14:31:07.214463 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:07.214391 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2bceb1-6f4bbf98ff-k6vzq" Nov 03 14:31:07.234309 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:31:07.234273 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:31:07.310408 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:07.310369 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92497d85-07e4-4286-aabc-6ad4036733e4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92497d85-07e4-4286-aabc-6ad4036733e4-plugins-config-volume\") pod \"92497d85-07e4-4286-aabc-6ad4036733e4\" (UID: \"92497d85-07e4-4286-aabc-6ad4036733e4\") " Nov 03 14:31:07.310646 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:07.310461 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92497d85-07e4-4286-aabc-6ad4036733e4-kube-api-access-58lbl\" (UniqueName: \"kubernetes.io/projected/92497d85-07e4-4286-aabc-6ad4036733e4-kube-api-access-58lbl\") pod \"92497d85-07e4-4286-aabc-6ad4036733e4\" (UID: \"92497d85-07e4-4286-aabc-6ad4036733e4\") " Nov 03 14:31:07.310807 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:07.310777 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92497d85-07e4-4286-aabc-6ad4036733e4-plugins-config-volume" pod "92497d85-07e4-4286-aabc-6ad4036733e4" (UID: "92497d85-07e4-4286-aabc-6ad4036733e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:31:07.312525 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:07.312504 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92497d85-07e4-4286-aabc-6ad4036733e4-kube-api-access-58lbl" pod "92497d85-07e4-4286-aabc-6ad4036733e4" (UID: "92497d85-07e4-4286-aabc-6ad4036733e4"). InnerVolumeSpecName "kube-api-access-58lbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:31:07.412366 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:07.412324 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92497d85-07e4-4286-aabc-6ad4036733e4-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:07.412366 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:07.412358 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58lbl\" (UniqueName: \"kubernetes.io/projected/92497d85-07e4-4286-aabc-6ad4036733e4-kube-api-access-58lbl\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:08.198361 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:08.198332 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2bceb1-6f4bbf98ff-k6vzq" Nov 03 14:31:08.198882 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:08.198366 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2bceb1-6f4bbf98ff-k6vzq" event={"ID":"92497d85-07e4-4286-aabc-6ad4036733e4","Type":"ContainerDied","Data":"99061d2e934f5ce18826b935dd2114ebb47d5c1b96cd21cc5365733dec72a0c8"} Nov 03 14:31:08.198882 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:08.198440 5979 scope.go:122] "RemoveContainer" containerID="83e5c9dfc47b7ae4cf91360c3bc28373737b434015b5ad435f80a721b81804fe" Nov 03 14:31:08.352294 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:08.352257 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-2bceb1-6f4bbf98ff-k6vzq"] Nov 03 14:31:08.398828 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:08.398799 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-2bceb1-6f4bbf98ff-k6vzq"] Nov 03 14:31:08.852200 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:08.852109 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:31:10.114567 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:10.114533 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92497d85-07e4-4286-aabc-6ad4036733e4" path="/var/lib/kubelet/pods/92497d85-07e4-4286-aabc-6ad4036733e4/volumes" Nov 03 14:31:21.620589 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:21.620555 5979 scope.go:122] "RemoveContainer" containerID="94551948ae7a4e2e6b4ac64d76360784bd4b18dd6bf4db51bd441903374d71e2" Nov 03 14:31:21.632116 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:21.632091 5979 scope.go:122] "RemoveContainer" containerID="7d06238fa5afcf9667e46f3717e58482c980bb978056d5526f733b88aa0d32d2" Nov 03 14:31:22.059599 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:22.059566 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d341af-7c4f477dcc-nvkxn"] Nov 03 14:31:22.172805 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:22.172770 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d341af-5c58b5c667-bnws4"] Nov 03 14:31:27.060060 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:27.059985 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d341af-7c4f477dcc-nvkxn" podUID="78dfe2a2-8938-4f14-a784-f07dbfcdd7f7" containerName="payload-processing" containerID="cri-o://34a4e8ddb6cccb8cbe3cfcc341953f72d4417c64f1c860b77829001987fb5db5" gracePeriod=25 Nov 03 14:31:27.173065 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:27.173026 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d341af-5c58b5c667-bnws4" podUID="f03e902b-a262-4d46-b393-dea566c264d2" containerName="payload-pre-processing" containerID="cri-o://80392b315f206fec3a2f219e29a41a44ec383771b0281661a7276d8bd95b4fc4" gracePeriod=25 Nov 03 14:31:29.364518 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:29.364463 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p"] Nov 03 14:31:29.364979 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:29.364725 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p" podUID="6974e682-76a0-438c-b303-9e498d41f0a0" containerName="istio-proxy" containerID="cri-o://781dcf3d32951ee0317b334db7a5d05730cff3d7310b9f4fd31ed29e67cc9f21" gracePeriod=30 Nov 03 14:31:29.437282 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:29.437247 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 14:31:29.437482 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:29.437316 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p" podUID="6974e682-76a0-438c-b303-9e498d41f0a0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 14:31:34.610746 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:34.610719 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p" Nov 03 14:31:34.689246 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:34.689157 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6974e682-76a0-438c-b303-9e498d41f0a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6974e682-76a0-438c-b303-9e498d41f0a0-istio-envoy\") pod \"6974e682-76a0-438c-b303-9e498d41f0a0\" (UID: \"6974e682-76a0-438c-b303-9e498d41f0a0\") " Nov 03 14:31:34.689246 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:34.689210 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6974e682-76a0-438c-b303-9e498d41f0a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6974e682-76a0-438c-b303-9e498d41f0a0-workload-socket\") pod \"6974e682-76a0-438c-b303-9e498d41f0a0\" (UID: \"6974e682-76a0-438c-b303-9e498d41f0a0\") " Nov 03 14:31:34.689511 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:34.689253 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6974e682-76a0-438c-b303-9e498d41f0a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6974e682-76a0-438c-b303-9e498d41f0a0-istio-podinfo\") pod \"6974e682-76a0-438c-b303-9e498d41f0a0\" (UID: \"6974e682-76a0-438c-b303-9e498d41f0a0\") " Nov 03 14:31:34.689511 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:34.689321 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6974e682-76a0-438c-b303-9e498d41f0a0-kube-api-access-24t9h\" (UniqueName: \"kubernetes.io/projected/6974e682-76a0-438c-b303-9e498d41f0a0-kube-api-access-24t9h\") pod \"6974e682-76a0-438c-b303-9e498d41f0a0\" (UID: \"6974e682-76a0-438c-b303-9e498d41f0a0\") " Nov 03 14:31:34.689511 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:34.689345 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6974e682-76a0-438c-b303-9e498d41f0a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6974e682-76a0-438c-b303-9e498d41f0a0-istiod-ca-cert\") pod \"6974e682-76a0-438c-b303-9e498d41f0a0\" (UID: \"6974e682-76a0-438c-b303-9e498d41f0a0\") " Nov 03 14:31:34.689511 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:34.689376 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6974e682-76a0-438c-b303-9e498d41f0a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6974e682-76a0-438c-b303-9e498d41f0a0-workload-certs\") pod \"6974e682-76a0-438c-b303-9e498d41f0a0\" (UID: \"6974e682-76a0-438c-b303-9e498d41f0a0\") " Nov 03 14:31:34.689511 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:34.689422 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6974e682-76a0-438c-b303-9e498d41f0a0-istio-token\" (UniqueName: \"kubernetes.io/projected/6974e682-76a0-438c-b303-9e498d41f0a0-istio-token\") pod \"6974e682-76a0-438c-b303-9e498d41f0a0\" (UID: \"6974e682-76a0-438c-b303-9e498d41f0a0\") " Nov 03 14:31:34.689511 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:34.689460 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6974e682-76a0-438c-b303-9e498d41f0a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6974e682-76a0-438c-b303-9e498d41f0a0-istio-data\") pod \"6974e682-76a0-438c-b303-9e498d41f0a0\" (UID: \"6974e682-76a0-438c-b303-9e498d41f0a0\") " Nov 03 14:31:34.689834 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:34.689526 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6974e682-76a0-438c-b303-9e498d41f0a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6974e682-76a0-438c-b303-9e498d41f0a0-credential-socket\") pod \"6974e682-76a0-438c-b303-9e498d41f0a0\" (UID: \"6974e682-76a0-438c-b303-9e498d41f0a0\") " Nov 03 14:31:34.689834 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:34.689551 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6974e682-76a0-438c-b303-9e498d41f0a0-workload-socket" pod "6974e682-76a0-438c-b303-9e498d41f0a0" (UID: "6974e682-76a0-438c-b303-9e498d41f0a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:31:34.690008 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:34.689835 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6974e682-76a0-438c-b303-9e498d41f0a0-istiod-ca-cert" pod "6974e682-76a0-438c-b303-9e498d41f0a0" (UID: "6974e682-76a0-438c-b303-9e498d41f0a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:31:34.690008 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:34.689895 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6974e682-76a0-438c-b303-9e498d41f0a0-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:34.690138 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:34.690059 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6974e682-76a0-438c-b303-9e498d41f0a0-workload-certs" pod "6974e682-76a0-438c-b303-9e498d41f0a0" (UID: "6974e682-76a0-438c-b303-9e498d41f0a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:31:34.690138 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:34.690128 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6974e682-76a0-438c-b303-9e498d41f0a0-credential-socket" pod "6974e682-76a0-438c-b303-9e498d41f0a0" (UID: "6974e682-76a0-438c-b303-9e498d41f0a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:31:34.690252 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:34.690150 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6974e682-76a0-438c-b303-9e498d41f0a0-istio-data" pod "6974e682-76a0-438c-b303-9e498d41f0a0" (UID: "6974e682-76a0-438c-b303-9e498d41f0a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:31:34.691748 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:34.691720 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6974e682-76a0-438c-b303-9e498d41f0a0-istio-envoy" pod "6974e682-76a0-438c-b303-9e498d41f0a0" (UID: "6974e682-76a0-438c-b303-9e498d41f0a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:31:34.691748 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:34.691741 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6974e682-76a0-438c-b303-9e498d41f0a0-kube-api-access-24t9h" pod "6974e682-76a0-438c-b303-9e498d41f0a0" (UID: "6974e682-76a0-438c-b303-9e498d41f0a0"). InnerVolumeSpecName "kube-api-access-24t9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:31:34.691929 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:34.691826 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6974e682-76a0-438c-b303-9e498d41f0a0-istio-podinfo" pod "6974e682-76a0-438c-b303-9e498d41f0a0" (UID: "6974e682-76a0-438c-b303-9e498d41f0a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:31:34.691929 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:34.691850 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6974e682-76a0-438c-b303-9e498d41f0a0-istio-token" pod "6974e682-76a0-438c-b303-9e498d41f0a0" (UID: "6974e682-76a0-438c-b303-9e498d41f0a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:31:34.790723 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:34.790681 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24t9h\" (UniqueName: \"kubernetes.io/projected/6974e682-76a0-438c-b303-9e498d41f0a0-kube-api-access-24t9h\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:34.790723 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:34.790725 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6974e682-76a0-438c-b303-9e498d41f0a0-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:34.790932 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:34.790742 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6974e682-76a0-438c-b303-9e498d41f0a0-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:34.790932 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:34.790755 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6974e682-76a0-438c-b303-9e498d41f0a0-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:34.790932 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:34.790770 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6974e682-76a0-438c-b303-9e498d41f0a0-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:34.790932 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:34.790784 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6974e682-76a0-438c-b303-9e498d41f0a0-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:34.790932 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:34.790800 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6974e682-76a0-438c-b303-9e498d41f0a0-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:34.790932 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:34.790812 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6974e682-76a0-438c-b303-9e498d41f0a0-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:35.347589 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:35.347552 5979 generic.go:356] "Generic (PLEG): container finished" podID="6974e682-76a0-438c-b303-9e498d41f0a0" containerID="781dcf3d32951ee0317b334db7a5d05730cff3d7310b9f4fd31ed29e67cc9f21" exitCode=0 Nov 03 14:31:35.347835 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:35.347601 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p" Nov 03 14:31:35.347835 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:35.347647 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p" event={"ID":"6974e682-76a0-438c-b303-9e498d41f0a0","Type":"ContainerDied","Data":"781dcf3d32951ee0317b334db7a5d05730cff3d7310b9f4fd31ed29e67cc9f21"} Nov 03 14:31:35.347835 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:35.347708 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p" event={"ID":"6974e682-76a0-438c-b303-9e498d41f0a0","Type":"ContainerDied","Data":"09fc2de535a8063d3b269537c2dcca704a3d5c5d8cda9478ea15a3b80fa4d602"} Nov 03 14:31:35.347835 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:35.347737 5979 scope.go:122] "RemoveContainer" containerID="781dcf3d32951ee0317b334db7a5d05730cff3d7310b9f4fd31ed29e67cc9f21" Nov 03 14:31:35.358344 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:35.358328 5979 scope.go:122] "RemoveContainer" containerID="781dcf3d32951ee0317b334db7a5d05730cff3d7310b9f4fd31ed29e67cc9f21" Nov 03 14:31:35.358627 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:31:35.358605 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"781dcf3d32951ee0317b334db7a5d05730cff3d7310b9f4fd31ed29e67cc9f21\": container with ID starting with 781dcf3d32951ee0317b334db7a5d05730cff3d7310b9f4fd31ed29e67cc9f21 not found: ID does not exist" containerID="781dcf3d32951ee0317b334db7a5d05730cff3d7310b9f4fd31ed29e67cc9f21" Nov 03 14:31:35.358688 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:35.358637 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"781dcf3d32951ee0317b334db7a5d05730cff3d7310b9f4fd31ed29e67cc9f21"} err="failed to get container status \"781dcf3d32951ee0317b334db7a5d05730cff3d7310b9f4fd31ed29e67cc9f21\": rpc error: code = NotFound desc = could not find container \"781dcf3d32951ee0317b334db7a5d05730cff3d7310b9f4fd31ed29e67cc9f21\": container with ID starting with 781dcf3d32951ee0317b334db7a5d05730cff3d7310b9f4fd31ed29e67cc9f21 not found: ID does not exist" Nov 03 14:31:35.404924 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:35.404894 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d341af-5c58b5c667-bnws4" podUID="f03e902b-a262-4d46-b393-dea566c264d2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:31:35.502753 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:35.502721 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p"] Nov 03 14:31:35.508176 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:35.508142 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-ngr9p"] Nov 03 14:31:36.114193 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:36.114158 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6974e682-76a0-438c-b303-9e498d41f0a0" path="/var/lib/kubelet/pods/6974e682-76a0-438c-b303-9e498d41f0a0/volumes" Nov 03 14:31:36.430032 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:36.429990 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d341af-7c4f477dcc-nvkxn" podUID="78dfe2a2-8938-4f14-a784-f07dbfcdd7f7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:31:37.233484 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:31:37.233432 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:31:41.374075 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:41.374039 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:31:44.049775 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:44.049736 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc"] Nov 03 14:31:44.050228 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:44.050023 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc" podUID="8df65c37-2cc2-4bd1-9d05-4576dfaec9a6" containerName="istio-proxy" containerID="cri-o://7e16e97308b61bf8f320f624ea933515c3d2663ac782cbd17a7417458f9d65dd" gracePeriod=30 Nov 03 14:31:45.216927 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:45.216889 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83dbd046-h65vn"] Nov 03 14:31:45.217370 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:45.217300 5979 cpu_manager.go:414] "containerMap: removing container" podUID="6974e682-76a0-438c-b303-9e498d41f0a0" containerName="istio-proxy" Nov 03 14:31:45.217370 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:45.217312 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6974e682-76a0-438c-b303-9e498d41f0a0" containerName="istio-proxy" Nov 03 14:31:45.217370 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:45.217322 5979 cpu_manager.go:414] "containerMap: removing container" podUID="8cf08c4a-4b04-4a73-a2dd-39bd58066be9" containerName="cleanup" Nov 03 14:31:45.217370 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:45.217327 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cf08c4a-4b04-4a73-a2dd-39bd58066be9" containerName="cleanup" Nov 03 14:31:45.217370 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:45.217346 5979 cpu_manager.go:414] "containerMap: removing container" podUID="92497d85-07e4-4286-aabc-6ad4036733e4" containerName="payload-processing" Nov 03 14:31:45.217370 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:45.217351 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92497d85-07e4-4286-aabc-6ad4036733e4" containerName="payload-processing" Nov 03 14:31:45.217370 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:45.217360 5979 cpu_manager.go:414] "containerMap: removing container" podUID="5bf7a9e5-acfd-4d55-9355-daf95a0797a1" containerName="istio-proxy" Nov 03 14:31:45.217370 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:45.217366 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bf7a9e5-acfd-4d55-9355-daf95a0797a1" containerName="istio-proxy" Nov 03 14:31:45.217738 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:45.217389 5979 cpu_manager.go:414] "containerMap: removing container" podUID="a0b8e1f3-1744-4fe0-8f2d-04b3ec7a13ad" containerName="cleanup" Nov 03 14:31:45.217738 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:45.217395 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0b8e1f3-1744-4fe0-8f2d-04b3ec7a13ad" containerName="cleanup" Nov 03 14:31:45.217738 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:45.217402 5979 cpu_manager.go:414] "containerMap: removing container" podUID="417d374a-e02d-4721-a1df-e2b912877928" containerName="cleanup" Nov 03 14:31:45.217738 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:45.217407 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="417d374a-e02d-4721-a1df-e2b912877928" containerName="cleanup" Nov 03 14:31:45.217738 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:45.217416 5979 cpu_manager.go:414] "containerMap: removing container" podUID="68fcc652-c24d-4b11-84a0-8e424f2e130c" containerName="cleanup" Nov 03 14:31:45.217738 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:45.217420 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68fcc652-c24d-4b11-84a0-8e424f2e130c" containerName="cleanup" Nov 03 14:31:45.217738 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:45.217505 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="6974e682-76a0-438c-b303-9e498d41f0a0" containerName="istio-proxy" Nov 03 14:31:45.217738 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:45.217514 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0b8e1f3-1744-4fe0-8f2d-04b3ec7a13ad" containerName="cleanup" Nov 03 14:31:45.217738 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:45.217519 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cf08c4a-4b04-4a73-a2dd-39bd58066be9" containerName="cleanup" Nov 03 14:31:45.217738 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:45.217526 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="041bbdde-47d2-46eb-a5d4-bae94c32990f" containerName="cleanup" Nov 03 14:31:45.217738 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:45.217532 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="92497d85-07e4-4286-aabc-6ad4036733e4" containerName="payload-processing" Nov 03 14:31:45.217738 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:45.217538 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bf7a9e5-acfd-4d55-9355-daf95a0797a1" containerName="istio-proxy" Nov 03 14:31:45.222324 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:45.222302 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dbd046-h65vn" Nov 03 14:31:45.271789 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:45.271754 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:31:45.301271 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:45.301231 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz48f\" (UniqueName: \"kubernetes.io/projected/712c3ae4-3332-4730-9553-d21963331b11-kube-api-access-fz48f\") pod \"maas-api-revoke-sub-83dbd046-h65vn\" (UID: \"712c3ae4-3332-4730-9553-d21963331b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dbd046-h65vn" Nov 03 14:31:45.301453 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:45.301427 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/712c3ae4-3332-4730-9553-d21963331b11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83dbd046-h65vn\" (UID: \"712c3ae4-3332-4730-9553-d21963331b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dbd046-h65vn" Nov 03 14:31:45.301540 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:45.301482 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/712c3ae4-3332-4730-9553-d21963331b11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83dbd046-h65vn\" (UID: \"712c3ae4-3332-4730-9553-d21963331b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dbd046-h65vn" Nov 03 14:31:45.318215 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:45.318168 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:31:45.318215 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:45.318192 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:31:45.318412 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:45.318254 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 14:31:45.323435 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:45.323401 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83dbd046-h65vn"] Nov 03 14:31:45.402383 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:45.402339 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/712c3ae4-3332-4730-9553-d21963331b11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83dbd046-h65vn\" (UID: \"712c3ae4-3332-4730-9553-d21963331b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dbd046-h65vn" Nov 03 14:31:45.402383 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:45.402385 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/712c3ae4-3332-4730-9553-d21963331b11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83dbd046-h65vn\" (UID: \"712c3ae4-3332-4730-9553-d21963331b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dbd046-h65vn" Nov 03 14:31:45.402651 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:45.402439 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz48f\" (UniqueName: \"kubernetes.io/projected/712c3ae4-3332-4730-9553-d21963331b11-kube-api-access-fz48f\") pod \"maas-api-revoke-sub-83dbd046-h65vn\" (UID: \"712c3ae4-3332-4730-9553-d21963331b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dbd046-h65vn" Nov 03 14:31:45.403061 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:45.403031 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/712c3ae4-3332-4730-9553-d21963331b11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83dbd046-h65vn\" (UID: \"712c3ae4-3332-4730-9553-d21963331b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dbd046-h65vn" Nov 03 14:31:45.405083 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:45.405057 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/712c3ae4-3332-4730-9553-d21963331b11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83dbd046-h65vn\" (UID: \"712c3ae4-3332-4730-9553-d21963331b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dbd046-h65vn" Nov 03 14:31:45.405279 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:45.405254 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d341af-5c58b5c667-bnws4" podUID="f03e902b-a262-4d46-b393-dea566c264d2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:31:45.456864 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:45.456830 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz48f\" (UniqueName: \"kubernetes.io/projected/712c3ae4-3332-4730-9553-d21963331b11-kube-api-access-fz48f\") pod \"maas-api-revoke-sub-83dbd046-h65vn\" (UID: \"712c3ae4-3332-4730-9553-d21963331b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dbd046-h65vn" Nov 03 14:31:45.535189 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:45.535104 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dbd046-h65vn" Nov 03 14:31:45.905695 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:45.905659 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83dbd046-h65vn"] Nov 03 14:31:45.913485 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:31:45.913448 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod712c3ae4_3332_4730_9553_d21963331b11.slice/crio-c5167c36126f473dd5dca7acd5e88f3b290d5ae611b5575d34a6cf54c6c6fc39 WatchSource:0}: Error finding container c5167c36126f473dd5dca7acd5e88f3b290d5ae611b5575d34a6cf54c6c6fc39: Status 404 returned error can't find the container with id c5167c36126f473dd5dca7acd5e88f3b290d5ae611b5575d34a6cf54c6c6fc39 Nov 03 14:31:46.417527 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:46.417423 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dbd046-h65vn" event={"ID":"712c3ae4-3332-4730-9553-d21963331b11","Type":"ContainerStarted","Data":"aff677a73e05914bf046b10107414425c0eddce8da9389984f46ac86a521da64"} Nov 03 14:31:46.417527 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:46.417507 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dbd046-h65vn" event={"ID":"712c3ae4-3332-4730-9553-d21963331b11","Type":"ContainerStarted","Data":"c5167c36126f473dd5dca7acd5e88f3b290d5ae611b5575d34a6cf54c6c6fc39"} Nov 03 14:31:46.430980 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:46.430947 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d341af-7c4f477dcc-nvkxn" podUID="78dfe2a2-8938-4f14-a784-f07dbfcdd7f7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:31:46.527560 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:46.527499 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dbd046-h65vn" podStartSLOduration=1.52745727 podStartE2EDuration="1.52745727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:31:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:31:46.526016643 +0000 UTC m=+5188.536704887" watchObservedRunningTime="2026-11-03 14:31:46.52745727 +0000 UTC m=+5188.538145523" Nov 03 14:31:47.305265 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:47.305239 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d341af-7c4f477dcc-nvkxn" Nov 03 14:31:47.405715 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:47.405687 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d341af-5c58b5c667-bnws4" Nov 03 14:31:47.422379 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:47.422344 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78dfe2a2-8938-4f14-a784-f07dbfcdd7f7-kube-api-access-stffb\" (UniqueName: \"kubernetes.io/projected/78dfe2a2-8938-4f14-a784-f07dbfcdd7f7-kube-api-access-stffb\") pod \"78dfe2a2-8938-4f14-a784-f07dbfcdd7f7\" (UID: \"78dfe2a2-8938-4f14-a784-f07dbfcdd7f7\") " Nov 03 14:31:47.422899 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:47.422496 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78dfe2a2-8938-4f14-a784-f07dbfcdd7f7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78dfe2a2-8938-4f14-a784-f07dbfcdd7f7-plugins-config-volume\") pod \"78dfe2a2-8938-4f14-a784-f07dbfcdd7f7\" (UID: \"78dfe2a2-8938-4f14-a784-f07dbfcdd7f7\") " Nov 03 14:31:47.422975 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:47.422949 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78dfe2a2-8938-4f14-a784-f07dbfcdd7f7-plugins-config-volume" pod "78dfe2a2-8938-4f14-a784-f07dbfcdd7f7" (UID: "78dfe2a2-8938-4f14-a784-f07dbfcdd7f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:31:47.426612 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:47.426585 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78dfe2a2-8938-4f14-a784-f07dbfcdd7f7-kube-api-access-stffb" pod "78dfe2a2-8938-4f14-a784-f07dbfcdd7f7" (UID: "78dfe2a2-8938-4f14-a784-f07dbfcdd7f7"). InnerVolumeSpecName "kube-api-access-stffb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:31:47.427516 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:47.427486 5979 generic.go:356] "Generic (PLEG): container finished" podID="78dfe2a2-8938-4f14-a784-f07dbfcdd7f7" containerID="34a4e8ddb6cccb8cbe3cfcc341953f72d4417c64f1c860b77829001987fb5db5" exitCode=0 Nov 03 14:31:47.427653 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:47.427534 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d341af-7c4f477dcc-nvkxn" Nov 03 14:31:47.427653 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:47.427583 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d341af-7c4f477dcc-nvkxn" event={"ID":"78dfe2a2-8938-4f14-a784-f07dbfcdd7f7","Type":"ContainerDied","Data":"34a4e8ddb6cccb8cbe3cfcc341953f72d4417c64f1c860b77829001987fb5db5"} Nov 03 14:31:47.427653 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:47.427640 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-d341af-7c4f477dcc-nvkxn" event={"ID":"78dfe2a2-8938-4f14-a784-f07dbfcdd7f7","Type":"ContainerDied","Data":"b099f22e9a2ae5aecee38e251047a8cd09dbf5e4d92d625ea94e8fb083593b04"} Nov 03 14:31:47.427841 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:47.427673 5979 scope.go:122] "RemoveContainer" containerID="34a4e8ddb6cccb8cbe3cfcc341953f72d4417c64f1c860b77829001987fb5db5" Nov 03 14:31:47.429840 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:47.429816 5979 generic.go:356] "Generic (PLEG): container finished" podID="712c3ae4-3332-4730-9553-d21963331b11" containerID="aff677a73e05914bf046b10107414425c0eddce8da9389984f46ac86a521da64" exitCode=0 Nov 03 14:31:47.429983 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:47.429911 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dbd046-h65vn" event={"ID":"712c3ae4-3332-4730-9553-d21963331b11","Type":"ContainerDied","Data":"aff677a73e05914bf046b10107414425c0eddce8da9389984f46ac86a521da64"} Nov 03 14:31:47.431632 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:47.431606 5979 generic.go:356] "Generic (PLEG): container finished" podID="f03e902b-a262-4d46-b393-dea566c264d2" containerID="80392b315f206fec3a2f219e29a41a44ec383771b0281661a7276d8bd95b4fc4" exitCode=0 Nov 03 14:31:47.431808 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:47.431672 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d341af-5c58b5c667-bnws4" Nov 03 14:31:47.431808 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:47.431695 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d341af-5c58b5c667-bnws4" event={"ID":"f03e902b-a262-4d46-b393-dea566c264d2","Type":"ContainerDied","Data":"80392b315f206fec3a2f219e29a41a44ec383771b0281661a7276d8bd95b4fc4"} Nov 03 14:31:47.431808 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:47.431740 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d341af-5c58b5c667-bnws4" event={"ID":"f03e902b-a262-4d46-b393-dea566c264d2","Type":"ContainerDied","Data":"32aee4de3dc45ecba210f688bda5e7689f20b62c3132c8ba50f27a203dec57c3"} Nov 03 14:31:47.440209 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:47.440189 5979 scope.go:122] "RemoveContainer" containerID="34a4e8ddb6cccb8cbe3cfcc341953f72d4417c64f1c860b77829001987fb5db5" Nov 03 14:31:47.440582 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:31:47.440508 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34a4e8ddb6cccb8cbe3cfcc341953f72d4417c64f1c860b77829001987fb5db5\": container with ID starting with 34a4e8ddb6cccb8cbe3cfcc341953f72d4417c64f1c860b77829001987fb5db5 not found: ID does not exist" containerID="34a4e8ddb6cccb8cbe3cfcc341953f72d4417c64f1c860b77829001987fb5db5" Nov 03 14:31:47.440582 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:47.440537 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34a4e8ddb6cccb8cbe3cfcc341953f72d4417c64f1c860b77829001987fb5db5"} err="failed to get container status \"34a4e8ddb6cccb8cbe3cfcc341953f72d4417c64f1c860b77829001987fb5db5\": rpc error: code = NotFound desc = could not find container \"34a4e8ddb6cccb8cbe3cfcc341953f72d4417c64f1c860b77829001987fb5db5\": container with ID starting with 34a4e8ddb6cccb8cbe3cfcc341953f72d4417c64f1c860b77829001987fb5db5 not found: ID does not exist" Nov 03 14:31:47.440582 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:47.440553 5979 scope.go:122] "RemoveContainer" containerID="80392b315f206fec3a2f219e29a41a44ec383771b0281661a7276d8bd95b4fc4" Nov 03 14:31:47.451276 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:47.451256 5979 scope.go:122] "RemoveContainer" containerID="80392b315f206fec3a2f219e29a41a44ec383771b0281661a7276d8bd95b4fc4" Nov 03 14:31:47.451624 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:31:47.451604 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80392b315f206fec3a2f219e29a41a44ec383771b0281661a7276d8bd95b4fc4\": container with ID starting with 80392b315f206fec3a2f219e29a41a44ec383771b0281661a7276d8bd95b4fc4 not found: ID does not exist" containerID="80392b315f206fec3a2f219e29a41a44ec383771b0281661a7276d8bd95b4fc4" Nov 03 14:31:47.451697 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:47.451632 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80392b315f206fec3a2f219e29a41a44ec383771b0281661a7276d8bd95b4fc4"} err="failed to get container status \"80392b315f206fec3a2f219e29a41a44ec383771b0281661a7276d8bd95b4fc4\": rpc error: code = NotFound desc = could not find container \"80392b315f206fec3a2f219e29a41a44ec383771b0281661a7276d8bd95b4fc4\": container with ID starting with 80392b315f206fec3a2f219e29a41a44ec383771b0281661a7276d8bd95b4fc4 not found: ID does not exist" Nov 03 14:31:47.524566 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:47.524140 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f03e902b-a262-4d46-b393-dea566c264d2-kube-api-access-9c2t2\" (UniqueName: \"kubernetes.io/projected/f03e902b-a262-4d46-b393-dea566c264d2-kube-api-access-9c2t2\") pod \"f03e902b-a262-4d46-b393-dea566c264d2\" (UID: \"f03e902b-a262-4d46-b393-dea566c264d2\") " Nov 03 14:31:47.524566 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:47.524222 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f03e902b-a262-4d46-b393-dea566c264d2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f03e902b-a262-4d46-b393-dea566c264d2-plugins-config-volume\") pod \"f03e902b-a262-4d46-b393-dea566c264d2\" (UID: \"f03e902b-a262-4d46-b393-dea566c264d2\") " Nov 03 14:31:47.524908 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:47.524702 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78dfe2a2-8938-4f14-a784-f07dbfcdd7f7-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:47.524908 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:47.524726 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stffb\" (UniqueName: \"kubernetes.io/projected/78dfe2a2-8938-4f14-a784-f07dbfcdd7f7-kube-api-access-stffb\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:47.524908 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:47.524786 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f03e902b-a262-4d46-b393-dea566c264d2-plugins-config-volume" pod "f03e902b-a262-4d46-b393-dea566c264d2" (UID: "f03e902b-a262-4d46-b393-dea566c264d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:31:47.526941 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:47.526911 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f03e902b-a262-4d46-b393-dea566c264d2-kube-api-access-9c2t2" pod "f03e902b-a262-4d46-b393-dea566c264d2" (UID: "f03e902b-a262-4d46-b393-dea566c264d2"). InnerVolumeSpecName "kube-api-access-9c2t2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:31:47.625562 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:47.625444 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c2t2\" (UniqueName: \"kubernetes.io/projected/f03e902b-a262-4d46-b393-dea566c264d2-kube-api-access-9c2t2\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:47.625562 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:47.625512 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f03e902b-a262-4d46-b393-dea566c264d2-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:47.784271 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:47.784233 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d341af-7c4f477dcc-nvkxn"] Nov 03 14:31:47.788230 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:47.788199 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-d341af-7c4f477dcc-nvkxn"] Nov 03 14:31:47.892886 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:47.892852 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d341af-5c58b5c667-bnws4"] Nov 03 14:31:47.898239 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:47.898215 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-d341af-5c58b5c667-bnws4"] Nov 03 14:31:48.114190 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:48.114146 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78dfe2a2-8938-4f14-a784-f07dbfcdd7f7" path="/var/lib/kubelet/pods/78dfe2a2-8938-4f14-a784-f07dbfcdd7f7/volumes" Nov 03 14:31:48.114673 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:48.114651 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f03e902b-a262-4d46-b393-dea566c264d2" path="/var/lib/kubelet/pods/f03e902b-a262-4d46-b393-dea566c264d2/volumes" Nov 03 14:31:48.576590 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:48.576564 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dbd046-h65vn" Nov 03 14:31:48.735265 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:48.735183 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/712c3ae4-3332-4730-9553-d21963331b11-kube-api-access-fz48f\" (UniqueName: \"kubernetes.io/projected/712c3ae4-3332-4730-9553-d21963331b11-kube-api-access-fz48f\") pod \"712c3ae4-3332-4730-9553-d21963331b11\" (UID: \"712c3ae4-3332-4730-9553-d21963331b11\") " Nov 03 14:31:48.735265 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:48.735254 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/712c3ae4-3332-4730-9553-d21963331b11-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/712c3ae4-3332-4730-9553-d21963331b11-maas-api-xks-ca\") pod \"712c3ae4-3332-4730-9553-d21963331b11\" (UID: \"712c3ae4-3332-4730-9553-d21963331b11\") " Nov 03 14:31:48.735519 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:48.735390 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/712c3ae4-3332-4730-9553-d21963331b11-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/712c3ae4-3332-4730-9553-d21963331b11-maas-api-openshift-ca\") pod \"712c3ae4-3332-4730-9553-d21963331b11\" (UID: \"712c3ae4-3332-4730-9553-d21963331b11\") " Nov 03 14:31:48.735768 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:48.735745 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/712c3ae4-3332-4730-9553-d21963331b11-maas-api-openshift-ca" pod "712c3ae4-3332-4730-9553-d21963331b11" (UID: "712c3ae4-3332-4730-9553-d21963331b11"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:31:48.737393 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:48.737366 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/712c3ae4-3332-4730-9553-d21963331b11-kube-api-access-fz48f" pod "712c3ae4-3332-4730-9553-d21963331b11" (UID: "712c3ae4-3332-4730-9553-d21963331b11"). InnerVolumeSpecName "kube-api-access-fz48f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:31:48.737393 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:48.737369 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/712c3ae4-3332-4730-9553-d21963331b11-maas-api-xks-ca" pod "712c3ae4-3332-4730-9553-d21963331b11" (UID: "712c3ae4-3332-4730-9553-d21963331b11"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:31:48.836966 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:48.836934 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/712c3ae4-3332-4730-9553-d21963331b11-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:48.836966 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:48.836964 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz48f\" (UniqueName: \"kubernetes.io/projected/712c3ae4-3332-4730-9553-d21963331b11-kube-api-access-fz48f\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:48.837184 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:48.836974 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/712c3ae4-3332-4730-9553-d21963331b11-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:49.290536 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:49.290510 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc" Nov 03 14:31:49.441494 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:49.441421 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-istio-podinfo\") pod \"8df65c37-2cc2-4bd1-9d05-4576dfaec9a6\" (UID: \"8df65c37-2cc2-4bd1-9d05-4576dfaec9a6\") " Nov 03 14:31:49.441494 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:49.441495 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-istiod-ca-cert\") pod \"8df65c37-2cc2-4bd1-9d05-4576dfaec9a6\" (UID: \"8df65c37-2cc2-4bd1-9d05-4576dfaec9a6\") " Nov 03 14:31:49.441824 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:49.441536 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-workload-socket\") pod \"8df65c37-2cc2-4bd1-9d05-4576dfaec9a6\" (UID: \"8df65c37-2cc2-4bd1-9d05-4576dfaec9a6\") " Nov 03 14:31:49.441824 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:49.441715 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-credential-socket\") pod \"8df65c37-2cc2-4bd1-9d05-4576dfaec9a6\" (UID: \"8df65c37-2cc2-4bd1-9d05-4576dfaec9a6\") " Nov 03 14:31:49.441824 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:49.441786 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-workload-certs\") pod \"8df65c37-2cc2-4bd1-9d05-4576dfaec9a6\" (UID: \"8df65c37-2cc2-4bd1-9d05-4576dfaec9a6\") " Nov 03 14:31:49.441992 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:49.441836 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-workload-socket" pod "8df65c37-2cc2-4bd1-9d05-4576dfaec9a6" (UID: "8df65c37-2cc2-4bd1-9d05-4576dfaec9a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:31:49.441992 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:49.441841 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-kube-api-access-gsxk4\" (UniqueName: \"kubernetes.io/projected/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-kube-api-access-gsxk4\") pod \"8df65c37-2cc2-4bd1-9d05-4576dfaec9a6\" (UID: \"8df65c37-2cc2-4bd1-9d05-4576dfaec9a6\") " Nov 03 14:31:49.441992 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:49.441915 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-istio-data\") pod \"8df65c37-2cc2-4bd1-9d05-4576dfaec9a6\" (UID: \"8df65c37-2cc2-4bd1-9d05-4576dfaec9a6\") " Nov 03 14:31:49.441992 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:49.441959 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-istiod-ca-cert" pod "8df65c37-2cc2-4bd1-9d05-4576dfaec9a6" (UID: "8df65c37-2cc2-4bd1-9d05-4576dfaec9a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:31:49.442220 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:49.441997 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-istio-token\" (UniqueName: \"kubernetes.io/projected/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-istio-token\") pod \"8df65c37-2cc2-4bd1-9d05-4576dfaec9a6\" (UID: \"8df65c37-2cc2-4bd1-9d05-4576dfaec9a6\") " Nov 03 14:31:49.442220 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:49.442036 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-istio-envoy\") pod \"8df65c37-2cc2-4bd1-9d05-4576dfaec9a6\" (UID: \"8df65c37-2cc2-4bd1-9d05-4576dfaec9a6\") " Nov 03 14:31:49.442220 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:49.442154 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-credential-socket" pod "8df65c37-2cc2-4bd1-9d05-4576dfaec9a6" (UID: "8df65c37-2cc2-4bd1-9d05-4576dfaec9a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:31:49.442220 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:49.442198 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-istio-data" pod "8df65c37-2cc2-4bd1-9d05-4576dfaec9a6" (UID: "8df65c37-2cc2-4bd1-9d05-4576dfaec9a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:31:49.442444 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:49.442267 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-workload-certs" pod "8df65c37-2cc2-4bd1-9d05-4576dfaec9a6" (UID: "8df65c37-2cc2-4bd1-9d05-4576dfaec9a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:31:49.442533 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:49.442457 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:49.442533 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:49.442511 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:49.442533 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:49.442527 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:49.442709 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:49.442542 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:49.442709 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:49.442560 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:49.444032 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:49.443992 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-kube-api-access-gsxk4" pod "8df65c37-2cc2-4bd1-9d05-4576dfaec9a6" (UID: "8df65c37-2cc2-4bd1-9d05-4576dfaec9a6"). InnerVolumeSpecName "kube-api-access-gsxk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:31:49.444161 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:49.444097 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-istio-podinfo" pod "8df65c37-2cc2-4bd1-9d05-4576dfaec9a6" (UID: "8df65c37-2cc2-4bd1-9d05-4576dfaec9a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:31:49.444315 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:49.444298 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-istio-token" pod "8df65c37-2cc2-4bd1-9d05-4576dfaec9a6" (UID: "8df65c37-2cc2-4bd1-9d05-4576dfaec9a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:31:49.444402 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:49.444385 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-istio-envoy" pod "8df65c37-2cc2-4bd1-9d05-4576dfaec9a6" (UID: "8df65c37-2cc2-4bd1-9d05-4576dfaec9a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:31:49.447655 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:49.447631 5979 generic.go:356] "Generic (PLEG): container finished" podID="8df65c37-2cc2-4bd1-9d05-4576dfaec9a6" containerID="7e16e97308b61bf8f320f624ea933515c3d2663ac782cbd17a7417458f9d65dd" exitCode=0 Nov 03 14:31:49.447774 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:49.447680 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc" Nov 03 14:31:49.447774 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:49.447714 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc" event={"ID":"8df65c37-2cc2-4bd1-9d05-4576dfaec9a6","Type":"ContainerDied","Data":"7e16e97308b61bf8f320f624ea933515c3d2663ac782cbd17a7417458f9d65dd"} Nov 03 14:31:49.447902 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:49.447779 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc" event={"ID":"8df65c37-2cc2-4bd1-9d05-4576dfaec9a6","Type":"ContainerDied","Data":"8bfbc78f08819bebec23464da6af413ce213c678322a5b08a3033beb15fe98f5"} Nov 03 14:31:49.447902 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:49.447812 5979 scope.go:122] "RemoveContainer" containerID="7e16e97308b61bf8f320f624ea933515c3d2663ac782cbd17a7417458f9d65dd" Nov 03 14:31:49.449138 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:49.449106 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dbd046-h65vn" event={"ID":"712c3ae4-3332-4730-9553-d21963331b11","Type":"ContainerDied","Data":"c5167c36126f473dd5dca7acd5e88f3b290d5ae611b5575d34a6cf54c6c6fc39"} Nov 03 14:31:49.449210 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:49.449144 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5167c36126f473dd5dca7acd5e88f3b290d5ae611b5575d34a6cf54c6c6fc39" Nov 03 14:31:49.449210 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:49.449149 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83dbd046-h65vn" Nov 03 14:31:49.459821 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:49.459761 5979 scope.go:122] "RemoveContainer" containerID="285bb6e9355b6d8fdcaf8c55057073e92fb5e1074e1af38f22f3036fa5464d1a" Nov 03 14:31:49.470793 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:49.470769 5979 scope.go:122] "RemoveContainer" containerID="7e16e97308b61bf8f320f624ea933515c3d2663ac782cbd17a7417458f9d65dd" Nov 03 14:31:49.471099 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:31:49.471077 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e16e97308b61bf8f320f624ea933515c3d2663ac782cbd17a7417458f9d65dd\": container with ID starting with 7e16e97308b61bf8f320f624ea933515c3d2663ac782cbd17a7417458f9d65dd not found: ID does not exist" containerID="7e16e97308b61bf8f320f624ea933515c3d2663ac782cbd17a7417458f9d65dd" Nov 03 14:31:49.471201 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:49.471105 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e16e97308b61bf8f320f624ea933515c3d2663ac782cbd17a7417458f9d65dd"} err="failed to get container status \"7e16e97308b61bf8f320f624ea933515c3d2663ac782cbd17a7417458f9d65dd\": rpc error: code = NotFound desc = could not find container \"7e16e97308b61bf8f320f624ea933515c3d2663ac782cbd17a7417458f9d65dd\": container with ID starting with 7e16e97308b61bf8f320f624ea933515c3d2663ac782cbd17a7417458f9d65dd not found: ID does not exist" Nov 03 14:31:49.471201 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:49.471122 5979 scope.go:122] "RemoveContainer" containerID="285bb6e9355b6d8fdcaf8c55057073e92fb5e1074e1af38f22f3036fa5464d1a" Nov 03 14:31:49.471385 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:31:49.471371 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"285bb6e9355b6d8fdcaf8c55057073e92fb5e1074e1af38f22f3036fa5464d1a\": container with ID starting with 285bb6e9355b6d8fdcaf8c55057073e92fb5e1074e1af38f22f3036fa5464d1a not found: ID does not exist" containerID="285bb6e9355b6d8fdcaf8c55057073e92fb5e1074e1af38f22f3036fa5464d1a" Nov 03 14:31:49.471437 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:49.471390 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"285bb6e9355b6d8fdcaf8c55057073e92fb5e1074e1af38f22f3036fa5464d1a"} err="failed to get container status \"285bb6e9355b6d8fdcaf8c55057073e92fb5e1074e1af38f22f3036fa5464d1a\": rpc error: code = NotFound desc = could not find container \"285bb6e9355b6d8fdcaf8c55057073e92fb5e1074e1af38f22f3036fa5464d1a\": container with ID starting with 285bb6e9355b6d8fdcaf8c55057073e92fb5e1074e1af38f22f3036fa5464d1a not found: ID does not exist" Nov 03 14:31:49.543590 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:49.543552 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:49.543590 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:49.543585 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:49.543590 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:49.543594 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:49.543850 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:49.543604 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsxk4\" (UniqueName: \"kubernetes.io/projected/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6-kube-api-access-gsxk4\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:31:49.657773 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:49.657735 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc"] Nov 03 14:31:49.662910 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:49.662861 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-d341af-openshift-default-5df74c59dc-75mnc"] Nov 03 14:31:50.113722 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:50.113680 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8df65c37-2cc2-4bd1-9d05-4576dfaec9a6" path="/var/lib/kubelet/pods/8df65c37-2cc2-4bd1-9d05-4576dfaec9a6/volumes" Nov 03 14:31:54.261612 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:54.261576 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:31:57.421395 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:57.421358 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58eb825e-t57b7"] Nov 03 14:31:57.421847 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:57.421813 5979 cpu_manager.go:414] "containerMap: removing container" podUID="f03e902b-a262-4d46-b393-dea566c264d2" containerName="payload-pre-processing" Nov 03 14:31:57.421847 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:57.421826 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f03e902b-a262-4d46-b393-dea566c264d2" containerName="payload-pre-processing" Nov 03 14:31:57.421925 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:57.421854 5979 cpu_manager.go:414] "containerMap: removing container" podUID="8df65c37-2cc2-4bd1-9d05-4576dfaec9a6" containerName="istio-proxy" Nov 03 14:31:57.421925 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:57.421867 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8df65c37-2cc2-4bd1-9d05-4576dfaec9a6" containerName="istio-proxy" Nov 03 14:31:57.421925 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:57.421888 5979 cpu_manager.go:414] "containerMap: removing container" podUID="78dfe2a2-8938-4f14-a784-f07dbfcdd7f7" containerName="payload-processing" Nov 03 14:31:57.421925 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:57.421893 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78dfe2a2-8938-4f14-a784-f07dbfcdd7f7" containerName="payload-processing" Nov 03 14:31:57.421925 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:57.421902 5979 cpu_manager.go:414] "containerMap: removing container" podUID="712c3ae4-3332-4730-9553-d21963331b11" containerName="revoke-keys" Nov 03 14:31:57.421925 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:57.421906 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="712c3ae4-3332-4730-9553-d21963331b11" containerName="revoke-keys" Nov 03 14:31:57.422121 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:57.421989 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="f03e902b-a262-4d46-b393-dea566c264d2" containerName="payload-pre-processing" Nov 03 14:31:57.422121 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:57.422007 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="8df65c37-2cc2-4bd1-9d05-4576dfaec9a6" containerName="istio-proxy" Nov 03 14:31:57.422121 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:57.422016 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="78dfe2a2-8938-4f14-a784-f07dbfcdd7f7" containerName="payload-processing" Nov 03 14:31:57.422121 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:57.422021 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="712c3ae4-3332-4730-9553-d21963331b11" containerName="revoke-keys" Nov 03 14:31:57.427116 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:57.427091 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58eb825e-t57b7" Nov 03 14:31:57.521885 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:57.521842 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92df4ddd-e105-482a-9a46-abf87a7c4033-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58eb825e-t57b7\" (UID: \"92df4ddd-e105-482a-9a46-abf87a7c4033\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58eb825e-t57b7" Nov 03 14:31:57.522039 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:57.521915 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt5qv\" (UniqueName: \"kubernetes.io/projected/92df4ddd-e105-482a-9a46-abf87a7c4033-kube-api-access-kt5qv\") pod \"maas-api-revoke-sub-58eb825e-t57b7\" (UID: \"92df4ddd-e105-482a-9a46-abf87a7c4033\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58eb825e-t57b7" Nov 03 14:31:57.522039 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:57.522008 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92df4ddd-e105-482a-9a46-abf87a7c4033-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58eb825e-t57b7\" (UID: \"92df4ddd-e105-482a-9a46-abf87a7c4033\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58eb825e-t57b7" Nov 03 14:31:57.522280 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:57.522251 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:31:57.522419 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:57.522290 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:31:57.522419 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:57.522316 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:31:57.522537 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:57.522429 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qzgsw\"" Nov 03 14:31:57.524731 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:57.524703 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58eb825e-t57b7"] Nov 03 14:31:57.623674 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:57.623634 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92df4ddd-e105-482a-9a46-abf87a7c4033-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58eb825e-t57b7\" (UID: \"92df4ddd-e105-482a-9a46-abf87a7c4033\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58eb825e-t57b7" Nov 03 14:31:57.623876 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:57.623703 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt5qv\" (UniqueName: \"kubernetes.io/projected/92df4ddd-e105-482a-9a46-abf87a7c4033-kube-api-access-kt5qv\") pod \"maas-api-revoke-sub-58eb825e-t57b7\" (UID: \"92df4ddd-e105-482a-9a46-abf87a7c4033\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58eb825e-t57b7" Nov 03 14:31:57.623876 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:57.623746 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92df4ddd-e105-482a-9a46-abf87a7c4033-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58eb825e-t57b7\" (UID: \"92df4ddd-e105-482a-9a46-abf87a7c4033\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58eb825e-t57b7" Nov 03 14:31:57.624488 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:57.624454 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92df4ddd-e105-482a-9a46-abf87a7c4033-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58eb825e-t57b7\" (UID: \"92df4ddd-e105-482a-9a46-abf87a7c4033\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58eb825e-t57b7" Nov 03 14:31:57.626084 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:57.626060 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92df4ddd-e105-482a-9a46-abf87a7c4033-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58eb825e-t57b7\" (UID: \"92df4ddd-e105-482a-9a46-abf87a7c4033\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58eb825e-t57b7" Nov 03 14:31:57.677760 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:57.677689 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt5qv\" (UniqueName: \"kubernetes.io/projected/92df4ddd-e105-482a-9a46-abf87a7c4033-kube-api-access-kt5qv\") pod \"maas-api-revoke-sub-58eb825e-t57b7\" (UID: \"92df4ddd-e105-482a-9a46-abf87a7c4033\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58eb825e-t57b7" Nov 03 14:31:57.739638 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:57.739604 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58eb825e-t57b7" Nov 03 14:31:58.116433 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:58.116348 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58eb825e-t57b7"] Nov 03 14:31:58.123777 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:31:58.123749 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92df4ddd_e105_482a_9a46_abf87a7c4033.slice/crio-f73bdff35f0a3a8262ac0a7b93c107e8db4f872aa8a8d5fd625792fc0bc05628 WatchSource:0}: Error finding container f73bdff35f0a3a8262ac0a7b93c107e8db4f872aa8a8d5fd625792fc0bc05628: Status 404 returned error can't find the container with id f73bdff35f0a3a8262ac0a7b93c107e8db4f872aa8a8d5fd625792fc0bc05628 Nov 03 14:31:58.502288 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:58.502240 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58eb825e-t57b7" event={"ID":"92df4ddd-e105-482a-9a46-abf87a7c4033","Type":"ContainerStarted","Data":"6fa41ed68ad8fa334f4d28ed7eda04a3caff41adf7d2a752bdd5a0c739613506"} Nov 03 14:31:58.502724 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:58.502306 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58eb825e-t57b7" event={"ID":"92df4ddd-e105-482a-9a46-abf87a7c4033","Type":"ContainerStarted","Data":"f73bdff35f0a3a8262ac0a7b93c107e8db4f872aa8a8d5fd625792fc0bc05628"} Nov 03 14:31:58.610019 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:58.609953 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58eb825e-t57b7" podStartSLOduration=1.609932389 podStartE2EDuration="1.609932389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:31:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:31:58.608737207 +0000 UTC m=+5200.619425448" watchObservedRunningTime="2026-11-03 14:31:58.609932389 +0000 UTC m=+5200.620620627" Nov 03 14:31:59.509037 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:59.508994 5979 generic.go:356] "Generic (PLEG): container finished" podID="92df4ddd-e105-482a-9a46-abf87a7c4033" containerID="6fa41ed68ad8fa334f4d28ed7eda04a3caff41adf7d2a752bdd5a0c739613506" exitCode=0 Nov 03 14:31:59.509446 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:31:59.509108 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58eb825e-t57b7" event={"ID":"92df4ddd-e105-482a-9a46-abf87a7c4033","Type":"ContainerDied","Data":"6fa41ed68ad8fa334f4d28ed7eda04a3caff41adf7d2a752bdd5a0c739613506"} Nov 03 14:32:00.649144 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:00.649116 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58eb825e-t57b7" Nov 03 14:32:00.756381 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:00.756334 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92df4ddd-e105-482a-9a46-abf87a7c4033-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92df4ddd-e105-482a-9a46-abf87a7c4033-maas-api-openshift-ca\") pod \"92df4ddd-e105-482a-9a46-abf87a7c4033\" (UID: \"92df4ddd-e105-482a-9a46-abf87a7c4033\") " Nov 03 14:32:00.756626 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:00.756403 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92df4ddd-e105-482a-9a46-abf87a7c4033-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92df4ddd-e105-482a-9a46-abf87a7c4033-maas-api-xks-ca\") pod \"92df4ddd-e105-482a-9a46-abf87a7c4033\" (UID: \"92df4ddd-e105-482a-9a46-abf87a7c4033\") " Nov 03 14:32:00.756714 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:00.756629 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92df4ddd-e105-482a-9a46-abf87a7c4033-kube-api-access-kt5qv\" (UniqueName: \"kubernetes.io/projected/92df4ddd-e105-482a-9a46-abf87a7c4033-kube-api-access-kt5qv\") pod \"92df4ddd-e105-482a-9a46-abf87a7c4033\" (UID: \"92df4ddd-e105-482a-9a46-abf87a7c4033\") " Nov 03 14:32:00.756803 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:00.756774 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92df4ddd-e105-482a-9a46-abf87a7c4033-maas-api-openshift-ca" pod "92df4ddd-e105-482a-9a46-abf87a7c4033" (UID: "92df4ddd-e105-482a-9a46-abf87a7c4033"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:32:00.757010 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:00.756988 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92df4ddd-e105-482a-9a46-abf87a7c4033-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:32:00.758801 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:00.758772 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92df4ddd-e105-482a-9a46-abf87a7c4033-kube-api-access-kt5qv" pod "92df4ddd-e105-482a-9a46-abf87a7c4033" (UID: "92df4ddd-e105-482a-9a46-abf87a7c4033"). InnerVolumeSpecName "kube-api-access-kt5qv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:32:00.758934 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:00.758799 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92df4ddd-e105-482a-9a46-abf87a7c4033-maas-api-xks-ca" pod "92df4ddd-e105-482a-9a46-abf87a7c4033" (UID: "92df4ddd-e105-482a-9a46-abf87a7c4033"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:32:00.857636 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:00.857541 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kt5qv\" (UniqueName: \"kubernetes.io/projected/92df4ddd-e105-482a-9a46-abf87a7c4033-kube-api-access-kt5qv\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:32:00.857636 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:00.857578 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92df4ddd-e105-482a-9a46-abf87a7c4033-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:32:01.522971 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:01.522944 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58eb825e-t57b7" Nov 03 14:32:01.523166 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:01.522936 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58eb825e-t57b7" event={"ID":"92df4ddd-e105-482a-9a46-abf87a7c4033","Type":"ContainerDied","Data":"f73bdff35f0a3a8262ac0a7b93c107e8db4f872aa8a8d5fd625792fc0bc05628"} Nov 03 14:32:01.523166 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:01.523077 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f73bdff35f0a3a8262ac0a7b93c107e8db4f872aa8a8d5fd625792fc0bc05628" Nov 03 14:32:07.233118 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:32:07.233083 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:32:11.878552 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:11.878516 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:32:24.791907 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:24.791873 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4"] Nov 03 14:32:24.792332 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:24.792317 5979 cpu_manager.go:414] "containerMap: removing container" podUID="8df65c37-2cc2-4bd1-9d05-4576dfaec9a6" containerName="istio-proxy" Nov 03 14:32:24.792380 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:24.792333 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8df65c37-2cc2-4bd1-9d05-4576dfaec9a6" containerName="istio-proxy" Nov 03 14:32:24.792380 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:24.792345 5979 cpu_manager.go:414] "containerMap: removing container" podUID="92df4ddd-e105-482a-9a46-abf87a7c4033" containerName="revoke-keys" Nov 03 14:32:24.792380 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:24.792351 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92df4ddd-e105-482a-9a46-abf87a7c4033" containerName="revoke-keys" Nov 03 14:32:24.792505 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:24.792426 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="8df65c37-2cc2-4bd1-9d05-4576dfaec9a6" containerName="istio-proxy" Nov 03 14:32:24.792505 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:24.792435 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="92df4ddd-e105-482a-9a46-abf87a7c4033" containerName="revoke-keys" Nov 03 14:32:24.797163 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:24.797142 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4" Nov 03 14:32:24.846307 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:24.846277 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-cdb532-openshift-default-dockercfg-q2fdz\"" Nov 03 14:32:24.888399 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:24.888352 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5065069-3e2e-4ead-89f9-a391e4d42b36-istio-envoy\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4\" (UID: \"f5065069-3e2e-4ead-89f9-a391e4d42b36\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4" Nov 03 14:32:24.888399 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:24.888400 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5065069-3e2e-4ead-89f9-a391e4d42b36-workload-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4\" (UID: \"f5065069-3e2e-4ead-89f9-a391e4d42b36\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4" Nov 03 14:32:24.888643 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:24.888419 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5065069-3e2e-4ead-89f9-a391e4d42b36-istio-token\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4\" (UID: \"f5065069-3e2e-4ead-89f9-a391e4d42b36\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4" Nov 03 14:32:24.888643 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:24.888550 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5065069-3e2e-4ead-89f9-a391e4d42b36-workload-certs\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4\" (UID: \"f5065069-3e2e-4ead-89f9-a391e4d42b36\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4" Nov 03 14:32:24.888643 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:24.888581 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5065069-3e2e-4ead-89f9-a391e4d42b36-istiod-ca-cert\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4\" (UID: \"f5065069-3e2e-4ead-89f9-a391e4d42b36\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4" Nov 03 14:32:24.888643 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:24.888605 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5065069-3e2e-4ead-89f9-a391e4d42b36-istio-data\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4\" (UID: \"f5065069-3e2e-4ead-89f9-a391e4d42b36\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4" Nov 03 14:32:24.888815 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:24.888642 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5065069-3e2e-4ead-89f9-a391e4d42b36-istio-podinfo\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4\" (UID: \"f5065069-3e2e-4ead-89f9-a391e4d42b36\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4" Nov 03 14:32:24.888815 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:24.888700 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5065069-3e2e-4ead-89f9-a391e4d42b36-credential-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4\" (UID: \"f5065069-3e2e-4ead-89f9-a391e4d42b36\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4" Nov 03 14:32:24.888815 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:24.888745 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28zxm\" (UniqueName: \"kubernetes.io/projected/f5065069-3e2e-4ead-89f9-a391e4d42b36-kube-api-access-28zxm\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4\" (UID: \"f5065069-3e2e-4ead-89f9-a391e4d42b36\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4" Nov 03 14:32:24.894842 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:24.894805 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4"] Nov 03 14:32:24.990200 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:24.990155 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5065069-3e2e-4ead-89f9-a391e4d42b36-workload-certs\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4\" (UID: \"f5065069-3e2e-4ead-89f9-a391e4d42b36\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4" Nov 03 14:32:24.990200 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:24.990204 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5065069-3e2e-4ead-89f9-a391e4d42b36-istiod-ca-cert\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4\" (UID: \"f5065069-3e2e-4ead-89f9-a391e4d42b36\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4" Nov 03 14:32:24.990507 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:24.990228 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5065069-3e2e-4ead-89f9-a391e4d42b36-istio-data\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4\" (UID: \"f5065069-3e2e-4ead-89f9-a391e4d42b36\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4" Nov 03 14:32:24.990507 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:24.990258 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5065069-3e2e-4ead-89f9-a391e4d42b36-istio-podinfo\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4\" (UID: \"f5065069-3e2e-4ead-89f9-a391e4d42b36\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4" Nov 03 14:32:24.990507 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:24.990277 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5065069-3e2e-4ead-89f9-a391e4d42b36-credential-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4\" (UID: \"f5065069-3e2e-4ead-89f9-a391e4d42b36\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4" Nov 03 14:32:24.990507 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:24.990311 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28zxm\" (UniqueName: \"kubernetes.io/projected/f5065069-3e2e-4ead-89f9-a391e4d42b36-kube-api-access-28zxm\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4\" (UID: \"f5065069-3e2e-4ead-89f9-a391e4d42b36\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4" Nov 03 14:32:24.990507 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:24.990394 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5065069-3e2e-4ead-89f9-a391e4d42b36-istio-envoy\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4\" (UID: \"f5065069-3e2e-4ead-89f9-a391e4d42b36\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4" Nov 03 14:32:24.990507 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:24.990430 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5065069-3e2e-4ead-89f9-a391e4d42b36-workload-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4\" (UID: \"f5065069-3e2e-4ead-89f9-a391e4d42b36\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4" Nov 03 14:32:24.990507 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:24.990457 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5065069-3e2e-4ead-89f9-a391e4d42b36-istio-token\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4\" (UID: \"f5065069-3e2e-4ead-89f9-a391e4d42b36\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4" Nov 03 14:32:24.990943 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:24.990683 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5065069-3e2e-4ead-89f9-a391e4d42b36-workload-certs\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4\" (UID: \"f5065069-3e2e-4ead-89f9-a391e4d42b36\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4" Nov 03 14:32:24.990943 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:24.990764 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5065069-3e2e-4ead-89f9-a391e4d42b36-credential-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4\" (UID: \"f5065069-3e2e-4ead-89f9-a391e4d42b36\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4" Nov 03 14:32:24.990943 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:24.990929 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5065069-3e2e-4ead-89f9-a391e4d42b36-workload-socket\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4\" (UID: \"f5065069-3e2e-4ead-89f9-a391e4d42b36\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4" Nov 03 14:32:24.991121 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:24.990963 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5065069-3e2e-4ead-89f9-a391e4d42b36-istio-data\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4\" (UID: \"f5065069-3e2e-4ead-89f9-a391e4d42b36\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4" Nov 03 14:32:24.991121 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:24.991046 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5065069-3e2e-4ead-89f9-a391e4d42b36-istiod-ca-cert\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4\" (UID: \"f5065069-3e2e-4ead-89f9-a391e4d42b36\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4" Nov 03 14:32:24.992713 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:24.992690 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5065069-3e2e-4ead-89f9-a391e4d42b36-istio-envoy\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4\" (UID: \"f5065069-3e2e-4ead-89f9-a391e4d42b36\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4" Nov 03 14:32:24.993010 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:24.992991 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5065069-3e2e-4ead-89f9-a391e4d42b36-istio-podinfo\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4\" (UID: \"f5065069-3e2e-4ead-89f9-a391e4d42b36\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4" Nov 03 14:32:25.046606 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:25.046531 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5065069-3e2e-4ead-89f9-a391e4d42b36-istio-token\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4\" (UID: \"f5065069-3e2e-4ead-89f9-a391e4d42b36\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4" Nov 03 14:32:25.046752 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:25.046618 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28zxm\" (UniqueName: \"kubernetes.io/projected/f5065069-3e2e-4ead-89f9-a391e4d42b36-kube-api-access-28zxm\") pod \"e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4\" (UID: \"f5065069-3e2e-4ead-89f9-a391e4d42b36\") " pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4" Nov 03 14:32:25.109625 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:25.109578 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4" Nov 03 14:32:25.535998 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:25.535952 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4"] Nov 03 14:32:25.542714 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:32:25.542687 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5065069_3e2e_4ead_89f9_a391e4d42b36.slice/crio-c8c51cdae1e704f7e8c409ed30fe145f43dfa71329130ca23d4d1ae659d4a333 WatchSource:0}: Error finding container c8c51cdae1e704f7e8c409ed30fe145f43dfa71329130ca23d4d1ae659d4a333: Status 404 returned error can't find the container with id c8c51cdae1e704f7e8c409ed30fe145f43dfa71329130ca23d4d1ae659d4a333 Nov 03 14:32:25.545041 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:25.545009 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:32:25.545152 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:25.545091 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:32:25.545152 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:25.545136 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:32:25.649332 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:25.649293 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4" event={"ID":"f5065069-3e2e-4ead-89f9-a391e4d42b36","Type":"ContainerStarted","Data":"06b48871718ae87648825514da0c872a0e924097b366f3636b0f5d108b04247b"} Nov 03 14:32:25.649542 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:25.649379 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4" event={"ID":"f5065069-3e2e-4ead-89f9-a391e4d42b36","Type":"ContainerStarted","Data":"c8c51cdae1e704f7e8c409ed30fe145f43dfa71329130ca23d4d1ae659d4a333"} Nov 03 14:32:25.762327 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:25.762277 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4" podStartSLOduration=1.762264506 podStartE2EDuration="1.762264506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:32:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:32:25.76168084 +0000 UTC m=+5227.772369081" watchObservedRunningTime="2026-11-03 14:32:25.762264506 +0000 UTC m=+5227.772952748" Nov 03 14:32:26.111511 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:26.111464 5979 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.126:15021/healthz/ready\": dial tcp 10.134.1.126:15021: connect: connection refused" start-of-body= Nov 03 14:32:26.112026 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:26.111525 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4" podUID="f5065069-3e2e-4ead-89f9-a391e4d42b36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.126:15021/healthz/ready\": dial tcp 10.134.1.126:15021: connect: connection refused" Nov 03 14:32:26.114169 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:26.114148 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4" Nov 03 14:32:27.110154 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:27.110120 5979 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.126:15021/healthz/ready\": dial tcp 10.134.1.126:15021: connect: connection refused" start-of-body= Nov 03 14:32:27.110350 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:27.110184 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4" podUID="f5065069-3e2e-4ead-89f9-a391e4d42b36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.126:15021/healthz/ready\": dial tcp 10.134.1.126:15021: connect: connection refused" Nov 03 14:32:28.110009 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:28.109973 5979 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.126:15021/healthz/ready\": dial tcp 10.134.1.126:15021: connect: connection refused" start-of-body= Nov 03 14:32:28.110482 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:28.110032 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4" podUID="f5065069-3e2e-4ead-89f9-a391e4d42b36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.126:15021/healthz/ready\": dial tcp 10.134.1.126:15021: connect: connection refused" Nov 03 14:32:29.042733 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:29.042649 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba40e004-z6twm"] Nov 03 14:32:29.089065 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:29.089029 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba40e004-z6twm"] Nov 03 14:32:29.110092 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:29.110060 5979 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.126:15021/healthz/ready\": dial tcp 10.134.1.126:15021: connect: connection refused" start-of-body= Nov 03 14:32:29.110490 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:29.110106 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4" podUID="f5065069-3e2e-4ead-89f9-a391e4d42b36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.126:15021/healthz/ready\": dial tcp 10.134.1.126:15021: connect: connection refused" Nov 03 14:32:30.110423 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:30.110376 5979 patch_prober.go:29] interesting pod/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.126:15021/healthz/ready\": dial tcp 10.134.1.126:15021: connect: connection refused" start-of-body= Nov 03 14:32:30.110908 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:30.110441 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4" podUID="f5065069-3e2e-4ead-89f9-a391e4d42b36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.126:15021/healthz/ready\": dial tcp 10.134.1.126:15021: connect: connection refused" Nov 03 14:32:30.114208 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:30.114183 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a792dc9-3da9-45e8-ba2b-71885ba2505c" path="/var/lib/kubelet/pods/5a792dc9-3da9-45e8-ba2b-71885ba2505c/volumes" Nov 03 14:32:30.615171 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:30.615126 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4"] Nov 03 14:32:30.615516 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:30.615444 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4" podUID="f5065069-3e2e-4ead-89f9-a391e4d42b36" containerName="istio-proxy" containerID="cri-o://06b48871718ae87648825514da0c872a0e924097b366f3636b0f5d108b04247b" gracePeriod=30 Nov 03 14:32:33.474731 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:33.474693 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-466qf_f31d565d-15b1-4b03-bbfc-ff43778c7a95/limitador/0.log" Nov 03 14:32:36.307056 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:36.307014 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-588d644cc-njxzq_445b3b20-c0f6-41d9-ab82-d1598427036b/manager/1.log" Nov 03 14:32:37.065547 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:37.065512 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-fwvg7_f33ad3b9-499c-41f5-9cf5-029709b82ca6/proxy/0.log" Nov 03 14:32:37.168192 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:37.168163 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4" Nov 03 14:32:37.233854 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:32:37.233763 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:32:37.337540 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:37.337506 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5065069-3e2e-4ead-89f9-a391e4d42b36-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5065069-3e2e-4ead-89f9-a391e4d42b36-istio-envoy\") pod \"f5065069-3e2e-4ead-89f9-a391e4d42b36\" (UID: \"f5065069-3e2e-4ead-89f9-a391e4d42b36\") " Nov 03 14:32:37.338034 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:37.337603 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f5065069-3e2e-4ead-89f9-a391e4d42b36-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5065069-3e2e-4ead-89f9-a391e4d42b36-istio-podinfo\") pod \"f5065069-3e2e-4ead-89f9-a391e4d42b36\" (UID: \"f5065069-3e2e-4ead-89f9-a391e4d42b36\") " Nov 03 14:32:37.338034 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:37.337644 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5065069-3e2e-4ead-89f9-a391e4d42b36-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5065069-3e2e-4ead-89f9-a391e4d42b36-workload-certs\") pod \"f5065069-3e2e-4ead-89f9-a391e4d42b36\" (UID: \"f5065069-3e2e-4ead-89f9-a391e4d42b36\") " Nov 03 14:32:37.338034 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:37.337693 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5065069-3e2e-4ead-89f9-a391e4d42b36-istio-token\" (UniqueName: \"kubernetes.io/projected/f5065069-3e2e-4ead-89f9-a391e4d42b36-istio-token\") pod \"f5065069-3e2e-4ead-89f9-a391e4d42b36\" (UID: \"f5065069-3e2e-4ead-89f9-a391e4d42b36\") " Nov 03 14:32:37.338215 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:37.338154 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5065069-3e2e-4ead-89f9-a391e4d42b36-workload-certs" pod "f5065069-3e2e-4ead-89f9-a391e4d42b36" (UID: "f5065069-3e2e-4ead-89f9-a391e4d42b36"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:32:37.338215 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:37.338206 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5065069-3e2e-4ead-89f9-a391e4d42b36-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5065069-3e2e-4ead-89f9-a391e4d42b36-istiod-ca-cert\") pod \"f5065069-3e2e-4ead-89f9-a391e4d42b36\" (UID: \"f5065069-3e2e-4ead-89f9-a391e4d42b36\") " Nov 03 14:32:37.338330 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:37.338270 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5065069-3e2e-4ead-89f9-a391e4d42b36-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5065069-3e2e-4ead-89f9-a391e4d42b36-workload-socket\") pod \"f5065069-3e2e-4ead-89f9-a391e4d42b36\" (UID: \"f5065069-3e2e-4ead-89f9-a391e4d42b36\") " Nov 03 14:32:37.338396 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:37.338328 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5065069-3e2e-4ead-89f9-a391e4d42b36-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5065069-3e2e-4ead-89f9-a391e4d42b36-credential-socket\") pod \"f5065069-3e2e-4ead-89f9-a391e4d42b36\" (UID: \"f5065069-3e2e-4ead-89f9-a391e4d42b36\") " Nov 03 14:32:37.338396 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:37.338372 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5065069-3e2e-4ead-89f9-a391e4d42b36-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5065069-3e2e-4ead-89f9-a391e4d42b36-istio-data\") pod \"f5065069-3e2e-4ead-89f9-a391e4d42b36\" (UID: \"f5065069-3e2e-4ead-89f9-a391e4d42b36\") " Nov 03 14:32:37.338540 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:37.338410 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5065069-3e2e-4ead-89f9-a391e4d42b36-kube-api-access-28zxm\" (UniqueName: \"kubernetes.io/projected/f5065069-3e2e-4ead-89f9-a391e4d42b36-kube-api-access-28zxm\") pod \"f5065069-3e2e-4ead-89f9-a391e4d42b36\" (UID: \"f5065069-3e2e-4ead-89f9-a391e4d42b36\") " Nov 03 14:32:37.338597 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:37.338572 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5065069-3e2e-4ead-89f9-a391e4d42b36-istiod-ca-cert" pod "f5065069-3e2e-4ead-89f9-a391e4d42b36" (UID: "f5065069-3e2e-4ead-89f9-a391e4d42b36"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:32:37.339116 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:37.339072 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5065069-3e2e-4ead-89f9-a391e4d42b36-istio-data" pod "f5065069-3e2e-4ead-89f9-a391e4d42b36" (UID: "f5065069-3e2e-4ead-89f9-a391e4d42b36"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:32:37.339271 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:37.339203 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5065069-3e2e-4ead-89f9-a391e4d42b36-workload-socket" pod "f5065069-3e2e-4ead-89f9-a391e4d42b36" (UID: "f5065069-3e2e-4ead-89f9-a391e4d42b36"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:32:37.339344 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:37.339270 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5065069-3e2e-4ead-89f9-a391e4d42b36-credential-socket" pod "f5065069-3e2e-4ead-89f9-a391e4d42b36" (UID: "f5065069-3e2e-4ead-89f9-a391e4d42b36"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:32:37.339446 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:37.339418 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5065069-3e2e-4ead-89f9-a391e4d42b36-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:32:37.339608 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:37.339457 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5065069-3e2e-4ead-89f9-a391e4d42b36-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:32:37.340486 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:37.340447 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5065069-3e2e-4ead-89f9-a391e4d42b36-istio-envoy" pod "f5065069-3e2e-4ead-89f9-a391e4d42b36" (UID: "f5065069-3e2e-4ead-89f9-a391e4d42b36"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:32:37.340594 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:37.340548 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5065069-3e2e-4ead-89f9-a391e4d42b36-istio-podinfo" pod "f5065069-3e2e-4ead-89f9-a391e4d42b36" (UID: "f5065069-3e2e-4ead-89f9-a391e4d42b36"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:32:37.340765 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:37.340745 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5065069-3e2e-4ead-89f9-a391e4d42b36-kube-api-access-28zxm" pod "f5065069-3e2e-4ead-89f9-a391e4d42b36" (UID: "f5065069-3e2e-4ead-89f9-a391e4d42b36"). InnerVolumeSpecName "kube-api-access-28zxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:32:37.340846 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:37.340747 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5065069-3e2e-4ead-89f9-a391e4d42b36-istio-token" pod "f5065069-3e2e-4ead-89f9-a391e4d42b36" (UID: "f5065069-3e2e-4ead-89f9-a391e4d42b36"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:32:37.440143 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:37.440111 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5065069-3e2e-4ead-89f9-a391e4d42b36-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:32:37.440143 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:37.440139 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5065069-3e2e-4ead-89f9-a391e4d42b36-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:32:37.440143 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:37.440148 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5065069-3e2e-4ead-89f9-a391e4d42b36-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:32:37.440393 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:37.440157 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5065069-3e2e-4ead-89f9-a391e4d42b36-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:32:37.440393 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:37.440169 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5065069-3e2e-4ead-89f9-a391e4d42b36-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:32:37.440393 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:37.440178 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5065069-3e2e-4ead-89f9-a391e4d42b36-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:32:37.440393 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:37.440187 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28zxm\" (UniqueName: \"kubernetes.io/projected/f5065069-3e2e-4ead-89f9-a391e4d42b36-kube-api-access-28zxm\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:32:37.697733 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:37.697697 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-588d644cc-njxzq_445b3b20-c0f6-41d9-ab82-d1598427036b/manager/0.log" Nov 03 14:32:37.738827 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:37.738795 5979 generic.go:356] "Generic (PLEG): container finished" podID="f5065069-3e2e-4ead-89f9-a391e4d42b36" containerID="06b48871718ae87648825514da0c872a0e924097b366f3636b0f5d108b04247b" exitCode=0 Nov 03 14:32:37.739027 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:37.738842 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4" Nov 03 14:32:37.739027 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:37.738890 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4" event={"ID":"f5065069-3e2e-4ead-89f9-a391e4d42b36","Type":"ContainerDied","Data":"06b48871718ae87648825514da0c872a0e924097b366f3636b0f5d108b04247b"} Nov 03 14:32:37.739027 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:37.738950 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4" event={"ID":"f5065069-3e2e-4ead-89f9-a391e4d42b36","Type":"ContainerDied","Data":"c8c51cdae1e704f7e8c409ed30fe145f43dfa71329130ca23d4d1ae659d4a333"} Nov 03 14:32:37.739027 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:37.738980 5979 scope.go:122] "RemoveContainer" containerID="06b48871718ae87648825514da0c872a0e924097b366f3636b0f5d108b04247b" Nov 03 14:32:37.761910 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:37.749743 5979 scope.go:122] "RemoveContainer" containerID="06b48871718ae87648825514da0c872a0e924097b366f3636b0f5d108b04247b" Nov 03 14:32:37.761910 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:32:37.750054 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06b48871718ae87648825514da0c872a0e924097b366f3636b0f5d108b04247b\": container with ID starting with 06b48871718ae87648825514da0c872a0e924097b366f3636b0f5d108b04247b not found: ID does not exist" containerID="06b48871718ae87648825514da0c872a0e924097b366f3636b0f5d108b04247b" Nov 03 14:32:37.761910 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:37.750082 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06b48871718ae87648825514da0c872a0e924097b366f3636b0f5d108b04247b"} err="failed to get container status \"06b48871718ae87648825514da0c872a0e924097b366f3636b0f5d108b04247b\": rpc error: code = NotFound desc = could not find container \"06b48871718ae87648825514da0c872a0e924097b366f3636b0f5d108b04247b\": container with ID starting with 06b48871718ae87648825514da0c872a0e924097b366f3636b0f5d108b04247b not found: ID does not exist" Nov 03 14:32:37.894243 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:37.894207 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4"] Nov 03 14:32:37.897315 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:37.897290 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-cdb532-openshift-default-8565bc776c-fqpz4"] Nov 03 14:32:38.114079 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:38.114005 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5065069-3e2e-4ead-89f9-a391e4d42b36" path="/var/lib/kubelet/pods/f5065069-3e2e-4ead-89f9-a391e4d42b36/volumes" Nov 03 14:32:40.047737 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:40.047695 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-675bbdab-xk5xj"] Nov 03 14:32:40.094144 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:40.094109 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-675bbdab-xk5xj"] Nov 03 14:32:40.114253 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:40.114218 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f552bb4d-99f9-4c4a-91ca-8ba563e522c4" path="/var/lib/kubelet/pods/f552bb4d-99f9-4c4a-91ca-8ba563e522c4/volumes" Nov 03 14:32:42.524333 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:42.524296 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm"] Nov 03 14:32:42.524846 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:42.524750 5979 cpu_manager.go:414] "containerMap: removing container" podUID="f5065069-3e2e-4ead-89f9-a391e4d42b36" containerName="istio-proxy" Nov 03 14:32:42.524846 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:42.524762 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5065069-3e2e-4ead-89f9-a391e4d42b36" containerName="istio-proxy" Nov 03 14:32:42.524846 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:42.524834 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5065069-3e2e-4ead-89f9-a391e4d42b36" containerName="istio-proxy" Nov 03 14:32:42.528614 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:42.528590 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm" Nov 03 14:32:42.577757 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:42.577721 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-f530cd7c-openshift-default-dockercfg-9kjzb\"" Nov 03 14:32:42.585380 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:42.585338 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1f2dcbe-0119-403d-b481-9fe2596bb889-workload-certs\") pod \"e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm\" (UID: \"c1f2dcbe-0119-403d-b481-9fe2596bb889\") " pod="openshift-ingress/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm" Nov 03 14:32:42.585574 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:42.585384 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1f2dcbe-0119-403d-b481-9fe2596bb889-istio-data\") pod \"e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm\" (UID: \"c1f2dcbe-0119-403d-b481-9fe2596bb889\") " pod="openshift-ingress/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm" Nov 03 14:32:42.585574 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:42.585417 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1f2dcbe-0119-403d-b481-9fe2596bb889-istio-token\") pod \"e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm\" (UID: \"c1f2dcbe-0119-403d-b481-9fe2596bb889\") " pod="openshift-ingress/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm" Nov 03 14:32:42.585669 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:42.585614 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1f2dcbe-0119-403d-b481-9fe2596bb889-workload-socket\") pod \"e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm\" (UID: \"c1f2dcbe-0119-403d-b481-9fe2596bb889\") " pod="openshift-ingress/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm" Nov 03 14:32:42.585669 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:42.585642 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1f2dcbe-0119-403d-b481-9fe2596bb889-credential-socket\") pod \"e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm\" (UID: \"c1f2dcbe-0119-403d-b481-9fe2596bb889\") " pod="openshift-ingress/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm" Nov 03 14:32:42.585751 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:42.585725 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1f2dcbe-0119-403d-b481-9fe2596bb889-istiod-ca-cert\") pod \"e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm\" (UID: \"c1f2dcbe-0119-403d-b481-9fe2596bb889\") " pod="openshift-ingress/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm" Nov 03 14:32:42.585751 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:42.585745 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1f2dcbe-0119-403d-b481-9fe2596bb889-istio-podinfo\") pod \"e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm\" (UID: \"c1f2dcbe-0119-403d-b481-9fe2596bb889\") " pod="openshift-ingress/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm" Nov 03 14:32:42.585844 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:42.585775 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1f2dcbe-0119-403d-b481-9fe2596bb889-istio-envoy\") pod \"e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm\" (UID: \"c1f2dcbe-0119-403d-b481-9fe2596bb889\") " pod="openshift-ingress/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm" Nov 03 14:32:42.585844 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:42.585818 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg4cr\" (UniqueName: \"kubernetes.io/projected/c1f2dcbe-0119-403d-b481-9fe2596bb889-kube-api-access-lg4cr\") pod \"e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm\" (UID: \"c1f2dcbe-0119-403d-b481-9fe2596bb889\") " pod="openshift-ingress/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm" Nov 03 14:32:42.626335 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:42.626302 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm"] Nov 03 14:32:42.686561 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:42.686525 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1f2dcbe-0119-403d-b481-9fe2596bb889-istio-envoy\") pod \"e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm\" (UID: \"c1f2dcbe-0119-403d-b481-9fe2596bb889\") " pod="openshift-ingress/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm" Nov 03 14:32:42.686561 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:42.686564 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg4cr\" (UniqueName: \"kubernetes.io/projected/c1f2dcbe-0119-403d-b481-9fe2596bb889-kube-api-access-lg4cr\") pod \"e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm\" (UID: \"c1f2dcbe-0119-403d-b481-9fe2596bb889\") " pod="openshift-ingress/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm" Nov 03 14:32:42.686838 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:42.686616 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1f2dcbe-0119-403d-b481-9fe2596bb889-workload-certs\") pod \"e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm\" (UID: \"c1f2dcbe-0119-403d-b481-9fe2596bb889\") " pod="openshift-ingress/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm" Nov 03 14:32:42.686838 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:42.686642 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1f2dcbe-0119-403d-b481-9fe2596bb889-istio-data\") pod \"e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm\" (UID: \"c1f2dcbe-0119-403d-b481-9fe2596bb889\") " pod="openshift-ingress/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm" Nov 03 14:32:42.686838 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:42.686674 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1f2dcbe-0119-403d-b481-9fe2596bb889-istio-token\") pod \"e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm\" (UID: \"c1f2dcbe-0119-403d-b481-9fe2596bb889\") " pod="openshift-ingress/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm" Nov 03 14:32:42.686838 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:42.686752 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1f2dcbe-0119-403d-b481-9fe2596bb889-workload-socket\") pod \"e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm\" (UID: \"c1f2dcbe-0119-403d-b481-9fe2596bb889\") " pod="openshift-ingress/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm" Nov 03 14:32:42.687088 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:42.686931 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1f2dcbe-0119-403d-b481-9fe2596bb889-credential-socket\") pod \"e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm\" (UID: \"c1f2dcbe-0119-403d-b481-9fe2596bb889\") " pod="openshift-ingress/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm" Nov 03 14:32:42.687346 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:42.687281 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1f2dcbe-0119-403d-b481-9fe2596bb889-istio-data\") pod \"e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm\" (UID: \"c1f2dcbe-0119-403d-b481-9fe2596bb889\") " pod="openshift-ingress/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm" Nov 03 14:32:42.687346 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:42.687315 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1f2dcbe-0119-403d-b481-9fe2596bb889-credential-socket\") pod \"e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm\" (UID: \"c1f2dcbe-0119-403d-b481-9fe2596bb889\") " pod="openshift-ingress/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm" Nov 03 14:32:42.687346 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:42.687336 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1f2dcbe-0119-403d-b481-9fe2596bb889-workload-socket\") pod \"e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm\" (UID: \"c1f2dcbe-0119-403d-b481-9fe2596bb889\") " pod="openshift-ingress/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm" Nov 03 14:32:42.687679 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:42.687435 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1f2dcbe-0119-403d-b481-9fe2596bb889-istiod-ca-cert\") pod \"e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm\" (UID: \"c1f2dcbe-0119-403d-b481-9fe2596bb889\") " pod="openshift-ingress/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm" Nov 03 14:32:42.687679 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:42.687458 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1f2dcbe-0119-403d-b481-9fe2596bb889-istio-podinfo\") pod \"e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm\" (UID: \"c1f2dcbe-0119-403d-b481-9fe2596bb889\") " pod="openshift-ingress/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm" Nov 03 14:32:42.687679 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:42.687512 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1f2dcbe-0119-403d-b481-9fe2596bb889-workload-certs\") pod \"e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm\" (UID: \"c1f2dcbe-0119-403d-b481-9fe2596bb889\") " pod="openshift-ingress/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm" Nov 03 14:32:42.688364 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:42.688343 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1f2dcbe-0119-403d-b481-9fe2596bb889-istiod-ca-cert\") pod \"e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm\" (UID: \"c1f2dcbe-0119-403d-b481-9fe2596bb889\") " pod="openshift-ingress/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm" Nov 03 14:32:42.689197 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:42.689169 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1f2dcbe-0119-403d-b481-9fe2596bb889-istio-envoy\") pod \"e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm\" (UID: \"c1f2dcbe-0119-403d-b481-9fe2596bb889\") " pod="openshift-ingress/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm" Nov 03 14:32:42.690100 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:42.690076 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1f2dcbe-0119-403d-b481-9fe2596bb889-istio-podinfo\") pod \"e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm\" (UID: \"c1f2dcbe-0119-403d-b481-9fe2596bb889\") " pod="openshift-ingress/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm" Nov 03 14:32:42.740406 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:42.740366 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1f2dcbe-0119-403d-b481-9fe2596bb889-istio-token\") pod \"e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm\" (UID: \"c1f2dcbe-0119-403d-b481-9fe2596bb889\") " pod="openshift-ingress/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm" Nov 03 14:32:42.740580 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:42.740531 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg4cr\" (UniqueName: \"kubernetes.io/projected/c1f2dcbe-0119-403d-b481-9fe2596bb889-kube-api-access-lg4cr\") pod \"e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm\" (UID: \"c1f2dcbe-0119-403d-b481-9fe2596bb889\") " pod="openshift-ingress/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm" Nov 03 14:32:42.841528 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:42.841417 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm" Nov 03 14:32:43.268600 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:43.268565 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm"] Nov 03 14:32:43.276519 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:32:43.276459 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1f2dcbe_0119_403d_b481_9fe2596bb889.slice/crio-37ae9b159da258c09c364d1a55dc4e4ff99eee216b4916a1fdf1df2133222c5e WatchSource:0}: Error finding container 37ae9b159da258c09c364d1a55dc4e4ff99eee216b4916a1fdf1df2133222c5e: Status 404 returned error can't find the container with id 37ae9b159da258c09c364d1a55dc4e4ff99eee216b4916a1fdf1df2133222c5e Nov 03 14:32:43.278705 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:43.278673 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:32:43.278787 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:43.278740 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:32:43.278787 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:43.278769 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:32:43.772710 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:43.772667 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm" event={"ID":"c1f2dcbe-0119-403d-b481-9fe2596bb889","Type":"ContainerStarted","Data":"51822c4bab41c3e1a188bcefb3cf28b9b44cb2a67fd3ce2dfd4f23dcd919b48e"} Nov 03 14:32:43.773177 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:43.772730 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm" event={"ID":"c1f2dcbe-0119-403d-b481-9fe2596bb889","Type":"ContainerStarted","Data":"37ae9b159da258c09c364d1a55dc4e4ff99eee216b4916a1fdf1df2133222c5e"} Nov 03 14:32:43.842299 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:43.842253 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm" Nov 03 14:32:43.885884 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:43.885837 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm" podStartSLOduration=1.8858225119999998 podStartE2EDuration="1.885822512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:32:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:32:43.88548524 +0000 UTC m=+5245.896173483" watchObservedRunningTime="2026-11-03 14:32:43.885822512 +0000 UTC m=+5245.896510756" Nov 03 14:32:44.847371 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:44.847341 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm" Nov 03 14:32:45.783628 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:45.783595 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm" Nov 03 14:32:45.784725 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:45.784698 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm" Nov 03 14:32:52.045254 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:52.045221 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0d285d0-hjshf"] Nov 03 14:32:52.091531 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:52.091456 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0d285d0-hjshf"] Nov 03 14:32:52.113939 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:52.113903 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2e3af4d-bd1b-450c-8306-048e2a720c93" path="/var/lib/kubelet/pods/f2e3af4d-bd1b-450c-8306-048e2a720c93/volumes" Nov 03 14:32:52.417930 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:52.417894 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f530cd7c-859d5c97c5-vdpgf"] Nov 03 14:32:52.421874 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:52.421851 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f530cd7c-859d5c97c5-vdpgf" Nov 03 14:32:52.486760 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:52.486723 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83fcd07e-8b31-4cf4-ada6-55c96671cd8a-maas-api-tls\") pod \"maas-api-e2e-ait-f530cd7c-859d5c97c5-vdpgf\" (UID: \"83fcd07e-8b31-4cf4-ada6-55c96671cd8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f530cd7c-859d5c97c5-vdpgf" Nov 03 14:32:52.486934 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:52.486800 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83fcd07e-8b31-4cf4-ada6-55c96671cd8a-metrics-certs\") pod \"maas-api-e2e-ait-f530cd7c-859d5c97c5-vdpgf\" (UID: \"83fcd07e-8b31-4cf4-ada6-55c96671cd8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f530cd7c-859d5c97c5-vdpgf" Nov 03 14:32:52.486934 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:52.486870 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83fcd07e-8b31-4cf4-ada6-55c96671cd8a-openshift-service-ca\") pod \"maas-api-e2e-ait-f530cd7c-859d5c97c5-vdpgf\" (UID: \"83fcd07e-8b31-4cf4-ada6-55c96671cd8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f530cd7c-859d5c97c5-vdpgf" Nov 03 14:32:52.487028 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:52.486959 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgtvt\" (UniqueName: \"kubernetes.io/projected/83fcd07e-8b31-4cf4-ada6-55c96671cd8a-kube-api-access-jgtvt\") pod \"maas-api-e2e-ait-f530cd7c-859d5c97c5-vdpgf\" (UID: \"83fcd07e-8b31-4cf4-ada6-55c96671cd8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f530cd7c-859d5c97c5-vdpgf" Nov 03 14:32:52.509455 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:52.509420 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:32:52.518725 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:52.518694 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f530cd7c-serving-cert\"" Nov 03 14:32:52.518869 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:52.518725 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-dgx9m\"" Nov 03 14:32:52.518869 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:52.518741 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:32:52.518869 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:52.518797 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 14:32:52.521389 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:52.521367 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f530cd7c-859d5c97c5-vdpgf"] Nov 03 14:32:52.588023 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:52.587981 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgtvt\" (UniqueName: \"kubernetes.io/projected/83fcd07e-8b31-4cf4-ada6-55c96671cd8a-kube-api-access-jgtvt\") pod \"maas-api-e2e-ait-f530cd7c-859d5c97c5-vdpgf\" (UID: \"83fcd07e-8b31-4cf4-ada6-55c96671cd8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f530cd7c-859d5c97c5-vdpgf" Nov 03 14:32:52.588209 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:52.588052 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83fcd07e-8b31-4cf4-ada6-55c96671cd8a-maas-api-tls\") pod \"maas-api-e2e-ait-f530cd7c-859d5c97c5-vdpgf\" (UID: \"83fcd07e-8b31-4cf4-ada6-55c96671cd8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f530cd7c-859d5c97c5-vdpgf" Nov 03 14:32:52.588209 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:52.588094 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83fcd07e-8b31-4cf4-ada6-55c96671cd8a-metrics-certs\") pod \"maas-api-e2e-ait-f530cd7c-859d5c97c5-vdpgf\" (UID: \"83fcd07e-8b31-4cf4-ada6-55c96671cd8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f530cd7c-859d5c97c5-vdpgf" Nov 03 14:32:52.588209 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:52.588123 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83fcd07e-8b31-4cf4-ada6-55c96671cd8a-openshift-service-ca\") pod \"maas-api-e2e-ait-f530cd7c-859d5c97c5-vdpgf\" (UID: \"83fcd07e-8b31-4cf4-ada6-55c96671cd8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f530cd7c-859d5c97c5-vdpgf" Nov 03 14:32:52.588356 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:32:52.588215 5979 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-f530cd7c-serving-cert: secret "maas-api-e2e-ait-f530cd7c-serving-cert" not found Nov 03 14:32:52.588356 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:32:52.588293 5979 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83fcd07e-8b31-4cf4-ada6-55c96671cd8a-maas-api-tls podName:83fcd07e-8b31-4cf4-ada6-55c96671cd8a nodeName:}" failed. No retries permitted until 2026-11-03 14:32:53.088272459 +0000 UTC m=+5255.098960682 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/83fcd07e-8b31-4cf4-ada6-55c96671cd8a-maas-api-tls") pod "maas-api-e2e-ait-f530cd7c-859d5c97c5-vdpgf" (UID: "83fcd07e-8b31-4cf4-ada6-55c96671cd8a") : secret "maas-api-e2e-ait-f530cd7c-serving-cert" not found Nov 03 14:32:52.588926 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:52.588902 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83fcd07e-8b31-4cf4-ada6-55c96671cd8a-openshift-service-ca\") pod \"maas-api-e2e-ait-f530cd7c-859d5c97c5-vdpgf\" (UID: \"83fcd07e-8b31-4cf4-ada6-55c96671cd8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f530cd7c-859d5c97c5-vdpgf" Nov 03 14:32:52.590601 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:52.590577 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83fcd07e-8b31-4cf4-ada6-55c96671cd8a-metrics-certs\") pod \"maas-api-e2e-ait-f530cd7c-859d5c97c5-vdpgf\" (UID: \"83fcd07e-8b31-4cf4-ada6-55c96671cd8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f530cd7c-859d5c97c5-vdpgf" Nov 03 14:32:52.643377 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:52.643342 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgtvt\" (UniqueName: \"kubernetes.io/projected/83fcd07e-8b31-4cf4-ada6-55c96671cd8a-kube-api-access-jgtvt\") pod \"maas-api-e2e-ait-f530cd7c-859d5c97c5-vdpgf\" (UID: \"83fcd07e-8b31-4cf4-ada6-55c96671cd8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f530cd7c-859d5c97c5-vdpgf" Nov 03 14:32:53.093753 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:53.093725 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83fcd07e-8b31-4cf4-ada6-55c96671cd8a-maas-api-tls\") pod \"maas-api-e2e-ait-f530cd7c-859d5c97c5-vdpgf\" (UID: \"83fcd07e-8b31-4cf4-ada6-55c96671cd8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f530cd7c-859d5c97c5-vdpgf" Nov 03 14:32:53.096138 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:53.096115 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83fcd07e-8b31-4cf4-ada6-55c96671cd8a-maas-api-tls\") pod \"maas-api-e2e-ait-f530cd7c-859d5c97c5-vdpgf\" (UID: \"83fcd07e-8b31-4cf4-ada6-55c96671cd8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f530cd7c-859d5c97c5-vdpgf" Nov 03 14:32:53.335414 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:53.335375 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f530cd7c-859d5c97c5-vdpgf" Nov 03 14:32:53.706272 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:53.706236 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f530cd7c-859d5c97c5-vdpgf"] Nov 03 14:32:53.714130 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:32:53.714101 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83fcd07e_8b31_4cf4_ada6_55c96671cd8a.slice/crio-f81e82b6019fcd7432d0b4a9a1e1f0eedc059dc82b2d3ab96930aab7c9b7b104 WatchSource:0}: Error finding container f81e82b6019fcd7432d0b4a9a1e1f0eedc059dc82b2d3ab96930aab7c9b7b104: Status 404 returned error can't find the container with id f81e82b6019fcd7432d0b4a9a1e1f0eedc059dc82b2d3ab96930aab7c9b7b104 Nov 03 14:32:53.715961 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:53.715942 5979 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 14:32:53.827112 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:53.827079 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f530cd7c-859d5c97c5-vdpgf" event={"ID":"83fcd07e-8b31-4cf4-ada6-55c96671cd8a","Type":"ContainerStarted","Data":"f81e82b6019fcd7432d0b4a9a1e1f0eedc059dc82b2d3ab96930aab7c9b7b104"} Nov 03 14:32:54.833917 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:54.833881 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f530cd7c-859d5c97c5-vdpgf" event={"ID":"83fcd07e-8b31-4cf4-ada6-55c96671cd8a","Type":"ContainerStarted","Data":"410c7e9390edde2c568fca3632e0a84fdfe14b080d7e56c375c1dcf582fa84d7"} Nov 03 14:32:54.834383 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:54.833982 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f530cd7c-859d5c97c5-vdpgf" Nov 03 14:32:54.944567 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:32:54.944438 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f530cd7c-859d5c97c5-vdpgf" podStartSLOduration=2.012069179 podStartE2EDuration="2.944422461s" totalImagesPullingTime="932.353282ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:32:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:32:54.943285142 +0000 UTC m=+5256.953973383" watchObservedRunningTime="2026-11-03 14:32:54.944422461 +0000 UTC m=+5256.955110703" Nov 03 14:33:00.370071 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:00.370030 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:33:00.845260 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:00.845229 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f530cd7c-859d5c97c5-vdpgf" Nov 03 14:33:03.055507 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:03.054513 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dded4219-2tjcj"] Nov 03 14:33:03.091248 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:03.091210 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dded4219-2tjcj"] Nov 03 14:33:04.115085 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:04.115051 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba6aa3ab-9c2c-4bca-8748-b6b5f1abc34c" path="/var/lib/kubelet/pods/ba6aa3ab-9c2c-4bca-8748-b6b5f1abc34c/volumes" Nov 03 14:33:07.233118 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:33:07.233080 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:33:10.926814 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:10.926782 5979 generic.go:356] "Generic (PLEG): container finished" podID="c1f2dcbe-0119-403d-b481-9fe2596bb889" containerID="51822c4bab41c3e1a188bcefb3cf28b9b44cb2a67fd3ce2dfd4f23dcd919b48e" exitCode=0 Nov 03 14:33:10.927352 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:10.926869 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm" event={"ID":"c1f2dcbe-0119-403d-b481-9fe2596bb889","Type":"ContainerDied","Data":"51822c4bab41c3e1a188bcefb3cf28b9b44cb2a67fd3ce2dfd4f23dcd919b48e"} Nov 03 14:33:10.927352 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:10.927304 5979 scope.go:122] "RemoveContainer" containerID="51822c4bab41c3e1a188bcefb3cf28b9b44cb2a67fd3ce2dfd4f23dcd919b48e" Nov 03 14:33:10.928549 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:10.928519 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:33:10.928663 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:10.928624 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:33:10.928735 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:10.928672 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:33:11.934241 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:11.934202 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm" event={"ID":"c1f2dcbe-0119-403d-b481-9fe2596bb889","Type":"ContainerStarted","Data":"34ace080a02eefe7bdcd06d29198a6384988c121f920874ff2b85253182e9a11"} Nov 03 14:33:12.842520 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:12.842459 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm" Nov 03 14:33:12.868514 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:12.860638 5979 patch_prober.go:29] interesting pod/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.127:15021/healthz/ready\": dial tcp 10.134.1.127:15021: connect: connection refused" start-of-body= Nov 03 14:33:12.868514 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:12.860778 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm" podUID="c1f2dcbe-0119-403d-b481-9fe2596bb889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.127:15021/healthz/ready\": dial tcp 10.134.1.127:15021: connect: connection refused" Nov 03 14:33:13.842702 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:13.842669 5979 patch_prober.go:29] interesting pod/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.127:15021/healthz/ready\": dial tcp 10.134.1.127:15021: connect: connection refused" start-of-body= Nov 03 14:33:13.843214 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:13.842731 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm" podUID="c1f2dcbe-0119-403d-b481-9fe2596bb889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.127:15021/healthz/ready\": dial tcp 10.134.1.127:15021: connect: connection refused" Nov 03 14:33:14.841890 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:14.841849 5979 patch_prober.go:29] interesting pod/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.127:15021/healthz/ready\": dial tcp 10.134.1.127:15021: connect: connection refused" start-of-body= Nov 03 14:33:14.842078 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:14.841903 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm" podUID="c1f2dcbe-0119-403d-b481-9fe2596bb889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.127:15021/healthz/ready\": dial tcp 10.134.1.127:15021: connect: connection refused" Nov 03 14:33:15.784394 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:15.784357 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm" Nov 03 14:33:15.842162 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:15.842131 5979 patch_prober.go:29] interesting pod/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.127:15021/healthz/ready\": dial tcp 10.134.1.127:15021: connect: connection refused" start-of-body= Nov 03 14:33:15.842329 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:15.842185 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm" podUID="c1f2dcbe-0119-403d-b481-9fe2596bb889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.127:15021/healthz/ready\": dial tcp 10.134.1.127:15021: connect: connection refused" Nov 03 14:33:16.841761 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:16.841718 5979 patch_prober.go:29] interesting pod/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.127:15021/healthz/ready\": dial tcp 10.134.1.127:15021: connect: connection refused" start-of-body= Nov 03 14:33:16.842176 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:16.841777 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm" podUID="c1f2dcbe-0119-403d-b481-9fe2596bb889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.127:15021/healthz/ready\": dial tcp 10.134.1.127:15021: connect: connection refused" Nov 03 14:33:17.846603 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:17.846570 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm" Nov 03 14:33:17.847542 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:17.847524 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm" Nov 03 14:33:34.183503 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:34.183457 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f530cd7c-859d5c97c5-vdpgf"] Nov 03 14:33:34.184017 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:34.183756 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f530cd7c-859d5c97c5-vdpgf" podUID="83fcd07e-8b31-4cf4-ada6-55c96671cd8a" containerName="maas-api" containerID="cri-o://410c7e9390edde2c568fca3632e0a84fdfe14b080d7e56c375c1dcf582fa84d7" gracePeriod=30 Nov 03 14:33:34.439098 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:34.439019 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f530cd7c-859d5c97c5-vdpgf" Nov 03 14:33:34.616940 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:34.616909 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83fcd07e-8b31-4cf4-ada6-55c96671cd8a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83fcd07e-8b31-4cf4-ada6-55c96671cd8a-maas-api-tls\") pod \"83fcd07e-8b31-4cf4-ada6-55c96671cd8a\" (UID: \"83fcd07e-8b31-4cf4-ada6-55c96671cd8a\") " Nov 03 14:33:34.617130 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:34.616961 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83fcd07e-8b31-4cf4-ada6-55c96671cd8a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/83fcd07e-8b31-4cf4-ada6-55c96671cd8a-metrics-certs\") pod \"83fcd07e-8b31-4cf4-ada6-55c96671cd8a\" (UID: \"83fcd07e-8b31-4cf4-ada6-55c96671cd8a\") " Nov 03 14:33:34.617130 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:34.617009 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83fcd07e-8b31-4cf4-ada6-55c96671cd8a-kube-api-access-jgtvt\" (UniqueName: \"kubernetes.io/projected/83fcd07e-8b31-4cf4-ada6-55c96671cd8a-kube-api-access-jgtvt\") pod \"83fcd07e-8b31-4cf4-ada6-55c96671cd8a\" (UID: \"83fcd07e-8b31-4cf4-ada6-55c96671cd8a\") " Nov 03 14:33:34.617130 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:34.617037 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83fcd07e-8b31-4cf4-ada6-55c96671cd8a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83fcd07e-8b31-4cf4-ada6-55c96671cd8a-openshift-service-ca\") pod \"83fcd07e-8b31-4cf4-ada6-55c96671cd8a\" (UID: \"83fcd07e-8b31-4cf4-ada6-55c96671cd8a\") " Nov 03 14:33:34.617465 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:34.617441 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83fcd07e-8b31-4cf4-ada6-55c96671cd8a-openshift-service-ca" pod "83fcd07e-8b31-4cf4-ada6-55c96671cd8a" (UID: "83fcd07e-8b31-4cf4-ada6-55c96671cd8a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:33:34.619361 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:34.619335 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83fcd07e-8b31-4cf4-ada6-55c96671cd8a-maas-api-tls" pod "83fcd07e-8b31-4cf4-ada6-55c96671cd8a" (UID: "83fcd07e-8b31-4cf4-ada6-55c96671cd8a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:33:34.619513 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:34.619365 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83fcd07e-8b31-4cf4-ada6-55c96671cd8a-kube-api-access-jgtvt" pod "83fcd07e-8b31-4cf4-ada6-55c96671cd8a" (UID: "83fcd07e-8b31-4cf4-ada6-55c96671cd8a"). InnerVolumeSpecName "kube-api-access-jgtvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:33:34.619513 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:34.619442 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83fcd07e-8b31-4cf4-ada6-55c96671cd8a-metrics-certs" pod "83fcd07e-8b31-4cf4-ada6-55c96671cd8a" (UID: "83fcd07e-8b31-4cf4-ada6-55c96671cd8a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:33:34.718168 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:34.718084 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83fcd07e-8b31-4cf4-ada6-55c96671cd8a-maas-api-tls\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:33:34.718168 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:34.718115 5979 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83fcd07e-8b31-4cf4-ada6-55c96671cd8a-metrics-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:33:34.718168 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:34.718129 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgtvt\" (UniqueName: \"kubernetes.io/projected/83fcd07e-8b31-4cf4-ada6-55c96671cd8a-kube-api-access-jgtvt\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:33:34.718168 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:34.718146 5979 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83fcd07e-8b31-4cf4-ada6-55c96671cd8a-openshift-service-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:33:35.085654 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:35.085571 5979 generic.go:356] "Generic (PLEG): container finished" podID="83fcd07e-8b31-4cf4-ada6-55c96671cd8a" containerID="410c7e9390edde2c568fca3632e0a84fdfe14b080d7e56c375c1dcf582fa84d7" exitCode=0 Nov 03 14:33:35.085654 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:35.085609 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f530cd7c-859d5c97c5-vdpgf" event={"ID":"83fcd07e-8b31-4cf4-ada6-55c96671cd8a","Type":"ContainerDied","Data":"410c7e9390edde2c568fca3632e0a84fdfe14b080d7e56c375c1dcf582fa84d7"} Nov 03 14:33:35.085654 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:35.085621 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f530cd7c-859d5c97c5-vdpgf" Nov 03 14:33:35.085654 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:35.085653 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f530cd7c-859d5c97c5-vdpgf" event={"ID":"83fcd07e-8b31-4cf4-ada6-55c96671cd8a","Type":"ContainerDied","Data":"f81e82b6019fcd7432d0b4a9a1e1f0eedc059dc82b2d3ab96930aab7c9b7b104"} Nov 03 14:33:35.085933 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:35.085675 5979 scope.go:122] "RemoveContainer" containerID="410c7e9390edde2c568fca3632e0a84fdfe14b080d7e56c375c1dcf582fa84d7" Nov 03 14:33:35.095552 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:35.095530 5979 scope.go:122] "RemoveContainer" containerID="410c7e9390edde2c568fca3632e0a84fdfe14b080d7e56c375c1dcf582fa84d7" Nov 03 14:33:35.095817 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:33:35.095800 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"410c7e9390edde2c568fca3632e0a84fdfe14b080d7e56c375c1dcf582fa84d7\": container with ID starting with 410c7e9390edde2c568fca3632e0a84fdfe14b080d7e56c375c1dcf582fa84d7 not found: ID does not exist" containerID="410c7e9390edde2c568fca3632e0a84fdfe14b080d7e56c375c1dcf582fa84d7" Nov 03 14:33:35.095871 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:35.095828 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"410c7e9390edde2c568fca3632e0a84fdfe14b080d7e56c375c1dcf582fa84d7"} err="failed to get container status \"410c7e9390edde2c568fca3632e0a84fdfe14b080d7e56c375c1dcf582fa84d7\": rpc error: code = NotFound desc = could not find container \"410c7e9390edde2c568fca3632e0a84fdfe14b080d7e56c375c1dcf582fa84d7\": container with ID starting with 410c7e9390edde2c568fca3632e0a84fdfe14b080d7e56c375c1dcf582fa84d7 not found: ID does not exist" Nov 03 14:33:35.239024 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:35.238988 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f530cd7c-859d5c97c5-vdpgf"] Nov 03 14:33:35.285555 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:35.285522 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f530cd7c-859d5c97c5-vdpgf"] Nov 03 14:33:36.114018 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:36.113985 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83fcd07e-8b31-4cf4-ada6-55c96671cd8a" path="/var/lib/kubelet/pods/83fcd07e-8b31-4cf4-ada6-55c96671cd8a/volumes" Nov 03 14:33:37.233378 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:33:37.233337 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:33:38.467807 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:38.467770 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:33:43.588842 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:43.587972 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm"] Nov 03 14:33:43.588842 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:43.588249 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm" podUID="c1f2dcbe-0119-403d-b481-9fe2596bb889" containerName="istio-proxy" containerID="cri-o://34ace080a02eefe7bdcd06d29198a6384988c121f920874ff2b85253182e9a11" gracePeriod=30 Nov 03 14:33:47.849714 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:47.849663 5979 patch_prober.go:29] interesting pod/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Nov 03 14:33:47.850220 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:47.849743 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm" podUID="c1f2dcbe-0119-403d-b481-9fe2596bb889" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 14:33:48.105623 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:48.105534 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-588d644cc-njxzq"] Nov 03 14:33:48.105793 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:48.105744 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-588d644cc-njxzq" podUID="445b3b20-c0f6-41d9-ab82-d1598427036b" containerName="manager" containerID="cri-o://26764003d8102fae4b01923267ff121e565bcc4811b8d3ab4a705779f260508e" gracePeriod=30 Nov 03 14:33:48.351136 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:48.351109 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-588d644cc-njxzq" Nov 03 14:33:48.463604 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:48.463569 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/445b3b20-c0f6-41d9-ab82-d1598427036b-webhook-cert\" (UniqueName: \"kubernetes.io/secret/445b3b20-c0f6-41d9-ab82-d1598427036b-webhook-cert\") pod \"445b3b20-c0f6-41d9-ab82-d1598427036b\" (UID: \"445b3b20-c0f6-41d9-ab82-d1598427036b\") " Nov 03 14:33:48.463815 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:48.463634 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/445b3b20-c0f6-41d9-ab82-d1598427036b-kube-api-access-phq2t\" (UniqueName: \"kubernetes.io/projected/445b3b20-c0f6-41d9-ab82-d1598427036b-kube-api-access-phq2t\") pod \"445b3b20-c0f6-41d9-ab82-d1598427036b\" (UID: \"445b3b20-c0f6-41d9-ab82-d1598427036b\") " Nov 03 14:33:48.463815 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:48.463671 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/445b3b20-c0f6-41d9-ab82-d1598427036b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/445b3b20-c0f6-41d9-ab82-d1598427036b-metrics-certs\") pod \"445b3b20-c0f6-41d9-ab82-d1598427036b\" (UID: \"445b3b20-c0f6-41d9-ab82-d1598427036b\") " Nov 03 14:33:48.465786 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:48.465755 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/445b3b20-c0f6-41d9-ab82-d1598427036b-kube-api-access-phq2t" pod "445b3b20-c0f6-41d9-ab82-d1598427036b" (UID: "445b3b20-c0f6-41d9-ab82-d1598427036b"). InnerVolumeSpecName "kube-api-access-phq2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:33:48.465786 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:48.465772 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/445b3b20-c0f6-41d9-ab82-d1598427036b-webhook-cert" pod "445b3b20-c0f6-41d9-ab82-d1598427036b" (UID: "445b3b20-c0f6-41d9-ab82-d1598427036b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:33:48.466167 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:48.466142 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/445b3b20-c0f6-41d9-ab82-d1598427036b-metrics-certs" pod "445b3b20-c0f6-41d9-ab82-d1598427036b" (UID: "445b3b20-c0f6-41d9-ab82-d1598427036b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:33:48.565178 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:48.565136 5979 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/445b3b20-c0f6-41d9-ab82-d1598427036b-metrics-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:33:48.565178 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:48.565174 5979 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/445b3b20-c0f6-41d9-ab82-d1598427036b-webhook-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:33:48.565394 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:48.565187 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phq2t\" (UniqueName: \"kubernetes.io/projected/445b3b20-c0f6-41d9-ab82-d1598427036b-kube-api-access-phq2t\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:33:48.823802 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:48.823777 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm" Nov 03 14:33:48.868063 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:48.868035 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1f2dcbe-0119-403d-b481-9fe2596bb889-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1f2dcbe-0119-403d-b481-9fe2596bb889-istiod-ca-cert\") pod \"c1f2dcbe-0119-403d-b481-9fe2596bb889\" (UID: \"c1f2dcbe-0119-403d-b481-9fe2596bb889\") " Nov 03 14:33:48.868398 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:48.868123 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c1f2dcbe-0119-403d-b481-9fe2596bb889-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1f2dcbe-0119-403d-b481-9fe2596bb889-istio-podinfo\") pod \"c1f2dcbe-0119-403d-b481-9fe2596bb889\" (UID: \"c1f2dcbe-0119-403d-b481-9fe2596bb889\") " Nov 03 14:33:48.868398 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:48.868154 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1f2dcbe-0119-403d-b481-9fe2596bb889-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1f2dcbe-0119-403d-b481-9fe2596bb889-workload-certs\") pod \"c1f2dcbe-0119-403d-b481-9fe2596bb889\" (UID: \"c1f2dcbe-0119-403d-b481-9fe2596bb889\") " Nov 03 14:33:48.868398 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:48.868203 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1f2dcbe-0119-403d-b481-9fe2596bb889-istio-token\" (UniqueName: \"kubernetes.io/projected/c1f2dcbe-0119-403d-b481-9fe2596bb889-istio-token\") pod \"c1f2dcbe-0119-403d-b481-9fe2596bb889\" (UID: \"c1f2dcbe-0119-403d-b481-9fe2596bb889\") " Nov 03 14:33:48.868398 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:48.868245 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1f2dcbe-0119-403d-b481-9fe2596bb889-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1f2dcbe-0119-403d-b481-9fe2596bb889-istio-data\") pod \"c1f2dcbe-0119-403d-b481-9fe2596bb889\" (UID: \"c1f2dcbe-0119-403d-b481-9fe2596bb889\") " Nov 03 14:33:48.868398 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:48.868302 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1f2dcbe-0119-403d-b481-9fe2596bb889-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1f2dcbe-0119-403d-b481-9fe2596bb889-istio-envoy\") pod \"c1f2dcbe-0119-403d-b481-9fe2596bb889\" (UID: \"c1f2dcbe-0119-403d-b481-9fe2596bb889\") " Nov 03 14:33:48.868398 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:48.868342 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1f2dcbe-0119-403d-b481-9fe2596bb889-kube-api-access-lg4cr\" (UniqueName: \"kubernetes.io/projected/c1f2dcbe-0119-403d-b481-9fe2596bb889-kube-api-access-lg4cr\") pod \"c1f2dcbe-0119-403d-b481-9fe2596bb889\" (UID: \"c1f2dcbe-0119-403d-b481-9fe2596bb889\") " Nov 03 14:33:48.868398 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:48.868373 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1f2dcbe-0119-403d-b481-9fe2596bb889-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1f2dcbe-0119-403d-b481-9fe2596bb889-workload-socket\") pod \"c1f2dcbe-0119-403d-b481-9fe2596bb889\" (UID: \"c1f2dcbe-0119-403d-b481-9fe2596bb889\") " Nov 03 14:33:48.868792 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:48.868402 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1f2dcbe-0119-403d-b481-9fe2596bb889-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1f2dcbe-0119-403d-b481-9fe2596bb889-credential-socket\") pod \"c1f2dcbe-0119-403d-b481-9fe2596bb889\" (UID: \"c1f2dcbe-0119-403d-b481-9fe2596bb889\") " Nov 03 14:33:48.868792 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:48.868426 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1f2dcbe-0119-403d-b481-9fe2596bb889-istiod-ca-cert" pod "c1f2dcbe-0119-403d-b481-9fe2596bb889" (UID: "c1f2dcbe-0119-403d-b481-9fe2596bb889"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:33:48.868792 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:48.868447 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1f2dcbe-0119-403d-b481-9fe2596bb889-workload-certs" pod "c1f2dcbe-0119-403d-b481-9fe2596bb889" (UID: "c1f2dcbe-0119-403d-b481-9fe2596bb889"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:33:48.868792 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:48.868626 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1f2dcbe-0119-403d-b481-9fe2596bb889-istio-data" pod "c1f2dcbe-0119-403d-b481-9fe2596bb889" (UID: "c1f2dcbe-0119-403d-b481-9fe2596bb889"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:33:48.869029 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:48.868818 5979 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1f2dcbe-0119-403d-b481-9fe2596bb889-istio-data\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:33:48.869029 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:48.868822 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1f2dcbe-0119-403d-b481-9fe2596bb889-workload-socket" pod "c1f2dcbe-0119-403d-b481-9fe2596bb889" (UID: "c1f2dcbe-0119-403d-b481-9fe2596bb889"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:33:48.869029 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:48.868838 5979 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1f2dcbe-0119-403d-b481-9fe2596bb889-istiod-ca-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:33:48.869029 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:48.868853 5979 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1f2dcbe-0119-403d-b481-9fe2596bb889-workload-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:33:48.869029 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:48.869020 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1f2dcbe-0119-403d-b481-9fe2596bb889-credential-socket" pod "c1f2dcbe-0119-403d-b481-9fe2596bb889" (UID: "c1f2dcbe-0119-403d-b481-9fe2596bb889"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:33:48.870343 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:48.870324 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1f2dcbe-0119-403d-b481-9fe2596bb889-istio-podinfo" pod "c1f2dcbe-0119-403d-b481-9fe2596bb889" (UID: "c1f2dcbe-0119-403d-b481-9fe2596bb889"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 14:33:48.870412 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:48.870365 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1f2dcbe-0119-403d-b481-9fe2596bb889-istio-token" pod "c1f2dcbe-0119-403d-b481-9fe2596bb889" (UID: "c1f2dcbe-0119-403d-b481-9fe2596bb889"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:33:48.870856 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:48.870838 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1f2dcbe-0119-403d-b481-9fe2596bb889-istio-envoy" pod "c1f2dcbe-0119-403d-b481-9fe2596bb889" (UID: "c1f2dcbe-0119-403d-b481-9fe2596bb889"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 14:33:48.870906 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:48.870857 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1f2dcbe-0119-403d-b481-9fe2596bb889-kube-api-access-lg4cr" pod "c1f2dcbe-0119-403d-b481-9fe2596bb889" (UID: "c1f2dcbe-0119-403d-b481-9fe2596bb889"). InnerVolumeSpecName "kube-api-access-lg4cr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:33:48.969929 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:48.969893 5979 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1f2dcbe-0119-403d-b481-9fe2596bb889-istio-envoy\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:33:48.969929 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:48.969923 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg4cr\" (UniqueName: \"kubernetes.io/projected/c1f2dcbe-0119-403d-b481-9fe2596bb889-kube-api-access-lg4cr\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:33:48.969929 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:48.969935 5979 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1f2dcbe-0119-403d-b481-9fe2596bb889-workload-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:33:48.970173 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:48.969948 5979 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1f2dcbe-0119-403d-b481-9fe2596bb889-credential-socket\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:33:48.970173 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:48.969957 5979 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1f2dcbe-0119-403d-b481-9fe2596bb889-istio-podinfo\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:33:48.970173 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:48.969966 5979 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1f2dcbe-0119-403d-b481-9fe2596bb889-istio-token\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:33:49.162745 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:49.162711 5979 generic.go:356] "Generic (PLEG): container finished" podID="c1f2dcbe-0119-403d-b481-9fe2596bb889" containerID="34ace080a02eefe7bdcd06d29198a6384988c121f920874ff2b85253182e9a11" exitCode=0 Nov 03 14:33:49.162951 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:49.162764 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm" Nov 03 14:33:49.162951 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:49.162797 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm" event={"ID":"c1f2dcbe-0119-403d-b481-9fe2596bb889","Type":"ContainerDied","Data":"34ace080a02eefe7bdcd06d29198a6384988c121f920874ff2b85253182e9a11"} Nov 03 14:33:49.162951 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:49.162847 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm" event={"ID":"c1f2dcbe-0119-403d-b481-9fe2596bb889","Type":"ContainerDied","Data":"37ae9b159da258c09c364d1a55dc4e4ff99eee216b4916a1fdf1df2133222c5e"} Nov 03 14:33:49.162951 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:49.162869 5979 scope.go:122] "RemoveContainer" containerID="34ace080a02eefe7bdcd06d29198a6384988c121f920874ff2b85253182e9a11" Nov 03 14:33:49.164432 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:49.164413 5979 generic.go:356] "Generic (PLEG): container finished" podID="445b3b20-c0f6-41d9-ab82-d1598427036b" containerID="26764003d8102fae4b01923267ff121e565bcc4811b8d3ab4a705779f260508e" exitCode=0 Nov 03 14:33:49.164567 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:49.164464 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-588d644cc-njxzq" event={"ID":"445b3b20-c0f6-41d9-ab82-d1598427036b","Type":"ContainerDied","Data":"26764003d8102fae4b01923267ff121e565bcc4811b8d3ab4a705779f260508e"} Nov 03 14:33:49.164567 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:49.164518 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-588d644cc-njxzq" event={"ID":"445b3b20-c0f6-41d9-ab82-d1598427036b","Type":"ContainerDied","Data":"44730c2bd8201ff2145cd2f6a2ea5b0eb36034353c6d74a015cddf44afeaa6fa"} Nov 03 14:33:49.164701 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:49.164464 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-588d644cc-njxzq" Nov 03 14:33:49.174100 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:49.173933 5979 scope.go:122] "RemoveContainer" containerID="51822c4bab41c3e1a188bcefb3cf28b9b44cb2a67fd3ce2dfd4f23dcd919b48e" Nov 03 14:33:49.182405 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:49.182388 5979 scope.go:122] "RemoveContainer" containerID="34ace080a02eefe7bdcd06d29198a6384988c121f920874ff2b85253182e9a11" Nov 03 14:33:49.182702 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:33:49.182683 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34ace080a02eefe7bdcd06d29198a6384988c121f920874ff2b85253182e9a11\": container with ID starting with 34ace080a02eefe7bdcd06d29198a6384988c121f920874ff2b85253182e9a11 not found: ID does not exist" containerID="34ace080a02eefe7bdcd06d29198a6384988c121f920874ff2b85253182e9a11" Nov 03 14:33:49.182763 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:49.182710 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34ace080a02eefe7bdcd06d29198a6384988c121f920874ff2b85253182e9a11"} err="failed to get container status \"34ace080a02eefe7bdcd06d29198a6384988c121f920874ff2b85253182e9a11\": rpc error: code = NotFound desc = could not find container \"34ace080a02eefe7bdcd06d29198a6384988c121f920874ff2b85253182e9a11\": container with ID starting with 34ace080a02eefe7bdcd06d29198a6384988c121f920874ff2b85253182e9a11 not found: ID does not exist" Nov 03 14:33:49.182763 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:49.182727 5979 scope.go:122] "RemoveContainer" containerID="51822c4bab41c3e1a188bcefb3cf28b9b44cb2a67fd3ce2dfd4f23dcd919b48e" Nov 03 14:33:49.182928 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:33:49.182912 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51822c4bab41c3e1a188bcefb3cf28b9b44cb2a67fd3ce2dfd4f23dcd919b48e\": container with ID starting with 51822c4bab41c3e1a188bcefb3cf28b9b44cb2a67fd3ce2dfd4f23dcd919b48e not found: ID does not exist" containerID="51822c4bab41c3e1a188bcefb3cf28b9b44cb2a67fd3ce2dfd4f23dcd919b48e" Nov 03 14:33:49.182973 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:49.182935 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51822c4bab41c3e1a188bcefb3cf28b9b44cb2a67fd3ce2dfd4f23dcd919b48e"} err="failed to get container status \"51822c4bab41c3e1a188bcefb3cf28b9b44cb2a67fd3ce2dfd4f23dcd919b48e\": rpc error: code = NotFound desc = could not find container \"51822c4bab41c3e1a188bcefb3cf28b9b44cb2a67fd3ce2dfd4f23dcd919b48e\": container with ID starting with 51822c4bab41c3e1a188bcefb3cf28b9b44cb2a67fd3ce2dfd4f23dcd919b48e not found: ID does not exist" Nov 03 14:33:49.182973 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:49.182951 5979 scope.go:122] "RemoveContainer" containerID="26764003d8102fae4b01923267ff121e565bcc4811b8d3ab4a705779f260508e" Nov 03 14:33:49.191864 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:49.191848 5979 scope.go:122] "RemoveContainer" containerID="d21e2d243f6298178d64e8d5ed1ddd9f2623beede4ff10ab99d739f1dd673579" Nov 03 14:33:49.199395 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:49.199377 5979 scope.go:122] "RemoveContainer" containerID="26764003d8102fae4b01923267ff121e565bcc4811b8d3ab4a705779f260508e" Nov 03 14:33:49.199685 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:33:49.199665 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26764003d8102fae4b01923267ff121e565bcc4811b8d3ab4a705779f260508e\": container with ID starting with 26764003d8102fae4b01923267ff121e565bcc4811b8d3ab4a705779f260508e not found: ID does not exist" containerID="26764003d8102fae4b01923267ff121e565bcc4811b8d3ab4a705779f260508e" Nov 03 14:33:49.199753 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:49.199691 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26764003d8102fae4b01923267ff121e565bcc4811b8d3ab4a705779f260508e"} err="failed to get container status \"26764003d8102fae4b01923267ff121e565bcc4811b8d3ab4a705779f260508e\": rpc error: code = NotFound desc = could not find container \"26764003d8102fae4b01923267ff121e565bcc4811b8d3ab4a705779f260508e\": container with ID starting with 26764003d8102fae4b01923267ff121e565bcc4811b8d3ab4a705779f260508e not found: ID does not exist" Nov 03 14:33:49.199753 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:49.199708 5979 scope.go:122] "RemoveContainer" containerID="d21e2d243f6298178d64e8d5ed1ddd9f2623beede4ff10ab99d739f1dd673579" Nov 03 14:33:49.199931 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:33:49.199913 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d21e2d243f6298178d64e8d5ed1ddd9f2623beede4ff10ab99d739f1dd673579\": container with ID starting with d21e2d243f6298178d64e8d5ed1ddd9f2623beede4ff10ab99d739f1dd673579 not found: ID does not exist" containerID="d21e2d243f6298178d64e8d5ed1ddd9f2623beede4ff10ab99d739f1dd673579" Nov 03 14:33:49.199984 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:49.199936 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d21e2d243f6298178d64e8d5ed1ddd9f2623beede4ff10ab99d739f1dd673579"} err="failed to get container status \"d21e2d243f6298178d64e8d5ed1ddd9f2623beede4ff10ab99d739f1dd673579\": rpc error: code = NotFound desc = could not find container \"d21e2d243f6298178d64e8d5ed1ddd9f2623beede4ff10ab99d739f1dd673579\": container with ID starting with d21e2d243f6298178d64e8d5ed1ddd9f2623beede4ff10ab99d739f1dd673579 not found: ID does not exist" Nov 03 14:33:49.346074 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:49.346037 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-588d644cc-njxzq"] Nov 03 14:33:49.361323 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:49.361289 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-588d644cc-njxzq"] Nov 03 14:33:49.479677 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:49.479595 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm"] Nov 03 14:33:49.490872 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:49.490844 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-f530cd7c-openshift-default-59df768d87-tvnsm"] Nov 03 14:33:50.113767 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:50.113733 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="445b3b20-c0f6-41d9-ab82-d1598427036b" path="/var/lib/kubelet/pods/445b3b20-c0f6-41d9-ab82-d1598427036b/volumes" Nov 03 14:33:50.114203 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:50.114189 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1f2dcbe-0119-403d-b481-9fe2596bb889" path="/var/lib/kubelet/pods/c1f2dcbe-0119-403d-b481-9fe2596bb889/volumes" Nov 03 14:33:50.305396 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:50.305361 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-588d644cc-lt76c"] Nov 03 14:33:50.305794 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:50.305780 5979 cpu_manager.go:414] "containerMap: removing container" podUID="c1f2dcbe-0119-403d-b481-9fe2596bb889" containerName="istio-proxy" Nov 03 14:33:50.305839 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:50.305795 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1f2dcbe-0119-403d-b481-9fe2596bb889" containerName="istio-proxy" Nov 03 14:33:50.305839 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:50.305809 5979 cpu_manager.go:414] "containerMap: removing container" podUID="445b3b20-c0f6-41d9-ab82-d1598427036b" containerName="manager" Nov 03 14:33:50.305839 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:50.305814 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="445b3b20-c0f6-41d9-ab82-d1598427036b" containerName="manager" Nov 03 14:33:50.305839 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:50.305823 5979 cpu_manager.go:414] "containerMap: removing container" podUID="83fcd07e-8b31-4cf4-ada6-55c96671cd8a" containerName="maas-api" Nov 03 14:33:50.305839 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:50.305828 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83fcd07e-8b31-4cf4-ada6-55c96671cd8a" containerName="maas-api" Nov 03 14:33:50.306016 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:50.305844 5979 cpu_manager.go:414] "containerMap: removing container" podUID="c1f2dcbe-0119-403d-b481-9fe2596bb889" containerName="istio-proxy" Nov 03 14:33:50.306016 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:50.305849 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1f2dcbe-0119-403d-b481-9fe2596bb889" containerName="istio-proxy" Nov 03 14:33:50.306016 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:50.305920 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1f2dcbe-0119-403d-b481-9fe2596bb889" containerName="istio-proxy" Nov 03 14:33:50.306016 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:50.305931 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="445b3b20-c0f6-41d9-ab82-d1598427036b" containerName="manager" Nov 03 14:33:50.306016 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:50.305940 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="445b3b20-c0f6-41d9-ab82-d1598427036b" containerName="manager" Nov 03 14:33:50.306016 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:50.305945 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="83fcd07e-8b31-4cf4-ada6-55c96671cd8a" containerName="maas-api" Nov 03 14:33:50.309336 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:50.309318 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-588d644cc-lt76c" Nov 03 14:33:50.358611 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:50.358576 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 14:33:50.385590 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:50.385548 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74ff5eff-a73d-4ba0-913f-ba087b52cd40-metrics-certs\") pod \"maas-controller-588d644cc-lt76c\" (UID: \"74ff5eff-a73d-4ba0-913f-ba087b52cd40\") " pod="opendatahub/maas-controller-588d644cc-lt76c" Nov 03 14:33:50.385786 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:50.385643 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tl64\" (UniqueName: \"kubernetes.io/projected/74ff5eff-a73d-4ba0-913f-ba087b52cd40-kube-api-access-5tl64\") pod \"maas-controller-588d644cc-lt76c\" (UID: \"74ff5eff-a73d-4ba0-913f-ba087b52cd40\") " pod="opendatahub/maas-controller-588d644cc-lt76c" Nov 03 14:33:50.385786 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:50.385721 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/74ff5eff-a73d-4ba0-913f-ba087b52cd40-webhook-cert\") pod \"maas-controller-588d644cc-lt76c\" (UID: \"74ff5eff-a73d-4ba0-913f-ba087b52cd40\") " pod="opendatahub/maas-controller-588d644cc-lt76c" Nov 03 14:33:50.405196 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:50.405172 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 14:33:50.405359 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:50.405233 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-rbwxk\"" Nov 03 14:33:50.405359 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:50.405289 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 14:33:50.415379 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:50.415351 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-588d644cc-lt76c"] Nov 03 14:33:50.486746 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:50.486697 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/74ff5eff-a73d-4ba0-913f-ba087b52cd40-webhook-cert\") pod \"maas-controller-588d644cc-lt76c\" (UID: \"74ff5eff-a73d-4ba0-913f-ba087b52cd40\") " pod="opendatahub/maas-controller-588d644cc-lt76c" Nov 03 14:33:50.486952 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:50.486867 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74ff5eff-a73d-4ba0-913f-ba087b52cd40-metrics-certs\") pod \"maas-controller-588d644cc-lt76c\" (UID: \"74ff5eff-a73d-4ba0-913f-ba087b52cd40\") " pod="opendatahub/maas-controller-588d644cc-lt76c" Nov 03 14:33:50.486952 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:50.486903 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tl64\" (UniqueName: \"kubernetes.io/projected/74ff5eff-a73d-4ba0-913f-ba087b52cd40-kube-api-access-5tl64\") pod \"maas-controller-588d644cc-lt76c\" (UID: \"74ff5eff-a73d-4ba0-913f-ba087b52cd40\") " pod="opendatahub/maas-controller-588d644cc-lt76c" Nov 03 14:33:50.489439 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:50.489411 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74ff5eff-a73d-4ba0-913f-ba087b52cd40-metrics-certs\") pod \"maas-controller-588d644cc-lt76c\" (UID: \"74ff5eff-a73d-4ba0-913f-ba087b52cd40\") " pod="opendatahub/maas-controller-588d644cc-lt76c" Nov 03 14:33:50.489831 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:50.489807 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/74ff5eff-a73d-4ba0-913f-ba087b52cd40-webhook-cert\") pod \"maas-controller-588d644cc-lt76c\" (UID: \"74ff5eff-a73d-4ba0-913f-ba087b52cd40\") " pod="opendatahub/maas-controller-588d644cc-lt76c" Nov 03 14:33:50.545400 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:50.545357 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tl64\" (UniqueName: \"kubernetes.io/projected/74ff5eff-a73d-4ba0-913f-ba087b52cd40-kube-api-access-5tl64\") pod \"maas-controller-588d644cc-lt76c\" (UID: \"74ff5eff-a73d-4ba0-913f-ba087b52cd40\") " pod="opendatahub/maas-controller-588d644cc-lt76c" Nov 03 14:33:50.623321 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:50.623271 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-588d644cc-lt76c" Nov 03 14:33:51.014398 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:51.014310 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-588d644cc-lt76c"] Nov 03 14:33:51.021305 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:33:51.021278 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74ff5eff_a73d_4ba0_913f_ba087b52cd40.slice/crio-a6caf118544b33ca30a4c182943c9ecb87e95225e6874eef416902ff3c1dee43 WatchSource:0}: Error finding container a6caf118544b33ca30a4c182943c9ecb87e95225e6874eef416902ff3c1dee43: Status 404 returned error can't find the container with id a6caf118544b33ca30a4c182943c9ecb87e95225e6874eef416902ff3c1dee43 Nov 03 14:33:51.182299 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:51.182250 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-588d644cc-lt76c" event={"ID":"74ff5eff-a73d-4ba0-913f-ba087b52cd40","Type":"ContainerStarted","Data":"a6caf118544b33ca30a4c182943c9ecb87e95225e6874eef416902ff3c1dee43"} Nov 03 14:33:51.338698 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:51.338599 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-dkw27 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:33:51.338698 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:51.338650 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dkw27" podUID="f313510b-02d8-4c68-accd-920343219917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:33:52.189362 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:52.189328 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-588d644cc-lt76c" event={"ID":"74ff5eff-a73d-4ba0-913f-ba087b52cd40","Type":"ContainerStarted","Data":"f1148e9351f574715382ace4f99adc0a2486f51c981028a9967492112a7d1285"} Nov 03 14:33:52.189768 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:52.189456 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-588d644cc-lt76c" Nov 03 14:33:52.311229 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:33:52.311127 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-588d644cc-lt76c" podStartSLOduration=1.372032019 podStartE2EDuration="2.311113557s" totalImagesPullingTime="939.081538ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:33:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:33:52.310653919 +0000 UTC m=+5314.321342162" watchObservedRunningTime="2026-11-03 14:33:52.311113557 +0000 UTC m=+5314.321801798" Nov 03 14:34:03.198872 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:03.198794 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-588d644cc-lt76c" Nov 03 14:34:07.233637 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:34:07.233598 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:34:10.834643 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:10.834606 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mxdbm"] Nov 03 14:34:10.835155 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:10.835004 5979 cpu_manager.go:414] "containerMap: removing container" podUID="445b3b20-c0f6-41d9-ab82-d1598427036b" containerName="manager" Nov 03 14:34:10.835155 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:10.835015 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="445b3b20-c0f6-41d9-ab82-d1598427036b" containerName="manager" Nov 03 14:34:10.835155 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:10.835093 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1f2dcbe-0119-403d-b481-9fe2596bb889" containerName="istio-proxy" Nov 03 14:34:10.838497 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:10.838456 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mxdbm" Nov 03 14:34:10.887845 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:10.887809 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:34:10.888362 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:10.888339 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbf4bb3f-88b7-449e-8d36-cd892e49bd86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-mxdbm\" (UID: \"fbf4bb3f-88b7-449e-8d36-cd892e49bd86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mxdbm" Nov 03 14:34:10.888427 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:10.888395 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hspq\" (UniqueName: \"kubernetes.io/projected/fbf4bb3f-88b7-449e-8d36-cd892e49bd86-kube-api-access-2hspq\") pod \"maas-api-revoke-sub-85237e45-mxdbm\" (UID: \"fbf4bb3f-88b7-449e-8d36-cd892e49bd86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mxdbm" Nov 03 14:34:10.888515 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:10.888491 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbf4bb3f-88b7-449e-8d36-cd892e49bd86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-mxdbm\" (UID: \"fbf4bb3f-88b7-449e-8d36-cd892e49bd86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mxdbm" Nov 03 14:34:10.934147 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:10.934112 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:34:10.934147 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:10.934142 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:34:10.934382 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:10.934280 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wgk87\"" Nov 03 14:34:10.936145 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:10.936123 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mxdbm"] Nov 03 14:34:10.989231 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:10.989193 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hspq\" (UniqueName: \"kubernetes.io/projected/fbf4bb3f-88b7-449e-8d36-cd892e49bd86-kube-api-access-2hspq\") pod \"maas-api-revoke-sub-85237e45-mxdbm\" (UID: \"fbf4bb3f-88b7-449e-8d36-cd892e49bd86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mxdbm" Nov 03 14:34:10.989464 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:10.989280 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbf4bb3f-88b7-449e-8d36-cd892e49bd86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-mxdbm\" (UID: \"fbf4bb3f-88b7-449e-8d36-cd892e49bd86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mxdbm" Nov 03 14:34:10.989464 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:10.989370 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbf4bb3f-88b7-449e-8d36-cd892e49bd86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-mxdbm\" (UID: \"fbf4bb3f-88b7-449e-8d36-cd892e49bd86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mxdbm" Nov 03 14:34:10.989974 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:10.989952 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbf4bb3f-88b7-449e-8d36-cd892e49bd86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-mxdbm\" (UID: \"fbf4bb3f-88b7-449e-8d36-cd892e49bd86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mxdbm" Nov 03 14:34:10.991711 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:10.991689 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbf4bb3f-88b7-449e-8d36-cd892e49bd86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-mxdbm\" (UID: \"fbf4bb3f-88b7-449e-8d36-cd892e49bd86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mxdbm" Nov 03 14:34:11.043623 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:11.043590 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hspq\" (UniqueName: \"kubernetes.io/projected/fbf4bb3f-88b7-449e-8d36-cd892e49bd86-kube-api-access-2hspq\") pod \"maas-api-revoke-sub-85237e45-mxdbm\" (UID: \"fbf4bb3f-88b7-449e-8d36-cd892e49bd86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mxdbm" Nov 03 14:34:11.150981 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:11.150943 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mxdbm" Nov 03 14:34:11.522829 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:11.522751 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mxdbm"] Nov 03 14:34:11.530952 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:34:11.530924 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbf4bb3f_88b7_449e_8d36_cd892e49bd86.slice/crio-c517b3da0a3f558c58516035eeb217434e87c803e20697f9fa535e745dcca357 WatchSource:0}: Error finding container c517b3da0a3f558c58516035eeb217434e87c803e20697f9fa535e745dcca357: Status 404 returned error can't find the container with id c517b3da0a3f558c58516035eeb217434e87c803e20697f9fa535e745dcca357 Nov 03 14:34:12.294692 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:12.294652 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mxdbm" event={"ID":"fbf4bb3f-88b7-449e-8d36-cd892e49bd86","Type":"ContainerStarted","Data":"933fa2d857a1ac4c9f703edb5c146419094b0fb834e43c63cc3ac10103abc8cb"} Nov 03 14:34:12.295083 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:12.294715 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mxdbm" event={"ID":"fbf4bb3f-88b7-449e-8d36-cd892e49bd86","Type":"ContainerStarted","Data":"c517b3da0a3f558c58516035eeb217434e87c803e20697f9fa535e745dcca357"} Nov 03 14:34:12.403078 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:12.403022 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mxdbm" podStartSLOduration=2.403008417 podStartE2EDuration="2.403008417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:34:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:34:12.402516437 +0000 UTC m=+5334.413204679" watchObservedRunningTime="2026-11-03 14:34:12.403008417 +0000 UTC m=+5334.413696656" Nov 03 14:34:13.302739 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:13.302701 5979 generic.go:356] "Generic (PLEG): container finished" podID="fbf4bb3f-88b7-449e-8d36-cd892e49bd86" containerID="933fa2d857a1ac4c9f703edb5c146419094b0fb834e43c63cc3ac10103abc8cb" exitCode=0 Nov 03 14:34:13.303147 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:13.302791 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mxdbm" event={"ID":"fbf4bb3f-88b7-449e-8d36-cd892e49bd86","Type":"ContainerDied","Data":"933fa2d857a1ac4c9f703edb5c146419094b0fb834e43c63cc3ac10103abc8cb"} Nov 03 14:34:14.443698 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:14.443671 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mxdbm" Nov 03 14:34:14.524054 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:14.524019 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbf4bb3f-88b7-449e-8d36-cd892e49bd86-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbf4bb3f-88b7-449e-8d36-cd892e49bd86-maas-api-openshift-ca\") pod \"fbf4bb3f-88b7-449e-8d36-cd892e49bd86\" (UID: \"fbf4bb3f-88b7-449e-8d36-cd892e49bd86\") " Nov 03 14:34:14.524228 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:14.524142 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbf4bb3f-88b7-449e-8d36-cd892e49bd86-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbf4bb3f-88b7-449e-8d36-cd892e49bd86-maas-api-xks-ca\") pod \"fbf4bb3f-88b7-449e-8d36-cd892e49bd86\" (UID: \"fbf4bb3f-88b7-449e-8d36-cd892e49bd86\") " Nov 03 14:34:14.524228 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:14.524177 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbf4bb3f-88b7-449e-8d36-cd892e49bd86-kube-api-access-2hspq\" (UniqueName: \"kubernetes.io/projected/fbf4bb3f-88b7-449e-8d36-cd892e49bd86-kube-api-access-2hspq\") pod \"fbf4bb3f-88b7-449e-8d36-cd892e49bd86\" (UID: \"fbf4bb3f-88b7-449e-8d36-cd892e49bd86\") " Nov 03 14:34:14.524535 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:14.524509 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbf4bb3f-88b7-449e-8d36-cd892e49bd86-maas-api-openshift-ca" pod "fbf4bb3f-88b7-449e-8d36-cd892e49bd86" (UID: "fbf4bb3f-88b7-449e-8d36-cd892e49bd86"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:34:14.526204 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:14.526176 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbf4bb3f-88b7-449e-8d36-cd892e49bd86-maas-api-xks-ca" pod "fbf4bb3f-88b7-449e-8d36-cd892e49bd86" (UID: "fbf4bb3f-88b7-449e-8d36-cd892e49bd86"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:34:14.526319 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:14.526280 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbf4bb3f-88b7-449e-8d36-cd892e49bd86-kube-api-access-2hspq" pod "fbf4bb3f-88b7-449e-8d36-cd892e49bd86" (UID: "fbf4bb3f-88b7-449e-8d36-cd892e49bd86"). InnerVolumeSpecName "kube-api-access-2hspq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:34:14.625589 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:14.625511 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbf4bb3f-88b7-449e-8d36-cd892e49bd86-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:34:14.625589 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:14.625540 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hspq\" (UniqueName: \"kubernetes.io/projected/fbf4bb3f-88b7-449e-8d36-cd892e49bd86-kube-api-access-2hspq\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:34:14.625589 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:14.625551 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbf4bb3f-88b7-449e-8d36-cd892e49bd86-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:34:15.315099 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:15.315009 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mxdbm" event={"ID":"fbf4bb3f-88b7-449e-8d36-cd892e49bd86","Type":"ContainerDied","Data":"c517b3da0a3f558c58516035eeb217434e87c803e20697f9fa535e745dcca357"} Nov 03 14:34:15.315099 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:15.315058 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c517b3da0a3f558c58516035eeb217434e87c803e20697f9fa535e745dcca357" Nov 03 14:34:15.315099 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:15.315065 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mxdbm" Nov 03 14:34:16.047616 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:16.047582 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43512dd5-7vqnz"] Nov 03 14:34:16.094135 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:16.094103 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43512dd5-7vqnz"] Nov 03 14:34:16.114160 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:16.114125 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e029367c-8b59-49eb-ad8f-60f024f5a5c7" path="/var/lib/kubelet/pods/e029367c-8b59-49eb-ad8f-60f024f5a5c7/volumes" Nov 03 14:34:21.709968 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:21.709934 5979 scope.go:122] "RemoveContainer" containerID="316d21df51b20b9f613b2943c8413e69172e09a75c0518acc3745e1049dbed03" Nov 03 14:34:21.718807 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:21.718786 5979 scope.go:122] "RemoveContainer" containerID="843e6994b70b03b67e8af0f4a6687b7145e1d350231b40905ca73246196045da" Nov 03 14:34:21.726673 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:21.726655 5979 scope.go:122] "RemoveContainer" containerID="49c128dcd4ec425f013d2eb199536329988b0c6ffaecfb9dd52d562fedcc3d68" Nov 03 14:34:21.734240 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:21.734215 5979 scope.go:122] "RemoveContainer" containerID="a64b215949e66fe31c04223ceda71908ef44b2571d34b348efc1582a35cc5715" Nov 03 14:34:23.044609 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:23.044568 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4834eea6-rxz6c"] Nov 03 14:34:23.091040 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:23.091006 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4834eea6-rxz6c"] Nov 03 14:34:23.485392 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:23.485345 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-588d644cc-lt76c"] Nov 03 14:34:23.485612 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:23.485575 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-588d644cc-lt76c" podUID="74ff5eff-a73d-4ba0-913f-ba087b52cd40" containerName="manager" containerID="cri-o://f1148e9351f574715382ace4f99adc0a2486f51c981028a9967492112a7d1285" gracePeriod=30 Nov 03 14:34:24.114148 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:24.114116 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b97ade37-319a-4885-ba04-9d943641ac1d" path="/var/lib/kubelet/pods/b97ade37-319a-4885-ba04-9d943641ac1d/volumes" Nov 03 14:34:24.235899 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:24.235873 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-588d644cc-lt76c" Nov 03 14:34:24.314527 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:24.314492 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74ff5eff-a73d-4ba0-913f-ba087b52cd40-kube-api-access-5tl64\" (UniqueName: \"kubernetes.io/projected/74ff5eff-a73d-4ba0-913f-ba087b52cd40-kube-api-access-5tl64\") pod \"74ff5eff-a73d-4ba0-913f-ba087b52cd40\" (UID: \"74ff5eff-a73d-4ba0-913f-ba087b52cd40\") " Nov 03 14:34:24.314720 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:24.314546 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74ff5eff-a73d-4ba0-913f-ba087b52cd40-webhook-cert\" (UniqueName: \"kubernetes.io/secret/74ff5eff-a73d-4ba0-913f-ba087b52cd40-webhook-cert\") pod \"74ff5eff-a73d-4ba0-913f-ba087b52cd40\" (UID: \"74ff5eff-a73d-4ba0-913f-ba087b52cd40\") " Nov 03 14:34:24.314720 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:24.314666 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74ff5eff-a73d-4ba0-913f-ba087b52cd40-metrics-certs\" (UniqueName: \"kubernetes.io/secret/74ff5eff-a73d-4ba0-913f-ba087b52cd40-metrics-certs\") pod \"74ff5eff-a73d-4ba0-913f-ba087b52cd40\" (UID: \"74ff5eff-a73d-4ba0-913f-ba087b52cd40\") " Nov 03 14:34:24.316787 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:24.316753 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74ff5eff-a73d-4ba0-913f-ba087b52cd40-webhook-cert" pod "74ff5eff-a73d-4ba0-913f-ba087b52cd40" (UID: "74ff5eff-a73d-4ba0-913f-ba087b52cd40"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:34:24.316787 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:24.316769 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74ff5eff-a73d-4ba0-913f-ba087b52cd40-metrics-certs" pod "74ff5eff-a73d-4ba0-913f-ba087b52cd40" (UID: "74ff5eff-a73d-4ba0-913f-ba087b52cd40"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:34:24.316947 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:24.316809 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74ff5eff-a73d-4ba0-913f-ba087b52cd40-kube-api-access-5tl64" pod "74ff5eff-a73d-4ba0-913f-ba087b52cd40" (UID: "74ff5eff-a73d-4ba0-913f-ba087b52cd40"). InnerVolumeSpecName "kube-api-access-5tl64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:34:24.364877 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:24.364791 5979 generic.go:356] "Generic (PLEG): container finished" podID="74ff5eff-a73d-4ba0-913f-ba087b52cd40" containerID="f1148e9351f574715382ace4f99adc0a2486f51c981028a9967492112a7d1285" exitCode=0 Nov 03 14:34:24.364877 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:24.364844 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-588d644cc-lt76c" Nov 03 14:34:24.365107 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:24.364879 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-588d644cc-lt76c" event={"ID":"74ff5eff-a73d-4ba0-913f-ba087b52cd40","Type":"ContainerDied","Data":"f1148e9351f574715382ace4f99adc0a2486f51c981028a9967492112a7d1285"} Nov 03 14:34:24.365107 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:24.364951 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-588d644cc-lt76c" event={"ID":"74ff5eff-a73d-4ba0-913f-ba087b52cd40","Type":"ContainerDied","Data":"a6caf118544b33ca30a4c182943c9ecb87e95225e6874eef416902ff3c1dee43"} Nov 03 14:34:24.365107 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:24.364985 5979 scope.go:122] "RemoveContainer" containerID="f1148e9351f574715382ace4f99adc0a2486f51c981028a9967492112a7d1285" Nov 03 14:34:24.379345 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:24.379322 5979 scope.go:122] "RemoveContainer" containerID="f1148e9351f574715382ace4f99adc0a2486f51c981028a9967492112a7d1285" Nov 03 14:34:24.379726 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:34:24.379697 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1148e9351f574715382ace4f99adc0a2486f51c981028a9967492112a7d1285\": container with ID starting with f1148e9351f574715382ace4f99adc0a2486f51c981028a9967492112a7d1285 not found: ID does not exist" containerID="f1148e9351f574715382ace4f99adc0a2486f51c981028a9967492112a7d1285" Nov 03 14:34:24.379800 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:24.379739 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1148e9351f574715382ace4f99adc0a2486f51c981028a9967492112a7d1285"} err="failed to get container status \"f1148e9351f574715382ace4f99adc0a2486f51c981028a9967492112a7d1285\": rpc error: code = NotFound desc = could not find container \"f1148e9351f574715382ace4f99adc0a2486f51c981028a9967492112a7d1285\": container with ID starting with f1148e9351f574715382ace4f99adc0a2486f51c981028a9967492112a7d1285 not found: ID does not exist" Nov 03 14:34:24.415561 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:24.415531 5979 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74ff5eff-a73d-4ba0-913f-ba087b52cd40-metrics-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:34:24.415561 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:24.415559 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tl64\" (UniqueName: \"kubernetes.io/projected/74ff5eff-a73d-4ba0-913f-ba087b52cd40-kube-api-access-5tl64\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:34:24.415757 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:24.415575 5979 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/74ff5eff-a73d-4ba0-913f-ba087b52cd40-webhook-cert\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:34:24.518911 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:24.518876 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-588d644cc-lt76c"] Nov 03 14:34:24.525199 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:24.524950 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-588d644cc-lt76c"] Nov 03 14:34:25.932822 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:25.932786 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-588d644cc-bhwpl"] Nov 03 14:34:25.933219 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:25.933197 5979 cpu_manager.go:414] "containerMap: removing container" podUID="fbf4bb3f-88b7-449e-8d36-cd892e49bd86" containerName="revoke-keys" Nov 03 14:34:25.933219 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:25.933209 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbf4bb3f-88b7-449e-8d36-cd892e49bd86" containerName="revoke-keys" Nov 03 14:34:25.933305 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:25.933242 5979 cpu_manager.go:414] "containerMap: removing container" podUID="74ff5eff-a73d-4ba0-913f-ba087b52cd40" containerName="manager" Nov 03 14:34:25.933305 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:25.933248 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74ff5eff-a73d-4ba0-913f-ba087b52cd40" containerName="manager" Nov 03 14:34:25.933373 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:25.933320 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="74ff5eff-a73d-4ba0-913f-ba087b52cd40" containerName="manager" Nov 03 14:34:25.933373 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:25.933332 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbf4bb3f-88b7-449e-8d36-cd892e49bd86" containerName="revoke-keys" Nov 03 14:34:25.937891 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:25.937867 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-588d644cc-bhwpl" Nov 03 14:34:25.987151 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:25.987119 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 14:34:26.032482 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:26.032446 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv4ps\" (UniqueName: \"kubernetes.io/projected/f6d109e0-3ef7-459b-bf9c-dea5de46b825-kube-api-access-xv4ps\") pod \"maas-controller-588d644cc-bhwpl\" (UID: \"f6d109e0-3ef7-459b-bf9c-dea5de46b825\") " pod="opendatahub/maas-controller-588d644cc-bhwpl" Nov 03 14:34:26.032671 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:26.032577 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f6d109e0-3ef7-459b-bf9c-dea5de46b825-webhook-cert\") pod \"maas-controller-588d644cc-bhwpl\" (UID: \"f6d109e0-3ef7-459b-bf9c-dea5de46b825\") " pod="opendatahub/maas-controller-588d644cc-bhwpl" Nov 03 14:34:26.032671 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:26.032645 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6d109e0-3ef7-459b-bf9c-dea5de46b825-metrics-certs\") pod \"maas-controller-588d644cc-bhwpl\" (UID: \"f6d109e0-3ef7-459b-bf9c-dea5de46b825\") " pod="opendatahub/maas-controller-588d644cc-bhwpl" Nov 03 14:34:26.033620 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:26.033602 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-rbwxk\"" Nov 03 14:34:26.033740 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:26.033659 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 14:34:26.033803 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:26.033740 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 14:34:26.042300 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:26.042280 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-588d644cc-bhwpl"] Nov 03 14:34:26.113154 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:26.113113 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74ff5eff-a73d-4ba0-913f-ba087b52cd40" path="/var/lib/kubelet/pods/74ff5eff-a73d-4ba0-913f-ba087b52cd40/volumes" Nov 03 14:34:26.133428 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:26.133397 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f6d109e0-3ef7-459b-bf9c-dea5de46b825-webhook-cert\") pod \"maas-controller-588d644cc-bhwpl\" (UID: \"f6d109e0-3ef7-459b-bf9c-dea5de46b825\") " pod="opendatahub/maas-controller-588d644cc-bhwpl" Nov 03 14:34:26.133595 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:26.133446 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6d109e0-3ef7-459b-bf9c-dea5de46b825-metrics-certs\") pod \"maas-controller-588d644cc-bhwpl\" (UID: \"f6d109e0-3ef7-459b-bf9c-dea5de46b825\") " pod="opendatahub/maas-controller-588d644cc-bhwpl" Nov 03 14:34:26.133595 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:26.133580 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv4ps\" (UniqueName: \"kubernetes.io/projected/f6d109e0-3ef7-459b-bf9c-dea5de46b825-kube-api-access-xv4ps\") pod \"maas-controller-588d644cc-bhwpl\" (UID: \"f6d109e0-3ef7-459b-bf9c-dea5de46b825\") " pod="opendatahub/maas-controller-588d644cc-bhwpl" Nov 03 14:34:26.135936 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:26.135915 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f6d109e0-3ef7-459b-bf9c-dea5de46b825-webhook-cert\") pod \"maas-controller-588d644cc-bhwpl\" (UID: \"f6d109e0-3ef7-459b-bf9c-dea5de46b825\") " pod="opendatahub/maas-controller-588d644cc-bhwpl" Nov 03 14:34:26.136059 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:26.136042 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6d109e0-3ef7-459b-bf9c-dea5de46b825-metrics-certs\") pod \"maas-controller-588d644cc-bhwpl\" (UID: \"f6d109e0-3ef7-459b-bf9c-dea5de46b825\") " pod="opendatahub/maas-controller-588d644cc-bhwpl" Nov 03 14:34:26.189211 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:26.189147 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv4ps\" (UniqueName: \"kubernetes.io/projected/f6d109e0-3ef7-459b-bf9c-dea5de46b825-kube-api-access-xv4ps\") pod \"maas-controller-588d644cc-bhwpl\" (UID: \"f6d109e0-3ef7-459b-bf9c-dea5de46b825\") " pod="opendatahub/maas-controller-588d644cc-bhwpl" Nov 03 14:34:26.250399 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:26.250366 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-588d644cc-bhwpl" Nov 03 14:34:26.621248 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:26.621163 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-588d644cc-bhwpl"] Nov 03 14:34:26.628710 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:34:26.628685 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6d109e0_3ef7_459b_bf9c_dea5de46b825.slice/crio-4a65ce0a3e37f64a8cffc7d092cc62ce131eeba82f47e02aef4b5725cc84a613 WatchSource:0}: Error finding container 4a65ce0a3e37f64a8cffc7d092cc62ce131eeba82f47e02aef4b5725cc84a613: Status 404 returned error can't find the container with id 4a65ce0a3e37f64a8cffc7d092cc62ce131eeba82f47e02aef4b5725cc84a613 Nov 03 14:34:27.045489 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:27.045444 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-420450a7-jp4nb"] Nov 03 14:34:27.092174 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:27.092143 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-420450a7-jp4nb"] Nov 03 14:34:27.382631 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:27.382548 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-588d644cc-bhwpl" event={"ID":"f6d109e0-3ef7-459b-bf9c-dea5de46b825","Type":"ContainerStarted","Data":"4a65ce0a3e37f64a8cffc7d092cc62ce131eeba82f47e02aef4b5725cc84a613"} Nov 03 14:34:28.114562 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:28.114521 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="062dae17-169d-4881-95ba-dd1d307af352" path="/var/lib/kubelet/pods/062dae17-169d-4881-95ba-dd1d307af352/volumes" Nov 03 14:34:28.388796 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:28.388756 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-588d644cc-bhwpl" event={"ID":"f6d109e0-3ef7-459b-bf9c-dea5de46b825","Type":"ContainerStarted","Data":"8588f737539c8c0e94e178f7b06cedbeffa6dfabd00a8760ae30ecfb358d41d4"} Nov 03 14:34:28.388979 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:28.388819 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-588d644cc-bhwpl" Nov 03 14:34:28.498013 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:28.497960 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-588d644cc-bhwpl" podStartSLOduration=2.669214087 podStartE2EDuration="3.497946582s" totalImagesPullingTime="828.732495ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:34:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:34:28.497569081 +0000 UTC m=+5350.508257336" watchObservedRunningTime="2026-11-03 14:34:28.497946582 +0000 UTC m=+5350.508634848" Nov 03 14:34:30.783928 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:30.783890 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8gjrw"] Nov 03 14:34:30.787782 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:30.787760 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8gjrw" Nov 03 14:34:30.879197 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:30.879155 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7e12233-9e76-4de5-b4cf-93644996783e-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-8gjrw\" (UID: \"e7e12233-9e76-4de5-b4cf-93644996783e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8gjrw" Nov 03 14:34:30.879197 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:30.879202 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7e12233-9e76-4de5-b4cf-93644996783e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-8gjrw\" (UID: \"e7e12233-9e76-4de5-b4cf-93644996783e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8gjrw" Nov 03 14:34:30.879451 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:30.879285 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sfk4\" (UniqueName: \"kubernetes.io/projected/e7e12233-9e76-4de5-b4cf-93644996783e-kube-api-access-5sfk4\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-8gjrw\" (UID: \"e7e12233-9e76-4de5-b4cf-93644996783e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8gjrw" Nov 03 14:34:30.879451 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:30.879322 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7e12233-9e76-4de5-b4cf-93644996783e-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-8gjrw\" (UID: \"e7e12233-9e76-4de5-b4cf-93644996783e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8gjrw" Nov 03 14:34:30.879451 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:30.879359 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7e12233-9e76-4de5-b4cf-93644996783e-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-8gjrw\" (UID: \"e7e12233-9e76-4de5-b4cf-93644996783e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8gjrw" Nov 03 14:34:30.879658 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:30.879513 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7e12233-9e76-4de5-b4cf-93644996783e-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-8gjrw\" (UID: \"e7e12233-9e76-4de5-b4cf-93644996783e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8gjrw" Nov 03 14:34:30.879658 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:30.879568 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7e12233-9e76-4de5-b4cf-93644996783e-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-8gjrw\" (UID: \"e7e12233-9e76-4de5-b4cf-93644996783e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8gjrw" Nov 03 14:34:30.879658 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:30.879606 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7e12233-9e76-4de5-b4cf-93644996783e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-8gjrw\" (UID: \"e7e12233-9e76-4de5-b4cf-93644996783e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8gjrw" Nov 03 14:34:30.879825 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:30.879718 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7e12233-9e76-4de5-b4cf-93644996783e-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-8gjrw\" (UID: \"e7e12233-9e76-4de5-b4cf-93644996783e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8gjrw" Nov 03 14:34:30.886289 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:30.886258 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8gjrw"] Nov 03 14:34:30.980461 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:30.980424 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7e12233-9e76-4de5-b4cf-93644996783e-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-8gjrw\" (UID: \"e7e12233-9e76-4de5-b4cf-93644996783e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8gjrw" Nov 03 14:34:30.980682 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:30.980503 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7e12233-9e76-4de5-b4cf-93644996783e-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-8gjrw\" (UID: \"e7e12233-9e76-4de5-b4cf-93644996783e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8gjrw" Nov 03 14:34:30.980682 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:30.980532 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7e12233-9e76-4de5-b4cf-93644996783e-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-8gjrw\" (UID: \"e7e12233-9e76-4de5-b4cf-93644996783e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8gjrw" Nov 03 14:34:30.980682 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:30.980547 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7e12233-9e76-4de5-b4cf-93644996783e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-8gjrw\" (UID: \"e7e12233-9e76-4de5-b4cf-93644996783e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8gjrw" Nov 03 14:34:30.980682 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:30.980592 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7e12233-9e76-4de5-b4cf-93644996783e-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-8gjrw\" (UID: \"e7e12233-9e76-4de5-b4cf-93644996783e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8gjrw" Nov 03 14:34:30.980682 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:30.980619 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7e12233-9e76-4de5-b4cf-93644996783e-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-8gjrw\" (UID: \"e7e12233-9e76-4de5-b4cf-93644996783e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8gjrw" Nov 03 14:34:30.980682 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:30.980643 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7e12233-9e76-4de5-b4cf-93644996783e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-8gjrw\" (UID: \"e7e12233-9e76-4de5-b4cf-93644996783e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8gjrw" Nov 03 14:34:30.981051 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:30.980788 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sfk4\" (UniqueName: \"kubernetes.io/projected/e7e12233-9e76-4de5-b4cf-93644996783e-kube-api-access-5sfk4\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-8gjrw\" (UID: \"e7e12233-9e76-4de5-b4cf-93644996783e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8gjrw" Nov 03 14:34:30.981051 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:30.980980 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7e12233-9e76-4de5-b4cf-93644996783e-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-8gjrw\" (UID: \"e7e12233-9e76-4de5-b4cf-93644996783e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8gjrw" Nov 03 14:34:30.981051 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:30.981017 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7e12233-9e76-4de5-b4cf-93644996783e-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-8gjrw\" (UID: \"e7e12233-9e76-4de5-b4cf-93644996783e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8gjrw" Nov 03 14:34:30.981193 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:30.981069 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7e12233-9e76-4de5-b4cf-93644996783e-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-8gjrw\" (UID: \"e7e12233-9e76-4de5-b4cf-93644996783e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8gjrw" Nov 03 14:34:30.981193 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:30.980821 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7e12233-9e76-4de5-b4cf-93644996783e-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-8gjrw\" (UID: \"e7e12233-9e76-4de5-b4cf-93644996783e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8gjrw" Nov 03 14:34:30.981343 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:30.981322 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7e12233-9e76-4de5-b4cf-93644996783e-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-8gjrw\" (UID: \"e7e12233-9e76-4de5-b4cf-93644996783e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8gjrw" Nov 03 14:34:30.981410 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:30.981391 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7e12233-9e76-4de5-b4cf-93644996783e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-8gjrw\" (UID: \"e7e12233-9e76-4de5-b4cf-93644996783e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8gjrw" Nov 03 14:34:30.982808 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:30.982787 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7e12233-9e76-4de5-b4cf-93644996783e-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-8gjrw\" (UID: \"e7e12233-9e76-4de5-b4cf-93644996783e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8gjrw" Nov 03 14:34:30.983043 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:30.983023 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7e12233-9e76-4de5-b4cf-93644996783e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-8gjrw\" (UID: \"e7e12233-9e76-4de5-b4cf-93644996783e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8gjrw" Nov 03 14:34:31.034326 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:31.034257 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7e12233-9e76-4de5-b4cf-93644996783e-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-8gjrw\" (UID: \"e7e12233-9e76-4de5-b4cf-93644996783e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8gjrw" Nov 03 14:34:31.034573 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:31.034548 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sfk4\" (UniqueName: \"kubernetes.io/projected/e7e12233-9e76-4de5-b4cf-93644996783e-kube-api-access-5sfk4\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-8gjrw\" (UID: \"e7e12233-9e76-4de5-b4cf-93644996783e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8gjrw" Nov 03 14:34:31.099998 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:31.099957 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8gjrw" Nov 03 14:34:31.525867 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:31.525831 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8gjrw"] Nov 03 14:34:31.533390 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:34:31.533363 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7e12233_9e76_4de5_b4cf_93644996783e.slice/crio-63e96ede6f8a3136b7024a53c568515d4820fabe04e2f506bed8edeab3742ad4 WatchSource:0}: Error finding container 63e96ede6f8a3136b7024a53c568515d4820fabe04e2f506bed8edeab3742ad4: Status 404 returned error can't find the container with id 63e96ede6f8a3136b7024a53c568515d4820fabe04e2f506bed8edeab3742ad4 Nov 03 14:34:31.535582 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:31.535549 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:34:31.535691 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:31.535627 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:34:31.535691 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:31.535673 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:34:32.410190 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:32.410145 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8gjrw" event={"ID":"e7e12233-9e76-4de5-b4cf-93644996783e","Type":"ContainerStarted","Data":"b9a574f8cc0e9945d1c1866805e4adbc8a589603f609d7a6cdcf1a80034e1386"} Nov 03 14:34:32.410615 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:32.410214 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8gjrw" event={"ID":"e7e12233-9e76-4de5-b4cf-93644996783e","Type":"ContainerStarted","Data":"63e96ede6f8a3136b7024a53c568515d4820fabe04e2f506bed8edeab3742ad4"} Nov 03 14:34:32.577600 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:32.577548 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8gjrw" podStartSLOduration=2.577532411 podStartE2EDuration="2.577532411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:34:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:34:32.576259369 +0000 UTC m=+5354.586947610" watchObservedRunningTime="2026-11-03 14:34:32.577532411 +0000 UTC m=+5354.588220718" Nov 03 14:34:33.100808 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:33.100765 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8gjrw" Nov 03 14:34:33.102305 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:33.102276 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-8gjrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.132:15021/healthz/ready\": dial tcp 10.134.1.132:15021: connect: connection refused" start-of-body= Nov 03 14:34:33.102443 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:33.102328 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8gjrw" podUID="e7e12233-9e76-4de5-b4cf-93644996783e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.132:15021/healthz/ready\": dial tcp 10.134.1.132:15021: connect: connection refused" Nov 03 14:34:34.101197 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:34.101156 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-8gjrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.132:15021/healthz/ready\": dial tcp 10.134.1.132:15021: connect: connection refused" start-of-body= Nov 03 14:34:34.101652 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:34.101215 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8gjrw" podUID="e7e12233-9e76-4de5-b4cf-93644996783e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.132:15021/healthz/ready\": dial tcp 10.134.1.132:15021: connect: connection refused" Nov 03 14:34:35.100943 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:35.100911 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-8gjrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.132:15021/healthz/ready\": dial tcp 10.134.1.132:15021: connect: connection refused" start-of-body= Nov 03 14:34:35.101133 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:35.100961 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8gjrw" podUID="e7e12233-9e76-4de5-b4cf-93644996783e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.132:15021/healthz/ready\": dial tcp 10.134.1.132:15021: connect: connection refused" Nov 03 14:34:36.042708 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:36.042674 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bba0b6c-rxsmw"] Nov 03 14:34:36.089341 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:36.089305 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bba0b6c-rxsmw"] Nov 03 14:34:36.101104 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:36.101076 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-8gjrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.132:15021/healthz/ready\": dial tcp 10.134.1.132:15021: connect: connection refused" start-of-body= Nov 03 14:34:36.101245 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:36.101124 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8gjrw" podUID="e7e12233-9e76-4de5-b4cf-93644996783e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.132:15021/healthz/ready\": dial tcp 10.134.1.132:15021: connect: connection refused" Nov 03 14:34:36.114248 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:36.114215 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6134692-b548-43a9-b097-6bce1f7ed9f9" path="/var/lib/kubelet/pods/e6134692-b548-43a9-b097-6bce1f7ed9f9/volumes" Nov 03 14:34:37.100396 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:37.100362 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-8gjrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.132:15021/healthz/ready\": dial tcp 10.134.1.132:15021: connect: connection refused" start-of-body= Nov 03 14:34:37.100865 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:37.100414 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8gjrw" podUID="e7e12233-9e76-4de5-b4cf-93644996783e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.132:15021/healthz/ready\": dial tcp 10.134.1.132:15021: connect: connection refused" Nov 03 14:34:37.233962 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:34:37.233925 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:34:38.047078 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:38.047031 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69d6d864-lqm5g"] Nov 03 14:34:38.093533 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:38.093461 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69d6d864-lqm5g"] Nov 03 14:34:38.101183 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:38.101150 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-8gjrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.132:15021/healthz/ready\": dial tcp 10.134.1.132:15021: connect: connection refused" start-of-body= Nov 03 14:34:38.101568 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:38.101213 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8gjrw" podUID="e7e12233-9e76-4de5-b4cf-93644996783e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.132:15021/healthz/ready\": dial tcp 10.134.1.132:15021: connect: connection refused" Nov 03 14:34:38.114892 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:38.114866 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8d3d4f9-8d57-4ff2-99fa-57ffeedfea04" path="/var/lib/kubelet/pods/a8d3d4f9-8d57-4ff2-99fa-57ffeedfea04/volumes" Nov 03 14:34:39.101012 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:39.100971 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-8gjrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.132:15021/healthz/ready\": dial tcp 10.134.1.132:15021: connect: connection refused" start-of-body= Nov 03 14:34:39.101207 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:39.101032 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8gjrw" podUID="e7e12233-9e76-4de5-b4cf-93644996783e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.132:15021/healthz/ready\": dial tcp 10.134.1.132:15021: connect: connection refused" Nov 03 14:34:39.396348 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:39.396316 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-588d644cc-bhwpl" Nov 03 14:34:40.105128 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:40.105094 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8gjrw" Nov 03 14:34:40.105588 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:40.105433 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8gjrw" Nov 03 14:34:40.106194 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:40.106169 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8gjrw" Nov 03 14:34:49.046241 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:49.046199 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a8bd0ab-d7tlh"] Nov 03 14:34:49.092761 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:49.092728 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a8bd0ab-d7tlh"] Nov 03 14:34:50.114329 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:50.114293 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09d30900-2c56-479b-96d7-b5da683a4356" path="/var/lib/kubelet/pods/09d30900-2c56-479b-96d7-b5da683a4356/volumes" Nov 03 14:34:51.180015 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:51.179973 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:34:54.043274 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:54.043231 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-578862a6-5fdq5"] Nov 03 14:34:54.089755 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:54.089714 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-578862a6-5fdq5"] Nov 03 14:34:54.114121 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:54.114087 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e1bdabe-7240-4fcb-b1d8-a95827a342cd" path="/var/lib/kubelet/pods/4e1bdabe-7240-4fcb-b1d8-a95827a342cd/volumes" Nov 03 14:34:56.695620 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:56.695582 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-4blnb"] Nov 03 14:34:56.700849 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:56.700825 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-4blnb" Nov 03 14:34:56.741393 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:56.741365 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae1ea258-2ad6-455c-81aa-86828bb969c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-4blnb\" (UID: \"ae1ea258-2ad6-455c-81aa-86828bb969c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-4blnb" Nov 03 14:34:56.741549 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:56.741404 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2qrg\" (UniqueName: \"kubernetes.io/projected/ae1ea258-2ad6-455c-81aa-86828bb969c9-kube-api-access-l2qrg\") pod \"maas-api-revoke-sub-f904bff9-4blnb\" (UID: \"ae1ea258-2ad6-455c-81aa-86828bb969c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-4blnb" Nov 03 14:34:56.741549 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:56.741432 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae1ea258-2ad6-455c-81aa-86828bb969c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-4blnb\" (UID: \"ae1ea258-2ad6-455c-81aa-86828bb969c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-4blnb" Nov 03 14:34:56.798396 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:56.798364 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:34:56.798396 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:56.798385 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:34:56.798679 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:56.798406 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:34:56.798679 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:56.798505 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wgk87\"" Nov 03 14:34:56.799936 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:56.799917 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-4blnb"] Nov 03 14:34:56.842575 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:56.842545 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae1ea258-2ad6-455c-81aa-86828bb969c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-4blnb\" (UID: \"ae1ea258-2ad6-455c-81aa-86828bb969c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-4blnb" Nov 03 14:34:56.842575 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:56.842577 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2qrg\" (UniqueName: \"kubernetes.io/projected/ae1ea258-2ad6-455c-81aa-86828bb969c9-kube-api-access-l2qrg\") pod \"maas-api-revoke-sub-f904bff9-4blnb\" (UID: \"ae1ea258-2ad6-455c-81aa-86828bb969c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-4blnb" Nov 03 14:34:56.842858 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:56.842598 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae1ea258-2ad6-455c-81aa-86828bb969c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-4blnb\" (UID: \"ae1ea258-2ad6-455c-81aa-86828bb969c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-4blnb" Nov 03 14:34:56.843297 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:56.843269 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae1ea258-2ad6-455c-81aa-86828bb969c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-4blnb\" (UID: \"ae1ea258-2ad6-455c-81aa-86828bb969c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-4blnb" Nov 03 14:34:56.844932 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:56.844909 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae1ea258-2ad6-455c-81aa-86828bb969c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-4blnb\" (UID: \"ae1ea258-2ad6-455c-81aa-86828bb969c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-4blnb" Nov 03 14:34:56.897252 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:56.897211 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2qrg\" (UniqueName: \"kubernetes.io/projected/ae1ea258-2ad6-455c-81aa-86828bb969c9-kube-api-access-l2qrg\") pod \"maas-api-revoke-sub-f904bff9-4blnb\" (UID: \"ae1ea258-2ad6-455c-81aa-86828bb969c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-4blnb" Nov 03 14:34:57.013785 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:57.013685 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-4blnb" Nov 03 14:34:57.382449 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:57.382360 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-4blnb"] Nov 03 14:34:57.389823 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:34:57.389796 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae1ea258_2ad6_455c_81aa_86828bb969c9.slice/crio-5ff7f80a5a17d768699bde40e23c0069288825c9ecd3f894449ff6bce9c586f1 WatchSource:0}: Error finding container 5ff7f80a5a17d768699bde40e23c0069288825c9ecd3f894449ff6bce9c586f1: Status 404 returned error can't find the container with id 5ff7f80a5a17d768699bde40e23c0069288825c9ecd3f894449ff6bce9c586f1 Nov 03 14:34:57.544820 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:57.544783 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-4blnb" event={"ID":"ae1ea258-2ad6-455c-81aa-86828bb969c9","Type":"ContainerStarted","Data":"5ff7f80a5a17d768699bde40e23c0069288825c9ecd3f894449ff6bce9c586f1"} Nov 03 14:34:58.551318 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:58.551264 5979 generic.go:356] "Generic (PLEG): container finished" podID="ae1ea258-2ad6-455c-81aa-86828bb969c9" containerID="b7a5745868fdf8dfe19129b166526b58e9411b04ce5e7afcd1f2c8971d10dab3" exitCode=0 Nov 03 14:34:58.551778 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:58.551343 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-4blnb" event={"ID":"ae1ea258-2ad6-455c-81aa-86828bb969c9","Type":"ContainerDied","Data":"b7a5745868fdf8dfe19129b166526b58e9411b04ce5e7afcd1f2c8971d10dab3"} Nov 03 14:34:59.705411 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:59.705374 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-4blnb" Nov 03 14:34:59.765129 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:59.765098 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae1ea258-2ad6-455c-81aa-86828bb969c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae1ea258-2ad6-455c-81aa-86828bb969c9-maas-api-xks-ca\") pod \"ae1ea258-2ad6-455c-81aa-86828bb969c9\" (UID: \"ae1ea258-2ad6-455c-81aa-86828bb969c9\") " Nov 03 14:34:59.765306 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:59.765140 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae1ea258-2ad6-455c-81aa-86828bb969c9-kube-api-access-l2qrg\" (UniqueName: \"kubernetes.io/projected/ae1ea258-2ad6-455c-81aa-86828bb969c9-kube-api-access-l2qrg\") pod \"ae1ea258-2ad6-455c-81aa-86828bb969c9\" (UID: \"ae1ea258-2ad6-455c-81aa-86828bb969c9\") " Nov 03 14:34:59.765306 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:59.765162 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae1ea258-2ad6-455c-81aa-86828bb969c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae1ea258-2ad6-455c-81aa-86828bb969c9-maas-api-openshift-ca\") pod \"ae1ea258-2ad6-455c-81aa-86828bb969c9\" (UID: \"ae1ea258-2ad6-455c-81aa-86828bb969c9\") " Nov 03 14:34:59.765597 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:59.765562 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae1ea258-2ad6-455c-81aa-86828bb969c9-maas-api-openshift-ca" pod "ae1ea258-2ad6-455c-81aa-86828bb969c9" (UID: "ae1ea258-2ad6-455c-81aa-86828bb969c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:34:59.767194 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:59.767161 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae1ea258-2ad6-455c-81aa-86828bb969c9-maas-api-xks-ca" pod "ae1ea258-2ad6-455c-81aa-86828bb969c9" (UID: "ae1ea258-2ad6-455c-81aa-86828bb969c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:34:59.767333 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:59.767314 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae1ea258-2ad6-455c-81aa-86828bb969c9-kube-api-access-l2qrg" pod "ae1ea258-2ad6-455c-81aa-86828bb969c9" (UID: "ae1ea258-2ad6-455c-81aa-86828bb969c9"). InnerVolumeSpecName "kube-api-access-l2qrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:34:59.866038 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:59.865940 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae1ea258-2ad6-455c-81aa-86828bb969c9-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:34:59.866038 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:59.865978 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2qrg\" (UniqueName: \"kubernetes.io/projected/ae1ea258-2ad6-455c-81aa-86828bb969c9-kube-api-access-l2qrg\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:34:59.866038 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:34:59.865993 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae1ea258-2ad6-455c-81aa-86828bb969c9-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:35:00.563091 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:00.563056 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-4blnb" event={"ID":"ae1ea258-2ad6-455c-81aa-86828bb969c9","Type":"ContainerDied","Data":"5ff7f80a5a17d768699bde40e23c0069288825c9ecd3f894449ff6bce9c586f1"} Nov 03 14:35:00.563091 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:00.563077 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-4blnb" Nov 03 14:35:00.563323 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:00.563104 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ff7f80a5a17d768699bde40e23c0069288825c9ecd3f894449ff6bce9c586f1" Nov 03 14:35:07.233659 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:35:07.233617 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:35:09.436457 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:09.436425 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:35:16.162810 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:16.162770 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lj2cw"] Nov 03 14:35:16.163439 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:16.163417 5979 cpu_manager.go:414] "containerMap: removing container" podUID="ae1ea258-2ad6-455c-81aa-86828bb969c9" containerName="revoke-keys" Nov 03 14:35:16.163546 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:16.163441 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae1ea258-2ad6-455c-81aa-86828bb969c9" containerName="revoke-keys" Nov 03 14:35:16.164862 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:16.164255 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae1ea258-2ad6-455c-81aa-86828bb969c9" containerName="revoke-keys" Nov 03 14:35:16.176379 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:16.176358 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lj2cw" Nov 03 14:35:16.225136 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:16.225095 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f4e540f-d1f9-4ffe-83ea-dfca92ce7233-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-lj2cw\" (UID: \"7f4e540f-d1f9-4ffe-83ea-dfca92ce7233\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lj2cw" Nov 03 14:35:16.225305 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:16.225152 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f4e540f-d1f9-4ffe-83ea-dfca92ce7233-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-lj2cw\" (UID: \"7f4e540f-d1f9-4ffe-83ea-dfca92ce7233\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lj2cw" Nov 03 14:35:16.225305 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:16.225253 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f4e540f-d1f9-4ffe-83ea-dfca92ce7233-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-lj2cw\" (UID: \"7f4e540f-d1f9-4ffe-83ea-dfca92ce7233\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lj2cw" Nov 03 14:35:16.225305 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:16.225301 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f4e540f-d1f9-4ffe-83ea-dfca92ce7233-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-lj2cw\" (UID: \"7f4e540f-d1f9-4ffe-83ea-dfca92ce7233\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lj2cw" Nov 03 14:35:16.225446 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:16.225385 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f4e540f-d1f9-4ffe-83ea-dfca92ce7233-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-lj2cw\" (UID: \"7f4e540f-d1f9-4ffe-83ea-dfca92ce7233\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lj2cw" Nov 03 14:35:16.225446 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:16.225424 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f4e540f-d1f9-4ffe-83ea-dfca92ce7233-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-lj2cw\" (UID: \"7f4e540f-d1f9-4ffe-83ea-dfca92ce7233\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lj2cw" Nov 03 14:35:16.225550 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:16.225490 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f4e540f-d1f9-4ffe-83ea-dfca92ce7233-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-lj2cw\" (UID: \"7f4e540f-d1f9-4ffe-83ea-dfca92ce7233\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lj2cw" Nov 03 14:35:16.225550 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:16.225534 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xtrw\" (UniqueName: \"kubernetes.io/projected/7f4e540f-d1f9-4ffe-83ea-dfca92ce7233-kube-api-access-2xtrw\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-lj2cw\" (UID: \"7f4e540f-d1f9-4ffe-83ea-dfca92ce7233\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lj2cw" Nov 03 14:35:16.225619 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:16.225560 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f4e540f-d1f9-4ffe-83ea-dfca92ce7233-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-lj2cw\" (UID: \"7f4e540f-d1f9-4ffe-83ea-dfca92ce7233\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lj2cw" Nov 03 14:35:16.326610 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:16.326570 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f4e540f-d1f9-4ffe-83ea-dfca92ce7233-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-lj2cw\" (UID: \"7f4e540f-d1f9-4ffe-83ea-dfca92ce7233\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lj2cw" Nov 03 14:35:16.326819 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:16.326651 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f4e540f-d1f9-4ffe-83ea-dfca92ce7233-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-lj2cw\" (UID: \"7f4e540f-d1f9-4ffe-83ea-dfca92ce7233\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lj2cw" Nov 03 14:35:16.326819 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:16.326681 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f4e540f-d1f9-4ffe-83ea-dfca92ce7233-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-lj2cw\" (UID: \"7f4e540f-d1f9-4ffe-83ea-dfca92ce7233\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lj2cw" Nov 03 14:35:16.326819 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:16.326707 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f4e540f-d1f9-4ffe-83ea-dfca92ce7233-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-lj2cw\" (UID: \"7f4e540f-d1f9-4ffe-83ea-dfca92ce7233\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lj2cw" Nov 03 14:35:16.326819 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:16.326754 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f4e540f-d1f9-4ffe-83ea-dfca92ce7233-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-lj2cw\" (UID: \"7f4e540f-d1f9-4ffe-83ea-dfca92ce7233\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lj2cw" Nov 03 14:35:16.326819 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:16.326777 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f4e540f-d1f9-4ffe-83ea-dfca92ce7233-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-lj2cw\" (UID: \"7f4e540f-d1f9-4ffe-83ea-dfca92ce7233\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lj2cw" Nov 03 14:35:16.327101 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:16.326824 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f4e540f-d1f9-4ffe-83ea-dfca92ce7233-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-lj2cw\" (UID: \"7f4e540f-d1f9-4ffe-83ea-dfca92ce7233\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lj2cw" Nov 03 14:35:16.327101 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:16.326869 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xtrw\" (UniqueName: \"kubernetes.io/projected/7f4e540f-d1f9-4ffe-83ea-dfca92ce7233-kube-api-access-2xtrw\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-lj2cw\" (UID: \"7f4e540f-d1f9-4ffe-83ea-dfca92ce7233\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lj2cw" Nov 03 14:35:16.327101 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:16.326911 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f4e540f-d1f9-4ffe-83ea-dfca92ce7233-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-lj2cw\" (UID: \"7f4e540f-d1f9-4ffe-83ea-dfca92ce7233\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lj2cw" Nov 03 14:35:16.327101 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:16.327077 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f4e540f-d1f9-4ffe-83ea-dfca92ce7233-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-lj2cw\" (UID: \"7f4e540f-d1f9-4ffe-83ea-dfca92ce7233\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lj2cw" Nov 03 14:35:16.327730 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:16.327417 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f4e540f-d1f9-4ffe-83ea-dfca92ce7233-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-lj2cw\" (UID: \"7f4e540f-d1f9-4ffe-83ea-dfca92ce7233\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lj2cw" Nov 03 14:35:16.327730 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:16.327443 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f4e540f-d1f9-4ffe-83ea-dfca92ce7233-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-lj2cw\" (UID: \"7f4e540f-d1f9-4ffe-83ea-dfca92ce7233\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lj2cw" Nov 03 14:35:16.327730 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:16.327515 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f4e540f-d1f9-4ffe-83ea-dfca92ce7233-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-lj2cw\" (UID: \"7f4e540f-d1f9-4ffe-83ea-dfca92ce7233\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lj2cw" Nov 03 14:35:16.327936 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:16.327747 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f4e540f-d1f9-4ffe-83ea-dfca92ce7233-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-lj2cw\" (UID: \"7f4e540f-d1f9-4ffe-83ea-dfca92ce7233\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lj2cw" Nov 03 14:35:16.330196 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:16.330174 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f4e540f-d1f9-4ffe-83ea-dfca92ce7233-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-lj2cw\" (UID: \"7f4e540f-d1f9-4ffe-83ea-dfca92ce7233\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lj2cw" Nov 03 14:35:16.330339 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:16.330322 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f4e540f-d1f9-4ffe-83ea-dfca92ce7233-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-lj2cw\" (UID: \"7f4e540f-d1f9-4ffe-83ea-dfca92ce7233\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lj2cw" Nov 03 14:35:16.379393 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:16.379363 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lj2cw"] Nov 03 14:35:16.382227 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:16.382168 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f4e540f-d1f9-4ffe-83ea-dfca92ce7233-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-lj2cw\" (UID: \"7f4e540f-d1f9-4ffe-83ea-dfca92ce7233\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lj2cw" Nov 03 14:35:16.385749 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:16.385708 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xtrw\" (UniqueName: \"kubernetes.io/projected/7f4e540f-d1f9-4ffe-83ea-dfca92ce7233-kube-api-access-2xtrw\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-lj2cw\" (UID: \"7f4e540f-d1f9-4ffe-83ea-dfca92ce7233\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lj2cw" Nov 03 14:35:16.496281 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:16.496187 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lj2cw" Nov 03 14:35:16.951908 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:16.951875 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lj2cw"] Nov 03 14:35:16.959401 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:35:16.959372 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f4e540f_d1f9_4ffe_83ea_dfca92ce7233.slice/crio-e0d5eec85015dd99c07e8e806ed0c4344eccafc739692a4110d8a555f56b2ae4 WatchSource:0}: Error finding container e0d5eec85015dd99c07e8e806ed0c4344eccafc739692a4110d8a555f56b2ae4: Status 404 returned error can't find the container with id e0d5eec85015dd99c07e8e806ed0c4344eccafc739692a4110d8a555f56b2ae4 Nov 03 14:35:16.961533 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:16.961503 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:35:16.961610 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:16.961565 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:35:16.961610 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:16.961596 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:35:17.677322 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:17.677274 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lj2cw" event={"ID":"7f4e540f-d1f9-4ffe-83ea-dfca92ce7233","Type":"ContainerStarted","Data":"db8e50addadce95ca216437fb22e90dd30515cb93e4ad47b7be25aae2cc8ec6f"} Nov 03 14:35:17.677983 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:17.677347 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lj2cw" event={"ID":"7f4e540f-d1f9-4ffe-83ea-dfca92ce7233","Type":"ContainerStarted","Data":"e0d5eec85015dd99c07e8e806ed0c4344eccafc739692a4110d8a555f56b2ae4"} Nov 03 14:35:17.840545 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:17.840463 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lj2cw" podStartSLOduration=1.840450266 podStartE2EDuration="1.840450266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:35:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:35:17.839312303 +0000 UTC m=+5399.850000545" watchObservedRunningTime="2026-11-03 14:35:17.840450266 +0000 UTC m=+5399.851138588" Nov 03 14:35:18.496550 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:18.496513 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lj2cw" Nov 03 14:35:18.498014 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:18.497991 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-lj2cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.134:15021/healthz/ready\": dial tcp 10.134.1.134:15021: connect: connection refused" start-of-body= Nov 03 14:35:18.498077 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:18.498037 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lj2cw" podUID="7f4e540f-d1f9-4ffe-83ea-dfca92ce7233" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.134:15021/healthz/ready\": dial tcp 10.134.1.134:15021: connect: connection refused" Nov 03 14:35:18.728447 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:18.728411 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:35:19.496910 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:19.496871 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-lj2cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.134:15021/healthz/ready\": dial tcp 10.134.1.134:15021: connect: connection refused" start-of-body= Nov 03 14:35:19.497136 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:19.496928 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lj2cw" podUID="7f4e540f-d1f9-4ffe-83ea-dfca92ce7233" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.134:15021/healthz/ready\": dial tcp 10.134.1.134:15021: connect: connection refused" Nov 03 14:35:20.497556 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:20.497521 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-lj2cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.134:15021/healthz/ready\": dial tcp 10.134.1.134:15021: connect: connection refused" start-of-body= Nov 03 14:35:20.497955 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:20.497584 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lj2cw" podUID="7f4e540f-d1f9-4ffe-83ea-dfca92ce7233" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.134:15021/healthz/ready\": dial tcp 10.134.1.134:15021: connect: connection refused" Nov 03 14:35:21.496594 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:21.496557 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-lj2cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.134:15021/healthz/ready\": dial tcp 10.134.1.134:15021: connect: connection refused" start-of-body= Nov 03 14:35:21.496789 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:21.496614 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lj2cw" podUID="7f4e540f-d1f9-4ffe-83ea-dfca92ce7233" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.134:15021/healthz/ready\": dial tcp 10.134.1.134:15021: connect: connection refused" Nov 03 14:35:21.775056 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:21.774965 5979 scope.go:122] "RemoveContainer" containerID="e16a312f87b7590b0f3d867d227ce5ee18364773c8cd27dbe6eeb5cc4d7a0f8e" Nov 03 14:35:21.783708 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:21.783685 5979 scope.go:122] "RemoveContainer" containerID="d4cbea2b2e3d98a7690384dda63efadd775aa70d4fc34146614de73a26e3deaa" Nov 03 14:35:22.497001 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:22.496959 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-lj2cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.134:15021/healthz/ready\": dial tcp 10.134.1.134:15021: connect: connection refused" start-of-body= Nov 03 14:35:22.497229 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:22.497038 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lj2cw" podUID="7f4e540f-d1f9-4ffe-83ea-dfca92ce7233" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.134:15021/healthz/ready\": dial tcp 10.134.1.134:15021: connect: connection refused" Nov 03 14:35:23.497285 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:23.497249 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-lj2cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.134:15021/healthz/ready\": dial tcp 10.134.1.134:15021: connect: connection refused" start-of-body= Nov 03 14:35:23.497801 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:23.497317 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lj2cw" podUID="7f4e540f-d1f9-4ffe-83ea-dfca92ce7233" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.134:15021/healthz/ready\": dial tcp 10.134.1.134:15021: connect: connection refused" Nov 03 14:35:25.497417 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:25.497378 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-lj2cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:35:25.498038 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:25.497441 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lj2cw" podUID="7f4e540f-d1f9-4ffe-83ea-dfca92ce7233" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:35:26.540644 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:26.540603 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-lj2cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:35:26.541078 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:26.540666 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lj2cw" podUID="7f4e540f-d1f9-4ffe-83ea-dfca92ce7233" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:35:27.581643 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:27.581597 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-lj2cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:35:27.582134 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:27.581676 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lj2cw" podUID="7f4e540f-d1f9-4ffe-83ea-dfca92ce7233" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:35:27.585162 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:27.585132 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:35:27.955666 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:27.955630 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lj2cw" Nov 03 14:35:27.956083 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:27.956054 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lj2cw" Nov 03 14:35:27.956679 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:27.956660 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lj2cw" Nov 03 14:35:29.448395 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:29.448318 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dnvk8"] Nov 03 14:35:29.454036 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:29.454013 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dnvk8" Nov 03 14:35:29.503139 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:29.503108 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:35:29.549724 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:29.549674 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:35:29.549724 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:29.549722 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:35:29.549986 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:29.549816 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wgk87\"" Nov 03 14:35:29.551371 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:29.551350 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dnvk8"] Nov 03 14:35:29.611969 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:29.611932 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/139c66ae-af59-4ea8-a319-ce31cde503f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-dnvk8\" (UID: \"139c66ae-af59-4ea8-a319-ce31cde503f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dnvk8" Nov 03 14:35:29.612180 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:29.612075 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsxmj\" (UniqueName: \"kubernetes.io/projected/139c66ae-af59-4ea8-a319-ce31cde503f4-kube-api-access-rsxmj\") pod \"maas-api-revoke-sub-f03629db-dnvk8\" (UID: \"139c66ae-af59-4ea8-a319-ce31cde503f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dnvk8" Nov 03 14:35:29.612180 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:29.612154 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/139c66ae-af59-4ea8-a319-ce31cde503f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-dnvk8\" (UID: \"139c66ae-af59-4ea8-a319-ce31cde503f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dnvk8" Nov 03 14:35:29.713367 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:29.713277 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/139c66ae-af59-4ea8-a319-ce31cde503f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-dnvk8\" (UID: \"139c66ae-af59-4ea8-a319-ce31cde503f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dnvk8" Nov 03 14:35:29.713367 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:29.713332 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsxmj\" (UniqueName: \"kubernetes.io/projected/139c66ae-af59-4ea8-a319-ce31cde503f4-kube-api-access-rsxmj\") pod \"maas-api-revoke-sub-f03629db-dnvk8\" (UID: \"139c66ae-af59-4ea8-a319-ce31cde503f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dnvk8" Nov 03 14:35:29.713619 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:29.713429 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/139c66ae-af59-4ea8-a319-ce31cde503f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-dnvk8\" (UID: \"139c66ae-af59-4ea8-a319-ce31cde503f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dnvk8" Nov 03 14:35:29.714018 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:29.713998 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/139c66ae-af59-4ea8-a319-ce31cde503f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-dnvk8\" (UID: \"139c66ae-af59-4ea8-a319-ce31cde503f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dnvk8" Nov 03 14:35:29.715822 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:29.715803 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/139c66ae-af59-4ea8-a319-ce31cde503f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-dnvk8\" (UID: \"139c66ae-af59-4ea8-a319-ce31cde503f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dnvk8" Nov 03 14:35:29.767098 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:29.767066 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsxmj\" (UniqueName: \"kubernetes.io/projected/139c66ae-af59-4ea8-a319-ce31cde503f4-kube-api-access-rsxmj\") pod \"maas-api-revoke-sub-f03629db-dnvk8\" (UID: \"139c66ae-af59-4ea8-a319-ce31cde503f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dnvk8" Nov 03 14:35:30.066767 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:30.066663 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dnvk8" Nov 03 14:35:30.433816 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:30.433786 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dnvk8"] Nov 03 14:35:30.441665 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:35:30.441635 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod139c66ae_af59_4ea8_a319_ce31cde503f4.slice/crio-d3ba5a3afa74edbd01ae84d2477360ce80e8c544dcbda625d2cd054945bba722 WatchSource:0}: Error finding container d3ba5a3afa74edbd01ae84d2477360ce80e8c544dcbda625d2cd054945bba722: Status 404 returned error can't find the container with id d3ba5a3afa74edbd01ae84d2477360ce80e8c544dcbda625d2cd054945bba722 Nov 03 14:35:30.755599 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:30.755564 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dnvk8" event={"ID":"139c66ae-af59-4ea8-a319-ce31cde503f4","Type":"ContainerStarted","Data":"d3ba5a3afa74edbd01ae84d2477360ce80e8c544dcbda625d2cd054945bba722"} Nov 03 14:35:31.762029 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:31.761997 5979 generic.go:356] "Generic (PLEG): container finished" podID="139c66ae-af59-4ea8-a319-ce31cde503f4" containerID="e6263b4cdde283940e8fa8aba4bdf05c19c4d935c781c6ef62b770bd58ac8a1e" exitCode=0 Nov 03 14:35:31.762447 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:31.762096 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dnvk8" event={"ID":"139c66ae-af59-4ea8-a319-ce31cde503f4","Type":"ContainerDied","Data":"e6263b4cdde283940e8fa8aba4bdf05c19c4d935c781c6ef62b770bd58ac8a1e"} Nov 03 14:35:32.898658 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:32.898630 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dnvk8" Nov 03 14:35:33.051271 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:33.051193 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/139c66ae-af59-4ea8-a319-ce31cde503f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/139c66ae-af59-4ea8-a319-ce31cde503f4-maas-api-openshift-ca\") pod \"139c66ae-af59-4ea8-a319-ce31cde503f4\" (UID: \"139c66ae-af59-4ea8-a319-ce31cde503f4\") " Nov 03 14:35:33.051439 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:33.051349 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/139c66ae-af59-4ea8-a319-ce31cde503f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/139c66ae-af59-4ea8-a319-ce31cde503f4-maas-api-xks-ca\") pod \"139c66ae-af59-4ea8-a319-ce31cde503f4\" (UID: \"139c66ae-af59-4ea8-a319-ce31cde503f4\") " Nov 03 14:35:33.051439 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:33.051397 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/139c66ae-af59-4ea8-a319-ce31cde503f4-kube-api-access-rsxmj\" (UniqueName: \"kubernetes.io/projected/139c66ae-af59-4ea8-a319-ce31cde503f4-kube-api-access-rsxmj\") pod \"139c66ae-af59-4ea8-a319-ce31cde503f4\" (UID: \"139c66ae-af59-4ea8-a319-ce31cde503f4\") " Nov 03 14:35:33.051647 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:33.051625 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/139c66ae-af59-4ea8-a319-ce31cde503f4-maas-api-openshift-ca" pod "139c66ae-af59-4ea8-a319-ce31cde503f4" (UID: "139c66ae-af59-4ea8-a319-ce31cde503f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:35:33.051765 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:33.051750 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/139c66ae-af59-4ea8-a319-ce31cde503f4-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:35:33.053535 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:33.053503 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/139c66ae-af59-4ea8-a319-ce31cde503f4-maas-api-xks-ca" pod "139c66ae-af59-4ea8-a319-ce31cde503f4" (UID: "139c66ae-af59-4ea8-a319-ce31cde503f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:35:33.053653 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:33.053506 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/139c66ae-af59-4ea8-a319-ce31cde503f4-kube-api-access-rsxmj" pod "139c66ae-af59-4ea8-a319-ce31cde503f4" (UID: "139c66ae-af59-4ea8-a319-ce31cde503f4"). InnerVolumeSpecName "kube-api-access-rsxmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:35:33.153308 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:33.153274 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/139c66ae-af59-4ea8-a319-ce31cde503f4-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:35:33.153308 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:33.153304 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsxmj\" (UniqueName: \"kubernetes.io/projected/139c66ae-af59-4ea8-a319-ce31cde503f4-kube-api-access-rsxmj\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:35:33.772349 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:33.772316 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dnvk8" event={"ID":"139c66ae-af59-4ea8-a319-ce31cde503f4","Type":"ContainerDied","Data":"d3ba5a3afa74edbd01ae84d2477360ce80e8c544dcbda625d2cd054945bba722"} Nov 03 14:35:33.772566 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:33.772356 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dnvk8" Nov 03 14:35:33.772566 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:33.772362 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3ba5a3afa74edbd01ae84d2477360ce80e8c544dcbda625d2cd054945bba722" Nov 03 14:35:36.047328 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:36.047292 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:35:37.234427 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:35:37.234385 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:35:45.030318 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:45.030280 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:35:46.416171 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:46.416124 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q"] Nov 03 14:35:46.417980 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:46.417948 5979 cpu_manager.go:414] "containerMap: removing container" podUID="139c66ae-af59-4ea8-a319-ce31cde503f4" containerName="revoke-keys" Nov 03 14:35:46.420494 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:46.418646 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="139c66ae-af59-4ea8-a319-ce31cde503f4" containerName="revoke-keys" Nov 03 14:35:46.420494 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:46.418895 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="139c66ae-af59-4ea8-a319-ce31cde503f4" containerName="revoke-keys" Nov 03 14:35:46.427934 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:46.427907 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q" Nov 03 14:35:46.491094 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:46.491050 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe445e0e-53c9-4543-9107-ff8f13cbd425-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j5s8q\" (UID: \"fe445e0e-53c9-4543-9107-ff8f13cbd425\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q" Nov 03 14:35:46.491456 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:46.491428 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe445e0e-53c9-4543-9107-ff8f13cbd425-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j5s8q\" (UID: \"fe445e0e-53c9-4543-9107-ff8f13cbd425\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q" Nov 03 14:35:46.491652 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:46.491631 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe445e0e-53c9-4543-9107-ff8f13cbd425-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j5s8q\" (UID: \"fe445e0e-53c9-4543-9107-ff8f13cbd425\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q" Nov 03 14:35:46.491777 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:46.491761 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twxnx\" (UniqueName: \"kubernetes.io/projected/fe445e0e-53c9-4543-9107-ff8f13cbd425-kube-api-access-twxnx\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j5s8q\" (UID: \"fe445e0e-53c9-4543-9107-ff8f13cbd425\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q" Nov 03 14:35:46.491923 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:46.491907 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe445e0e-53c9-4543-9107-ff8f13cbd425-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j5s8q\" (UID: \"fe445e0e-53c9-4543-9107-ff8f13cbd425\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q" Nov 03 14:35:46.492053 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:46.492036 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe445e0e-53c9-4543-9107-ff8f13cbd425-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j5s8q\" (UID: \"fe445e0e-53c9-4543-9107-ff8f13cbd425\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q" Nov 03 14:35:46.492182 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:46.492165 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe445e0e-53c9-4543-9107-ff8f13cbd425-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j5s8q\" (UID: \"fe445e0e-53c9-4543-9107-ff8f13cbd425\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q" Nov 03 14:35:46.492305 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:46.492289 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe445e0e-53c9-4543-9107-ff8f13cbd425-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j5s8q\" (UID: \"fe445e0e-53c9-4543-9107-ff8f13cbd425\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q" Nov 03 14:35:46.492420 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:46.492403 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe445e0e-53c9-4543-9107-ff8f13cbd425-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j5s8q\" (UID: \"fe445e0e-53c9-4543-9107-ff8f13cbd425\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q" Nov 03 14:35:46.519494 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:46.518577 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q"] Nov 03 14:35:46.607088 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:46.607038 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe445e0e-53c9-4543-9107-ff8f13cbd425-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j5s8q\" (UID: \"fe445e0e-53c9-4543-9107-ff8f13cbd425\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q" Nov 03 14:35:46.607493 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:46.607445 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe445e0e-53c9-4543-9107-ff8f13cbd425-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j5s8q\" (UID: \"fe445e0e-53c9-4543-9107-ff8f13cbd425\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q" Nov 03 14:35:46.607671 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:46.607650 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe445e0e-53c9-4543-9107-ff8f13cbd425-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j5s8q\" (UID: \"fe445e0e-53c9-4543-9107-ff8f13cbd425\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q" Nov 03 14:35:46.607808 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:46.607789 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twxnx\" (UniqueName: \"kubernetes.io/projected/fe445e0e-53c9-4543-9107-ff8f13cbd425-kube-api-access-twxnx\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j5s8q\" (UID: \"fe445e0e-53c9-4543-9107-ff8f13cbd425\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q" Nov 03 14:35:46.607962 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:46.607943 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe445e0e-53c9-4543-9107-ff8f13cbd425-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j5s8q\" (UID: \"fe445e0e-53c9-4543-9107-ff8f13cbd425\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q" Nov 03 14:35:46.608101 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:46.608083 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe445e0e-53c9-4543-9107-ff8f13cbd425-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j5s8q\" (UID: \"fe445e0e-53c9-4543-9107-ff8f13cbd425\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q" Nov 03 14:35:46.608233 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:46.608218 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe445e0e-53c9-4543-9107-ff8f13cbd425-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j5s8q\" (UID: \"fe445e0e-53c9-4543-9107-ff8f13cbd425\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q" Nov 03 14:35:46.608359 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:46.608344 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe445e0e-53c9-4543-9107-ff8f13cbd425-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j5s8q\" (UID: \"fe445e0e-53c9-4543-9107-ff8f13cbd425\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q" Nov 03 14:35:46.608505 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:46.608485 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe445e0e-53c9-4543-9107-ff8f13cbd425-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j5s8q\" (UID: \"fe445e0e-53c9-4543-9107-ff8f13cbd425\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q" Nov 03 14:35:46.622831 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:46.613091 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe445e0e-53c9-4543-9107-ff8f13cbd425-credential-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j5s8q\" (UID: \"fe445e0e-53c9-4543-9107-ff8f13cbd425\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q" Nov 03 14:35:46.622831 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:46.613360 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe445e0e-53c9-4543-9107-ff8f13cbd425-workload-socket\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j5s8q\" (UID: \"fe445e0e-53c9-4543-9107-ff8f13cbd425\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q" Nov 03 14:35:46.622831 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:46.614029 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe445e0e-53c9-4543-9107-ff8f13cbd425-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j5s8q\" (UID: \"fe445e0e-53c9-4543-9107-ff8f13cbd425\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q" Nov 03 14:35:46.622831 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:46.614287 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe445e0e-53c9-4543-9107-ff8f13cbd425-workload-certs\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j5s8q\" (UID: \"fe445e0e-53c9-4543-9107-ff8f13cbd425\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q" Nov 03 14:35:46.625879 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:46.625851 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe445e0e-53c9-4543-9107-ff8f13cbd425-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j5s8q\" (UID: \"fe445e0e-53c9-4543-9107-ff8f13cbd425\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q" Nov 03 14:35:46.626072 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:46.625873 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe445e0e-53c9-4543-9107-ff8f13cbd425-istio-data\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j5s8q\" (UID: \"fe445e0e-53c9-4543-9107-ff8f13cbd425\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q" Nov 03 14:35:46.626289 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:46.626006 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe445e0e-53c9-4543-9107-ff8f13cbd425-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j5s8q\" (UID: \"fe445e0e-53c9-4543-9107-ff8f13cbd425\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q" Nov 03 14:35:46.679549 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:46.673187 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twxnx\" (UniqueName: \"kubernetes.io/projected/fe445e0e-53c9-4543-9107-ff8f13cbd425-kube-api-access-twxnx\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j5s8q\" (UID: \"fe445e0e-53c9-4543-9107-ff8f13cbd425\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q" Nov 03 14:35:46.680620 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:46.674355 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe445e0e-53c9-4543-9107-ff8f13cbd425-istio-token\") pod \"maas-default-gateway-openshift-default-8b9c49cd7-j5s8q\" (UID: \"fe445e0e-53c9-4543-9107-ff8f13cbd425\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q" Nov 03 14:35:46.759715 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:46.759674 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q" Nov 03 14:35:47.317507 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:47.317385 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q"] Nov 03 14:35:47.325219 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:47.325184 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:35:47.325352 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:47.325271 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:35:47.325352 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:47.325315 5979 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 03 14:35:47.911494 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:47.910622 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q" event={"ID":"fe445e0e-53c9-4543-9107-ff8f13cbd425","Type":"ContainerStarted","Data":"1b7b795a1225e833cf0a314f08597c93f77f3d527f21ecb3bf755c6bc3a12dfa"} Nov 03 14:35:47.911494 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:47.910700 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q" event={"ID":"fe445e0e-53c9-4543-9107-ff8f13cbd425","Type":"ContainerStarted","Data":"f914c8498a19e3e68132fd65b6c5f40e0c0209b92e933e3d5768d3ee270f7b00"} Nov 03 14:35:48.032496 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:48.028207 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q" podStartSLOduration=2.028185927 podStartE2EDuration="2.028185927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:35:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:35:48.026142494 +0000 UTC m=+5430.036830739" watchObservedRunningTime="2026-11-03 14:35:48.028185927 +0000 UTC m=+5430.038874170" Nov 03 14:35:48.765438 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:48.765397 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.136:15021/healthz/ready\": dial tcp 10.134.1.136:15021: connect: connection refused" start-of-body= Nov 03 14:35:48.765765 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:48.765731 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q" podUID="fe445e0e-53c9-4543-9107-ff8f13cbd425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.136:15021/healthz/ready\": dial tcp 10.134.1.136:15021: connect: connection refused" Nov 03 14:35:48.765937 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:48.765523 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q" Nov 03 14:35:48.964495 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:48.963631 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d6flt"] Nov 03 14:35:48.971916 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:48.971885 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d6flt" Nov 03 14:35:49.018534 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:49.018449 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d6flt"] Nov 03 14:35:49.064514 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:49.063697 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:35:49.064514 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:49.063826 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:35:49.064514 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:49.063927 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:35:49.068708 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:49.068673 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wgk87\"" Nov 03 14:35:49.071440 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:49.071408 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/705a8a0d-4d1a-485d-9f82-e908515d848c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-d6flt\" (UID: \"705a8a0d-4d1a-485d-9f82-e908515d848c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d6flt" Nov 03 14:35:49.071612 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:49.071456 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/705a8a0d-4d1a-485d-9f82-e908515d848c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-d6flt\" (UID: \"705a8a0d-4d1a-485d-9f82-e908515d848c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d6flt" Nov 03 14:35:49.071695 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:49.071641 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsd7c\" (UniqueName: \"kubernetes.io/projected/705a8a0d-4d1a-485d-9f82-e908515d848c-kube-api-access-xsd7c\") pod \"maas-api-revoke-sub-0165d7d2-d6flt\" (UID: \"705a8a0d-4d1a-485d-9f82-e908515d848c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d6flt" Nov 03 14:35:49.172999 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:49.172947 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/705a8a0d-4d1a-485d-9f82-e908515d848c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-d6flt\" (UID: \"705a8a0d-4d1a-485d-9f82-e908515d848c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d6flt" Nov 03 14:35:49.174180 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:49.174153 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/705a8a0d-4d1a-485d-9f82-e908515d848c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-d6flt\" (UID: \"705a8a0d-4d1a-485d-9f82-e908515d848c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d6flt" Nov 03 14:35:49.174517 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:49.174491 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsd7c\" (UniqueName: \"kubernetes.io/projected/705a8a0d-4d1a-485d-9f82-e908515d848c-kube-api-access-xsd7c\") pod \"maas-api-revoke-sub-0165d7d2-d6flt\" (UID: \"705a8a0d-4d1a-485d-9f82-e908515d848c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d6flt" Nov 03 14:35:49.175239 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:49.175213 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/705a8a0d-4d1a-485d-9f82-e908515d848c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-d6flt\" (UID: \"705a8a0d-4d1a-485d-9f82-e908515d848c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d6flt" Nov 03 14:35:49.176089 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:49.176072 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/705a8a0d-4d1a-485d-9f82-e908515d848c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-d6flt\" (UID: \"705a8a0d-4d1a-485d-9f82-e908515d848c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d6flt" Nov 03 14:35:49.228412 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:49.228377 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsd7c\" (UniqueName: \"kubernetes.io/projected/705a8a0d-4d1a-485d-9f82-e908515d848c-kube-api-access-xsd7c\") pod \"maas-api-revoke-sub-0165d7d2-d6flt\" (UID: \"705a8a0d-4d1a-485d-9f82-e908515d848c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d6flt" Nov 03 14:35:49.290390 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:49.290308 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d6flt" Nov 03 14:35:49.679869 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:49.679831 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d6flt"] Nov 03 14:35:49.763497 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:49.762897 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.136:15021/healthz/ready\": dial tcp 10.134.1.136:15021: connect: connection refused" start-of-body= Nov 03 14:35:49.763497 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:49.762961 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q" podUID="fe445e0e-53c9-4543-9107-ff8f13cbd425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.136:15021/healthz/ready\": dial tcp 10.134.1.136:15021: connect: connection refused" Nov 03 14:35:49.921724 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:49.921683 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d6flt" event={"ID":"705a8a0d-4d1a-485d-9f82-e908515d848c","Type":"ContainerStarted","Data":"3d04799291c909b5080207ff0b0e5f8f9029e8fd60262790460b040655ed2986"} Nov 03 14:35:50.765507 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:50.763965 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.136:15021/healthz/ready\": dial tcp 10.134.1.136:15021: connect: connection refused" start-of-body= Nov 03 14:35:50.765507 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:50.764042 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q" podUID="fe445e0e-53c9-4543-9107-ff8f13cbd425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.136:15021/healthz/ready\": dial tcp 10.134.1.136:15021: connect: connection refused" Nov 03 14:35:50.928283 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:50.928245 5979 generic.go:356] "Generic (PLEG): container finished" podID="705a8a0d-4d1a-485d-9f82-e908515d848c" containerID="0db707d8847d4b5fdb0f6f5a38afcc7f979253dacf575473fd67f30df114fd83" exitCode=0 Nov 03 14:35:50.928464 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:50.928336 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d6flt" event={"ID":"705a8a0d-4d1a-485d-9f82-e908515d848c","Type":"ContainerDied","Data":"0db707d8847d4b5fdb0f6f5a38afcc7f979253dacf575473fd67f30df114fd83"} Nov 03 14:35:51.768944 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:51.768896 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.136:15021/healthz/ready\": dial tcp 10.134.1.136:15021: connect: connection refused" start-of-body= Nov 03 14:35:51.769384 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:51.768989 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q" podUID="fe445e0e-53c9-4543-9107-ff8f13cbd425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.136:15021/healthz/ready\": dial tcp 10.134.1.136:15021: connect: connection refused" Nov 03 14:35:52.118021 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:52.117997 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d6flt" Nov 03 14:35:52.205602 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:52.205568 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/705a8a0d-4d1a-485d-9f82-e908515d848c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/705a8a0d-4d1a-485d-9f82-e908515d848c-maas-api-openshift-ca\") pod \"705a8a0d-4d1a-485d-9f82-e908515d848c\" (UID: \"705a8a0d-4d1a-485d-9f82-e908515d848c\") " Nov 03 14:35:52.205602 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:52.205611 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/705a8a0d-4d1a-485d-9f82-e908515d848c-kube-api-access-xsd7c\" (UniqueName: \"kubernetes.io/projected/705a8a0d-4d1a-485d-9f82-e908515d848c-kube-api-access-xsd7c\") pod \"705a8a0d-4d1a-485d-9f82-e908515d848c\" (UID: \"705a8a0d-4d1a-485d-9f82-e908515d848c\") " Nov 03 14:35:52.205859 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:52.205691 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/705a8a0d-4d1a-485d-9f82-e908515d848c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/705a8a0d-4d1a-485d-9f82-e908515d848c-maas-api-xks-ca\") pod \"705a8a0d-4d1a-485d-9f82-e908515d848c\" (UID: \"705a8a0d-4d1a-485d-9f82-e908515d848c\") " Nov 03 14:35:52.206010 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:52.205979 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/705a8a0d-4d1a-485d-9f82-e908515d848c-maas-api-openshift-ca" pod "705a8a0d-4d1a-485d-9f82-e908515d848c" (UID: "705a8a0d-4d1a-485d-9f82-e908515d848c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:35:52.207706 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:52.207677 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/705a8a0d-4d1a-485d-9f82-e908515d848c-kube-api-access-xsd7c" pod "705a8a0d-4d1a-485d-9f82-e908515d848c" (UID: "705a8a0d-4d1a-485d-9f82-e908515d848c"). InnerVolumeSpecName "kube-api-access-xsd7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:35:52.207820 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:52.207713 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/705a8a0d-4d1a-485d-9f82-e908515d848c-maas-api-xks-ca" pod "705a8a0d-4d1a-485d-9f82-e908515d848c" (UID: "705a8a0d-4d1a-485d-9f82-e908515d848c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:35:52.307274 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:52.307234 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/705a8a0d-4d1a-485d-9f82-e908515d848c-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:35:52.307274 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:52.307266 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsd7c\" (UniqueName: \"kubernetes.io/projected/705a8a0d-4d1a-485d-9f82-e908515d848c-kube-api-access-xsd7c\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:35:52.307274 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:52.307276 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/705a8a0d-4d1a-485d-9f82-e908515d848c-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:35:52.762912 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:52.762871 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.136:15021/healthz/ready\": dial tcp 10.134.1.136:15021: connect: connection refused" start-of-body= Nov 03 14:35:52.763193 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:52.763167 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q" podUID="fe445e0e-53c9-4543-9107-ff8f13cbd425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.136:15021/healthz/ready\": dial tcp 10.134.1.136:15021: connect: connection refused" Nov 03 14:35:52.939656 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:52.939627 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d6flt" Nov 03 14:35:52.940072 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:52.939625 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d6flt" event={"ID":"705a8a0d-4d1a-485d-9f82-e908515d848c","Type":"ContainerDied","Data":"3d04799291c909b5080207ff0b0e5f8f9029e8fd60262790460b040655ed2986"} Nov 03 14:35:52.940072 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:52.939758 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d04799291c909b5080207ff0b0e5f8f9029e8fd60262790460b040655ed2986" Nov 03 14:35:53.761244 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:53.761192 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.136:15021/healthz/ready\": dial tcp 10.134.1.136:15021: connect: connection refused" start-of-body= Nov 03 14:35:53.761244 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:53.761245 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q" podUID="fe445e0e-53c9-4543-9107-ff8f13cbd425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.136:15021/healthz/ready\": dial tcp 10.134.1.136:15021: connect: connection refused" Nov 03 14:35:54.760517 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:54.760464 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.136:15021/healthz/ready\": dial tcp 10.134.1.136:15021: connect: connection refused" start-of-body= Nov 03 14:35:54.760902 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:54.760537 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q" podUID="fe445e0e-53c9-4543-9107-ff8f13cbd425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.136:15021/healthz/ready\": dial tcp 10.134.1.136:15021: connect: connection refused" Nov 03 14:35:55.234032 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:55.233990 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:35:56.761233 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:56.761193 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:35:56.761864 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:56.761261 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q" podUID="fe445e0e-53c9-4543-9107-ff8f13cbd425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:35:57.802665 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:57.802621 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:35:57.803187 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:57.802704 5979 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q" podUID="fe445e0e-53c9-4543-9107-ff8f13cbd425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:35:58.409551 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:58.409520 5979 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q" Nov 03 14:35:58.410051 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:58.410024 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q" Nov 03 14:35:58.410664 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:35:58.410643 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q" Nov 03 14:36:00.821076 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:00.821037 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8hpsc"] Nov 03 14:36:00.821558 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:00.821543 5979 cpu_manager.go:414] "containerMap: removing container" podUID="705a8a0d-4d1a-485d-9f82-e908515d848c" containerName="revoke-keys" Nov 03 14:36:00.821612 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:00.821558 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="705a8a0d-4d1a-485d-9f82-e908515d848c" containerName="revoke-keys" Nov 03 14:36:00.821671 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:00.821661 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="705a8a0d-4d1a-485d-9f82-e908515d848c" containerName="revoke-keys" Nov 03 14:36:00.824973 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:00.824952 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8hpsc" Nov 03 14:36:00.895869 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:00.895823 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dac3f92-c7ef-4844-9dc5-d3f4980f688e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-8hpsc\" (UID: \"5dac3f92-c7ef-4844-9dc5-d3f4980f688e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8hpsc" Nov 03 14:36:00.895869 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:00.895876 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s5pw\" (UniqueName: \"kubernetes.io/projected/5dac3f92-c7ef-4844-9dc5-d3f4980f688e-kube-api-access-9s5pw\") pod \"maas-api-revoke-sub-6b19338f-8hpsc\" (UID: \"5dac3f92-c7ef-4844-9dc5-d3f4980f688e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8hpsc" Nov 03 14:36:00.896139 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:00.896015 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dac3f92-c7ef-4844-9dc5-d3f4980f688e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-8hpsc\" (UID: \"5dac3f92-c7ef-4844-9dc5-d3f4980f688e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8hpsc" Nov 03 14:36:00.920838 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:00.920805 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:36:00.920838 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:00.920808 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:36:00.921110 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:00.920817 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:36:00.921110 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:00.920853 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wgk87\"" Nov 03 14:36:00.922868 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:00.922846 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8hpsc"] Nov 03 14:36:00.997006 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:00.996977 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dac3f92-c7ef-4844-9dc5-d3f4980f688e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-8hpsc\" (UID: \"5dac3f92-c7ef-4844-9dc5-d3f4980f688e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8hpsc" Nov 03 14:36:00.997184 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:00.997015 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s5pw\" (UniqueName: \"kubernetes.io/projected/5dac3f92-c7ef-4844-9dc5-d3f4980f688e-kube-api-access-9s5pw\") pod \"maas-api-revoke-sub-6b19338f-8hpsc\" (UID: \"5dac3f92-c7ef-4844-9dc5-d3f4980f688e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8hpsc" Nov 03 14:36:00.997184 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:00.997058 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dac3f92-c7ef-4844-9dc5-d3f4980f688e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-8hpsc\" (UID: \"5dac3f92-c7ef-4844-9dc5-d3f4980f688e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8hpsc" Nov 03 14:36:00.997775 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:00.997755 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dac3f92-c7ef-4844-9dc5-d3f4980f688e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-8hpsc\" (UID: \"5dac3f92-c7ef-4844-9dc5-d3f4980f688e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8hpsc" Nov 03 14:36:00.999382 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:00.999364 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dac3f92-c7ef-4844-9dc5-d3f4980f688e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-8hpsc\" (UID: \"5dac3f92-c7ef-4844-9dc5-d3f4980f688e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8hpsc" Nov 03 14:36:01.051888 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:01.051851 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s5pw\" (UniqueName: \"kubernetes.io/projected/5dac3f92-c7ef-4844-9dc5-d3f4980f688e-kube-api-access-9s5pw\") pod \"maas-api-revoke-sub-6b19338f-8hpsc\" (UID: \"5dac3f92-c7ef-4844-9dc5-d3f4980f688e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8hpsc" Nov 03 14:36:01.140059 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:01.140022 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8hpsc" Nov 03 14:36:01.508748 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:01.508657 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8hpsc"] Nov 03 14:36:01.516272 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:36:01.516247 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dac3f92_c7ef_4844_9dc5_d3f4980f688e.slice/crio-941ed750b657d6d3909a08e9e64391625a456338c0851e71dcd5866c3c4d5143 WatchSource:0}: Error finding container 941ed750b657d6d3909a08e9e64391625a456338c0851e71dcd5866c3c4d5143: Status 404 returned error can't find the container with id 941ed750b657d6d3909a08e9e64391625a456338c0851e71dcd5866c3c4d5143 Nov 03 14:36:01.994939 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:01.994901 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8hpsc" event={"ID":"5dac3f92-c7ef-4844-9dc5-d3f4980f688e","Type":"ContainerStarted","Data":"9ebaf36d8b463ed57dc7092742968e04e13643221f4a009528dadbeb6a88bcb1"} Nov 03 14:36:01.995353 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:01.994958 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8hpsc" event={"ID":"5dac3f92-c7ef-4844-9dc5-d3f4980f688e","Type":"ContainerStarted","Data":"941ed750b657d6d3909a08e9e64391625a456338c0851e71dcd5866c3c4d5143"} Nov 03 14:36:02.105230 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:02.105130 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8hpsc" podStartSLOduration=2.105115694 podStartE2EDuration="2.105115694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:36:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:36:02.103164126 +0000 UTC m=+5444.113852368" watchObservedRunningTime="2026-11-03 14:36:02.105115694 +0000 UTC m=+5444.115803973" Nov 03 14:36:03.001808 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:03.001775 5979 generic.go:356] "Generic (PLEG): container finished" podID="5dac3f92-c7ef-4844-9dc5-d3f4980f688e" containerID="9ebaf36d8b463ed57dc7092742968e04e13643221f4a009528dadbeb6a88bcb1" exitCode=0 Nov 03 14:36:03.002303 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:03.001821 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8hpsc" event={"ID":"5dac3f92-c7ef-4844-9dc5-d3f4980f688e","Type":"ContainerDied","Data":"9ebaf36d8b463ed57dc7092742968e04e13643221f4a009528dadbeb6a88bcb1"} Nov 03 14:36:04.143299 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:04.143271 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8hpsc" Nov 03 14:36:04.232371 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:04.232339 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dac3f92-c7ef-4844-9dc5-d3f4980f688e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dac3f92-c7ef-4844-9dc5-d3f4980f688e-maas-api-openshift-ca\") pod \"5dac3f92-c7ef-4844-9dc5-d3f4980f688e\" (UID: \"5dac3f92-c7ef-4844-9dc5-d3f4980f688e\") " Nov 03 14:36:04.232568 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:04.232437 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5dac3f92-c7ef-4844-9dc5-d3f4980f688e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dac3f92-c7ef-4844-9dc5-d3f4980f688e-maas-api-xks-ca\") pod \"5dac3f92-c7ef-4844-9dc5-d3f4980f688e\" (UID: \"5dac3f92-c7ef-4844-9dc5-d3f4980f688e\") " Nov 03 14:36:04.232568 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:04.232531 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dac3f92-c7ef-4844-9dc5-d3f4980f688e-kube-api-access-9s5pw\" (UniqueName: \"kubernetes.io/projected/5dac3f92-c7ef-4844-9dc5-d3f4980f688e-kube-api-access-9s5pw\") pod \"5dac3f92-c7ef-4844-9dc5-d3f4980f688e\" (UID: \"5dac3f92-c7ef-4844-9dc5-d3f4980f688e\") " Nov 03 14:36:04.232752 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:04.232728 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dac3f92-c7ef-4844-9dc5-d3f4980f688e-maas-api-openshift-ca" pod "5dac3f92-c7ef-4844-9dc5-d3f4980f688e" (UID: "5dac3f92-c7ef-4844-9dc5-d3f4980f688e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:36:04.232988 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:04.232974 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5dac3f92-c7ef-4844-9dc5-d3f4980f688e-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:36:04.234598 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:04.234574 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dac3f92-c7ef-4844-9dc5-d3f4980f688e-maas-api-xks-ca" pod "5dac3f92-c7ef-4844-9dc5-d3f4980f688e" (UID: "5dac3f92-c7ef-4844-9dc5-d3f4980f688e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:36:04.234598 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:04.234593 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dac3f92-c7ef-4844-9dc5-d3f4980f688e-kube-api-access-9s5pw" pod "5dac3f92-c7ef-4844-9dc5-d3f4980f688e" (UID: "5dac3f92-c7ef-4844-9dc5-d3f4980f688e"). InnerVolumeSpecName "kube-api-access-9s5pw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:36:04.334434 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:04.334355 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5dac3f92-c7ef-4844-9dc5-d3f4980f688e-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:36:04.334434 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:04.334383 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s5pw\" (UniqueName: \"kubernetes.io/projected/5dac3f92-c7ef-4844-9dc5-d3f4980f688e-kube-api-access-9s5pw\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:36:05.013174 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:05.013140 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8hpsc" Nov 03 14:36:05.013174 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:05.013162 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8hpsc" event={"ID":"5dac3f92-c7ef-4844-9dc5-d3f4980f688e","Type":"ContainerDied","Data":"941ed750b657d6d3909a08e9e64391625a456338c0851e71dcd5866c3c4d5143"} Nov 03 14:36:05.013449 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:05.013209 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="941ed750b657d6d3909a08e9e64391625a456338c0851e71dcd5866c3c4d5143" Nov 03 14:36:07.233781 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:36:07.233741 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:36:13.561864 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:13.561831 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:36:17.070291 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:17.070252 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-cb156fca-kserve-router-scheduler-6dd746467-g4pnh"] Nov 03 14:36:17.070976 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:17.070959 5979 cpu_manager.go:414] "containerMap: removing container" podUID="5dac3f92-c7ef-4844-9dc5-d3f4980f688e" containerName="revoke-keys" Nov 03 14:36:17.071032 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:17.070979 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dac3f92-c7ef-4844-9dc5-d3f4980f688e" containerName="revoke-keys" Nov 03 14:36:17.071092 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:17.071080 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dac3f92-c7ef-4844-9dc5-d3f4980f688e" containerName="revoke-keys" Nov 03 14:36:17.074375 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:17.074355 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-cb156fca-kserve-router-scheduler-6dd746467-g4pnh" Nov 03 14:36:17.131139 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:17.131101 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-667xm\" (UniqueName: \"kubernetes.io/projected/21b4fa89-810b-4c87-97e4-a0cc5e5afd90-kube-api-access-667xm\") pod \"e2e-prio-pool-cb156fca-kserve-router-scheduler-6dd746467-g4pnh\" (UID: \"21b4fa89-810b-4c87-97e4-a0cc5e5afd90\") " pod="llm/e2e-prio-pool-cb156fca-kserve-router-scheduler-6dd746467-g4pnh" Nov 03 14:36:17.131330 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:17.131181 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21b4fa89-810b-4c87-97e4-a0cc5e5afd90-tls-certs\") pod \"e2e-prio-pool-cb156fca-kserve-router-scheduler-6dd746467-g4pnh\" (UID: \"21b4fa89-810b-4c87-97e4-a0cc5e5afd90\") " pod="llm/e2e-prio-pool-cb156fca-kserve-router-scheduler-6dd746467-g4pnh" Nov 03 14:36:17.147687 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:17.147656 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-cb156fca-epp-sa-dockercfg-t7hmv\"" Nov 03 14:36:17.147687 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:17.147680 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-cb156fca-kserve-self-signed-certs\"" Nov 03 14:36:17.172769 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:17.172740 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-cb156fca-kserve-router-scheduler-6dd746467-g4pnh"] Nov 03 14:36:17.232533 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:17.232463 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-667xm\" (UniqueName: \"kubernetes.io/projected/21b4fa89-810b-4c87-97e4-a0cc5e5afd90-kube-api-access-667xm\") pod \"e2e-prio-pool-cb156fca-kserve-router-scheduler-6dd746467-g4pnh\" (UID: \"21b4fa89-810b-4c87-97e4-a0cc5e5afd90\") " pod="llm/e2e-prio-pool-cb156fca-kserve-router-scheduler-6dd746467-g4pnh" Nov 03 14:36:17.232872 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:17.232620 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21b4fa89-810b-4c87-97e4-a0cc5e5afd90-tls-certs\") pod \"e2e-prio-pool-cb156fca-kserve-router-scheduler-6dd746467-g4pnh\" (UID: \"21b4fa89-810b-4c87-97e4-a0cc5e5afd90\") " pod="llm/e2e-prio-pool-cb156fca-kserve-router-scheduler-6dd746467-g4pnh" Nov 03 14:36:17.246968 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:17.246926 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21b4fa89-810b-4c87-97e4-a0cc5e5afd90-tls-certs\") pod \"e2e-prio-pool-cb156fca-kserve-router-scheduler-6dd746467-g4pnh\" (UID: \"21b4fa89-810b-4c87-97e4-a0cc5e5afd90\") " pod="llm/e2e-prio-pool-cb156fca-kserve-router-scheduler-6dd746467-g4pnh" Nov 03 14:36:17.288244 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:17.288208 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-667xm\" (UniqueName: \"kubernetes.io/projected/21b4fa89-810b-4c87-97e4-a0cc5e5afd90-kube-api-access-667xm\") pod \"e2e-prio-pool-cb156fca-kserve-router-scheduler-6dd746467-g4pnh\" (UID: \"21b4fa89-810b-4c87-97e4-a0cc5e5afd90\") " pod="llm/e2e-prio-pool-cb156fca-kserve-router-scheduler-6dd746467-g4pnh" Nov 03 14:36:17.387267 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:17.387237 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-cb156fca-kserve-router-scheduler-6dd746467-g4pnh" Nov 03 14:36:17.761372 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:17.761285 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-cb156fca-kserve-router-scheduler-6dd746467-g4pnh"] Nov 03 14:36:17.768992 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:36:17.768966 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21b4fa89_810b_4c87_97e4_a0cc5e5afd90.slice/crio-cff9c3eb52cc2e462d510855accb9db04be06a4b06e997e3750abad70273a672 WatchSource:0}: Error finding container cff9c3eb52cc2e462d510855accb9db04be06a4b06e997e3750abad70273a672: Status 404 returned error can't find the container with id cff9c3eb52cc2e462d510855accb9db04be06a4b06e997e3750abad70273a672 Nov 03 14:36:18.115448 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:18.115364 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-cb156fca-kserve-router-scheduler-6dd746467-g4pnh" event={"ID":"21b4fa89-810b-4c87-97e4-a0cc5e5afd90","Type":"ContainerStarted","Data":"cff9c3eb52cc2e462d510855accb9db04be06a4b06e997e3750abad70273a672"} Nov 03 14:36:20.129527 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:20.129407 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-cb156fca-kserve-router-scheduler-6dd746467-g4pnh" event={"ID":"21b4fa89-810b-4c87-97e4-a0cc5e5afd90","Type":"ContainerStarted","Data":"98e1163a715bb90b2bf73587a9990b76f8c34bb00741154f6081a3198c10d1cb"} Nov 03 14:36:20.129901 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:20.129583 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-cb156fca-kserve-router-scheduler-6dd746467-g4pnh" Nov 03 14:36:20.237867 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:20.237816 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-cb156fca-kserve-router-scheduler-6dd746467-g4pnh" podStartSLOduration=1.220954777 podStartE2EDuration="3.237803593s" totalImagesPullingTime="2.016848816s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:36:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:36:20.236577736 +0000 UTC m=+5462.247265975" watchObservedRunningTime="2026-11-03 14:36:20.237803593 +0000 UTC m=+5462.248491836" Nov 03 14:36:21.817357 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:21.817315 5979 scope.go:122] "RemoveContainer" containerID="1fa37934ff204b2bfcda7431e40b2f793c44579c54b68473d0ea7d6fb71630a0" Nov 03 14:36:21.881734 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:21.881690 5979 scope.go:122] "RemoveContainer" containerID="58a47de9fc5ea66812ade3ab8bf57b07b36b89e7f464c1fbbeb88ba77092f029" Nov 03 14:36:21.945959 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:21.945920 5979 scope.go:122] "RemoveContainer" containerID="7486f3f6396e27c5c863fb860ca99043da5eb539411b09e5ca3abbfb00acea5b" Nov 03 14:36:22.016494 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:22.013682 5979 scope.go:122] "RemoveContainer" containerID="3bfe311038d0eaecbc8ce66e3024b3833c4ee147728d54d8e68f7e57c9a392bf" Nov 03 14:36:22.062863 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:22.062832 5979 scope.go:122] "RemoveContainer" containerID="6e77c5d70e66e509f48ef38c56a18862088221cd413b20a3ded830f39f04c78d" Nov 03 14:36:26.988977 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:26.988937 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:36:37.234088 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:36:37.234043 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:36:49.043059 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:49.043021 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83dbd046-h65vn"] Nov 03 14:36:49.089568 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:49.089533 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83dbd046-h65vn"] Nov 03 14:36:50.114410 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:50.114376 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="712c3ae4-3332-4730-9553-d21963331b11" path="/var/lib/kubelet/pods/712c3ae4-3332-4730-9553-d21963331b11/volumes" Nov 03 14:36:51.140367 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:51.140334 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-cb156fca-kserve-router-scheduler-6dd746467-g4pnh" Nov 03 14:36:59.650130 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:36:59.650050 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:37:01.071495 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:37:01.063866 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58eb825e-t57b7"] Nov 03 14:37:01.112495 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:37:01.111702 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58eb825e-t57b7"] Nov 03 14:37:02.136627 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:37:02.136593 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92df4ddd-e105-482a-9a46-abf87a7c4033" path="/var/lib/kubelet/pods/92df4ddd-e105-482a-9a46-abf87a7c4033/volumes" Nov 03 14:37:07.233784 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:37:07.233745 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:37:26.082854 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:37:26.082821 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:37:26.421974 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:37:26.421937 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-cb156fca-kserve-router-scheduler-6dd746467-g4pnh"] Nov 03 14:37:31.140346 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:37:31.140304 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-cb156fca-kserve-router-scheduler-6dd746467-g4pnh" podUID="21b4fa89-810b-4c87-97e4-a0cc5e5afd90" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:37:37.234183 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:37:37.234147 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:37:38.629584 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:37:38.629548 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:37:41.140278 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:37:41.140236 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-cb156fca-kserve-router-scheduler-6dd746467-g4pnh" podUID="21b4fa89-810b-4c87-97e4-a0cc5e5afd90" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:37:41.443278 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:37:41.443188 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-cb156fca-kserve-router-scheduler-6dd746467-g4pnh" podUID="21b4fa89-810b-4c87-97e4-a0cc5e5afd90" containerName="main" containerID="cri-o://98e1163a715bb90b2bf73587a9990b76f8c34bb00741154f6081a3198c10d1cb" gracePeriod=45 Nov 03 14:37:51.140836 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:37:51.140798 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-cb156fca-kserve-router-scheduler-6dd746467-g4pnh" podUID="21b4fa89-810b-4c87-97e4-a0cc5e5afd90" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:37:51.141258 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:37:51.140906 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-cb156fca-kserve-router-scheduler-6dd746467-g4pnh" Nov 03 14:38:01.140929 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:38:01.140891 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-cb156fca-kserve-router-scheduler-6dd746467-g4pnh" podUID="21b4fa89-810b-4c87-97e4-a0cc5e5afd90" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:38:04.384262 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:38:04.384225 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:38:07.257273 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:38:07.257241 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:38:11.140844 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:38:11.140810 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-cb156fca-kserve-router-scheduler-6dd746467-g4pnh" podUID="21b4fa89-810b-4c87-97e4-a0cc5e5afd90" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:38:11.739926 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:38:11.739899 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-cb156fca-kserve-router-scheduler-6dd746467-g4pnh" Nov 03 14:38:11.843576 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:38:11.843533 5979 generic.go:356] "Generic (PLEG): container finished" podID="21b4fa89-810b-4c87-97e4-a0cc5e5afd90" containerID="98e1163a715bb90b2bf73587a9990b76f8c34bb00741154f6081a3198c10d1cb" exitCode=0 Nov 03 14:38:11.843576 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:38:11.843565 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-cb156fca-kserve-router-scheduler-6dd746467-g4pnh" event={"ID":"21b4fa89-810b-4c87-97e4-a0cc5e5afd90","Type":"ContainerDied","Data":"98e1163a715bb90b2bf73587a9990b76f8c34bb00741154f6081a3198c10d1cb"} Nov 03 14:38:11.843809 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:38:11.843586 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-cb156fca-kserve-router-scheduler-6dd746467-g4pnh" Nov 03 14:38:11.843809 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:38:11.843626 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-cb156fca-kserve-router-scheduler-6dd746467-g4pnh" event={"ID":"21b4fa89-810b-4c87-97e4-a0cc5e5afd90","Type":"ContainerDied","Data":"cff9c3eb52cc2e462d510855accb9db04be06a4b06e997e3750abad70273a672"} Nov 03 14:38:11.843809 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:38:11.843651 5979 scope.go:122] "RemoveContainer" containerID="98e1163a715bb90b2bf73587a9990b76f8c34bb00741154f6081a3198c10d1cb" Nov 03 14:38:11.850773 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:38:11.850740 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21b4fa89-810b-4c87-97e4-a0cc5e5afd90-kube-api-access-667xm\" (UniqueName: \"kubernetes.io/projected/21b4fa89-810b-4c87-97e4-a0cc5e5afd90-kube-api-access-667xm\") pod \"21b4fa89-810b-4c87-97e4-a0cc5e5afd90\" (UID: \"21b4fa89-810b-4c87-97e4-a0cc5e5afd90\") " Nov 03 14:38:11.851016 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:38:11.850949 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21b4fa89-810b-4c87-97e4-a0cc5e5afd90-tls-certs\" (UniqueName: \"kubernetes.io/secret/21b4fa89-810b-4c87-97e4-a0cc5e5afd90-tls-certs\") pod \"21b4fa89-810b-4c87-97e4-a0cc5e5afd90\" (UID: \"21b4fa89-810b-4c87-97e4-a0cc5e5afd90\") " Nov 03 14:38:11.853045 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:38:11.853016 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21b4fa89-810b-4c87-97e4-a0cc5e5afd90-tls-certs" pod "21b4fa89-810b-4c87-97e4-a0cc5e5afd90" (UID: "21b4fa89-810b-4c87-97e4-a0cc5e5afd90"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:38:11.853116 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:38:11.853049 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21b4fa89-810b-4c87-97e4-a0cc5e5afd90-kube-api-access-667xm" pod "21b4fa89-810b-4c87-97e4-a0cc5e5afd90" (UID: "21b4fa89-810b-4c87-97e4-a0cc5e5afd90"). InnerVolumeSpecName "kube-api-access-667xm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:38:11.853116 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:38:11.853061 5979 scope.go:122] "RemoveContainer" containerID="98e1163a715bb90b2bf73587a9990b76f8c34bb00741154f6081a3198c10d1cb" Nov 03 14:38:11.853382 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:38:11.853359 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98e1163a715bb90b2bf73587a9990b76f8c34bb00741154f6081a3198c10d1cb\": container with ID starting with 98e1163a715bb90b2bf73587a9990b76f8c34bb00741154f6081a3198c10d1cb not found: ID does not exist" containerID="98e1163a715bb90b2bf73587a9990b76f8c34bb00741154f6081a3198c10d1cb" Nov 03 14:38:11.853423 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:38:11.853388 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98e1163a715bb90b2bf73587a9990b76f8c34bb00741154f6081a3198c10d1cb"} err="failed to get container status \"98e1163a715bb90b2bf73587a9990b76f8c34bb00741154f6081a3198c10d1cb\": rpc error: code = NotFound desc = could not find container \"98e1163a715bb90b2bf73587a9990b76f8c34bb00741154f6081a3198c10d1cb\": container with ID starting with 98e1163a715bb90b2bf73587a9990b76f8c34bb00741154f6081a3198c10d1cb not found: ID does not exist" Nov 03 14:38:11.952159 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:38:11.952126 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-667xm\" (UniqueName: \"kubernetes.io/projected/21b4fa89-810b-4c87-97e4-a0cc5e5afd90-kube-api-access-667xm\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:38:11.952159 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:38:11.952157 5979 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21b4fa89-810b-4c87-97e4-a0cc5e5afd90-tls-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:38:12.266823 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:38:12.266789 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-cb156fca-kserve-router-scheduler-6dd746467-g4pnh"] Nov 03 14:38:12.313033 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:38:12.313002 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-cb156fca-kserve-router-scheduler-6dd746467-g4pnh"] Nov 03 14:38:14.114407 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:38:14.114371 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21b4fa89-810b-4c87-97e4-a0cc5e5afd90" path="/var/lib/kubelet/pods/21b4fa89-810b-4c87-97e4-a0cc5e5afd90/volumes" Nov 03 14:38:22.433138 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:38:22.433105 5979 scope.go:122] "RemoveContainer" containerID="6fa41ed68ad8fa334f4d28ed7eda04a3caff41adf7d2a752bdd5a0c739613506" Nov 03 14:38:22.441446 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:38:22.441426 5979 scope.go:122] "RemoveContainer" containerID="aff677a73e05914bf046b10107414425c0eddce8da9389984f46ac86a521da64" Nov 03 14:38:22.979302 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:38:22.979270 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:38:37.233786 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:38:37.233745 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:38:48.155085 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:38:48.155053 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:39:03.219257 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:39:03.219217 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-626c17cb-th2l5"] Nov 03 14:39:03.219903 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:39:03.219881 5979 cpu_manager.go:414] "containerMap: removing container" podUID="21b4fa89-810b-4c87-97e4-a0cc5e5afd90" containerName="main" Nov 03 14:39:03.220008 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:39:03.219905 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21b4fa89-810b-4c87-97e4-a0cc5e5afd90" containerName="main" Nov 03 14:39:03.220073 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:39:03.220012 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="21b4fa89-810b-4c87-97e4-a0cc5e5afd90" containerName="main" Nov 03 14:39:03.223488 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:39:03.223456 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-626c17cb-th2l5" Nov 03 14:39:03.272639 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:39:03.272600 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:39:03.272828 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:39:03.272638 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:39:03.292267 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:39:03.292233 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/673fcc1f-a5c3-4ea3-a87e-82357a1f2711-maas-api-xks-ca\") pod \"maas-api-revoke-sub-626c17cb-th2l5\" (UID: \"673fcc1f-a5c3-4ea3-a87e-82357a1f2711\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-626c17cb-th2l5" Nov 03 14:39:03.292496 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:39:03.292448 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5gk8\" (UniqueName: \"kubernetes.io/projected/673fcc1f-a5c3-4ea3-a87e-82357a1f2711-kube-api-access-x5gk8\") pod \"maas-api-revoke-sub-626c17cb-th2l5\" (UID: \"673fcc1f-a5c3-4ea3-a87e-82357a1f2711\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-626c17cb-th2l5" Nov 03 14:39:03.292574 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:39:03.292551 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/673fcc1f-a5c3-4ea3-a87e-82357a1f2711-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-626c17cb-th2l5\" (UID: \"673fcc1f-a5c3-4ea3-a87e-82357a1f2711\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-626c17cb-th2l5" Nov 03 14:39:03.319173 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:39:03.319132 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:39:03.319173 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:39:03.319170 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wgk87\"" Nov 03 14:39:03.321223 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:39:03.321201 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-626c17cb-th2l5"] Nov 03 14:39:03.393250 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:39:03.393212 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5gk8\" (UniqueName: \"kubernetes.io/projected/673fcc1f-a5c3-4ea3-a87e-82357a1f2711-kube-api-access-x5gk8\") pod \"maas-api-revoke-sub-626c17cb-th2l5\" (UID: \"673fcc1f-a5c3-4ea3-a87e-82357a1f2711\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-626c17cb-th2l5" Nov 03 14:39:03.393441 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:39:03.393259 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/673fcc1f-a5c3-4ea3-a87e-82357a1f2711-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-626c17cb-th2l5\" (UID: \"673fcc1f-a5c3-4ea3-a87e-82357a1f2711\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-626c17cb-th2l5" Nov 03 14:39:03.393526 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:39:03.393438 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/673fcc1f-a5c3-4ea3-a87e-82357a1f2711-maas-api-xks-ca\") pod \"maas-api-revoke-sub-626c17cb-th2l5\" (UID: \"673fcc1f-a5c3-4ea3-a87e-82357a1f2711\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-626c17cb-th2l5" Nov 03 14:39:03.393925 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:39:03.393908 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/673fcc1f-a5c3-4ea3-a87e-82357a1f2711-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-626c17cb-th2l5\" (UID: \"673fcc1f-a5c3-4ea3-a87e-82357a1f2711\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-626c17cb-th2l5" Nov 03 14:39:03.395709 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:39:03.395685 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/673fcc1f-a5c3-4ea3-a87e-82357a1f2711-maas-api-xks-ca\") pod \"maas-api-revoke-sub-626c17cb-th2l5\" (UID: \"673fcc1f-a5c3-4ea3-a87e-82357a1f2711\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-626c17cb-th2l5" Nov 03 14:39:03.447305 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:39:03.447274 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5gk8\" (UniqueName: \"kubernetes.io/projected/673fcc1f-a5c3-4ea3-a87e-82357a1f2711-kube-api-access-x5gk8\") pod \"maas-api-revoke-sub-626c17cb-th2l5\" (UID: \"673fcc1f-a5c3-4ea3-a87e-82357a1f2711\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-626c17cb-th2l5" Nov 03 14:39:03.536733 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:39:03.536652 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-626c17cb-th2l5" Nov 03 14:39:03.906555 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:39:03.906517 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-626c17cb-th2l5"] Nov 03 14:39:03.914181 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:39:03.914145 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod673fcc1f_a5c3_4ea3_a87e_82357a1f2711.slice/crio-3a8dae8dbc59210197f09790b10fc1190735309c326268ac806406cf75445b17 WatchSource:0}: Error finding container 3a8dae8dbc59210197f09790b10fc1190735309c326268ac806406cf75445b17: Status 404 returned error can't find the container with id 3a8dae8dbc59210197f09790b10fc1190735309c326268ac806406cf75445b17 Nov 03 14:39:04.117401 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:39:04.117365 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-626c17cb-th2l5" event={"ID":"673fcc1f-a5c3-4ea3-a87e-82357a1f2711","Type":"ContainerStarted","Data":"3a8dae8dbc59210197f09790b10fc1190735309c326268ac806406cf75445b17"} Nov 03 14:39:05.124152 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:39:05.124054 5979 generic.go:356] "Generic (PLEG): container finished" podID="673fcc1f-a5c3-4ea3-a87e-82357a1f2711" containerID="a53f40e2f671ac1a63c13fa708b01c26afb76d1b4986d42d46f7560bda6f3ada" exitCode=0 Nov 03 14:39:05.124621 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:39:05.124144 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-626c17cb-th2l5" event={"ID":"673fcc1f-a5c3-4ea3-a87e-82357a1f2711","Type":"ContainerDied","Data":"a53f40e2f671ac1a63c13fa708b01c26afb76d1b4986d42d46f7560bda6f3ada"} Nov 03 14:39:06.257740 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:39:06.257713 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-626c17cb-th2l5" Nov 03 14:39:06.325535 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:39:06.325504 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/673fcc1f-a5c3-4ea3-a87e-82357a1f2711-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/673fcc1f-a5c3-4ea3-a87e-82357a1f2711-maas-api-xks-ca\") pod \"673fcc1f-a5c3-4ea3-a87e-82357a1f2711\" (UID: \"673fcc1f-a5c3-4ea3-a87e-82357a1f2711\") " Nov 03 14:39:06.325726 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:39:06.325646 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/673fcc1f-a5c3-4ea3-a87e-82357a1f2711-kube-api-access-x5gk8\" (UniqueName: \"kubernetes.io/projected/673fcc1f-a5c3-4ea3-a87e-82357a1f2711-kube-api-access-x5gk8\") pod \"673fcc1f-a5c3-4ea3-a87e-82357a1f2711\" (UID: \"673fcc1f-a5c3-4ea3-a87e-82357a1f2711\") " Nov 03 14:39:06.325726 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:39:06.325681 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/673fcc1f-a5c3-4ea3-a87e-82357a1f2711-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/673fcc1f-a5c3-4ea3-a87e-82357a1f2711-maas-api-openshift-ca\") pod \"673fcc1f-a5c3-4ea3-a87e-82357a1f2711\" (UID: \"673fcc1f-a5c3-4ea3-a87e-82357a1f2711\") " Nov 03 14:39:06.326128 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:39:06.326101 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/673fcc1f-a5c3-4ea3-a87e-82357a1f2711-maas-api-openshift-ca" pod "673fcc1f-a5c3-4ea3-a87e-82357a1f2711" (UID: "673fcc1f-a5c3-4ea3-a87e-82357a1f2711"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:39:06.327761 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:39:06.327733 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/673fcc1f-a5c3-4ea3-a87e-82357a1f2711-maas-api-xks-ca" pod "673fcc1f-a5c3-4ea3-a87e-82357a1f2711" (UID: "673fcc1f-a5c3-4ea3-a87e-82357a1f2711"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:39:06.327870 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:39:06.327780 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/673fcc1f-a5c3-4ea3-a87e-82357a1f2711-kube-api-access-x5gk8" pod "673fcc1f-a5c3-4ea3-a87e-82357a1f2711" (UID: "673fcc1f-a5c3-4ea3-a87e-82357a1f2711"). InnerVolumeSpecName "kube-api-access-x5gk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:39:06.427282 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:39:06.427244 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/673fcc1f-a5c3-4ea3-a87e-82357a1f2711-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:39:06.427282 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:39:06.427276 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/673fcc1f-a5c3-4ea3-a87e-82357a1f2711-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:39:06.427282 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:39:06.427286 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5gk8\" (UniqueName: \"kubernetes.io/projected/673fcc1f-a5c3-4ea3-a87e-82357a1f2711-kube-api-access-x5gk8\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:39:07.134778 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:39:07.134744 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-626c17cb-th2l5" Nov 03 14:39:07.134963 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:39:07.134773 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-626c17cb-th2l5" event={"ID":"673fcc1f-a5c3-4ea3-a87e-82357a1f2711","Type":"ContainerDied","Data":"3a8dae8dbc59210197f09790b10fc1190735309c326268ac806406cf75445b17"} Nov 03 14:39:07.134963 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:39:07.134821 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a8dae8dbc59210197f09790b10fc1190735309c326268ac806406cf75445b17" Nov 03 14:39:07.233826 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:39:07.233788 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:39:15.045771 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:39:15.045733 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mxdbm"] Nov 03 14:39:15.092310 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:39:15.092268 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mxdbm"] Nov 03 14:39:16.114155 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:39:16.114114 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbf4bb3f-88b7-449e-8d36-cd892e49bd86" path="/var/lib/kubelet/pods/fbf4bb3f-88b7-449e-8d36-cd892e49bd86/volumes" Nov 03 14:39:16.240030 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:39:16.239989 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:39:34.351831 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:39:34.351795 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:39:37.233720 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:39:37.233676 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:40:00.047427 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:40:00.047349 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-4blnb"] Nov 03 14:40:00.093956 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:40:00.093921 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-4blnb"] Nov 03 14:40:00.114110 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:40:00.114072 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae1ea258-2ad6-455c-81aa-86828bb969c9" path="/var/lib/kubelet/pods/ae1ea258-2ad6-455c-81aa-86828bb969c9/volumes" Nov 03 14:40:07.234322 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:40:07.234280 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:40:12.746981 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:40:12.746947 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:40:22.475571 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:40:22.475538 5979 scope.go:122] "RemoveContainer" containerID="933fa2d857a1ac4c9f703edb5c146419094b0fb834e43c63cc3ac10103abc8cb" Nov 03 14:40:34.045162 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:40:34.045119 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dnvk8"] Nov 03 14:40:34.091413 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:40:34.091377 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-dnvk8"] Nov 03 14:40:34.114249 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:40:34.114204 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="139c66ae-af59-4ea8-a319-ce31cde503f4" path="/var/lib/kubelet/pods/139c66ae-af59-4ea8-a319-ce31cde503f4/volumes" Nov 03 14:40:37.234215 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:40:37.234176 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:40:39.376119 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:40:39.376083 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:40:53.049092 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:40:53.049055 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d6flt"] Nov 03 14:40:53.095393 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:40:53.095356 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-d6flt"] Nov 03 14:40:54.114318 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:40:54.114280 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="705a8a0d-4d1a-485d-9f82-e908515d848c" path="/var/lib/kubelet/pods/705a8a0d-4d1a-485d-9f82-e908515d848c/volumes" Nov 03 14:41:05.044879 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:05.044843 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8hpsc"] Nov 03 14:41:05.091265 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:05.091228 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-8hpsc"] Nov 03 14:41:06.114160 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:06.114124 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dac3f92-c7ef-4844-9dc5-d3f4980f688e" path="/var/lib/kubelet/pods/5dac3f92-c7ef-4844-9dc5-d3f4980f688e/volumes" Nov 03 14:41:07.234199 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:41:07.234161 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:41:11.377803 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:11.377761 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sgx99"] Nov 03 14:41:11.378231 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:11.378181 5979 cpu_manager.go:414] "containerMap: removing container" podUID="673fcc1f-a5c3-4ea3-a87e-82357a1f2711" containerName="revoke-keys" Nov 03 14:41:11.378231 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:11.378191 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="673fcc1f-a5c3-4ea3-a87e-82357a1f2711" containerName="revoke-keys" Nov 03 14:41:11.378328 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:11.378279 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="673fcc1f-a5c3-4ea3-a87e-82357a1f2711" containerName="revoke-keys" Nov 03 14:41:11.381424 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:11.381373 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sgx99" Nov 03 14:41:11.433634 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:11.433600 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:41:11.433840 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:11.433685 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:41:11.433840 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:11.433762 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wgk87\"" Nov 03 14:41:11.480382 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:11.480339 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:41:11.482167 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:11.482140 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sgx99"] Nov 03 14:41:11.526298 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:11.526266 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9hw9\" (UniqueName: \"kubernetes.io/projected/74fcaea3-7dd0-47e8-88b1-995b55f49e02-kube-api-access-h9hw9\") pod \"maas-api-revoke-sub-452b6d7b-sgx99\" (UID: \"74fcaea3-7dd0-47e8-88b1-995b55f49e02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sgx99" Nov 03 14:41:11.526450 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:11.526307 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74fcaea3-7dd0-47e8-88b1-995b55f49e02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-sgx99\" (UID: \"74fcaea3-7dd0-47e8-88b1-995b55f49e02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sgx99" Nov 03 14:41:11.526548 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:11.526525 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74fcaea3-7dd0-47e8-88b1-995b55f49e02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-sgx99\" (UID: \"74fcaea3-7dd0-47e8-88b1-995b55f49e02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sgx99" Nov 03 14:41:11.627323 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:11.627281 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74fcaea3-7dd0-47e8-88b1-995b55f49e02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-sgx99\" (UID: \"74fcaea3-7dd0-47e8-88b1-995b55f49e02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sgx99" Nov 03 14:41:11.627569 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:11.627393 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9hw9\" (UniqueName: \"kubernetes.io/projected/74fcaea3-7dd0-47e8-88b1-995b55f49e02-kube-api-access-h9hw9\") pod \"maas-api-revoke-sub-452b6d7b-sgx99\" (UID: \"74fcaea3-7dd0-47e8-88b1-995b55f49e02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sgx99" Nov 03 14:41:11.627569 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:11.627419 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74fcaea3-7dd0-47e8-88b1-995b55f49e02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-sgx99\" (UID: \"74fcaea3-7dd0-47e8-88b1-995b55f49e02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sgx99" Nov 03 14:41:11.628192 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:11.628132 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74fcaea3-7dd0-47e8-88b1-995b55f49e02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-sgx99\" (UID: \"74fcaea3-7dd0-47e8-88b1-995b55f49e02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sgx99" Nov 03 14:41:11.629778 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:11.629756 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74fcaea3-7dd0-47e8-88b1-995b55f49e02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-sgx99\" (UID: \"74fcaea3-7dd0-47e8-88b1-995b55f49e02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sgx99" Nov 03 14:41:11.682216 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:11.682181 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9hw9\" (UniqueName: \"kubernetes.io/projected/74fcaea3-7dd0-47e8-88b1-995b55f49e02-kube-api-access-h9hw9\") pod \"maas-api-revoke-sub-452b6d7b-sgx99\" (UID: \"74fcaea3-7dd0-47e8-88b1-995b55f49e02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sgx99" Nov 03 14:41:11.694811 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:11.694779 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sgx99" Nov 03 14:41:12.064389 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:12.064356 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sgx99"] Nov 03 14:41:12.072849 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:41:12.072819 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74fcaea3_7dd0_47e8_88b1_995b55f49e02.slice/crio-5312b190d94452e6e2da7baa8d8ccb11d10bc8d74703434c1ffef773e6abcf2a WatchSource:0}: Error finding container 5312b190d94452e6e2da7baa8d8ccb11d10bc8d74703434c1ffef773e6abcf2a: Status 404 returned error can't find the container with id 5312b190d94452e6e2da7baa8d8ccb11d10bc8d74703434c1ffef773e6abcf2a Nov 03 14:41:12.643484 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:12.643447 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:41:12.849903 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:12.849861 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sgx99" event={"ID":"74fcaea3-7dd0-47e8-88b1-995b55f49e02","Type":"ContainerStarted","Data":"7ce7ca58326e1f9d1a17e3d043558ef2c0ec484cdbedefdb31ef3b987f36397a"} Nov 03 14:41:12.850090 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:12.849932 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sgx99" event={"ID":"74fcaea3-7dd0-47e8-88b1-995b55f49e02","Type":"ContainerStarted","Data":"5312b190d94452e6e2da7baa8d8ccb11d10bc8d74703434c1ffef773e6abcf2a"} Nov 03 14:41:12.960149 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:12.960089 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sgx99" podStartSLOduration=1.9600706730000002 podStartE2EDuration="1.960070673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:41:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:41:12.958244292 +0000 UTC m=+5754.968932565" watchObservedRunningTime="2026-11-03 14:41:12.960070673 +0000 UTC m=+5754.970758918" Nov 03 14:41:13.864924 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:13.864882 5979 generic.go:356] "Generic (PLEG): container finished" podID="74fcaea3-7dd0-47e8-88b1-995b55f49e02" containerID="7ce7ca58326e1f9d1a17e3d043558ef2c0ec484cdbedefdb31ef3b987f36397a" exitCode=0 Nov 03 14:41:13.865502 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:13.865004 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sgx99" event={"ID":"74fcaea3-7dd0-47e8-88b1-995b55f49e02","Type":"ContainerDied","Data":"7ce7ca58326e1f9d1a17e3d043558ef2c0ec484cdbedefdb31ef3b987f36397a"} Nov 03 14:41:15.613675 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:15.613643 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sgx99" Nov 03 14:41:15.666501 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:15.653008 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74fcaea3-7dd0-47e8-88b1-995b55f49e02-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74fcaea3-7dd0-47e8-88b1-995b55f49e02-maas-api-xks-ca\") pod \"74fcaea3-7dd0-47e8-88b1-995b55f49e02\" (UID: \"74fcaea3-7dd0-47e8-88b1-995b55f49e02\") " Nov 03 14:41:15.666501 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:15.653074 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74fcaea3-7dd0-47e8-88b1-995b55f49e02-kube-api-access-h9hw9\" (UniqueName: \"kubernetes.io/projected/74fcaea3-7dd0-47e8-88b1-995b55f49e02-kube-api-access-h9hw9\") pod \"74fcaea3-7dd0-47e8-88b1-995b55f49e02\" (UID: \"74fcaea3-7dd0-47e8-88b1-995b55f49e02\") " Nov 03 14:41:15.666501 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:15.653146 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74fcaea3-7dd0-47e8-88b1-995b55f49e02-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74fcaea3-7dd0-47e8-88b1-995b55f49e02-maas-api-openshift-ca\") pod \"74fcaea3-7dd0-47e8-88b1-995b55f49e02\" (UID: \"74fcaea3-7dd0-47e8-88b1-995b55f49e02\") " Nov 03 14:41:15.666501 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:15.657665 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74fcaea3-7dd0-47e8-88b1-995b55f49e02-maas-api-openshift-ca" pod "74fcaea3-7dd0-47e8-88b1-995b55f49e02" (UID: "74fcaea3-7dd0-47e8-88b1-995b55f49e02"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:41:15.666501 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:15.661910 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74fcaea3-7dd0-47e8-88b1-995b55f49e02-kube-api-access-h9hw9" pod "74fcaea3-7dd0-47e8-88b1-995b55f49e02" (UID: "74fcaea3-7dd0-47e8-88b1-995b55f49e02"). InnerVolumeSpecName "kube-api-access-h9hw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:41:15.678924 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:15.678883 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74fcaea3-7dd0-47e8-88b1-995b55f49e02-maas-api-xks-ca" pod "74fcaea3-7dd0-47e8-88b1-995b55f49e02" (UID: "74fcaea3-7dd0-47e8-88b1-995b55f49e02"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:41:15.773677 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:15.773561 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9hw9\" (UniqueName: \"kubernetes.io/projected/74fcaea3-7dd0-47e8-88b1-995b55f49e02-kube-api-access-h9hw9\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:41:15.773924 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:15.773902 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74fcaea3-7dd0-47e8-88b1-995b55f49e02-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:41:15.774076 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:15.774060 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74fcaea3-7dd0-47e8-88b1-995b55f49e02-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:41:15.940851 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:15.940812 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sgx99" event={"ID":"74fcaea3-7dd0-47e8-88b1-995b55f49e02","Type":"ContainerDied","Data":"5312b190d94452e6e2da7baa8d8ccb11d10bc8d74703434c1ffef773e6abcf2a"} Nov 03 14:41:15.941128 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:15.941110 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5312b190d94452e6e2da7baa8d8ccb11d10bc8d74703434c1ffef773e6abcf2a" Nov 03 14:41:15.941312 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:15.941296 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sgx99" Nov 03 14:41:22.495861 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:22.495819 5979 scope.go:122] "RemoveContainer" containerID="b7a5745868fdf8dfe19129b166526b58e9411b04ce5e7afcd1f2c8971d10dab3" Nov 03 14:41:27.861856 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:27.861818 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:41:37.233663 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:41:37.233623 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:41:42.361202 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:42.360204 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:41:42.498060 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:42.498022 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-j4png"] Nov 03 14:41:42.498461 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:42.498449 5979 cpu_manager.go:414] "containerMap: removing container" podUID="74fcaea3-7dd0-47e8-88b1-995b55f49e02" containerName="revoke-keys" Nov 03 14:41:42.498526 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:42.498462 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74fcaea3-7dd0-47e8-88b1-995b55f49e02" containerName="revoke-keys" Nov 03 14:41:42.498568 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:42.498555 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="74fcaea3-7dd0-47e8-88b1-995b55f49e02" containerName="revoke-keys" Nov 03 14:41:42.501818 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:42.501784 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-j4png" Nov 03 14:41:42.555922 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:42.555887 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1305c97a-0ac3-4f7a-9735-d78406162d1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-j4png\" (UID: \"1305c97a-0ac3-4f7a-9735-d78406162d1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-j4png" Nov 03 14:41:42.556098 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:42.555936 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2fbg\" (UniqueName: \"kubernetes.io/projected/1305c97a-0ac3-4f7a-9735-d78406162d1e-kube-api-access-k2fbg\") pod \"maas-api-revoke-sub-3e31a506-j4png\" (UID: \"1305c97a-0ac3-4f7a-9735-d78406162d1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-j4png" Nov 03 14:41:42.556147 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:42.556105 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1305c97a-0ac3-4f7a-9735-d78406162d1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-j4png\" (UID: \"1305c97a-0ac3-4f7a-9735-d78406162d1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-j4png" Nov 03 14:41:42.597424 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:42.597377 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:41:42.597424 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:42.597404 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:41:42.597680 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:42.597438 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:41:42.597680 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:42.597497 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wgk87\"" Nov 03 14:41:42.598999 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:42.598978 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-j4png"] Nov 03 14:41:42.657214 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:42.657174 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1305c97a-0ac3-4f7a-9735-d78406162d1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-j4png\" (UID: \"1305c97a-0ac3-4f7a-9735-d78406162d1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-j4png" Nov 03 14:41:42.657427 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:42.657307 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1305c97a-0ac3-4f7a-9735-d78406162d1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-j4png\" (UID: \"1305c97a-0ac3-4f7a-9735-d78406162d1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-j4png" Nov 03 14:41:42.657427 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:42.657344 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2fbg\" (UniqueName: \"kubernetes.io/projected/1305c97a-0ac3-4f7a-9735-d78406162d1e-kube-api-access-k2fbg\") pod \"maas-api-revoke-sub-3e31a506-j4png\" (UID: \"1305c97a-0ac3-4f7a-9735-d78406162d1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-j4png" Nov 03 14:41:42.657993 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:42.657968 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1305c97a-0ac3-4f7a-9735-d78406162d1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-j4png\" (UID: \"1305c97a-0ac3-4f7a-9735-d78406162d1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-j4png" Nov 03 14:41:42.659848 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:42.659823 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1305c97a-0ac3-4f7a-9735-d78406162d1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-j4png\" (UID: \"1305c97a-0ac3-4f7a-9735-d78406162d1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-j4png" Nov 03 14:41:42.711613 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:42.711579 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2fbg\" (UniqueName: \"kubernetes.io/projected/1305c97a-0ac3-4f7a-9735-d78406162d1e-kube-api-access-k2fbg\") pod \"maas-api-revoke-sub-3e31a506-j4png\" (UID: \"1305c97a-0ac3-4f7a-9735-d78406162d1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-j4png" Nov 03 14:41:42.814998 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:42.814688 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-j4png" Nov 03 14:41:43.231198 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:43.231087 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-j4png"] Nov 03 14:41:44.125904 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:44.125563 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-j4png" event={"ID":"1305c97a-0ac3-4f7a-9735-d78406162d1e","Type":"ContainerStarted","Data":"638031b4d9f73a32ac77f15cb95ab31c41dcbd22618b83193e6be96ed82ffb40"} Nov 03 14:41:44.125904 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:44.125631 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-j4png" event={"ID":"1305c97a-0ac3-4f7a-9735-d78406162d1e","Type":"ContainerStarted","Data":"adf9979fcf131c0392b3fc2f5b432c1a077662ff74737c6f0ad6f868461897b3"} Nov 03 14:41:44.233603 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:44.233539 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-j4png" podStartSLOduration=2.233518622 podStartE2EDuration="2.233518622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:41:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:41:44.231520907 +0000 UTC m=+5786.242209151" watchObservedRunningTime="2026-11-03 14:41:44.233518622 +0000 UTC m=+5786.244206880" Nov 03 14:41:45.139989 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:45.139940 5979 generic.go:356] "Generic (PLEG): container finished" podID="1305c97a-0ac3-4f7a-9735-d78406162d1e" containerID="638031b4d9f73a32ac77f15cb95ab31c41dcbd22618b83193e6be96ed82ffb40" exitCode=0 Nov 03 14:41:45.140566 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:45.140068 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-j4png" event={"ID":"1305c97a-0ac3-4f7a-9735-d78406162d1e","Type":"ContainerDied","Data":"638031b4d9f73a32ac77f15cb95ab31c41dcbd22618b83193e6be96ed82ffb40"} Nov 03 14:41:46.279224 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:46.279203 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-j4png" Nov 03 14:41:46.343044 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:46.343013 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1305c97a-0ac3-4f7a-9735-d78406162d1e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1305c97a-0ac3-4f7a-9735-d78406162d1e-maas-api-openshift-ca\") pod \"1305c97a-0ac3-4f7a-9735-d78406162d1e\" (UID: \"1305c97a-0ac3-4f7a-9735-d78406162d1e\") " Nov 03 14:41:46.343196 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:46.343062 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1305c97a-0ac3-4f7a-9735-d78406162d1e-kube-api-access-k2fbg\" (UniqueName: \"kubernetes.io/projected/1305c97a-0ac3-4f7a-9735-d78406162d1e-kube-api-access-k2fbg\") pod \"1305c97a-0ac3-4f7a-9735-d78406162d1e\" (UID: \"1305c97a-0ac3-4f7a-9735-d78406162d1e\") " Nov 03 14:41:46.343196 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:46.343150 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1305c97a-0ac3-4f7a-9735-d78406162d1e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1305c97a-0ac3-4f7a-9735-d78406162d1e-maas-api-xks-ca\") pod \"1305c97a-0ac3-4f7a-9735-d78406162d1e\" (UID: \"1305c97a-0ac3-4f7a-9735-d78406162d1e\") " Nov 03 14:41:46.343386 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:46.343363 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1305c97a-0ac3-4f7a-9735-d78406162d1e-maas-api-openshift-ca" pod "1305c97a-0ac3-4f7a-9735-d78406162d1e" (UID: "1305c97a-0ac3-4f7a-9735-d78406162d1e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:41:46.345092 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:46.345060 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1305c97a-0ac3-4f7a-9735-d78406162d1e-maas-api-xks-ca" pod "1305c97a-0ac3-4f7a-9735-d78406162d1e" (UID: "1305c97a-0ac3-4f7a-9735-d78406162d1e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:41:46.345183 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:46.345144 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1305c97a-0ac3-4f7a-9735-d78406162d1e-kube-api-access-k2fbg" pod "1305c97a-0ac3-4f7a-9735-d78406162d1e" (UID: "1305c97a-0ac3-4f7a-9735-d78406162d1e"). InnerVolumeSpecName "kube-api-access-k2fbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:41:46.444095 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:46.444018 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1305c97a-0ac3-4f7a-9735-d78406162d1e-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:41:46.444095 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:46.444049 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1305c97a-0ac3-4f7a-9735-d78406162d1e-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:41:46.444095 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:46.444062 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2fbg\" (UniqueName: \"kubernetes.io/projected/1305c97a-0ac3-4f7a-9735-d78406162d1e-kube-api-access-k2fbg\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:41:47.151433 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:47.151397 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-j4png" event={"ID":"1305c97a-0ac3-4f7a-9735-d78406162d1e","Type":"ContainerDied","Data":"adf9979fcf131c0392b3fc2f5b432c1a077662ff74737c6f0ad6f868461897b3"} Nov 03 14:41:47.151433 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:47.151424 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-j4png" Nov 03 14:41:47.151682 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:47.151445 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="adf9979fcf131c0392b3fc2f5b432c1a077662ff74737c6f0ad6f868461897b3" Nov 03 14:41:55.249393 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:41:55.249361 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:42:06.131149 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:42:06.131110 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:42:07.233736 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:42:07.233699 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:42:07.802284 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:42:07.802251 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c10bde9-2z295"] Nov 03 14:42:07.802687 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:42:07.802674 5979 cpu_manager.go:414] "containerMap: removing container" podUID="1305c97a-0ac3-4f7a-9735-d78406162d1e" containerName="revoke-keys" Nov 03 14:42:07.802744 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:42:07.802688 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1305c97a-0ac3-4f7a-9735-d78406162d1e" containerName="revoke-keys" Nov 03 14:42:07.802783 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:42:07.802774 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="1305c97a-0ac3-4f7a-9735-d78406162d1e" containerName="revoke-keys" Nov 03 14:42:07.805921 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:42:07.805895 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c10bde9-2z295" Nov 03 14:42:07.855079 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:42:07.855048 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:42:07.901515 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:42:07.901458 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:42:07.901515 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:42:07.901503 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:42:07.901515 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:42:07.901464 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wgk87\"" Nov 03 14:42:07.903084 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:42:07.903064 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c10bde9-2z295"] Nov 03 14:42:07.956549 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:42:07.956515 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20381d2b-0de9-4b7b-96f9-507c3bc3e365-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c10bde9-2z295\" (UID: \"20381d2b-0de9-4b7b-96f9-507c3bc3e365\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c10bde9-2z295" Nov 03 14:42:07.956753 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:42:07.956625 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shrh9\" (UniqueName: \"kubernetes.io/projected/20381d2b-0de9-4b7b-96f9-507c3bc3e365-kube-api-access-shrh9\") pod \"maas-api-revoke-sub-3c10bde9-2z295\" (UID: \"20381d2b-0de9-4b7b-96f9-507c3bc3e365\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c10bde9-2z295" Nov 03 14:42:07.956753 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:42:07.956661 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20381d2b-0de9-4b7b-96f9-507c3bc3e365-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c10bde9-2z295\" (UID: \"20381d2b-0de9-4b7b-96f9-507c3bc3e365\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c10bde9-2z295" Nov 03 14:42:08.058077 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:42:08.057986 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shrh9\" (UniqueName: \"kubernetes.io/projected/20381d2b-0de9-4b7b-96f9-507c3bc3e365-kube-api-access-shrh9\") pod \"maas-api-revoke-sub-3c10bde9-2z295\" (UID: \"20381d2b-0de9-4b7b-96f9-507c3bc3e365\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c10bde9-2z295" Nov 03 14:42:08.058077 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:42:08.058030 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20381d2b-0de9-4b7b-96f9-507c3bc3e365-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c10bde9-2z295\" (UID: \"20381d2b-0de9-4b7b-96f9-507c3bc3e365\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c10bde9-2z295" Nov 03 14:42:08.058330 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:42:08.058130 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20381d2b-0de9-4b7b-96f9-507c3bc3e365-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c10bde9-2z295\" (UID: \"20381d2b-0de9-4b7b-96f9-507c3bc3e365\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c10bde9-2z295" Nov 03 14:42:08.058802 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:42:08.058774 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20381d2b-0de9-4b7b-96f9-507c3bc3e365-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c10bde9-2z295\" (UID: \"20381d2b-0de9-4b7b-96f9-507c3bc3e365\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c10bde9-2z295" Nov 03 14:42:08.060545 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:42:08.060524 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20381d2b-0de9-4b7b-96f9-507c3bc3e365-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c10bde9-2z295\" (UID: \"20381d2b-0de9-4b7b-96f9-507c3bc3e365\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c10bde9-2z295" Nov 03 14:42:08.111882 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:42:08.111832 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shrh9\" (UniqueName: \"kubernetes.io/projected/20381d2b-0de9-4b7b-96f9-507c3bc3e365-kube-api-access-shrh9\") pod \"maas-api-revoke-sub-3c10bde9-2z295\" (UID: \"20381d2b-0de9-4b7b-96f9-507c3bc3e365\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c10bde9-2z295" Nov 03 14:42:08.118995 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:42:08.118909 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c10bde9-2z295" Nov 03 14:42:08.487543 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:42:08.487508 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c10bde9-2z295"] Nov 03 14:42:08.495048 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:42:08.495020 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20381d2b_0de9_4b7b_96f9_507c3bc3e365.slice/crio-6c0a660a05324cfcbde6e2f493e7273c4788f8dd70eaf7d42bd07f361719be59 WatchSource:0}: Error finding container 6c0a660a05324cfcbde6e2f493e7273c4788f8dd70eaf7d42bd07f361719be59: Status 404 returned error can't find the container with id 6c0a660a05324cfcbde6e2f493e7273c4788f8dd70eaf7d42bd07f361719be59 Nov 03 14:42:09.313351 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:42:09.313317 5979 generic.go:356] "Generic (PLEG): container finished" podID="20381d2b-0de9-4b7b-96f9-507c3bc3e365" containerID="ac5a3e02a26927cf82388e2c830577b8fb53b93cb24082d0a3953cb991e1d728" exitCode=0 Nov 03 14:42:09.313564 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:42:09.313395 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c10bde9-2z295" event={"ID":"20381d2b-0de9-4b7b-96f9-507c3bc3e365","Type":"ContainerDied","Data":"ac5a3e02a26927cf82388e2c830577b8fb53b93cb24082d0a3953cb991e1d728"} Nov 03 14:42:09.313564 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:42:09.313449 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c10bde9-2z295" event={"ID":"20381d2b-0de9-4b7b-96f9-507c3bc3e365","Type":"ContainerStarted","Data":"6c0a660a05324cfcbde6e2f493e7273c4788f8dd70eaf7d42bd07f361719be59"} Nov 03 14:42:10.459107 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:42:10.459079 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c10bde9-2z295" Nov 03 14:42:10.585633 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:42:10.585555 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20381d2b-0de9-4b7b-96f9-507c3bc3e365-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20381d2b-0de9-4b7b-96f9-507c3bc3e365-maas-api-xks-ca\") pod \"20381d2b-0de9-4b7b-96f9-507c3bc3e365\" (UID: \"20381d2b-0de9-4b7b-96f9-507c3bc3e365\") " Nov 03 14:42:10.585820 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:42:10.585645 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20381d2b-0de9-4b7b-96f9-507c3bc3e365-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20381d2b-0de9-4b7b-96f9-507c3bc3e365-maas-api-openshift-ca\") pod \"20381d2b-0de9-4b7b-96f9-507c3bc3e365\" (UID: \"20381d2b-0de9-4b7b-96f9-507c3bc3e365\") " Nov 03 14:42:10.585820 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:42:10.585678 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20381d2b-0de9-4b7b-96f9-507c3bc3e365-kube-api-access-shrh9\" (UniqueName: \"kubernetes.io/projected/20381d2b-0de9-4b7b-96f9-507c3bc3e365-kube-api-access-shrh9\") pod \"20381d2b-0de9-4b7b-96f9-507c3bc3e365\" (UID: \"20381d2b-0de9-4b7b-96f9-507c3bc3e365\") " Nov 03 14:42:10.586059 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:42:10.586033 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20381d2b-0de9-4b7b-96f9-507c3bc3e365-maas-api-openshift-ca" pod "20381d2b-0de9-4b7b-96f9-507c3bc3e365" (UID: "20381d2b-0de9-4b7b-96f9-507c3bc3e365"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:42:10.587851 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:42:10.587824 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20381d2b-0de9-4b7b-96f9-507c3bc3e365-kube-api-access-shrh9" pod "20381d2b-0de9-4b7b-96f9-507c3bc3e365" (UID: "20381d2b-0de9-4b7b-96f9-507c3bc3e365"). InnerVolumeSpecName "kube-api-access-shrh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:42:10.587959 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:42:10.587908 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20381d2b-0de9-4b7b-96f9-507c3bc3e365-maas-api-xks-ca" pod "20381d2b-0de9-4b7b-96f9-507c3bc3e365" (UID: "20381d2b-0de9-4b7b-96f9-507c3bc3e365"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:42:10.686894 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:42:10.686848 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20381d2b-0de9-4b7b-96f9-507c3bc3e365-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:42:10.686894 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:42:10.686888 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shrh9\" (UniqueName: \"kubernetes.io/projected/20381d2b-0de9-4b7b-96f9-507c3bc3e365-kube-api-access-shrh9\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:42:10.686894 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:42:10.686900 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20381d2b-0de9-4b7b-96f9-507c3bc3e365-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:42:11.325306 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:42:11.325276 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c10bde9-2z295" Nov 03 14:42:11.325532 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:42:11.325309 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c10bde9-2z295" event={"ID":"20381d2b-0de9-4b7b-96f9-507c3bc3e365","Type":"ContainerDied","Data":"6c0a660a05324cfcbde6e2f493e7273c4788f8dd70eaf7d42bd07f361719be59"} Nov 03 14:42:11.325532 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:42:11.325366 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c0a660a05324cfcbde6e2f493e7273c4788f8dd70eaf7d42bd07f361719be59" Nov 03 14:42:20.647135 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:42:20.647100 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:42:22.517634 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:42:22.517600 5979 scope.go:122] "RemoveContainer" containerID="0db707d8847d4b5fdb0f6f5a38afcc7f979253dacf575473fd67f30df114fd83" Nov 03 14:42:22.526222 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:42:22.526198 5979 scope.go:122] "RemoveContainer" containerID="e6263b4cdde283940e8fa8aba4bdf05c19c4d935c781c6ef62b770bd58ac8a1e" Nov 03 14:42:22.533815 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:42:22.533794 5979 scope.go:122] "RemoveContainer" containerID="9ebaf36d8b463ed57dc7092742968e04e13643221f4a009528dadbeb6a88bcb1" Nov 03 14:42:37.232652 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:42:37.232608 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:42:53.469226 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:42:53.469188 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-68d6ccb6b8-8tq9t"] Nov 03 14:42:53.469779 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:42:53.469649 5979 cpu_manager.go:414] "containerMap: removing container" podUID="20381d2b-0de9-4b7b-96f9-507c3bc3e365" containerName="revoke-keys" Nov 03 14:42:53.469779 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:42:53.469662 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20381d2b-0de9-4b7b-96f9-507c3bc3e365" containerName="revoke-keys" Nov 03 14:42:53.469779 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:42:53.469732 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="20381d2b-0de9-4b7b-96f9-507c3bc3e365" containerName="revoke-keys" Nov 03 14:42:53.473005 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:42:53.472983 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-68d6ccb6b8-8tq9t" Nov 03 14:42:53.522343 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:42:53.522310 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-lpgh7\"" Nov 03 14:42:53.522343 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:42:53.522310 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 03 14:42:53.532449 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:42:53.532427 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/706b4485-a4d4-4aa7-a942-9bd37ef907e9-plugins-config-volume\") pod \"payload-pre-processing-68d6ccb6b8-8tq9t\" (UID: \"706b4485-a4d4-4aa7-a942-9bd37ef907e9\") " pod="openshift-ingress/payload-pre-processing-68d6ccb6b8-8tq9t" Nov 03 14:42:53.532658 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:42:53.532491 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd82q\" (UniqueName: \"kubernetes.io/projected/706b4485-a4d4-4aa7-a942-9bd37ef907e9-kube-api-access-pd82q\") pod \"payload-pre-processing-68d6ccb6b8-8tq9t\" (UID: \"706b4485-a4d4-4aa7-a942-9bd37ef907e9\") " pod="openshift-ingress/payload-pre-processing-68d6ccb6b8-8tq9t" Nov 03 14:42:53.570816 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:42:53.570786 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-68d6ccb6b8-8tq9t"] Nov 03 14:42:53.633699 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:42:53.633663 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/706b4485-a4d4-4aa7-a942-9bd37ef907e9-plugins-config-volume\") pod \"payload-pre-processing-68d6ccb6b8-8tq9t\" (UID: \"706b4485-a4d4-4aa7-a942-9bd37ef907e9\") " pod="openshift-ingress/payload-pre-processing-68d6ccb6b8-8tq9t" Nov 03 14:42:53.633879 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:42:53.633719 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd82q\" (UniqueName: \"kubernetes.io/projected/706b4485-a4d4-4aa7-a942-9bd37ef907e9-kube-api-access-pd82q\") pod \"payload-pre-processing-68d6ccb6b8-8tq9t\" (UID: \"706b4485-a4d4-4aa7-a942-9bd37ef907e9\") " pod="openshift-ingress/payload-pre-processing-68d6ccb6b8-8tq9t" Nov 03 14:42:53.634425 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:42:53.634402 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/706b4485-a4d4-4aa7-a942-9bd37ef907e9-plugins-config-volume\") pod \"payload-pre-processing-68d6ccb6b8-8tq9t\" (UID: \"706b4485-a4d4-4aa7-a942-9bd37ef907e9\") " pod="openshift-ingress/payload-pre-processing-68d6ccb6b8-8tq9t" Nov 03 14:42:53.687615 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:42:53.687581 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd82q\" (UniqueName: \"kubernetes.io/projected/706b4485-a4d4-4aa7-a942-9bd37ef907e9-kube-api-access-pd82q\") pod \"payload-pre-processing-68d6ccb6b8-8tq9t\" (UID: \"706b4485-a4d4-4aa7-a942-9bd37ef907e9\") " pod="openshift-ingress/payload-pre-processing-68d6ccb6b8-8tq9t" Nov 03 14:42:53.786330 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:42:53.786251 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-68d6ccb6b8-8tq9t" Nov 03 14:42:54.155531 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:42:54.155497 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-68d6ccb6b8-8tq9t"] Nov 03 14:42:54.165116 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:42:54.165081 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod706b4485_a4d4_4aa7_a942_9bd37ef907e9.slice/crio-6c0ac24f2ac9db8c92a03faf18bbb591f6bbafb6dfdbfba288e529c5b877a7c2 WatchSource:0}: Error finding container 6c0ac24f2ac9db8c92a03faf18bbb591f6bbafb6dfdbfba288e529c5b877a7c2: Status 404 returned error can't find the container with id 6c0ac24f2ac9db8c92a03faf18bbb591f6bbafb6dfdbfba288e529c5b877a7c2 Nov 03 14:42:54.542557 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:42:54.542514 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-68d6ccb6b8-8tq9t" event={"ID":"706b4485-a4d4-4aa7-a942-9bd37ef907e9","Type":"ContainerStarted","Data":"43725f03f91ecd60d250ab6340565438a9017047480136018e53485128804ef0"} Nov 03 14:42:54.542984 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:42:54.542588 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-68d6ccb6b8-8tq9t" Nov 03 14:42:54.542984 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:42:54.542611 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-68d6ccb6b8-8tq9t" event={"ID":"706b4485-a4d4-4aa7-a942-9bd37ef907e9","Type":"ContainerStarted","Data":"6c0ac24f2ac9db8c92a03faf18bbb591f6bbafb6dfdbfba288e529c5b877a7c2"} Nov 03 14:42:54.653518 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:42:54.653439 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-68d6ccb6b8-8tq9t" podStartSLOduration=1.653420402 podStartE2EDuration="1.653420402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:42:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:42:54.650466894 +0000 UTC m=+5856.661155135" watchObservedRunningTime="2026-11-03 14:42:54.653420402 +0000 UTC m=+5856.664108646" Nov 03 14:43:05.549571 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:43:05.549466 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-68d6ccb6b8-8tq9t" Nov 03 14:43:07.233747 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:43:07.233712 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:43:13.555366 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:43:13.555330 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:43:37.233709 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:43:37.233667 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:44:07.047872 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:44:07.047834 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-626c17cb-th2l5"] Nov 03 14:44:07.093765 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:44:07.093728 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-626c17cb-th2l5"] Nov 03 14:44:07.233809 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:44:07.233770 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:44:08.114889 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:44:08.114854 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="673fcc1f-a5c3-4ea3-a87e-82357a1f2711" path="/var/lib/kubelet/pods/673fcc1f-a5c3-4ea3-a87e-82357a1f2711/volumes" Nov 03 14:44:37.233876 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:44:37.233835 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:44:48.095391 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:44:48.095350 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-pgjzx"] Nov 03 14:44:48.099049 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:44:48.099027 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-pgjzx" Nov 03 14:44:48.148048 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:44:48.148018 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:44:48.194446 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:44:48.194414 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:44:48.194644 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:44:48.194449 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:44:48.194703 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:44:48.194648 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wgk87\"" Nov 03 14:44:48.196275 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:44:48.196257 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-pgjzx"] Nov 03 14:44:48.203986 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:44:48.203956 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8znn\" (UniqueName: \"kubernetes.io/projected/4c6f24ba-019e-47bc-b173-668b20ed1689-kube-api-access-v8znn\") pod \"maas-api-revoke-sub-e2547622-pgjzx\" (UID: \"4c6f24ba-019e-47bc-b173-668b20ed1689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-pgjzx" Nov 03 14:44:48.204136 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:44:48.204116 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c6f24ba-019e-47bc-b173-668b20ed1689-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-pgjzx\" (UID: \"4c6f24ba-019e-47bc-b173-668b20ed1689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-pgjzx" Nov 03 14:44:48.204256 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:44:48.204241 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c6f24ba-019e-47bc-b173-668b20ed1689-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-pgjzx\" (UID: \"4c6f24ba-019e-47bc-b173-668b20ed1689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-pgjzx" Nov 03 14:44:48.305285 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:44:48.305246 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8znn\" (UniqueName: \"kubernetes.io/projected/4c6f24ba-019e-47bc-b173-668b20ed1689-kube-api-access-v8znn\") pod \"maas-api-revoke-sub-e2547622-pgjzx\" (UID: \"4c6f24ba-019e-47bc-b173-668b20ed1689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-pgjzx" Nov 03 14:44:48.305512 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:44:48.305311 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c6f24ba-019e-47bc-b173-668b20ed1689-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-pgjzx\" (UID: \"4c6f24ba-019e-47bc-b173-668b20ed1689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-pgjzx" Nov 03 14:44:48.305512 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:44:48.305335 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c6f24ba-019e-47bc-b173-668b20ed1689-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-pgjzx\" (UID: \"4c6f24ba-019e-47bc-b173-668b20ed1689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-pgjzx" Nov 03 14:44:48.306154 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:44:48.306116 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c6f24ba-019e-47bc-b173-668b20ed1689-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-pgjzx\" (UID: \"4c6f24ba-019e-47bc-b173-668b20ed1689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-pgjzx" Nov 03 14:44:48.308123 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:44:48.308102 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c6f24ba-019e-47bc-b173-668b20ed1689-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-pgjzx\" (UID: \"4c6f24ba-019e-47bc-b173-668b20ed1689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-pgjzx" Nov 03 14:44:48.367294 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:44:48.367219 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8znn\" (UniqueName: \"kubernetes.io/projected/4c6f24ba-019e-47bc-b173-668b20ed1689-kube-api-access-v8znn\") pod \"maas-api-revoke-sub-e2547622-pgjzx\" (UID: \"4c6f24ba-019e-47bc-b173-668b20ed1689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-pgjzx" Nov 03 14:44:48.413331 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:44:48.413296 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-pgjzx" Nov 03 14:44:48.786053 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:44:48.786016 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-pgjzx"] Nov 03 14:44:48.794054 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:44:48.794024 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c6f24ba_019e_47bc_b173_668b20ed1689.slice/crio-89be0f6f321964a078adcec690d0967c67bb230477c210b54c956d3a8b11759c WatchSource:0}: Error finding container 89be0f6f321964a078adcec690d0967c67bb230477c210b54c956d3a8b11759c: Status 404 returned error can't find the container with id 89be0f6f321964a078adcec690d0967c67bb230477c210b54c956d3a8b11759c Nov 03 14:44:49.150574 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:44:49.150539 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-pgjzx" event={"ID":"4c6f24ba-019e-47bc-b173-668b20ed1689","Type":"ContainerStarted","Data":"f37437f74b8b954a63655f41f07796199fd72a73d7da80aba265e0caa51ca8ed"} Nov 03 14:44:49.150987 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:44:49.150589 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-pgjzx" event={"ID":"4c6f24ba-019e-47bc-b173-668b20ed1689","Type":"ContainerStarted","Data":"89be0f6f321964a078adcec690d0967c67bb230477c210b54c956d3a8b11759c"} Nov 03 14:44:49.258088 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:44:49.258029 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-pgjzx" podStartSLOduration=1.258015787 podStartE2EDuration="1.258015787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:44:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:44:49.257274695 +0000 UTC m=+5971.267962937" watchObservedRunningTime="2026-11-03 14:44:49.258015787 +0000 UTC m=+5971.268704028" Nov 03 14:44:50.156961 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:44:50.156925 5979 generic.go:356] "Generic (PLEG): container finished" podID="4c6f24ba-019e-47bc-b173-668b20ed1689" containerID="f37437f74b8b954a63655f41f07796199fd72a73d7da80aba265e0caa51ca8ed" exitCode=0 Nov 03 14:44:50.157385 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:44:50.156975 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-pgjzx" event={"ID":"4c6f24ba-019e-47bc-b173-668b20ed1689","Type":"ContainerDied","Data":"f37437f74b8b954a63655f41f07796199fd72a73d7da80aba265e0caa51ca8ed"} Nov 03 14:44:51.298524 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:44:51.298502 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-pgjzx" Nov 03 14:44:51.333800 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:44:51.333773 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c6f24ba-019e-47bc-b173-668b20ed1689-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c6f24ba-019e-47bc-b173-668b20ed1689-maas-api-openshift-ca\") pod \"4c6f24ba-019e-47bc-b173-668b20ed1689\" (UID: \"4c6f24ba-019e-47bc-b173-668b20ed1689\") " Nov 03 14:44:51.333932 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:44:51.333809 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c6f24ba-019e-47bc-b173-668b20ed1689-kube-api-access-v8znn\" (UniqueName: \"kubernetes.io/projected/4c6f24ba-019e-47bc-b173-668b20ed1689-kube-api-access-v8znn\") pod \"4c6f24ba-019e-47bc-b173-668b20ed1689\" (UID: \"4c6f24ba-019e-47bc-b173-668b20ed1689\") " Nov 03 14:44:51.333932 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:44:51.333896 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c6f24ba-019e-47bc-b173-668b20ed1689-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c6f24ba-019e-47bc-b173-668b20ed1689-maas-api-xks-ca\") pod \"4c6f24ba-019e-47bc-b173-668b20ed1689\" (UID: \"4c6f24ba-019e-47bc-b173-668b20ed1689\") " Nov 03 14:44:51.334174 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:44:51.334149 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c6f24ba-019e-47bc-b173-668b20ed1689-maas-api-openshift-ca" pod "4c6f24ba-019e-47bc-b173-668b20ed1689" (UID: "4c6f24ba-019e-47bc-b173-668b20ed1689"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:44:51.335867 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:44:51.335843 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c6f24ba-019e-47bc-b173-668b20ed1689-kube-api-access-v8znn" pod "4c6f24ba-019e-47bc-b173-668b20ed1689" (UID: "4c6f24ba-019e-47bc-b173-668b20ed1689"). InnerVolumeSpecName "kube-api-access-v8znn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:44:51.335867 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:44:51.335848 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c6f24ba-019e-47bc-b173-668b20ed1689-maas-api-xks-ca" pod "4c6f24ba-019e-47bc-b173-668b20ed1689" (UID: "4c6f24ba-019e-47bc-b173-668b20ed1689"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:44:51.435140 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:44:51.435055 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c6f24ba-019e-47bc-b173-668b20ed1689-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:44:51.435140 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:44:51.435087 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8znn\" (UniqueName: \"kubernetes.io/projected/4c6f24ba-019e-47bc-b173-668b20ed1689-kube-api-access-v8znn\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:44:51.435140 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:44:51.435101 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c6f24ba-019e-47bc-b173-668b20ed1689-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:44:52.169033 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:44:52.169002 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-pgjzx" Nov 03 14:44:52.169181 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:44:52.169005 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-pgjzx" event={"ID":"4c6f24ba-019e-47bc-b173-668b20ed1689","Type":"ContainerDied","Data":"89be0f6f321964a078adcec690d0967c67bb230477c210b54c956d3a8b11759c"} Nov 03 14:44:52.169181 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:44:52.169113 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89be0f6f321964a078adcec690d0967c67bb230477c210b54c956d3a8b11759c" Nov 03 14:45:00.171880 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:00.171833 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895285-92p9q"] Nov 03 14:45:00.172597 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:00.172575 5979 cpu_manager.go:414] "containerMap: removing container" podUID="4c6f24ba-019e-47bc-b173-668b20ed1689" containerName="revoke-keys" Nov 03 14:45:00.172665 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:00.172599 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c6f24ba-019e-47bc-b173-668b20ed1689" containerName="revoke-keys" Nov 03 14:45:00.172795 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:00.172782 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c6f24ba-019e-47bc-b173-668b20ed1689" containerName="revoke-keys" Nov 03 14:45:00.176666 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:00.176647 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895285-92p9q" Nov 03 14:45:00.226352 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:00.226310 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895285-92p9q"] Nov 03 14:45:00.232342 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:00.232300 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wh5v\" (UniqueName: \"kubernetes.io/projected/bfde0656-f585-42b0-80e5-4634f5dc5d71-kube-api-access-5wh5v\") pod \"maas-api-key-cleanup-29895285-92p9q\" (UID: \"bfde0656-f585-42b0-80e5-4634f5dc5d71\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895285-92p9q" Nov 03 14:45:00.232526 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:00.232439 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfde0656-f585-42b0-80e5-4634f5dc5d71-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895285-92p9q\" (UID: \"bfde0656-f585-42b0-80e5-4634f5dc5d71\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895285-92p9q" Nov 03 14:45:00.275746 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:00.274755 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:45:00.275746 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:00.275487 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wgk87\"" Nov 03 14:45:00.275746 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:00.275619 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:45:00.333651 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:00.333605 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wh5v\" (UniqueName: \"kubernetes.io/projected/bfde0656-f585-42b0-80e5-4634f5dc5d71-kube-api-access-5wh5v\") pod \"maas-api-key-cleanup-29895285-92p9q\" (UID: \"bfde0656-f585-42b0-80e5-4634f5dc5d71\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895285-92p9q" Nov 03 14:45:00.333854 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:00.333666 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfde0656-f585-42b0-80e5-4634f5dc5d71-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895285-92p9q\" (UID: \"bfde0656-f585-42b0-80e5-4634f5dc5d71\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895285-92p9q" Nov 03 14:45:00.334282 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:00.334261 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfde0656-f585-42b0-80e5-4634f5dc5d71-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895285-92p9q\" (UID: \"bfde0656-f585-42b0-80e5-4634f5dc5d71\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895285-92p9q" Nov 03 14:45:00.387203 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:00.387167 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wh5v\" (UniqueName: \"kubernetes.io/projected/bfde0656-f585-42b0-80e5-4634f5dc5d71-kube-api-access-5wh5v\") pod \"maas-api-key-cleanup-29895285-92p9q\" (UID: \"bfde0656-f585-42b0-80e5-4634f5dc5d71\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895285-92p9q" Nov 03 14:45:00.489905 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:00.489813 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895285-92p9q" Nov 03 14:45:00.857774 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:00.857694 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895285-92p9q"] Nov 03 14:45:00.864714 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:45:00.864681 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfde0656_f585_42b0_80e5_4634f5dc5d71.slice/crio-9ecbc43276a208fd0b331f4ff29e30f71e2d910b80c513c691903094467db96d WatchSource:0}: Error finding container 9ecbc43276a208fd0b331f4ff29e30f71e2d910b80c513c691903094467db96d: Status 404 returned error can't find the container with id 9ecbc43276a208fd0b331f4ff29e30f71e2d910b80c513c691903094467db96d Nov 03 14:45:01.076891 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:01.076860 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:45:01.213917 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:01.213880 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895285-92p9q" event={"ID":"bfde0656-f585-42b0-80e5-4634f5dc5d71","Type":"ContainerStarted","Data":"f44d9127b485e83cffa76f934939f67beb27a46cc36a8097fed74d5fb7d6354d"} Nov 03 14:45:01.214366 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:01.213951 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895285-92p9q" event={"ID":"bfde0656-f585-42b0-80e5-4634f5dc5d71","Type":"ContainerStarted","Data":"9ecbc43276a208fd0b331f4ff29e30f71e2d910b80c513c691903094467db96d"} Nov 03 14:45:01.325435 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:01.325380 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895285-92p9q" podStartSLOduration=1.325365266 podStartE2EDuration="1.325365266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:45:01.323857469 +0000 UTC m=+5983.334545711" watchObservedRunningTime="2026-11-03 14:45:01.325365266 +0000 UTC m=+5983.336053508" Nov 03 14:45:02.242251 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:02.242211 5979 generic.go:356] "Generic (PLEG): container finished" podID="bfde0656-f585-42b0-80e5-4634f5dc5d71" containerID="f44d9127b485e83cffa76f934939f67beb27a46cc36a8097fed74d5fb7d6354d" exitCode=60 Nov 03 14:45:02.242933 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:02.242605 5979 scope.go:122] "RemoveContainer" containerID="f44d9127b485e83cffa76f934939f67beb27a46cc36a8097fed74d5fb7d6354d" Nov 03 14:45:02.243129 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:02.242628 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895285-92p9q" event={"ID":"bfde0656-f585-42b0-80e5-4634f5dc5d71","Type":"ContainerDied","Data":"f44d9127b485e83cffa76f934939f67beb27a46cc36a8097fed74d5fb7d6354d"} Nov 03 14:45:03.307899 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:03.307860 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895285-92p9q" event={"ID":"bfde0656-f585-42b0-80e5-4634f5dc5d71","Type":"ContainerStarted","Data":"9737040ab299c68ead532fa23d3fcfcddff12c91489a232506d423efbfe93804"} Nov 03 14:45:04.319536 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:04.319505 5979 generic.go:356] "Generic (PLEG): container finished" podID="bfde0656-f585-42b0-80e5-4634f5dc5d71" containerID="9737040ab299c68ead532fa23d3fcfcddff12c91489a232506d423efbfe93804" exitCode=60 Nov 03 14:45:04.320105 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:04.319588 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895285-92p9q" event={"ID":"bfde0656-f585-42b0-80e5-4634f5dc5d71","Type":"ContainerDied","Data":"9737040ab299c68ead532fa23d3fcfcddff12c91489a232506d423efbfe93804"} Nov 03 14:45:04.320229 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:04.319803 5979 scope.go:122] "RemoveContainer" containerID="9737040ab299c68ead532fa23d3fcfcddff12c91489a232506d423efbfe93804" Nov 03 14:45:04.320535 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:04.320383 5979 scope.go:122] "RemoveContainer" containerID="f44d9127b485e83cffa76f934939f67beb27a46cc36a8097fed74d5fb7d6354d" Nov 03 14:45:04.320657 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:45:04.320599 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895285-92p9q_odh-ai-gateway-infra(bfde0656-f585-42b0-80e5-4634f5dc5d71)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895285-92p9q" podUID="bfde0656-f585-42b0-80e5-4634f5dc5d71" Nov 03 14:45:05.327031 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:05.327004 5979 scope.go:122] "RemoveContainer" containerID="9737040ab299c68ead532fa23d3fcfcddff12c91489a232506d423efbfe93804" Nov 03 14:45:05.327414 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:45:05.327257 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895285-92p9q_odh-ai-gateway-infra(bfde0656-f585-42b0-80e5-4634f5dc5d71)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895285-92p9q" podUID="bfde0656-f585-42b0-80e5-4634f5dc5d71" Nov 03 14:45:07.324985 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:45:07.324908 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:45:12.450498 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:12.450434 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:45:17.107879 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:17.107847 5979 scope.go:122] "RemoveContainer" containerID="9737040ab299c68ead532fa23d3fcfcddff12c91489a232506d423efbfe93804" Nov 03 14:45:18.467432 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:18.467400 5979 generic.go:356] "Generic (PLEG): container finished" podID="bfde0656-f585-42b0-80e5-4634f5dc5d71" containerID="deb8df7e56f3bcd0e43aa5a2df51d6026b0a729611020e045bb323237e1f273e" exitCode=60 Nov 03 14:45:18.467899 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:18.467498 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895285-92p9q" event={"ID":"bfde0656-f585-42b0-80e5-4634f5dc5d71","Type":"ContainerDied","Data":"deb8df7e56f3bcd0e43aa5a2df51d6026b0a729611020e045bb323237e1f273e"} Nov 03 14:45:18.467899 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:18.467567 5979 scope.go:122] "RemoveContainer" containerID="9737040ab299c68ead532fa23d3fcfcddff12c91489a232506d423efbfe93804" Nov 03 14:45:18.468001 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:18.467931 5979 scope.go:122] "RemoveContainer" containerID="deb8df7e56f3bcd0e43aa5a2df51d6026b0a729611020e045bb323237e1f273e" Nov 03 14:45:18.468227 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:45:18.468200 5979 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895285-92p9q_odh-ai-gateway-infra(bfde0656-f585-42b0-80e5-4634f5dc5d71)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895285-92p9q" podUID="bfde0656-f585-42b0-80e5-4634f5dc5d71" Nov 03 14:45:19.589975 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:19.589944 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895285-92p9q"] Nov 03 14:45:19.729344 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:19.729322 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895285-92p9q" Nov 03 14:45:19.899990 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:19.899943 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfde0656-f585-42b0-80e5-4634f5dc5d71-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfde0656-f585-42b0-80e5-4634f5dc5d71-maas-api-service-ca\") pod \"bfde0656-f585-42b0-80e5-4634f5dc5d71\" (UID: \"bfde0656-f585-42b0-80e5-4634f5dc5d71\") " Nov 03 14:45:19.900191 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:19.900065 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfde0656-f585-42b0-80e5-4634f5dc5d71-kube-api-access-5wh5v\" (UniqueName: \"kubernetes.io/projected/bfde0656-f585-42b0-80e5-4634f5dc5d71-kube-api-access-5wh5v\") pod \"bfde0656-f585-42b0-80e5-4634f5dc5d71\" (UID: \"bfde0656-f585-42b0-80e5-4634f5dc5d71\") " Nov 03 14:45:19.900393 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:19.900363 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfde0656-f585-42b0-80e5-4634f5dc5d71-maas-api-service-ca" pod "bfde0656-f585-42b0-80e5-4634f5dc5d71" (UID: "bfde0656-f585-42b0-80e5-4634f5dc5d71"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:45:19.900599 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:19.900578 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfde0656-f585-42b0-80e5-4634f5dc5d71-maas-api-service-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:45:19.902143 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:19.902122 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfde0656-f585-42b0-80e5-4634f5dc5d71-kube-api-access-5wh5v" pod "bfde0656-f585-42b0-80e5-4634f5dc5d71" (UID: "bfde0656-f585-42b0-80e5-4634f5dc5d71"). InnerVolumeSpecName "kube-api-access-5wh5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:45:20.002206 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:20.002169 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wh5v\" (UniqueName: \"kubernetes.io/projected/bfde0656-f585-42b0-80e5-4634f5dc5d71-kube-api-access-5wh5v\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:45:20.481101 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:20.481065 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895285-92p9q" event={"ID":"bfde0656-f585-42b0-80e5-4634f5dc5d71","Type":"ContainerDied","Data":"9ecbc43276a208fd0b331f4ff29e30f71e2d910b80c513c691903094467db96d"} Nov 03 14:45:20.481301 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:20.481132 5979 scope.go:122] "RemoveContainer" containerID="deb8df7e56f3bcd0e43aa5a2df51d6026b0a729611020e045bb323237e1f273e" Nov 03 14:45:20.481301 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:20.481076 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895285-92p9q" Nov 03 14:45:20.634494 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:20.634449 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895285-92p9q"] Nov 03 14:45:20.640801 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:20.640775 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895285-92p9q"] Nov 03 14:45:20.680928 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:20.680899 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:45:22.114384 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:22.114351 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfde0656-f585-42b0-80e5-4634f5dc5d71" path="/var/lib/kubelet/pods/bfde0656-f585-42b0-80e5-4634f5dc5d71/volumes" Nov 03 14:45:22.575486 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:22.575435 5979 scope.go:122] "RemoveContainer" containerID="a53f40e2f671ac1a63c13fa708b01c26afb76d1b4986d42d46f7560bda6f3ada" Nov 03 14:45:23.150819 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:23.150791 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vjdcb"] Nov 03 14:45:23.151231 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:23.151216 5979 cpu_manager.go:414] "containerMap: removing container" podUID="bfde0656-f585-42b0-80e5-4634f5dc5d71" containerName="cleanup" Nov 03 14:45:23.151281 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:23.151231 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfde0656-f585-42b0-80e5-4634f5dc5d71" containerName="cleanup" Nov 03 14:45:23.151281 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:23.151248 5979 cpu_manager.go:414] "containerMap: removing container" podUID="bfde0656-f585-42b0-80e5-4634f5dc5d71" containerName="cleanup" Nov 03 14:45:23.151281 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:23.151253 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfde0656-f585-42b0-80e5-4634f5dc5d71" containerName="cleanup" Nov 03 14:45:23.151382 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:23.151331 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfde0656-f585-42b0-80e5-4634f5dc5d71" containerName="cleanup" Nov 03 14:45:23.151382 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:23.151341 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfde0656-f585-42b0-80e5-4634f5dc5d71" containerName="cleanup" Nov 03 14:45:23.156222 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:23.156200 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vjdcb" Nov 03 14:45:23.205927 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:23.205890 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:45:23.206096 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:23.205991 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:45:23.252395 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:23.252363 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wgk87\"" Nov 03 14:45:23.252395 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:23.252363 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:45:23.254303 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:23.254279 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vjdcb"] Nov 03 14:45:23.335615 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:23.335583 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01c5a8f4-4b2d-4d24-a719-ea02e2d7cbc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-vjdcb\" (UID: \"01c5a8f4-4b2d-4d24-a719-ea02e2d7cbc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vjdcb" Nov 03 14:45:23.335615 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:23.335619 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01c5a8f4-4b2d-4d24-a719-ea02e2d7cbc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-vjdcb\" (UID: \"01c5a8f4-4b2d-4d24-a719-ea02e2d7cbc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vjdcb" Nov 03 14:45:23.335818 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:23.335788 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7b5h\" (UniqueName: \"kubernetes.io/projected/01c5a8f4-4b2d-4d24-a719-ea02e2d7cbc1-kube-api-access-z7b5h\") pod \"maas-api-revoke-sub-c1e271a9-vjdcb\" (UID: \"01c5a8f4-4b2d-4d24-a719-ea02e2d7cbc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vjdcb" Nov 03 14:45:23.436456 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:23.436382 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7b5h\" (UniqueName: \"kubernetes.io/projected/01c5a8f4-4b2d-4d24-a719-ea02e2d7cbc1-kube-api-access-z7b5h\") pod \"maas-api-revoke-sub-c1e271a9-vjdcb\" (UID: \"01c5a8f4-4b2d-4d24-a719-ea02e2d7cbc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vjdcb" Nov 03 14:45:23.436456 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:23.436444 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01c5a8f4-4b2d-4d24-a719-ea02e2d7cbc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-vjdcb\" (UID: \"01c5a8f4-4b2d-4d24-a719-ea02e2d7cbc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vjdcb" Nov 03 14:45:23.436667 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:23.436493 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01c5a8f4-4b2d-4d24-a719-ea02e2d7cbc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-vjdcb\" (UID: \"01c5a8f4-4b2d-4d24-a719-ea02e2d7cbc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vjdcb" Nov 03 14:45:23.437166 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:23.437142 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01c5a8f4-4b2d-4d24-a719-ea02e2d7cbc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-vjdcb\" (UID: \"01c5a8f4-4b2d-4d24-a719-ea02e2d7cbc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vjdcb" Nov 03 14:45:23.438810 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:23.438786 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01c5a8f4-4b2d-4d24-a719-ea02e2d7cbc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-vjdcb\" (UID: \"01c5a8f4-4b2d-4d24-a719-ea02e2d7cbc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vjdcb" Nov 03 14:45:23.490237 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:23.490202 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7b5h\" (UniqueName: \"kubernetes.io/projected/01c5a8f4-4b2d-4d24-a719-ea02e2d7cbc1-kube-api-access-z7b5h\") pod \"maas-api-revoke-sub-c1e271a9-vjdcb\" (UID: \"01c5a8f4-4b2d-4d24-a719-ea02e2d7cbc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vjdcb" Nov 03 14:45:23.769270 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:23.769190 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vjdcb" Nov 03 14:45:24.140337 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:24.140309 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vjdcb"] Nov 03 14:45:24.148418 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:45:24.148389 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01c5a8f4_4b2d_4d24_a719_ea02e2d7cbc1.slice/crio-6f15e5ca24b9301b6eb4f634d496be81dc6d2c28c8fadf389b3a1b6ab95bbb78 WatchSource:0}: Error finding container 6f15e5ca24b9301b6eb4f634d496be81dc6d2c28c8fadf389b3a1b6ab95bbb78: Status 404 returned error can't find the container with id 6f15e5ca24b9301b6eb4f634d496be81dc6d2c28c8fadf389b3a1b6ab95bbb78 Nov 03 14:45:24.503708 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:24.503677 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vjdcb" event={"ID":"01c5a8f4-4b2d-4d24-a719-ea02e2d7cbc1","Type":"ContainerStarted","Data":"bdfeca87978993ca8849654c5de837ebfe754222dbe3c0043f87ccc18738ce2d"} Nov 03 14:45:24.504094 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:24.503731 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vjdcb" event={"ID":"01c5a8f4-4b2d-4d24-a719-ea02e2d7cbc1","Type":"ContainerStarted","Data":"6f15e5ca24b9301b6eb4f634d496be81dc6d2c28c8fadf389b3a1b6ab95bbb78"} Nov 03 14:45:24.611724 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:24.611672 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vjdcb" podStartSLOduration=1.611657327 podStartE2EDuration="1.611657327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:45:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:45:24.610419943 +0000 UTC m=+6006.621108185" watchObservedRunningTime="2026-11-03 14:45:24.611657327 +0000 UTC m=+6006.622345569" Nov 03 14:45:25.509568 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:25.509536 5979 generic.go:356] "Generic (PLEG): container finished" podID="01c5a8f4-4b2d-4d24-a719-ea02e2d7cbc1" containerID="bdfeca87978993ca8849654c5de837ebfe754222dbe3c0043f87ccc18738ce2d" exitCode=0 Nov 03 14:45:25.509974 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:25.509620 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vjdcb" event={"ID":"01c5a8f4-4b2d-4d24-a719-ea02e2d7cbc1","Type":"ContainerDied","Data":"bdfeca87978993ca8849654c5de837ebfe754222dbe3c0043f87ccc18738ce2d"} Nov 03 14:45:26.650349 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:26.650325 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vjdcb" Nov 03 14:45:26.773415 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:26.773326 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01c5a8f4-4b2d-4d24-a719-ea02e2d7cbc1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01c5a8f4-4b2d-4d24-a719-ea02e2d7cbc1-maas-api-xks-ca\") pod \"01c5a8f4-4b2d-4d24-a719-ea02e2d7cbc1\" (UID: \"01c5a8f4-4b2d-4d24-a719-ea02e2d7cbc1\") " Nov 03 14:45:26.773415 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:26.773399 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01c5a8f4-4b2d-4d24-a719-ea02e2d7cbc1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01c5a8f4-4b2d-4d24-a719-ea02e2d7cbc1-maas-api-openshift-ca\") pod \"01c5a8f4-4b2d-4d24-a719-ea02e2d7cbc1\" (UID: \"01c5a8f4-4b2d-4d24-a719-ea02e2d7cbc1\") " Nov 03 14:45:26.773712 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:26.773522 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01c5a8f4-4b2d-4d24-a719-ea02e2d7cbc1-kube-api-access-z7b5h\" (UniqueName: \"kubernetes.io/projected/01c5a8f4-4b2d-4d24-a719-ea02e2d7cbc1-kube-api-access-z7b5h\") pod \"01c5a8f4-4b2d-4d24-a719-ea02e2d7cbc1\" (UID: \"01c5a8f4-4b2d-4d24-a719-ea02e2d7cbc1\") " Nov 03 14:45:26.773799 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:26.773764 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01c5a8f4-4b2d-4d24-a719-ea02e2d7cbc1-maas-api-openshift-ca" pod "01c5a8f4-4b2d-4d24-a719-ea02e2d7cbc1" (UID: "01c5a8f4-4b2d-4d24-a719-ea02e2d7cbc1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:45:26.773900 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:26.773877 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01c5a8f4-4b2d-4d24-a719-ea02e2d7cbc1-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:45:26.775708 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:26.775679 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01c5a8f4-4b2d-4d24-a719-ea02e2d7cbc1-kube-api-access-z7b5h" pod "01c5a8f4-4b2d-4d24-a719-ea02e2d7cbc1" (UID: "01c5a8f4-4b2d-4d24-a719-ea02e2d7cbc1"). InnerVolumeSpecName "kube-api-access-z7b5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:45:26.775958 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:26.775936 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01c5a8f4-4b2d-4d24-a719-ea02e2d7cbc1-maas-api-xks-ca" pod "01c5a8f4-4b2d-4d24-a719-ea02e2d7cbc1" (UID: "01c5a8f4-4b2d-4d24-a719-ea02e2d7cbc1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:45:26.874594 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:26.874561 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01c5a8f4-4b2d-4d24-a719-ea02e2d7cbc1-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:45:26.874594 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:26.874589 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7b5h\" (UniqueName: \"kubernetes.io/projected/01c5a8f4-4b2d-4d24-a719-ea02e2d7cbc1-kube-api-access-z7b5h\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:45:27.520887 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:27.520854 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vjdcb" Nov 03 14:45:27.521065 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:27.520858 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vjdcb" event={"ID":"01c5a8f4-4b2d-4d24-a719-ea02e2d7cbc1","Type":"ContainerDied","Data":"6f15e5ca24b9301b6eb4f634d496be81dc6d2c28c8fadf389b3a1b6ab95bbb78"} Nov 03 14:45:27.521065 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:27.520977 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f15e5ca24b9301b6eb4f634d496be81dc6d2c28c8fadf389b3a1b6ab95bbb78" Nov 03 14:45:35.861682 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:35.861647 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:45:37.233643 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:45:37.233607 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:45:44.458495 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:44.458387 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:45:44.458495 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:44.458488 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q" podUID="fe445e0e-53c9-4543-9107-ff8f13cbd425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:45:47.856463 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:47.856424 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:45:49.362971 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:49.362937 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-xqc9n"] Nov 03 14:45:49.363446 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:49.363358 5979 cpu_manager.go:414] "containerMap: removing container" podUID="bfde0656-f585-42b0-80e5-4634f5dc5d71" containerName="cleanup" Nov 03 14:45:49.363446 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:49.363369 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfde0656-f585-42b0-80e5-4634f5dc5d71" containerName="cleanup" Nov 03 14:45:49.363446 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:49.363399 5979 cpu_manager.go:414] "containerMap: removing container" podUID="01c5a8f4-4b2d-4d24-a719-ea02e2d7cbc1" containerName="revoke-keys" Nov 03 14:45:49.363446 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:49.363405 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01c5a8f4-4b2d-4d24-a719-ea02e2d7cbc1" containerName="revoke-keys" Nov 03 14:45:49.363629 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:49.363491 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="01c5a8f4-4b2d-4d24-a719-ea02e2d7cbc1" containerName="revoke-keys" Nov 03 14:45:49.363629 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:49.363505 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfde0656-f585-42b0-80e5-4634f5dc5d71" containerName="cleanup" Nov 03 14:45:49.367291 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:49.367270 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-xqc9n" Nov 03 14:45:49.441201 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:49.441157 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22j7p\" (UniqueName: \"kubernetes.io/projected/081897b6-db58-4a1b-85f1-852046335915-kube-api-access-22j7p\") pod \"maas-api-revoke-sub-2e1f5b5f-xqc9n\" (UID: \"081897b6-db58-4a1b-85f1-852046335915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-xqc9n" Nov 03 14:45:49.441401 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:49.441229 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/081897b6-db58-4a1b-85f1-852046335915-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-xqc9n\" (UID: \"081897b6-db58-4a1b-85f1-852046335915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-xqc9n" Nov 03 14:45:49.441401 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:49.441261 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/081897b6-db58-4a1b-85f1-852046335915-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-xqc9n\" (UID: \"081897b6-db58-4a1b-85f1-852046335915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-xqc9n" Nov 03 14:45:49.462736 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:49.462697 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:45:49.462736 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:49.462713 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:45:49.462937 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:49.462739 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:45:49.462937 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:49.462784 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wgk87\"" Nov 03 14:45:49.464582 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:49.464560 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-xqc9n"] Nov 03 14:45:49.542221 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:49.542181 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22j7p\" (UniqueName: \"kubernetes.io/projected/081897b6-db58-4a1b-85f1-852046335915-kube-api-access-22j7p\") pod \"maas-api-revoke-sub-2e1f5b5f-xqc9n\" (UID: \"081897b6-db58-4a1b-85f1-852046335915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-xqc9n" Nov 03 14:45:49.542396 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:49.542225 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/081897b6-db58-4a1b-85f1-852046335915-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-xqc9n\" (UID: \"081897b6-db58-4a1b-85f1-852046335915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-xqc9n" Nov 03 14:45:49.542396 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:49.542278 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/081897b6-db58-4a1b-85f1-852046335915-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-xqc9n\" (UID: \"081897b6-db58-4a1b-85f1-852046335915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-xqc9n" Nov 03 14:45:49.543052 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:49.543029 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/081897b6-db58-4a1b-85f1-852046335915-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-xqc9n\" (UID: \"081897b6-db58-4a1b-85f1-852046335915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-xqc9n" Nov 03 14:45:49.544683 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:49.544661 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/081897b6-db58-4a1b-85f1-852046335915-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-xqc9n\" (UID: \"081897b6-db58-4a1b-85f1-852046335915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-xqc9n" Nov 03 14:45:49.602091 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:49.602061 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22j7p\" (UniqueName: \"kubernetes.io/projected/081897b6-db58-4a1b-85f1-852046335915-kube-api-access-22j7p\") pod \"maas-api-revoke-sub-2e1f5b5f-xqc9n\" (UID: \"081897b6-db58-4a1b-85f1-852046335915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-xqc9n" Nov 03 14:45:49.681819 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:49.681784 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-xqc9n" Nov 03 14:45:50.049636 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:50.049555 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-xqc9n"] Nov 03 14:45:50.056778 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:45:50.056750 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod081897b6_db58_4a1b_85f1_852046335915.slice/crio-b123d6b3d493e06290b065ab78a483a50141157158cfddfddf01e3699bb534cb WatchSource:0}: Error finding container b123d6b3d493e06290b065ab78a483a50141157158cfddfddf01e3699bb534cb: Status 404 returned error can't find the container with id b123d6b3d493e06290b065ab78a483a50141157158cfddfddf01e3699bb534cb Nov 03 14:45:50.682532 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:50.682494 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-xqc9n" event={"ID":"081897b6-db58-4a1b-85f1-852046335915","Type":"ContainerStarted","Data":"d861dce9a8f5bea87891b8fbc22c119502cbaf9c416c5604b4d7547ad1495d96"} Nov 03 14:45:50.683045 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:50.682555 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-xqc9n" event={"ID":"081897b6-db58-4a1b-85f1-852046335915","Type":"ContainerStarted","Data":"b123d6b3d493e06290b065ab78a483a50141157158cfddfddf01e3699bb534cb"} Nov 03 14:45:50.792056 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:50.792001 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-xqc9n" podStartSLOduration=1.791986635 podStartE2EDuration="1.791986635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:45:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:45:50.79069946 +0000 UTC m=+6032.801387703" watchObservedRunningTime="2026-11-03 14:45:50.791986635 +0000 UTC m=+6032.802674912" Nov 03 14:45:51.690035 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:51.690001 5979 generic.go:356] "Generic (PLEG): container finished" podID="081897b6-db58-4a1b-85f1-852046335915" containerID="d861dce9a8f5bea87891b8fbc22c119502cbaf9c416c5604b4d7547ad1495d96" exitCode=0 Nov 03 14:45:51.690436 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:51.690083 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-xqc9n" event={"ID":"081897b6-db58-4a1b-85f1-852046335915","Type":"ContainerDied","Data":"d861dce9a8f5bea87891b8fbc22c119502cbaf9c416c5604b4d7547ad1495d96"} Nov 03 14:45:52.827395 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:52.827370 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-xqc9n" Nov 03 14:45:52.975666 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:52.975587 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/081897b6-db58-4a1b-85f1-852046335915-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/081897b6-db58-4a1b-85f1-852046335915-maas-api-openshift-ca\") pod \"081897b6-db58-4a1b-85f1-852046335915\" (UID: \"081897b6-db58-4a1b-85f1-852046335915\") " Nov 03 14:45:52.975666 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:52.975655 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/081897b6-db58-4a1b-85f1-852046335915-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/081897b6-db58-4a1b-85f1-852046335915-maas-api-xks-ca\") pod \"081897b6-db58-4a1b-85f1-852046335915\" (UID: \"081897b6-db58-4a1b-85f1-852046335915\") " Nov 03 14:45:52.975873 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:52.975715 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/081897b6-db58-4a1b-85f1-852046335915-kube-api-access-22j7p\" (UniqueName: \"kubernetes.io/projected/081897b6-db58-4a1b-85f1-852046335915-kube-api-access-22j7p\") pod \"081897b6-db58-4a1b-85f1-852046335915\" (UID: \"081897b6-db58-4a1b-85f1-852046335915\") " Nov 03 14:45:52.975976 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:52.975947 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/081897b6-db58-4a1b-85f1-852046335915-maas-api-openshift-ca" pod "081897b6-db58-4a1b-85f1-852046335915" (UID: "081897b6-db58-4a1b-85f1-852046335915"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:45:52.976089 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:52.976064 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/081897b6-db58-4a1b-85f1-852046335915-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:45:52.977887 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:52.977865 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/081897b6-db58-4a1b-85f1-852046335915-maas-api-xks-ca" pod "081897b6-db58-4a1b-85f1-852046335915" (UID: "081897b6-db58-4a1b-85f1-852046335915"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:45:52.977964 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:52.977925 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/081897b6-db58-4a1b-85f1-852046335915-kube-api-access-22j7p" pod "081897b6-db58-4a1b-85f1-852046335915" (UID: "081897b6-db58-4a1b-85f1-852046335915"). InnerVolumeSpecName "kube-api-access-22j7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:45:53.077309 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:53.077276 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/081897b6-db58-4a1b-85f1-852046335915-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:45:53.077309 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:53.077309 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22j7p\" (UniqueName: \"kubernetes.io/projected/081897b6-db58-4a1b-85f1-852046335915-kube-api-access-22j7p\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:45:53.702221 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:53.702184 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-xqc9n" event={"ID":"081897b6-db58-4a1b-85f1-852046335915","Type":"ContainerDied","Data":"b123d6b3d493e06290b065ab78a483a50141157158cfddfddf01e3699bb534cb"} Nov 03 14:45:53.702429 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:53.702253 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b123d6b3d493e06290b065ab78a483a50141157158cfddfddf01e3699bb534cb" Nov 03 14:45:53.702429 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:45:53.702194 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-xqc9n" Nov 03 14:46:01.993623 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:01.993529 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r8gdk"] Nov 03 14:46:01.994044 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:01.994026 5979 cpu_manager.go:414] "containerMap: removing container" podUID="081897b6-db58-4a1b-85f1-852046335915" containerName="revoke-keys" Nov 03 14:46:01.994044 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:01.994041 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="081897b6-db58-4a1b-85f1-852046335915" containerName="revoke-keys" Nov 03 14:46:01.994138 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:01.994109 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="081897b6-db58-4a1b-85f1-852046335915" containerName="revoke-keys" Nov 03 14:46:01.997700 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:01.997678 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r8gdk" Nov 03 14:46:02.093531 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:02.093493 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:46:02.093531 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:02.093503 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wgk87\"" Nov 03 14:46:02.093531 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:02.093500 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:46:02.093813 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:02.093554 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:46:02.095500 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:02.095466 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r8gdk"] Nov 03 14:46:02.169941 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:02.169901 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdtzr\" (UniqueName: \"kubernetes.io/projected/1e9295fd-84fd-4ae3-88f5-70dedac0bcb3-kube-api-access-qdtzr\") pod \"maas-api-revoke-sub-ec843018-r8gdk\" (UID: \"1e9295fd-84fd-4ae3-88f5-70dedac0bcb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r8gdk" Nov 03 14:46:02.170135 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:02.169965 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e9295fd-84fd-4ae3-88f5-70dedac0bcb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-r8gdk\" (UID: \"1e9295fd-84fd-4ae3-88f5-70dedac0bcb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r8gdk" Nov 03 14:46:02.170135 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:02.170075 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e9295fd-84fd-4ae3-88f5-70dedac0bcb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-r8gdk\" (UID: \"1e9295fd-84fd-4ae3-88f5-70dedac0bcb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r8gdk" Nov 03 14:46:02.250176 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:02.250088 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:46:02.271154 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:02.271116 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdtzr\" (UniqueName: \"kubernetes.io/projected/1e9295fd-84fd-4ae3-88f5-70dedac0bcb3-kube-api-access-qdtzr\") pod \"maas-api-revoke-sub-ec843018-r8gdk\" (UID: \"1e9295fd-84fd-4ae3-88f5-70dedac0bcb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r8gdk" Nov 03 14:46:02.271316 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:02.271200 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e9295fd-84fd-4ae3-88f5-70dedac0bcb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-r8gdk\" (UID: \"1e9295fd-84fd-4ae3-88f5-70dedac0bcb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r8gdk" Nov 03 14:46:02.271316 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:02.271259 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e9295fd-84fd-4ae3-88f5-70dedac0bcb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-r8gdk\" (UID: \"1e9295fd-84fd-4ae3-88f5-70dedac0bcb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r8gdk" Nov 03 14:46:02.271960 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:02.271935 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e9295fd-84fd-4ae3-88f5-70dedac0bcb3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-r8gdk\" (UID: \"1e9295fd-84fd-4ae3-88f5-70dedac0bcb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r8gdk" Nov 03 14:46:02.273501 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:02.273465 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e9295fd-84fd-4ae3-88f5-70dedac0bcb3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-r8gdk\" (UID: \"1e9295fd-84fd-4ae3-88f5-70dedac0bcb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r8gdk" Nov 03 14:46:02.325627 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:02.325596 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdtzr\" (UniqueName: \"kubernetes.io/projected/1e9295fd-84fd-4ae3-88f5-70dedac0bcb3-kube-api-access-qdtzr\") pod \"maas-api-revoke-sub-ec843018-r8gdk\" (UID: \"1e9295fd-84fd-4ae3-88f5-70dedac0bcb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r8gdk" Nov 03 14:46:02.610713 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:02.610613 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r8gdk" Nov 03 14:46:02.993080 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:02.993041 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r8gdk"] Nov 03 14:46:03.000165 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:46:03.000136 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e9295fd_84fd_4ae3_88f5_70dedac0bcb3.slice/crio-e1ad21284c755e87af8da8a47f22969212683f80acff510380d9d1491900bcd7 WatchSource:0}: Error finding container e1ad21284c755e87af8da8a47f22969212683f80acff510380d9d1491900bcd7: Status 404 returned error can't find the container with id e1ad21284c755e87af8da8a47f22969212683f80acff510380d9d1491900bcd7 Nov 03 14:46:03.758005 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:03.757965 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r8gdk" event={"ID":"1e9295fd-84fd-4ae3-88f5-70dedac0bcb3","Type":"ContainerStarted","Data":"6be055b85bc6a256dc6a68ba7405c86d858a44cfc88994207e2617c1356458a2"} Nov 03 14:46:03.758218 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:03.758020 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r8gdk" event={"ID":"1e9295fd-84fd-4ae3-88f5-70dedac0bcb3","Type":"ContainerStarted","Data":"e1ad21284c755e87af8da8a47f22969212683f80acff510380d9d1491900bcd7"} Nov 03 14:46:03.867892 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:03.867836 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r8gdk" podStartSLOduration=2.867821301 podStartE2EDuration="2.867821301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:46:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:46:03.866658891 +0000 UTC m=+6045.877347157" watchObservedRunningTime="2026-11-03 14:46:03.867821301 +0000 UTC m=+6045.878509582" Nov 03 14:46:04.764170 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:04.764133 5979 generic.go:356] "Generic (PLEG): container finished" podID="1e9295fd-84fd-4ae3-88f5-70dedac0bcb3" containerID="6be055b85bc6a256dc6a68ba7405c86d858a44cfc88994207e2617c1356458a2" exitCode=0 Nov 03 14:46:04.764615 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:04.764200 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r8gdk" event={"ID":"1e9295fd-84fd-4ae3-88f5-70dedac0bcb3","Type":"ContainerDied","Data":"6be055b85bc6a256dc6a68ba7405c86d858a44cfc88994207e2617c1356458a2"} Nov 03 14:46:05.907665 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:05.907637 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r8gdk" Nov 03 14:46:06.012591 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:06.012555 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e9295fd-84fd-4ae3-88f5-70dedac0bcb3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e9295fd-84fd-4ae3-88f5-70dedac0bcb3-maas-api-xks-ca\") pod \"1e9295fd-84fd-4ae3-88f5-70dedac0bcb3\" (UID: \"1e9295fd-84fd-4ae3-88f5-70dedac0bcb3\") " Nov 03 14:46:06.012794 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:06.012626 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e9295fd-84fd-4ae3-88f5-70dedac0bcb3-kube-api-access-qdtzr\" (UniqueName: \"kubernetes.io/projected/1e9295fd-84fd-4ae3-88f5-70dedac0bcb3-kube-api-access-qdtzr\") pod \"1e9295fd-84fd-4ae3-88f5-70dedac0bcb3\" (UID: \"1e9295fd-84fd-4ae3-88f5-70dedac0bcb3\") " Nov 03 14:46:06.012794 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:06.012686 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e9295fd-84fd-4ae3-88f5-70dedac0bcb3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e9295fd-84fd-4ae3-88f5-70dedac0bcb3-maas-api-openshift-ca\") pod \"1e9295fd-84fd-4ae3-88f5-70dedac0bcb3\" (UID: \"1e9295fd-84fd-4ae3-88f5-70dedac0bcb3\") " Nov 03 14:46:06.013055 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:06.013030 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e9295fd-84fd-4ae3-88f5-70dedac0bcb3-maas-api-openshift-ca" pod "1e9295fd-84fd-4ae3-88f5-70dedac0bcb3" (UID: "1e9295fd-84fd-4ae3-88f5-70dedac0bcb3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:46:06.014811 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:06.014782 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e9295fd-84fd-4ae3-88f5-70dedac0bcb3-maas-api-xks-ca" pod "1e9295fd-84fd-4ae3-88f5-70dedac0bcb3" (UID: "1e9295fd-84fd-4ae3-88f5-70dedac0bcb3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:46:06.014937 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:06.014843 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e9295fd-84fd-4ae3-88f5-70dedac0bcb3-kube-api-access-qdtzr" pod "1e9295fd-84fd-4ae3-88f5-70dedac0bcb3" (UID: "1e9295fd-84fd-4ae3-88f5-70dedac0bcb3"). InnerVolumeSpecName "kube-api-access-qdtzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:46:06.113697 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:06.113661 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e9295fd-84fd-4ae3-88f5-70dedac0bcb3-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:46:06.113697 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:06.113688 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e9295fd-84fd-4ae3-88f5-70dedac0bcb3-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:46:06.113697 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:06.113698 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdtzr\" (UniqueName: \"kubernetes.io/projected/1e9295fd-84fd-4ae3-88f5-70dedac0bcb3-kube-api-access-qdtzr\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:46:06.774777 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:06.774745 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r8gdk" Nov 03 14:46:06.774962 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:06.774783 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r8gdk" event={"ID":"1e9295fd-84fd-4ae3-88f5-70dedac0bcb3","Type":"ContainerDied","Data":"e1ad21284c755e87af8da8a47f22969212683f80acff510380d9d1491900bcd7"} Nov 03 14:46:06.774962 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:06.774830 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1ad21284c755e87af8da8a47f22969212683f80acff510380d9d1491900bcd7" Nov 03 14:46:07.234052 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:46:07.234011 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:46:15.043693 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:15.043653 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sgx99"] Nov 03 14:46:15.090296 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:15.090265 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-sgx99"] Nov 03 14:46:16.114186 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:16.114147 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74fcaea3-7dd0-47e8-88b1-995b55f49e02" path="/var/lib/kubelet/pods/74fcaea3-7dd0-47e8-88b1-995b55f49e02/volumes" Nov 03 14:46:19.280236 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:19.280203 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:46:37.233417 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:46:37.233378 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:46:37.514883 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:37.514790 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vbcp9"] Nov 03 14:46:37.515318 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:37.515298 5979 cpu_manager.go:414] "containerMap: removing container" podUID="1e9295fd-84fd-4ae3-88f5-70dedac0bcb3" containerName="revoke-keys" Nov 03 14:46:37.515412 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:37.515320 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e9295fd-84fd-4ae3-88f5-70dedac0bcb3" containerName="revoke-keys" Nov 03 14:46:37.515513 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:37.515486 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e9295fd-84fd-4ae3-88f5-70dedac0bcb3" containerName="revoke-keys" Nov 03 14:46:37.520166 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:37.520142 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vbcp9" Nov 03 14:46:37.595147 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:37.595102 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8750bbf1-49f2-45ef-ad4a-33724ccfd791-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-vbcp9\" (UID: \"8750bbf1-49f2-45ef-ad4a-33724ccfd791\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vbcp9" Nov 03 14:46:37.595366 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:37.595215 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8750bbf1-49f2-45ef-ad4a-33724ccfd791-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-vbcp9\" (UID: \"8750bbf1-49f2-45ef-ad4a-33724ccfd791\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vbcp9" Nov 03 14:46:37.595366 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:37.595273 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlq9r\" (UniqueName: \"kubernetes.io/projected/8750bbf1-49f2-45ef-ad4a-33724ccfd791-kube-api-access-nlq9r\") pod \"maas-api-revoke-sub-046e1343-vbcp9\" (UID: \"8750bbf1-49f2-45ef-ad4a-33724ccfd791\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vbcp9" Nov 03 14:46:37.616814 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:37.616774 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:46:37.617027 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:37.616816 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:46:37.617027 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:37.616888 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wgk87\"" Nov 03 14:46:37.617027 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:37.616891 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:46:37.619160 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:37.619137 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vbcp9"] Nov 03 14:46:37.696783 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:37.696750 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8750bbf1-49f2-45ef-ad4a-33724ccfd791-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-vbcp9\" (UID: \"8750bbf1-49f2-45ef-ad4a-33724ccfd791\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vbcp9" Nov 03 14:46:37.696992 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:37.696902 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8750bbf1-49f2-45ef-ad4a-33724ccfd791-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-vbcp9\" (UID: \"8750bbf1-49f2-45ef-ad4a-33724ccfd791\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vbcp9" Nov 03 14:46:37.696992 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:37.696936 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlq9r\" (UniqueName: \"kubernetes.io/projected/8750bbf1-49f2-45ef-ad4a-33724ccfd791-kube-api-access-nlq9r\") pod \"maas-api-revoke-sub-046e1343-vbcp9\" (UID: \"8750bbf1-49f2-45ef-ad4a-33724ccfd791\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vbcp9" Nov 03 14:46:37.697490 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:37.697453 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8750bbf1-49f2-45ef-ad4a-33724ccfd791-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-vbcp9\" (UID: \"8750bbf1-49f2-45ef-ad4a-33724ccfd791\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vbcp9" Nov 03 14:46:37.699216 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:37.699195 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8750bbf1-49f2-45ef-ad4a-33724ccfd791-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-vbcp9\" (UID: \"8750bbf1-49f2-45ef-ad4a-33724ccfd791\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vbcp9" Nov 03 14:46:37.750945 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:37.750905 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlq9r\" (UniqueName: \"kubernetes.io/projected/8750bbf1-49f2-45ef-ad4a-33724ccfd791-kube-api-access-nlq9r\") pod \"maas-api-revoke-sub-046e1343-vbcp9\" (UID: \"8750bbf1-49f2-45ef-ad4a-33724ccfd791\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vbcp9" Nov 03 14:46:37.832911 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:37.832833 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vbcp9" Nov 03 14:46:38.202675 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:38.202640 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vbcp9"] Nov 03 14:46:38.209298 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:46:38.209264 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8750bbf1_49f2_45ef_ad4a_33724ccfd791.slice/crio-26bd654d090b8284bc54990835607e0ccb1da935f1e12b903f7b2f5116fdfe1d WatchSource:0}: Error finding container 26bd654d090b8284bc54990835607e0ccb1da935f1e12b903f7b2f5116fdfe1d: Status 404 returned error can't find the container with id 26bd654d090b8284bc54990835607e0ccb1da935f1e12b903f7b2f5116fdfe1d Nov 03 14:46:38.962424 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:38.962390 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vbcp9" event={"ID":"8750bbf1-49f2-45ef-ad4a-33724ccfd791","Type":"ContainerStarted","Data":"8a1651867885d5c9e1fdecf63705a7a11f118ac710cb209dbbd8a284cb13c6ed"} Nov 03 14:46:38.962856 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:38.962452 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vbcp9" event={"ID":"8750bbf1-49f2-45ef-ad4a-33724ccfd791","Type":"ContainerStarted","Data":"26bd654d090b8284bc54990835607e0ccb1da935f1e12b903f7b2f5116fdfe1d"} Nov 03 14:46:39.071193 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:39.071138 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vbcp9" podStartSLOduration=2.071123896 podStartE2EDuration="2.071123896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:46:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:46:39.069389632 +0000 UTC m=+6081.080077876" watchObservedRunningTime="2026-11-03 14:46:39.071123896 +0000 UTC m=+6081.081812137" Nov 03 14:46:39.968599 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:39.968563 5979 generic.go:356] "Generic (PLEG): container finished" podID="8750bbf1-49f2-45ef-ad4a-33724ccfd791" containerID="8a1651867885d5c9e1fdecf63705a7a11f118ac710cb209dbbd8a284cb13c6ed" exitCode=0 Nov 03 14:46:39.969074 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:39.968652 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vbcp9" event={"ID":"8750bbf1-49f2-45ef-ad4a-33724ccfd791","Type":"ContainerDied","Data":"8a1651867885d5c9e1fdecf63705a7a11f118ac710cb209dbbd8a284cb13c6ed"} Nov 03 14:46:41.110543 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:41.110518 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vbcp9" Nov 03 14:46:41.130398 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:41.130369 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8750bbf1-49f2-45ef-ad4a-33724ccfd791-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8750bbf1-49f2-45ef-ad4a-33724ccfd791-maas-api-xks-ca\") pod \"8750bbf1-49f2-45ef-ad4a-33724ccfd791\" (UID: \"8750bbf1-49f2-45ef-ad4a-33724ccfd791\") " Nov 03 14:46:41.130550 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:41.130513 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8750bbf1-49f2-45ef-ad4a-33724ccfd791-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8750bbf1-49f2-45ef-ad4a-33724ccfd791-maas-api-openshift-ca\") pod \"8750bbf1-49f2-45ef-ad4a-33724ccfd791\" (UID: \"8750bbf1-49f2-45ef-ad4a-33724ccfd791\") " Nov 03 14:46:41.130599 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:41.130587 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8750bbf1-49f2-45ef-ad4a-33724ccfd791-kube-api-access-nlq9r\" (UniqueName: \"kubernetes.io/projected/8750bbf1-49f2-45ef-ad4a-33724ccfd791-kube-api-access-nlq9r\") pod \"8750bbf1-49f2-45ef-ad4a-33724ccfd791\" (UID: \"8750bbf1-49f2-45ef-ad4a-33724ccfd791\") " Nov 03 14:46:41.130854 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:41.130831 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8750bbf1-49f2-45ef-ad4a-33724ccfd791-maas-api-openshift-ca" pod "8750bbf1-49f2-45ef-ad4a-33724ccfd791" (UID: "8750bbf1-49f2-45ef-ad4a-33724ccfd791"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:46:41.131146 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:41.130969 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8750bbf1-49f2-45ef-ad4a-33724ccfd791-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:46:41.132649 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:41.132627 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8750bbf1-49f2-45ef-ad4a-33724ccfd791-maas-api-xks-ca" pod "8750bbf1-49f2-45ef-ad4a-33724ccfd791" (UID: "8750bbf1-49f2-45ef-ad4a-33724ccfd791"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:46:41.132738 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:41.132663 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8750bbf1-49f2-45ef-ad4a-33724ccfd791-kube-api-access-nlq9r" pod "8750bbf1-49f2-45ef-ad4a-33724ccfd791" (UID: "8750bbf1-49f2-45ef-ad4a-33724ccfd791"). InnerVolumeSpecName "kube-api-access-nlq9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:46:41.232068 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:41.231970 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlq9r\" (UniqueName: \"kubernetes.io/projected/8750bbf1-49f2-45ef-ad4a-33724ccfd791-kube-api-access-nlq9r\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:46:41.232068 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:41.232006 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8750bbf1-49f2-45ef-ad4a-33724ccfd791-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:46:41.979110 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:41.979067 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vbcp9" event={"ID":"8750bbf1-49f2-45ef-ad4a-33724ccfd791","Type":"ContainerDied","Data":"26bd654d090b8284bc54990835607e0ccb1da935f1e12b903f7b2f5116fdfe1d"} Nov 03 14:46:41.979110 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:41.979116 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26bd654d090b8284bc54990835607e0ccb1da935f1e12b903f7b2f5116fdfe1d" Nov 03 14:46:41.979335 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:41.979083 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vbcp9" Nov 03 14:46:47.044775 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:47.044735 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-j4png"] Nov 03 14:46:47.091103 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:47.091069 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-j4png"] Nov 03 14:46:48.113912 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:48.113879 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1305c97a-0ac3-4f7a-9735-d78406162d1e" path="/var/lib/kubelet/pods/1305c97a-0ac3-4f7a-9735-d78406162d1e/volumes" Nov 03 14:46:48.327321 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:48.327285 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v6c4j"] Nov 03 14:46:48.327734 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:48.327721 5979 cpu_manager.go:414] "containerMap: removing container" podUID="8750bbf1-49f2-45ef-ad4a-33724ccfd791" containerName="revoke-keys" Nov 03 14:46:48.327796 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:48.327735 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8750bbf1-49f2-45ef-ad4a-33724ccfd791" containerName="revoke-keys" Nov 03 14:46:48.327837 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:48.327819 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="8750bbf1-49f2-45ef-ad4a-33724ccfd791" containerName="revoke-keys" Nov 03 14:46:48.331483 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:48.331454 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v6c4j" Nov 03 14:46:48.380649 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:48.380578 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:46:48.403493 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:48.402277 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dhfl\" (UniqueName: \"kubernetes.io/projected/04264336-4e68-424f-93a9-190b9bfe5606-kube-api-access-6dhfl\") pod \"maas-api-revoke-sub-d3fba056-v6c4j\" (UID: \"04264336-4e68-424f-93a9-190b9bfe5606\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v6c4j" Nov 03 14:46:48.403493 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:48.402438 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04264336-4e68-424f-93a9-190b9bfe5606-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-v6c4j\" (UID: \"04264336-4e68-424f-93a9-190b9bfe5606\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v6c4j" Nov 03 14:46:48.403493 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:48.402501 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04264336-4e68-424f-93a9-190b9bfe5606-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-v6c4j\" (UID: \"04264336-4e68-424f-93a9-190b9bfe5606\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v6c4j" Nov 03 14:46:48.427188 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:48.427157 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:46:48.427344 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:48.427156 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:46:48.427344 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:48.427163 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wgk87\"" Nov 03 14:46:48.429112 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:48.429091 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v6c4j"] Nov 03 14:46:48.504020 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:48.503981 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04264336-4e68-424f-93a9-190b9bfe5606-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-v6c4j\" (UID: \"04264336-4e68-424f-93a9-190b9bfe5606\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v6c4j" Nov 03 14:46:48.504230 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:48.504039 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04264336-4e68-424f-93a9-190b9bfe5606-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-v6c4j\" (UID: \"04264336-4e68-424f-93a9-190b9bfe5606\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v6c4j" Nov 03 14:46:48.504230 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:48.504102 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dhfl\" (UniqueName: \"kubernetes.io/projected/04264336-4e68-424f-93a9-190b9bfe5606-kube-api-access-6dhfl\") pod \"maas-api-revoke-sub-d3fba056-v6c4j\" (UID: \"04264336-4e68-424f-93a9-190b9bfe5606\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v6c4j" Nov 03 14:46:48.504637 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:48.504613 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04264336-4e68-424f-93a9-190b9bfe5606-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-v6c4j\" (UID: \"04264336-4e68-424f-93a9-190b9bfe5606\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v6c4j" Nov 03 14:46:48.506378 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:48.506358 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04264336-4e68-424f-93a9-190b9bfe5606-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-v6c4j\" (UID: \"04264336-4e68-424f-93a9-190b9bfe5606\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v6c4j" Nov 03 14:46:48.559963 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:48.559938 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dhfl\" (UniqueName: \"kubernetes.io/projected/04264336-4e68-424f-93a9-190b9bfe5606-kube-api-access-6dhfl\") pod \"maas-api-revoke-sub-d3fba056-v6c4j\" (UID: \"04264336-4e68-424f-93a9-190b9bfe5606\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v6c4j" Nov 03 14:46:48.644068 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:48.644037 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v6c4j" Nov 03 14:46:49.012326 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:49.012218 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v6c4j"] Nov 03 14:46:49.019438 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:46:49.019409 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04264336_4e68_424f_93a9_190b9bfe5606.slice/crio-ec873437ea0f3be595f7e1177e4f2ed60bd3eaeb598f4e72778429e553f992d7 WatchSource:0}: Error finding container ec873437ea0f3be595f7e1177e4f2ed60bd3eaeb598f4e72778429e553f992d7: Status 404 returned error can't find the container with id ec873437ea0f3be595f7e1177e4f2ed60bd3eaeb598f4e72778429e553f992d7 Nov 03 14:46:50.024577 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:50.024541 5979 generic.go:356] "Generic (PLEG): container finished" podID="04264336-4e68-424f-93a9-190b9bfe5606" containerID="f800bad9ce14929b16fb99c223119a4d178c448ecec653bfff48c4b73ac78269" exitCode=0 Nov 03 14:46:50.025022 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:50.024629 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v6c4j" event={"ID":"04264336-4e68-424f-93a9-190b9bfe5606","Type":"ContainerDied","Data":"f800bad9ce14929b16fb99c223119a4d178c448ecec653bfff48c4b73ac78269"} Nov 03 14:46:50.025022 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:50.024697 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v6c4j" event={"ID":"04264336-4e68-424f-93a9-190b9bfe5606","Type":"ContainerStarted","Data":"ec873437ea0f3be595f7e1177e4f2ed60bd3eaeb598f4e72778429e553f992d7"} Nov 03 14:46:50.348337 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:50.348297 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:46:51.240661 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:51.240623 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v6c4j" Nov 03 14:46:51.337293 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:51.337184 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04264336-4e68-424f-93a9-190b9bfe5606-kube-api-access-6dhfl\" (UniqueName: \"kubernetes.io/projected/04264336-4e68-424f-93a9-190b9bfe5606-kube-api-access-6dhfl\") pod \"04264336-4e68-424f-93a9-190b9bfe5606\" (UID: \"04264336-4e68-424f-93a9-190b9bfe5606\") " Nov 03 14:46:51.337293 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:51.337263 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04264336-4e68-424f-93a9-190b9bfe5606-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04264336-4e68-424f-93a9-190b9bfe5606-maas-api-openshift-ca\") pod \"04264336-4e68-424f-93a9-190b9bfe5606\" (UID: \"04264336-4e68-424f-93a9-190b9bfe5606\") " Nov 03 14:46:51.337595 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:51.337364 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04264336-4e68-424f-93a9-190b9bfe5606-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04264336-4e68-424f-93a9-190b9bfe5606-maas-api-xks-ca\") pod \"04264336-4e68-424f-93a9-190b9bfe5606\" (UID: \"04264336-4e68-424f-93a9-190b9bfe5606\") " Nov 03 14:46:51.338059 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:51.338027 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04264336-4e68-424f-93a9-190b9bfe5606-maas-api-openshift-ca" pod "04264336-4e68-424f-93a9-190b9bfe5606" (UID: "04264336-4e68-424f-93a9-190b9bfe5606"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:46:51.340295 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:51.340052 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04264336-4e68-424f-93a9-190b9bfe5606-kube-api-access-6dhfl" pod "04264336-4e68-424f-93a9-190b9bfe5606" (UID: "04264336-4e68-424f-93a9-190b9bfe5606"). InnerVolumeSpecName "kube-api-access-6dhfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:46:51.344698 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:51.344664 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04264336-4e68-424f-93a9-190b9bfe5606-maas-api-xks-ca" pod "04264336-4e68-424f-93a9-190b9bfe5606" (UID: "04264336-4e68-424f-93a9-190b9bfe5606"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:46:51.439491 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:51.439427 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dhfl\" (UniqueName: \"kubernetes.io/projected/04264336-4e68-424f-93a9-190b9bfe5606-kube-api-access-6dhfl\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:46:51.439713 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:51.439510 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04264336-4e68-424f-93a9-190b9bfe5606-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:46:51.439713 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:51.439531 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04264336-4e68-424f-93a9-190b9bfe5606-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:46:52.051494 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:52.051016 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v6c4j" event={"ID":"04264336-4e68-424f-93a9-190b9bfe5606","Type":"ContainerDied","Data":"ec873437ea0f3be595f7e1177e4f2ed60bd3eaeb598f4e72778429e553f992d7"} Nov 03 14:46:52.051494 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:52.051079 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec873437ea0f3be595f7e1177e4f2ed60bd3eaeb598f4e72778429e553f992d7" Nov 03 14:46:52.051494 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:46:52.051149 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v6c4j" Nov 03 14:47:01.077870 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:01.077832 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:47:07.233072 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:47:07.233031 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:47:11.045099 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:11.045061 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c10bde9-2z295"] Nov 03 14:47:11.091575 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:11.091545 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c10bde9-2z295"] Nov 03 14:47:12.114620 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:12.114587 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20381d2b-0de9-4b7b-96f9-507c3bc3e365" path="/var/lib/kubelet/pods/20381d2b-0de9-4b7b-96f9-507c3bc3e365/volumes" Nov 03 14:47:13.056943 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:13.056906 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:47:22.623071 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:22.623038 5979 scope.go:122] "RemoveContainer" containerID="7ce7ca58326e1f9d1a17e3d043558ef2c0ec484cdbedefdb31ef3b987f36397a" Nov 03 14:47:23.370990 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:23.370953 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5sj28"] Nov 03 14:47:23.371398 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:23.371386 5979 cpu_manager.go:414] "containerMap: removing container" podUID="04264336-4e68-424f-93a9-190b9bfe5606" containerName="revoke-keys" Nov 03 14:47:23.371446 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:23.371399 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04264336-4e68-424f-93a9-190b9bfe5606" containerName="revoke-keys" Nov 03 14:47:23.371510 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:23.371481 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="04264336-4e68-424f-93a9-190b9bfe5606" containerName="revoke-keys" Nov 03 14:47:23.376205 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:23.376183 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5sj28" Nov 03 14:47:23.425524 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:23.425495 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:47:23.425815 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:23.425761 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:47:23.456737 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:23.456706 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b59pt\" (UniqueName: \"kubernetes.io/projected/1888aae5-3017-43bd-baa1-57623738a1a0-kube-api-access-b59pt\") pod \"maas-api-revoke-sub-ae594979-5sj28\" (UID: \"1888aae5-3017-43bd-baa1-57623738a1a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5sj28" Nov 03 14:47:23.456915 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:23.456762 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1888aae5-3017-43bd-baa1-57623738a1a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-5sj28\" (UID: \"1888aae5-3017-43bd-baa1-57623738a1a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5sj28" Nov 03 14:47:23.456915 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:23.456889 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1888aae5-3017-43bd-baa1-57623738a1a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-5sj28\" (UID: \"1888aae5-3017-43bd-baa1-57623738a1a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5sj28" Nov 03 14:47:23.472184 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:23.472150 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wgk87\"" Nov 03 14:47:23.472184 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:23.472150 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:47:23.474060 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:23.474037 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5sj28"] Nov 03 14:47:23.557967 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:23.557938 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1888aae5-3017-43bd-baa1-57623738a1a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-5sj28\" (UID: \"1888aae5-3017-43bd-baa1-57623738a1a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5sj28" Nov 03 14:47:23.558150 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:23.558007 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1888aae5-3017-43bd-baa1-57623738a1a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-5sj28\" (UID: \"1888aae5-3017-43bd-baa1-57623738a1a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5sj28" Nov 03 14:47:23.558150 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:23.558069 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b59pt\" (UniqueName: \"kubernetes.io/projected/1888aae5-3017-43bd-baa1-57623738a1a0-kube-api-access-b59pt\") pod \"maas-api-revoke-sub-ae594979-5sj28\" (UID: \"1888aae5-3017-43bd-baa1-57623738a1a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5sj28" Nov 03 14:47:23.558589 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:23.558569 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1888aae5-3017-43bd-baa1-57623738a1a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-5sj28\" (UID: \"1888aae5-3017-43bd-baa1-57623738a1a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5sj28" Nov 03 14:47:23.560263 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:23.560244 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1888aae5-3017-43bd-baa1-57623738a1a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-5sj28\" (UID: \"1888aae5-3017-43bd-baa1-57623738a1a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5sj28" Nov 03 14:47:23.612073 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:23.612042 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b59pt\" (UniqueName: \"kubernetes.io/projected/1888aae5-3017-43bd-baa1-57623738a1a0-kube-api-access-b59pt\") pod \"maas-api-revoke-sub-ae594979-5sj28\" (UID: \"1888aae5-3017-43bd-baa1-57623738a1a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5sj28" Nov 03 14:47:23.690177 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:23.690145 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5sj28" Nov 03 14:47:24.059650 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:24.059565 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5sj28"] Nov 03 14:47:24.066648 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:47:24.066617 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1888aae5_3017_43bd_baa1_57623738a1a0.slice/crio-69f06b89c154fb84360e5e9d17266a0f167ccdea8839693ea221eaf5a73726ff WatchSource:0}: Error finding container 69f06b89c154fb84360e5e9d17266a0f167ccdea8839693ea221eaf5a73726ff: Status 404 returned error can't find the container with id 69f06b89c154fb84360e5e9d17266a0f167ccdea8839693ea221eaf5a73726ff Nov 03 14:47:24.274013 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:24.273980 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5sj28" event={"ID":"1888aae5-3017-43bd-baa1-57623738a1a0","Type":"ContainerStarted","Data":"69f06b89c154fb84360e5e9d17266a0f167ccdea8839693ea221eaf5a73726ff"} Nov 03 14:47:25.280222 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:25.280138 5979 generic.go:356] "Generic (PLEG): container finished" podID="1888aae5-3017-43bd-baa1-57623738a1a0" containerID="9dc508f8b15c9eb4cb7a842b6b07b2655a8599c0e52f43e94a94fbe131f8002c" exitCode=0 Nov 03 14:47:25.280621 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:25.280225 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5sj28" event={"ID":"1888aae5-3017-43bd-baa1-57623738a1a0","Type":"ContainerDied","Data":"9dc508f8b15c9eb4cb7a842b6b07b2655a8599c0e52f43e94a94fbe131f8002c"} Nov 03 14:47:26.418466 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:26.418440 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5sj28" Nov 03 14:47:26.486689 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:26.486658 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1888aae5-3017-43bd-baa1-57623738a1a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1888aae5-3017-43bd-baa1-57623738a1a0-maas-api-xks-ca\") pod \"1888aae5-3017-43bd-baa1-57623738a1a0\" (UID: \"1888aae5-3017-43bd-baa1-57623738a1a0\") " Nov 03 14:47:26.486864 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:26.486740 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1888aae5-3017-43bd-baa1-57623738a1a0-kube-api-access-b59pt\" (UniqueName: \"kubernetes.io/projected/1888aae5-3017-43bd-baa1-57623738a1a0-kube-api-access-b59pt\") pod \"1888aae5-3017-43bd-baa1-57623738a1a0\" (UID: \"1888aae5-3017-43bd-baa1-57623738a1a0\") " Nov 03 14:47:26.486864 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:26.486768 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1888aae5-3017-43bd-baa1-57623738a1a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1888aae5-3017-43bd-baa1-57623738a1a0-maas-api-openshift-ca\") pod \"1888aae5-3017-43bd-baa1-57623738a1a0\" (UID: \"1888aae5-3017-43bd-baa1-57623738a1a0\") " Nov 03 14:47:26.487186 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:26.487164 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1888aae5-3017-43bd-baa1-57623738a1a0-maas-api-openshift-ca" pod "1888aae5-3017-43bd-baa1-57623738a1a0" (UID: "1888aae5-3017-43bd-baa1-57623738a1a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:47:26.488866 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:26.488843 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1888aae5-3017-43bd-baa1-57623738a1a0-kube-api-access-b59pt" pod "1888aae5-3017-43bd-baa1-57623738a1a0" (UID: "1888aae5-3017-43bd-baa1-57623738a1a0"). InnerVolumeSpecName "kube-api-access-b59pt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:47:26.488990 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:26.488863 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1888aae5-3017-43bd-baa1-57623738a1a0-maas-api-xks-ca" pod "1888aae5-3017-43bd-baa1-57623738a1a0" (UID: "1888aae5-3017-43bd-baa1-57623738a1a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:47:26.588165 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:26.588082 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1888aae5-3017-43bd-baa1-57623738a1a0-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:47:26.588165 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:26.588117 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1888aae5-3017-43bd-baa1-57623738a1a0-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:47:26.588165 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:26.588127 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b59pt\" (UniqueName: \"kubernetes.io/projected/1888aae5-3017-43bd-baa1-57623738a1a0-kube-api-access-b59pt\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:47:27.291064 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:27.291035 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5sj28" Nov 03 14:47:27.291254 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:27.291035 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5sj28" event={"ID":"1888aae5-3017-43bd-baa1-57623738a1a0","Type":"ContainerDied","Data":"69f06b89c154fb84360e5e9d17266a0f167ccdea8839693ea221eaf5a73726ff"} Nov 03 14:47:27.291254 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:27.291162 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69f06b89c154fb84360e5e9d17266a0f167ccdea8839693ea221eaf5a73726ff" Nov 03 14:47:36.167082 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:36.167044 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:47:37.237112 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:47:37.237065 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:47:42.295490 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:42.294791 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-xwwdw"] Nov 03 14:47:42.315034 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:42.315000 5979 cpu_manager.go:414] "containerMap: removing container" podUID="1888aae5-3017-43bd-baa1-57623738a1a0" containerName="revoke-keys" Nov 03 14:47:42.315287 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:42.315265 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1888aae5-3017-43bd-baa1-57623738a1a0" containerName="revoke-keys" Nov 03 14:47:42.315610 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:42.315590 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="1888aae5-3017-43bd-baa1-57623738a1a0" containerName="revoke-keys" Nov 03 14:47:42.330863 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:42.330841 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-xwwdw" Nov 03 14:47:42.396510 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:42.396392 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-xwwdw"] Nov 03 14:47:42.413283 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:42.410796 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:47:42.413283 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:42.410962 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wgk87\"" Nov 03 14:47:42.413283 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:42.411034 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:47:42.467535 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:42.462765 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:47:42.638492 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:42.633667 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cea64d6-4955-4f2c-a08c-0f5d7724927b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-xwwdw\" (UID: \"5cea64d6-4955-4f2c-a08c-0f5d7724927b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-xwwdw" Nov 03 14:47:42.638492 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:42.633780 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qrtt\" (UniqueName: \"kubernetes.io/projected/5cea64d6-4955-4f2c-a08c-0f5d7724927b-kube-api-access-4qrtt\") pod \"maas-api-revoke-sub-a4aef7d7-xwwdw\" (UID: \"5cea64d6-4955-4f2c-a08c-0f5d7724927b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-xwwdw" Nov 03 14:47:42.638492 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:42.633825 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cea64d6-4955-4f2c-a08c-0f5d7724927b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-xwwdw\" (UID: \"5cea64d6-4955-4f2c-a08c-0f5d7724927b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-xwwdw" Nov 03 14:47:42.739113 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:42.739065 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cea64d6-4955-4f2c-a08c-0f5d7724927b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-xwwdw\" (UID: \"5cea64d6-4955-4f2c-a08c-0f5d7724927b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-xwwdw" Nov 03 14:47:42.739464 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:42.739441 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qrtt\" (UniqueName: \"kubernetes.io/projected/5cea64d6-4955-4f2c-a08c-0f5d7724927b-kube-api-access-4qrtt\") pod \"maas-api-revoke-sub-a4aef7d7-xwwdw\" (UID: \"5cea64d6-4955-4f2c-a08c-0f5d7724927b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-xwwdw" Nov 03 14:47:42.739661 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:42.739639 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cea64d6-4955-4f2c-a08c-0f5d7724927b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-xwwdw\" (UID: \"5cea64d6-4955-4f2c-a08c-0f5d7724927b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-xwwdw" Nov 03 14:47:42.742246 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:42.742223 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cea64d6-4955-4f2c-a08c-0f5d7724927b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-xwwdw\" (UID: \"5cea64d6-4955-4f2c-a08c-0f5d7724927b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-xwwdw" Nov 03 14:47:42.770494 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:42.760944 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cea64d6-4955-4f2c-a08c-0f5d7724927b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-xwwdw\" (UID: \"5cea64d6-4955-4f2c-a08c-0f5d7724927b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-xwwdw" Nov 03 14:47:42.818274 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:42.818237 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qrtt\" (UniqueName: \"kubernetes.io/projected/5cea64d6-4955-4f2c-a08c-0f5d7724927b-kube-api-access-4qrtt\") pod \"maas-api-revoke-sub-a4aef7d7-xwwdw\" (UID: \"5cea64d6-4955-4f2c-a08c-0f5d7724927b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-xwwdw" Nov 03 14:47:42.986732 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:42.986637 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-xwwdw" Nov 03 14:47:43.773094 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:43.773055 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-xwwdw"] Nov 03 14:47:44.006621 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:44.006578 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-lj2cw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:47:44.006919 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:44.006880 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lj2cw" podUID="7f4e540f-d1f9-4ffe-83ea-dfca92ce7233" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:47:44.483765 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:44.483670 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-xwwdw" event={"ID":"5cea64d6-4955-4f2c-a08c-0f5d7724927b","Type":"ContainerStarted","Data":"5ff445b9529969f61c99d18bf19dc90d3f0e74f5780c729f0c8cab0ced212d8d"} Nov 03 14:47:44.483765 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:44.483734 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-xwwdw" event={"ID":"5cea64d6-4955-4f2c-a08c-0f5d7724927b","Type":"ContainerStarted","Data":"bd258cdb834f6318bb574825fbceb2566699829b7f91ca1076db90c3d2829fe0"} Nov 03 14:47:44.591824 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:44.591775 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-xwwdw" podStartSLOduration=2.591761308 podStartE2EDuration="2.591761308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:47:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:47:44.590908554 +0000 UTC m=+6146.601596795" watchObservedRunningTime="2026-11-03 14:47:44.591761308 +0000 UTC m=+6146.602449550" Nov 03 14:47:45.489882 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:45.489851 5979 generic.go:356] "Generic (PLEG): container finished" podID="5cea64d6-4955-4f2c-a08c-0f5d7724927b" containerID="5ff445b9529969f61c99d18bf19dc90d3f0e74f5780c729f0c8cab0ced212d8d" exitCode=0 Nov 03 14:47:45.490274 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:45.489940 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-xwwdw" event={"ID":"5cea64d6-4955-4f2c-a08c-0f5d7724927b","Type":"ContainerDied","Data":"5ff445b9529969f61c99d18bf19dc90d3f0e74f5780c729f0c8cab0ced212d8d"} Nov 03 14:47:45.587344 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:45.587311 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:47:46.660012 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:46.659986 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-xwwdw" Nov 03 14:47:46.834030 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:46.833954 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cea64d6-4955-4f2c-a08c-0f5d7724927b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cea64d6-4955-4f2c-a08c-0f5d7724927b-maas-api-xks-ca\") pod \"5cea64d6-4955-4f2c-a08c-0f5d7724927b\" (UID: \"5cea64d6-4955-4f2c-a08c-0f5d7724927b\") " Nov 03 14:47:46.834030 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:46.834007 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cea64d6-4955-4f2c-a08c-0f5d7724927b-kube-api-access-4qrtt\" (UniqueName: \"kubernetes.io/projected/5cea64d6-4955-4f2c-a08c-0f5d7724927b-kube-api-access-4qrtt\") pod \"5cea64d6-4955-4f2c-a08c-0f5d7724927b\" (UID: \"5cea64d6-4955-4f2c-a08c-0f5d7724927b\") " Nov 03 14:47:46.834233 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:46.834126 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cea64d6-4955-4f2c-a08c-0f5d7724927b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cea64d6-4955-4f2c-a08c-0f5d7724927b-maas-api-openshift-ca\") pod \"5cea64d6-4955-4f2c-a08c-0f5d7724927b\" (UID: \"5cea64d6-4955-4f2c-a08c-0f5d7724927b\") " Nov 03 14:47:46.834550 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:46.834527 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cea64d6-4955-4f2c-a08c-0f5d7724927b-maas-api-openshift-ca" pod "5cea64d6-4955-4f2c-a08c-0f5d7724927b" (UID: "5cea64d6-4955-4f2c-a08c-0f5d7724927b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:47:46.836107 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:46.836078 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cea64d6-4955-4f2c-a08c-0f5d7724927b-maas-api-xks-ca" pod "5cea64d6-4955-4f2c-a08c-0f5d7724927b" (UID: "5cea64d6-4955-4f2c-a08c-0f5d7724927b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:47:46.836262 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:46.836206 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cea64d6-4955-4f2c-a08c-0f5d7724927b-kube-api-access-4qrtt" pod "5cea64d6-4955-4f2c-a08c-0f5d7724927b" (UID: "5cea64d6-4955-4f2c-a08c-0f5d7724927b"). InnerVolumeSpecName "kube-api-access-4qrtt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:47:46.935355 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:46.935318 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cea64d6-4955-4f2c-a08c-0f5d7724927b-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:47:46.935355 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:46.935351 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cea64d6-4955-4f2c-a08c-0f5d7724927b-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:47:46.935599 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:46.935362 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qrtt\" (UniqueName: \"kubernetes.io/projected/5cea64d6-4955-4f2c-a08c-0f5d7724927b-kube-api-access-4qrtt\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:47:47.502481 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:47.502448 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-xwwdw" Nov 03 14:47:47.502696 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:47.502454 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-xwwdw" event={"ID":"5cea64d6-4955-4f2c-a08c-0f5d7724927b","Type":"ContainerDied","Data":"bd258cdb834f6318bb574825fbceb2566699829b7f91ca1076db90c3d2829fe0"} Nov 03 14:47:47.502696 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:47.502606 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd258cdb834f6318bb574825fbceb2566699829b7f91ca1076db90c3d2829fe0" Nov 03 14:47:53.645894 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:47:53.645858 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:48:02.276005 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:02.275962 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:48:03.622136 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:03.622092 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wzslf"] Nov 03 14:48:03.641040 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:03.640999 5979 cpu_manager.go:414] "containerMap: removing container" podUID="5cea64d6-4955-4f2c-a08c-0f5d7724927b" containerName="revoke-keys" Nov 03 14:48:03.641290 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:03.641270 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cea64d6-4955-4f2c-a08c-0f5d7724927b" containerName="revoke-keys" Nov 03 14:48:03.641558 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:03.641541 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cea64d6-4955-4f2c-a08c-0f5d7724927b" containerName="revoke-keys" Nov 03 14:48:03.662493 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:03.649163 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wzslf" Nov 03 14:48:03.711504 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:03.711393 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9e1f087-e6e6-4cc7-b0c9-6ba850c9881f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-wzslf\" (UID: \"b9e1f087-e6e6-4cc7-b0c9-6ba850c9881f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wzslf" Nov 03 14:48:03.711790 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:03.711765 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhc44\" (UniqueName: \"kubernetes.io/projected/b9e1f087-e6e6-4cc7-b0c9-6ba850c9881f-kube-api-access-nhc44\") pod \"maas-api-revoke-sub-79805ca9-wzslf\" (UID: \"b9e1f087-e6e6-4cc7-b0c9-6ba850c9881f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wzslf" Nov 03 14:48:03.712124 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:03.712102 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9e1f087-e6e6-4cc7-b0c9-6ba850c9881f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-wzslf\" (UID: \"b9e1f087-e6e6-4cc7-b0c9-6ba850c9881f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wzslf" Nov 03 14:48:03.736810 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:03.715245 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wzslf"] Nov 03 14:48:03.736810 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:03.736133 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:48:03.736810 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:03.736208 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:48:03.736810 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:03.736251 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:48:03.762195 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:03.761646 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wgk87\"" Nov 03 14:48:03.909495 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:03.885806 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9e1f087-e6e6-4cc7-b0c9-6ba850c9881f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-wzslf\" (UID: \"b9e1f087-e6e6-4cc7-b0c9-6ba850c9881f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wzslf" Nov 03 14:48:03.937227 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:03.909842 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhc44\" (UniqueName: \"kubernetes.io/projected/b9e1f087-e6e6-4cc7-b0c9-6ba850c9881f-kube-api-access-nhc44\") pod \"maas-api-revoke-sub-79805ca9-wzslf\" (UID: \"b9e1f087-e6e6-4cc7-b0c9-6ba850c9881f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wzslf" Nov 03 14:48:03.937681 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:03.937646 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9e1f087-e6e6-4cc7-b0c9-6ba850c9881f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-wzslf\" (UID: \"b9e1f087-e6e6-4cc7-b0c9-6ba850c9881f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wzslf" Nov 03 14:48:03.937990 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:03.918946 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9e1f087-e6e6-4cc7-b0c9-6ba850c9881f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-wzslf\" (UID: \"b9e1f087-e6e6-4cc7-b0c9-6ba850c9881f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wzslf" Nov 03 14:48:03.938746 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:03.938712 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9e1f087-e6e6-4cc7-b0c9-6ba850c9881f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-wzslf\" (UID: \"b9e1f087-e6e6-4cc7-b0c9-6ba850c9881f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wzslf" Nov 03 14:48:04.023988 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:04.023943 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhc44\" (UniqueName: \"kubernetes.io/projected/b9e1f087-e6e6-4cc7-b0c9-6ba850c9881f-kube-api-access-nhc44\") pod \"maas-api-revoke-sub-79805ca9-wzslf\" (UID: \"b9e1f087-e6e6-4cc7-b0c9-6ba850c9881f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wzslf" Nov 03 14:48:04.041904 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:04.041641 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wzslf" Nov 03 14:48:05.016842 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:05.015032 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wzslf"] Nov 03 14:48:05.736648 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:05.736602 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wzslf" event={"ID":"b9e1f087-e6e6-4cc7-b0c9-6ba850c9881f","Type":"ContainerStarted","Data":"03ff0b999dc001e4d4980720924609cfba413eab1fb612667a50bd74b69999e8"} Nov 03 14:48:06.743725 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:06.743690 5979 generic.go:356] "Generic (PLEG): container finished" podID="b9e1f087-e6e6-4cc7-b0c9-6ba850c9881f" containerID="211464ba6c647fc16ae174d9227a8b10b6015be5956aef449dafd2d0d8bd0236" exitCode=0 Nov 03 14:48:06.744172 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:06.743785 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wzslf" event={"ID":"b9e1f087-e6e6-4cc7-b0c9-6ba850c9881f","Type":"ContainerDied","Data":"211464ba6c647fc16ae174d9227a8b10b6015be5956aef449dafd2d0d8bd0236"} Nov 03 14:48:07.233106 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:48:07.233071 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:48:07.888958 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:07.888933 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wzslf" Nov 03 14:48:07.962448 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:07.962417 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9e1f087-e6e6-4cc7-b0c9-6ba850c9881f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9e1f087-e6e6-4cc7-b0c9-6ba850c9881f-maas-api-xks-ca\") pod \"b9e1f087-e6e6-4cc7-b0c9-6ba850c9881f\" (UID: \"b9e1f087-e6e6-4cc7-b0c9-6ba850c9881f\") " Nov 03 14:48:07.962608 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:07.962467 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9e1f087-e6e6-4cc7-b0c9-6ba850c9881f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9e1f087-e6e6-4cc7-b0c9-6ba850c9881f-maas-api-openshift-ca\") pod \"b9e1f087-e6e6-4cc7-b0c9-6ba850c9881f\" (UID: \"b9e1f087-e6e6-4cc7-b0c9-6ba850c9881f\") " Nov 03 14:48:07.962608 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:07.962595 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9e1f087-e6e6-4cc7-b0c9-6ba850c9881f-kube-api-access-nhc44\" (UniqueName: \"kubernetes.io/projected/b9e1f087-e6e6-4cc7-b0c9-6ba850c9881f-kube-api-access-nhc44\") pod \"b9e1f087-e6e6-4cc7-b0c9-6ba850c9881f\" (UID: \"b9e1f087-e6e6-4cc7-b0c9-6ba850c9881f\") " Nov 03 14:48:07.962883 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:07.962857 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9e1f087-e6e6-4cc7-b0c9-6ba850c9881f-maas-api-openshift-ca" pod "b9e1f087-e6e6-4cc7-b0c9-6ba850c9881f" (UID: "b9e1f087-e6e6-4cc7-b0c9-6ba850c9881f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:48:07.964581 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:07.964553 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9e1f087-e6e6-4cc7-b0c9-6ba850c9881f-kube-api-access-nhc44" pod "b9e1f087-e6e6-4cc7-b0c9-6ba850c9881f" (UID: "b9e1f087-e6e6-4cc7-b0c9-6ba850c9881f"). InnerVolumeSpecName "kube-api-access-nhc44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:48:07.964668 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:07.964605 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9e1f087-e6e6-4cc7-b0c9-6ba850c9881f-maas-api-xks-ca" pod "b9e1f087-e6e6-4cc7-b0c9-6ba850c9881f" (UID: "b9e1f087-e6e6-4cc7-b0c9-6ba850c9881f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:48:08.064171 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:08.064080 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhc44\" (UniqueName: \"kubernetes.io/projected/b9e1f087-e6e6-4cc7-b0c9-6ba850c9881f-kube-api-access-nhc44\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:48:08.064171 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:08.064115 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9e1f087-e6e6-4cc7-b0c9-6ba850c9881f-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:48:08.064171 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:08.064125 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9e1f087-e6e6-4cc7-b0c9-6ba850c9881f-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:48:08.755661 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:08.755628 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wzslf" Nov 03 14:48:08.755851 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:08.755635 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wzslf" event={"ID":"b9e1f087-e6e6-4cc7-b0c9-6ba850c9881f","Type":"ContainerDied","Data":"03ff0b999dc001e4d4980720924609cfba413eab1fb612667a50bd74b69999e8"} Nov 03 14:48:08.755851 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:08.755749 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03ff0b999dc001e4d4980720924609cfba413eab1fb612667a50bd74b69999e8" Nov 03 14:48:11.268091 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:11.268056 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:48:19.830135 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:19.830103 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:48:21.163941 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:21.163894 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jxkv2"] Nov 03 14:48:21.164737 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:21.164693 5979 cpu_manager.go:414] "containerMap: removing container" podUID="b9e1f087-e6e6-4cc7-b0c9-6ba850c9881f" containerName="revoke-keys" Nov 03 14:48:21.164926 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:21.164905 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9e1f087-e6e6-4cc7-b0c9-6ba850c9881f" containerName="revoke-keys" Nov 03 14:48:21.165237 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:21.165219 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9e1f087-e6e6-4cc7-b0c9-6ba850c9881f" containerName="revoke-keys" Nov 03 14:48:21.169607 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:21.169584 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jxkv2" Nov 03 14:48:21.288518 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:21.283843 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:48:21.290648 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:21.290619 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:48:21.316638 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:21.290923 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:48:21.321539 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:21.292198 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jxkv2"] Nov 03 14:48:21.321808 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:21.303138 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f8bed76-f71e-46ec-ab8f-ebca5a00f714-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-jxkv2\" (UID: \"0f8bed76-f71e-46ec-ab8f-ebca5a00f714\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jxkv2" Nov 03 14:48:21.331627 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:21.331595 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-968l5\" (UniqueName: \"kubernetes.io/projected/0f8bed76-f71e-46ec-ab8f-ebca5a00f714-kube-api-access-968l5\") pod \"maas-api-revoke-sub-0b7e8edb-jxkv2\" (UID: \"0f8bed76-f71e-46ec-ab8f-ebca5a00f714\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jxkv2" Nov 03 14:48:21.355494 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:21.331895 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f8bed76-f71e-46ec-ab8f-ebca5a00f714-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-jxkv2\" (UID: \"0f8bed76-f71e-46ec-ab8f-ebca5a00f714\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jxkv2" Nov 03 14:48:21.355815 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:21.316585 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wgk87\"" Nov 03 14:48:21.490546 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:21.460083 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f8bed76-f71e-46ec-ab8f-ebca5a00f714-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-jxkv2\" (UID: \"0f8bed76-f71e-46ec-ab8f-ebca5a00f714\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jxkv2" Nov 03 14:48:21.490546 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:21.460192 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-968l5\" (UniqueName: \"kubernetes.io/projected/0f8bed76-f71e-46ec-ab8f-ebca5a00f714-kube-api-access-968l5\") pod \"maas-api-revoke-sub-0b7e8edb-jxkv2\" (UID: \"0f8bed76-f71e-46ec-ab8f-ebca5a00f714\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jxkv2" Nov 03 14:48:21.490546 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:21.460231 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f8bed76-f71e-46ec-ab8f-ebca5a00f714-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-jxkv2\" (UID: \"0f8bed76-f71e-46ec-ab8f-ebca5a00f714\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jxkv2" Nov 03 14:48:21.490546 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:21.461145 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f8bed76-f71e-46ec-ab8f-ebca5a00f714-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-jxkv2\" (UID: \"0f8bed76-f71e-46ec-ab8f-ebca5a00f714\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jxkv2" Nov 03 14:48:21.490546 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:21.488659 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f8bed76-f71e-46ec-ab8f-ebca5a00f714-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-jxkv2\" (UID: \"0f8bed76-f71e-46ec-ab8f-ebca5a00f714\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jxkv2" Nov 03 14:48:21.607506 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:21.578201 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-968l5\" (UniqueName: \"kubernetes.io/projected/0f8bed76-f71e-46ec-ab8f-ebca5a00f714-kube-api-access-968l5\") pod \"maas-api-revoke-sub-0b7e8edb-jxkv2\" (UID: \"0f8bed76-f71e-46ec-ab8f-ebca5a00f714\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jxkv2" Nov 03 14:48:21.792554 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:21.792402 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jxkv2" Nov 03 14:48:22.604561 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:22.604016 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jxkv2"] Nov 03 14:48:22.770496 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:22.769845 5979 scope.go:122] "RemoveContainer" containerID="ac5a3e02a26927cf82388e2c830577b8fb53b93cb24082d0a3953cb991e1d728" Nov 03 14:48:22.860698 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:22.860599 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jxkv2" event={"ID":"0f8bed76-f71e-46ec-ab8f-ebca5a00f714","Type":"ContainerStarted","Data":"54710f9f44fd05aa07b4937cbb070b6c9c3af813a88bcae36f73f3c1218d62af"} Nov 03 14:48:22.932708 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:22.931719 5979 scope.go:122] "RemoveContainer" containerID="638031b4d9f73a32ac77f15cb95ab31c41dcbd22618b83193e6be96ed82ffb40" Nov 03 14:48:23.867028 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:23.866996 5979 generic.go:356] "Generic (PLEG): container finished" podID="0f8bed76-f71e-46ec-ab8f-ebca5a00f714" containerID="6828f10bf3f227098cd4bd30f041b29fdc122fc71675c4b6f8d1fe8f0ee59175" exitCode=0 Nov 03 14:48:23.867374 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:23.867075 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jxkv2" event={"ID":"0f8bed76-f71e-46ec-ab8f-ebca5a00f714","Type":"ContainerDied","Data":"6828f10bf3f227098cd4bd30f041b29fdc122fc71675c4b6f8d1fe8f0ee59175"} Nov 03 14:48:25.009748 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:25.009719 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jxkv2" Nov 03 14:48:25.111541 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:25.111509 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f8bed76-f71e-46ec-ab8f-ebca5a00f714-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f8bed76-f71e-46ec-ab8f-ebca5a00f714-maas-api-openshift-ca\") pod \"0f8bed76-f71e-46ec-ab8f-ebca5a00f714\" (UID: \"0f8bed76-f71e-46ec-ab8f-ebca5a00f714\") " Nov 03 14:48:25.111744 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:25.111647 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f8bed76-f71e-46ec-ab8f-ebca5a00f714-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f8bed76-f71e-46ec-ab8f-ebca5a00f714-maas-api-xks-ca\") pod \"0f8bed76-f71e-46ec-ab8f-ebca5a00f714\" (UID: \"0f8bed76-f71e-46ec-ab8f-ebca5a00f714\") " Nov 03 14:48:25.111744 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:25.111678 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f8bed76-f71e-46ec-ab8f-ebca5a00f714-kube-api-access-968l5\" (UniqueName: \"kubernetes.io/projected/0f8bed76-f71e-46ec-ab8f-ebca5a00f714-kube-api-access-968l5\") pod \"0f8bed76-f71e-46ec-ab8f-ebca5a00f714\" (UID: \"0f8bed76-f71e-46ec-ab8f-ebca5a00f714\") " Nov 03 14:48:25.111879 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:25.111843 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f8bed76-f71e-46ec-ab8f-ebca5a00f714-maas-api-openshift-ca" pod "0f8bed76-f71e-46ec-ab8f-ebca5a00f714" (UID: "0f8bed76-f71e-46ec-ab8f-ebca5a00f714"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:48:25.112010 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:25.111992 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f8bed76-f71e-46ec-ab8f-ebca5a00f714-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:48:25.113700 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:25.113671 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f8bed76-f71e-46ec-ab8f-ebca5a00f714-maas-api-xks-ca" pod "0f8bed76-f71e-46ec-ab8f-ebca5a00f714" (UID: "0f8bed76-f71e-46ec-ab8f-ebca5a00f714"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:48:25.113892 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:25.113872 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f8bed76-f71e-46ec-ab8f-ebca5a00f714-kube-api-access-968l5" pod "0f8bed76-f71e-46ec-ab8f-ebca5a00f714" (UID: "0f8bed76-f71e-46ec-ab8f-ebca5a00f714"). InnerVolumeSpecName "kube-api-access-968l5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:48:25.213346 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:25.213304 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f8bed76-f71e-46ec-ab8f-ebca5a00f714-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:48:25.213346 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:25.213346 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-968l5\" (UniqueName: \"kubernetes.io/projected/0f8bed76-f71e-46ec-ab8f-ebca5a00f714-kube-api-access-968l5\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:48:25.878485 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:25.878443 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jxkv2" Nov 03 14:48:25.878485 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:25.878454 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jxkv2" event={"ID":"0f8bed76-f71e-46ec-ab8f-ebca5a00f714","Type":"ContainerDied","Data":"54710f9f44fd05aa07b4937cbb070b6c9c3af813a88bcae36f73f3c1218d62af"} Nov 03 14:48:25.878715 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:25.878509 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54710f9f44fd05aa07b4937cbb070b6c9c3af813a88bcae36f73f3c1218d62af" Nov 03 14:48:28.972029 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:28.971992 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:48:32.453788 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:32.453752 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lkmqm"] Nov 03 14:48:32.454207 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:32.454178 5979 cpu_manager.go:414] "containerMap: removing container" podUID="0f8bed76-f71e-46ec-ab8f-ebca5a00f714" containerName="revoke-keys" Nov 03 14:48:32.454207 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:32.454188 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f8bed76-f71e-46ec-ab8f-ebca5a00f714" containerName="revoke-keys" Nov 03 14:48:32.454289 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:32.454275 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f8bed76-f71e-46ec-ab8f-ebca5a00f714" containerName="revoke-keys" Nov 03 14:48:32.457716 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:32.457696 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lkmqm" Nov 03 14:48:32.507030 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:32.506993 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:48:32.507030 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:32.507002 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:48:32.507257 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:32.507075 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:48:32.553784 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:32.553728 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wgk87\"" Nov 03 14:48:32.555727 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:32.555701 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lkmqm"] Nov 03 14:48:32.589741 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:32.589694 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55f9bb16-497f-48b6-a114-2143018b8ca1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-lkmqm\" (UID: \"55f9bb16-497f-48b6-a114-2143018b8ca1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lkmqm" Nov 03 14:48:32.589921 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:32.589771 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xhck\" (UniqueName: \"kubernetes.io/projected/55f9bb16-497f-48b6-a114-2143018b8ca1-kube-api-access-5xhck\") pod \"maas-api-revoke-sub-3d4156f1-lkmqm\" (UID: \"55f9bb16-497f-48b6-a114-2143018b8ca1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lkmqm" Nov 03 14:48:32.589921 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:32.589824 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55f9bb16-497f-48b6-a114-2143018b8ca1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-lkmqm\" (UID: \"55f9bb16-497f-48b6-a114-2143018b8ca1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lkmqm" Nov 03 14:48:32.691686 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:32.691651 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55f9bb16-497f-48b6-a114-2143018b8ca1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-lkmqm\" (UID: \"55f9bb16-497f-48b6-a114-2143018b8ca1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lkmqm" Nov 03 14:48:32.691897 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:32.691772 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55f9bb16-497f-48b6-a114-2143018b8ca1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-lkmqm\" (UID: \"55f9bb16-497f-48b6-a114-2143018b8ca1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lkmqm" Nov 03 14:48:32.691897 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:32.691811 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xhck\" (UniqueName: \"kubernetes.io/projected/55f9bb16-497f-48b6-a114-2143018b8ca1-kube-api-access-5xhck\") pod \"maas-api-revoke-sub-3d4156f1-lkmqm\" (UID: \"55f9bb16-497f-48b6-a114-2143018b8ca1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lkmqm" Nov 03 14:48:32.692444 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:32.692419 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55f9bb16-497f-48b6-a114-2143018b8ca1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-lkmqm\" (UID: \"55f9bb16-497f-48b6-a114-2143018b8ca1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lkmqm" Nov 03 14:48:32.694058 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:32.694036 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55f9bb16-497f-48b6-a114-2143018b8ca1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-lkmqm\" (UID: \"55f9bb16-497f-48b6-a114-2143018b8ca1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lkmqm" Nov 03 14:48:32.746833 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:32.746750 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xhck\" (UniqueName: \"kubernetes.io/projected/55f9bb16-497f-48b6-a114-2143018b8ca1-kube-api-access-5xhck\") pod \"maas-api-revoke-sub-3d4156f1-lkmqm\" (UID: \"55f9bb16-497f-48b6-a114-2143018b8ca1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lkmqm" Nov 03 14:48:32.770864 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:32.770829 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lkmqm" Nov 03 14:48:33.140552 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:33.140524 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lkmqm"] Nov 03 14:48:33.148097 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:48:33.148069 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55f9bb16_497f_48b6_a114_2143018b8ca1.slice/crio-7343fceb0407897387f0bd0808e11881711b244e499e8c2e2e969d31d99c6e24 WatchSource:0}: Error finding container 7343fceb0407897387f0bd0808e11881711b244e499e8c2e2e969d31d99c6e24: Status 404 returned error can't find the container with id 7343fceb0407897387f0bd0808e11881711b244e499e8c2e2e969d31d99c6e24 Nov 03 14:48:33.929510 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:33.929464 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lkmqm" event={"ID":"55f9bb16-497f-48b6-a114-2143018b8ca1","Type":"ContainerStarted","Data":"4dfdff6521e41e11d658f0c4c2bd7f80cbe93c19848b9e68f7754f4a555ac5c0"} Nov 03 14:48:33.929921 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:33.929532 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lkmqm" event={"ID":"55f9bb16-497f-48b6-a114-2143018b8ca1","Type":"ContainerStarted","Data":"7343fceb0407897387f0bd0808e11881711b244e499e8c2e2e969d31d99c6e24"} Nov 03 14:48:34.039308 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:34.039268 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lkmqm" podStartSLOduration=2.039253948 podStartE2EDuration="2.039253948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:48:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:48:34.038208424 +0000 UTC m=+6196.048896668" watchObservedRunningTime="2026-11-03 14:48:34.039253948 +0000 UTC m=+6196.049942191" Nov 03 14:48:34.936077 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:34.936041 5979 generic.go:356] "Generic (PLEG): container finished" podID="55f9bb16-497f-48b6-a114-2143018b8ca1" containerID="4dfdff6521e41e11d658f0c4c2bd7f80cbe93c19848b9e68f7754f4a555ac5c0" exitCode=0 Nov 03 14:48:34.936541 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:34.936128 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lkmqm" event={"ID":"55f9bb16-497f-48b6-a114-2143018b8ca1","Type":"ContainerDied","Data":"4dfdff6521e41e11d658f0c4c2bd7f80cbe93c19848b9e68f7754f4a555ac5c0"} Nov 03 14:48:36.073672 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:36.073646 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lkmqm" Nov 03 14:48:36.124440 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:36.124414 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55f9bb16-497f-48b6-a114-2143018b8ca1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55f9bb16-497f-48b6-a114-2143018b8ca1-maas-api-openshift-ca\") pod \"55f9bb16-497f-48b6-a114-2143018b8ca1\" (UID: \"55f9bb16-497f-48b6-a114-2143018b8ca1\") " Nov 03 14:48:36.124622 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:36.124539 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55f9bb16-497f-48b6-a114-2143018b8ca1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55f9bb16-497f-48b6-a114-2143018b8ca1-maas-api-xks-ca\") pod \"55f9bb16-497f-48b6-a114-2143018b8ca1\" (UID: \"55f9bb16-497f-48b6-a114-2143018b8ca1\") " Nov 03 14:48:36.124622 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:36.124561 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55f9bb16-497f-48b6-a114-2143018b8ca1-kube-api-access-5xhck\" (UniqueName: \"kubernetes.io/projected/55f9bb16-497f-48b6-a114-2143018b8ca1-kube-api-access-5xhck\") pod \"55f9bb16-497f-48b6-a114-2143018b8ca1\" (UID: \"55f9bb16-497f-48b6-a114-2143018b8ca1\") " Nov 03 14:48:36.124805 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:36.124778 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55f9bb16-497f-48b6-a114-2143018b8ca1-maas-api-openshift-ca" pod "55f9bb16-497f-48b6-a114-2143018b8ca1" (UID: "55f9bb16-497f-48b6-a114-2143018b8ca1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:48:36.126604 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:36.126580 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55f9bb16-497f-48b6-a114-2143018b8ca1-maas-api-xks-ca" pod "55f9bb16-497f-48b6-a114-2143018b8ca1" (UID: "55f9bb16-497f-48b6-a114-2143018b8ca1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:48:36.126759 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:36.126681 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55f9bb16-497f-48b6-a114-2143018b8ca1-kube-api-access-5xhck" pod "55f9bb16-497f-48b6-a114-2143018b8ca1" (UID: "55f9bb16-497f-48b6-a114-2143018b8ca1"). InnerVolumeSpecName "kube-api-access-5xhck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:48:36.225880 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:36.225786 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55f9bb16-497f-48b6-a114-2143018b8ca1-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:48:36.225880 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:36.225816 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55f9bb16-497f-48b6-a114-2143018b8ca1-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:48:36.225880 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:36.225827 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xhck\" (UniqueName: \"kubernetes.io/projected/55f9bb16-497f-48b6-a114-2143018b8ca1-kube-api-access-5xhck\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:48:36.947543 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:36.947505 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lkmqm" event={"ID":"55f9bb16-497f-48b6-a114-2143018b8ca1","Type":"ContainerDied","Data":"7343fceb0407897387f0bd0808e11881711b244e499e8c2e2e969d31d99c6e24"} Nov 03 14:48:36.947543 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:36.947528 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lkmqm" Nov 03 14:48:36.947776 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:36.947556 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7343fceb0407897387f0bd0808e11881711b244e499e8c2e2e969d31d99c6e24" Nov 03 14:48:37.233429 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:48:37.233348 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:48:46.852362 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:46.852318 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:48:56.158665 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:56.158614 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-8gjrw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:48:56.159364 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:56.159327 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8gjrw" podUID="e7e12233-9e76-4de5-b4cf-93644996783e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:48:56.400656 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:56.400623 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:48:56.772438 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:56.772403 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lkmqm"] Nov 03 14:48:56.819312 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:56.819276 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-lkmqm"] Nov 03 14:48:56.899957 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:56.899914 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgt6c"] Nov 03 14:48:56.900434 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:56.900415 5979 cpu_manager.go:414] "containerMap: removing container" podUID="55f9bb16-497f-48b6-a114-2143018b8ca1" containerName="revoke-keys" Nov 03 14:48:56.900556 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:56.900436 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55f9bb16-497f-48b6-a114-2143018b8ca1" containerName="revoke-keys" Nov 03 14:48:56.900644 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:56.900629 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="55f9bb16-497f-48b6-a114-2143018b8ca1" containerName="revoke-keys" Nov 03 14:48:56.904080 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:56.904059 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgt6c" Nov 03 14:48:56.946276 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:56.946248 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3867f370-8190-4932-b69f-5d4f7fb2b488-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-cgt6c\" (UID: \"3867f370-8190-4932-b69f-5d4f7fb2b488\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgt6c" Nov 03 14:48:56.946416 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:56.946297 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3867f370-8190-4932-b69f-5d4f7fb2b488-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-cgt6c\" (UID: \"3867f370-8190-4932-b69f-5d4f7fb2b488\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgt6c" Nov 03 14:48:56.946518 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:56.946502 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87x7d\" (UniqueName: \"kubernetes.io/projected/3867f370-8190-4932-b69f-5d4f7fb2b488-kube-api-access-87x7d\") pod \"maas-api-revoke-sub-3d4156f1-cgt6c\" (UID: \"3867f370-8190-4932-b69f-5d4f7fb2b488\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgt6c" Nov 03 14:48:56.953388 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:56.953365 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:48:56.953594 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:56.953551 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:48:56.953594 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:56.953565 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:48:56.967456 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:56.967436 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wgk87\"" Nov 03 14:48:56.975483 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:56.975443 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgt6c"] Nov 03 14:48:57.047619 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:57.047525 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3867f370-8190-4932-b69f-5d4f7fb2b488-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-cgt6c\" (UID: \"3867f370-8190-4932-b69f-5d4f7fb2b488\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgt6c" Nov 03 14:48:57.047787 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:57.047682 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87x7d\" (UniqueName: \"kubernetes.io/projected/3867f370-8190-4932-b69f-5d4f7fb2b488-kube-api-access-87x7d\") pod \"maas-api-revoke-sub-3d4156f1-cgt6c\" (UID: \"3867f370-8190-4932-b69f-5d4f7fb2b488\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgt6c" Nov 03 14:48:57.047787 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:57.047725 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3867f370-8190-4932-b69f-5d4f7fb2b488-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-cgt6c\" (UID: \"3867f370-8190-4932-b69f-5d4f7fb2b488\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgt6c" Nov 03 14:48:57.048220 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:57.048198 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3867f370-8190-4932-b69f-5d4f7fb2b488-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-cgt6c\" (UID: \"3867f370-8190-4932-b69f-5d4f7fb2b488\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgt6c" Nov 03 14:48:57.050005 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:57.049977 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3867f370-8190-4932-b69f-5d4f7fb2b488-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-cgt6c\" (UID: \"3867f370-8190-4932-b69f-5d4f7fb2b488\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgt6c" Nov 03 14:48:57.101623 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:57.101595 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87x7d\" (UniqueName: \"kubernetes.io/projected/3867f370-8190-4932-b69f-5d4f7fb2b488-kube-api-access-87x7d\") pod \"maas-api-revoke-sub-3d4156f1-cgt6c\" (UID: \"3867f370-8190-4932-b69f-5d4f7fb2b488\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgt6c" Nov 03 14:48:57.216373 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:57.216347 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgt6c" Nov 03 14:48:57.585529 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:57.585488 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgt6c"] Nov 03 14:48:57.593391 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:48:57.593358 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3867f370_8190_4932_b69f_5d4f7fb2b488.slice/crio-1863e92b3c23211ff55b5cf12496ae1d0bd2e621089c885bac73164fff9b8fad WatchSource:0}: Error finding container 1863e92b3c23211ff55b5cf12496ae1d0bd2e621089c885bac73164fff9b8fad: Status 404 returned error can't find the container with id 1863e92b3c23211ff55b5cf12496ae1d0bd2e621089c885bac73164fff9b8fad Nov 03 14:48:58.115417 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:58.115330 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55f9bb16-497f-48b6-a114-2143018b8ca1" path="/var/lib/kubelet/pods/55f9bb16-497f-48b6-a114-2143018b8ca1/volumes" Nov 03 14:48:58.137929 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:58.137892 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgt6c" event={"ID":"3867f370-8190-4932-b69f-5d4f7fb2b488","Type":"ContainerStarted","Data":"6aaa38469308c8464360ad20356102e844673f8d254feb2889c031d448835ee3"} Nov 03 14:48:58.138102 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:58.137953 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgt6c" event={"ID":"3867f370-8190-4932-b69f-5d4f7fb2b488","Type":"ContainerStarted","Data":"1863e92b3c23211ff55b5cf12496ae1d0bd2e621089c885bac73164fff9b8fad"} Nov 03 14:48:58.246544 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:58.246492 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgt6c" podStartSLOduration=2.246464084 podStartE2EDuration="2.246464084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:48:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:48:58.245572052 +0000 UTC m=+6220.256260296" watchObservedRunningTime="2026-11-03 14:48:58.246464084 +0000 UTC m=+6220.257152326" Nov 03 14:48:59.144837 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:59.144798 5979 generic.go:356] "Generic (PLEG): container finished" podID="3867f370-8190-4932-b69f-5d4f7fb2b488" containerID="6aaa38469308c8464360ad20356102e844673f8d254feb2889c031d448835ee3" exitCode=0 Nov 03 14:48:59.145035 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:48:59.144879 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgt6c" event={"ID":"3867f370-8190-4932-b69f-5d4f7fb2b488","Type":"ContainerDied","Data":"6aaa38469308c8464360ad20356102e844673f8d254feb2889c031d448835ee3"} Nov 03 14:49:00.285223 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:00.285194 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgt6c" Nov 03 14:49:00.381131 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:00.381091 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3867f370-8190-4932-b69f-5d4f7fb2b488-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3867f370-8190-4932-b69f-5d4f7fb2b488-maas-api-openshift-ca\") pod \"3867f370-8190-4932-b69f-5d4f7fb2b488\" (UID: \"3867f370-8190-4932-b69f-5d4f7fb2b488\") " Nov 03 14:49:00.381338 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:00.381138 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3867f370-8190-4932-b69f-5d4f7fb2b488-kube-api-access-87x7d\" (UniqueName: \"kubernetes.io/projected/3867f370-8190-4932-b69f-5d4f7fb2b488-kube-api-access-87x7d\") pod \"3867f370-8190-4932-b69f-5d4f7fb2b488\" (UID: \"3867f370-8190-4932-b69f-5d4f7fb2b488\") " Nov 03 14:49:00.381338 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:00.381282 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3867f370-8190-4932-b69f-5d4f7fb2b488-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3867f370-8190-4932-b69f-5d4f7fb2b488-maas-api-xks-ca\") pod \"3867f370-8190-4932-b69f-5d4f7fb2b488\" (UID: \"3867f370-8190-4932-b69f-5d4f7fb2b488\") " Nov 03 14:49:00.381492 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:00.381420 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3867f370-8190-4932-b69f-5d4f7fb2b488-maas-api-openshift-ca" pod "3867f370-8190-4932-b69f-5d4f7fb2b488" (UID: "3867f370-8190-4932-b69f-5d4f7fb2b488"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:49:00.381632 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:00.381606 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3867f370-8190-4932-b69f-5d4f7fb2b488-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:49:00.383262 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:00.383238 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3867f370-8190-4932-b69f-5d4f7fb2b488-kube-api-access-87x7d" pod "3867f370-8190-4932-b69f-5d4f7fb2b488" (UID: "3867f370-8190-4932-b69f-5d4f7fb2b488"). InnerVolumeSpecName "kube-api-access-87x7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:49:00.383389 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:00.383310 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3867f370-8190-4932-b69f-5d4f7fb2b488-maas-api-xks-ca" pod "3867f370-8190-4932-b69f-5d4f7fb2b488" (UID: "3867f370-8190-4932-b69f-5d4f7fb2b488"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:49:00.482821 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:00.482738 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3867f370-8190-4932-b69f-5d4f7fb2b488-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:49:00.482821 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:00.482770 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87x7d\" (UniqueName: \"kubernetes.io/projected/3867f370-8190-4932-b69f-5d4f7fb2b488-kube-api-access-87x7d\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:49:01.158526 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:01.158493 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgt6c" Nov 03 14:49:01.158726 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:01.158494 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgt6c" event={"ID":"3867f370-8190-4932-b69f-5d4f7fb2b488","Type":"ContainerDied","Data":"1863e92b3c23211ff55b5cf12496ae1d0bd2e621089c885bac73164fff9b8fad"} Nov 03 14:49:01.158726 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:01.158632 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1863e92b3c23211ff55b5cf12496ae1d0bd2e621089c885bac73164fff9b8fad" Nov 03 14:49:07.233719 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:49:07.233675 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:49:11.348772 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:11.348734 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:49:21.362283 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:21.362246 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:49:31.111271 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:31.111229 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:49:37.233756 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:49:37.233716 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:49:38.347181 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:38.347140 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w27bv"] Nov 03 14:49:38.347623 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:38.347572 5979 cpu_manager.go:414] "containerMap: removing container" podUID="3867f370-8190-4932-b69f-5d4f7fb2b488" containerName="revoke-keys" Nov 03 14:49:38.347623 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:38.347584 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3867f370-8190-4932-b69f-5d4f7fb2b488" containerName="revoke-keys" Nov 03 14:49:38.347702 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:38.347681 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="3867f370-8190-4932-b69f-5d4f7fb2b488" containerName="revoke-keys" Nov 03 14:49:38.351353 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:38.351331 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w27bv" Nov 03 14:49:38.447555 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:38.447510 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:49:38.447771 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:38.447756 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:49:38.447866 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:38.447769 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:49:38.447866 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:38.447784 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wgk87\"" Nov 03 14:49:38.449825 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:38.449800 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w27bv"] Nov 03 14:49:38.458406 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:38.458383 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5cpz\" (UniqueName: \"kubernetes.io/projected/0ced6c9f-849c-48a3-99c3-bf54e17c6231-kube-api-access-t5cpz\") pod \"maas-api-revoke-sub-e2fc9c5b-w27bv\" (UID: \"0ced6c9f-849c-48a3-99c3-bf54e17c6231\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w27bv" Nov 03 14:49:38.458575 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:38.458523 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ced6c9f-849c-48a3-99c3-bf54e17c6231-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-w27bv\" (UID: \"0ced6c9f-849c-48a3-99c3-bf54e17c6231\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w27bv" Nov 03 14:49:38.458637 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:38.458620 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ced6c9f-849c-48a3-99c3-bf54e17c6231-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-w27bv\" (UID: \"0ced6c9f-849c-48a3-99c3-bf54e17c6231\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w27bv" Nov 03 14:49:38.560015 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:38.559977 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5cpz\" (UniqueName: \"kubernetes.io/projected/0ced6c9f-849c-48a3-99c3-bf54e17c6231-kube-api-access-t5cpz\") pod \"maas-api-revoke-sub-e2fc9c5b-w27bv\" (UID: \"0ced6c9f-849c-48a3-99c3-bf54e17c6231\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w27bv" Nov 03 14:49:38.560210 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:38.560099 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ced6c9f-849c-48a3-99c3-bf54e17c6231-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-w27bv\" (UID: \"0ced6c9f-849c-48a3-99c3-bf54e17c6231\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w27bv" Nov 03 14:49:38.560210 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:38.560159 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ced6c9f-849c-48a3-99c3-bf54e17c6231-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-w27bv\" (UID: \"0ced6c9f-849c-48a3-99c3-bf54e17c6231\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w27bv" Nov 03 14:49:38.560773 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:38.560748 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ced6c9f-849c-48a3-99c3-bf54e17c6231-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-w27bv\" (UID: \"0ced6c9f-849c-48a3-99c3-bf54e17c6231\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w27bv" Nov 03 14:49:38.562516 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:38.562496 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ced6c9f-849c-48a3-99c3-bf54e17c6231-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-w27bv\" (UID: \"0ced6c9f-849c-48a3-99c3-bf54e17c6231\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w27bv" Nov 03 14:49:38.614794 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:38.614721 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5cpz\" (UniqueName: \"kubernetes.io/projected/0ced6c9f-849c-48a3-99c3-bf54e17c6231-kube-api-access-t5cpz\") pod \"maas-api-revoke-sub-e2fc9c5b-w27bv\" (UID: \"0ced6c9f-849c-48a3-99c3-bf54e17c6231\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w27bv" Nov 03 14:49:38.665146 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:38.665113 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w27bv" Nov 03 14:49:39.033356 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:39.033316 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w27bv"] Nov 03 14:49:39.040637 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:49:39.040600 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ced6c9f_849c_48a3_99c3_bf54e17c6231.slice/crio-fac61d636aee3b24bd96419f4f2c8b41a4cd8cc0c024a1a701d91520b44850c2 WatchSource:0}: Error finding container fac61d636aee3b24bd96419f4f2c8b41a4cd8cc0c024a1a701d91520b44850c2: Status 404 returned error can't find the container with id fac61d636aee3b24bd96419f4f2c8b41a4cd8cc0c024a1a701d91520b44850c2 Nov 03 14:49:39.395057 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:39.395020 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w27bv" event={"ID":"0ced6c9f-849c-48a3-99c3-bf54e17c6231","Type":"ContainerStarted","Data":"b62d5bdf0a6c2f9d3e7b92976611bf70315c59da9ee826e88405bc5e6b2f6b04"} Nov 03 14:49:39.395534 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:39.395078 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w27bv" event={"ID":"0ced6c9f-849c-48a3-99c3-bf54e17c6231","Type":"ContainerStarted","Data":"fac61d636aee3b24bd96419f4f2c8b41a4cd8cc0c024a1a701d91520b44850c2"} Nov 03 14:49:39.507712 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:39.507662 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w27bv" podStartSLOduration=1.507648485 podStartE2EDuration="1.507648485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:49:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:49:39.506955272 +0000 UTC m=+6261.517643518" watchObservedRunningTime="2026-11-03 14:49:39.507648485 +0000 UTC m=+6261.518336726" Nov 03 14:49:40.401677 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:40.401642 5979 generic.go:356] "Generic (PLEG): container finished" podID="0ced6c9f-849c-48a3-99c3-bf54e17c6231" containerID="b62d5bdf0a6c2f9d3e7b92976611bf70315c59da9ee826e88405bc5e6b2f6b04" exitCode=0 Nov 03 14:49:40.402108 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:40.401741 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w27bv" event={"ID":"0ced6c9f-849c-48a3-99c3-bf54e17c6231","Type":"ContainerDied","Data":"b62d5bdf0a6c2f9d3e7b92976611bf70315c59da9ee826e88405bc5e6b2f6b04"} Nov 03 14:49:41.548888 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:41.548858 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w27bv" Nov 03 14:49:41.593180 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:41.593151 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ced6c9f-849c-48a3-99c3-bf54e17c6231-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ced6c9f-849c-48a3-99c3-bf54e17c6231-maas-api-xks-ca\") pod \"0ced6c9f-849c-48a3-99c3-bf54e17c6231\" (UID: \"0ced6c9f-849c-48a3-99c3-bf54e17c6231\") " Nov 03 14:49:41.593348 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:41.593197 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ced6c9f-849c-48a3-99c3-bf54e17c6231-kube-api-access-t5cpz\" (UniqueName: \"kubernetes.io/projected/0ced6c9f-849c-48a3-99c3-bf54e17c6231-kube-api-access-t5cpz\") pod \"0ced6c9f-849c-48a3-99c3-bf54e17c6231\" (UID: \"0ced6c9f-849c-48a3-99c3-bf54e17c6231\") " Nov 03 14:49:41.593348 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:41.593271 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ced6c9f-849c-48a3-99c3-bf54e17c6231-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ced6c9f-849c-48a3-99c3-bf54e17c6231-maas-api-openshift-ca\") pod \"0ced6c9f-849c-48a3-99c3-bf54e17c6231\" (UID: \"0ced6c9f-849c-48a3-99c3-bf54e17c6231\") " Nov 03 14:49:41.593703 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:41.593679 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ced6c9f-849c-48a3-99c3-bf54e17c6231-maas-api-openshift-ca" pod "0ced6c9f-849c-48a3-99c3-bf54e17c6231" (UID: "0ced6c9f-849c-48a3-99c3-bf54e17c6231"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:49:41.595328 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:41.595300 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ced6c9f-849c-48a3-99c3-bf54e17c6231-kube-api-access-t5cpz" pod "0ced6c9f-849c-48a3-99c3-bf54e17c6231" (UID: "0ced6c9f-849c-48a3-99c3-bf54e17c6231"). InnerVolumeSpecName "kube-api-access-t5cpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:49:41.595409 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:41.595323 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ced6c9f-849c-48a3-99c3-bf54e17c6231-maas-api-xks-ca" pod "0ced6c9f-849c-48a3-99c3-bf54e17c6231" (UID: "0ced6c9f-849c-48a3-99c3-bf54e17c6231"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:49:41.694598 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:41.694522 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ced6c9f-849c-48a3-99c3-bf54e17c6231-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:49:41.694598 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:41.694552 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5cpz\" (UniqueName: \"kubernetes.io/projected/0ced6c9f-849c-48a3-99c3-bf54e17c6231-kube-api-access-t5cpz\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:49:41.694598 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:41.694563 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ced6c9f-849c-48a3-99c3-bf54e17c6231-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:49:42.414708 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:42.414674 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w27bv" event={"ID":"0ced6c9f-849c-48a3-99c3-bf54e17c6231","Type":"ContainerDied","Data":"fac61d636aee3b24bd96419f4f2c8b41a4cd8cc0c024a1a701d91520b44850c2"} Nov 03 14:49:42.414884 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:42.414726 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fac61d636aee3b24bd96419f4f2c8b41a4cd8cc0c024a1a701d91520b44850c2" Nov 03 14:49:42.414884 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:42.414685 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w27bv" Nov 03 14:49:49.117194 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:49.117159 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g5jp5"] Nov 03 14:49:49.117639 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:49.117598 5979 cpu_manager.go:414] "containerMap: removing container" podUID="0ced6c9f-849c-48a3-99c3-bf54e17c6231" containerName="revoke-keys" Nov 03 14:49:49.117639 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:49.117610 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ced6c9f-849c-48a3-99c3-bf54e17c6231" containerName="revoke-keys" Nov 03 14:49:49.117717 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:49.117699 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ced6c9f-849c-48a3-99c3-bf54e17c6231" containerName="revoke-keys" Nov 03 14:49:49.120903 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:49.120882 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g5jp5" Nov 03 14:49:49.170319 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:49.170283 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:49:49.216570 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:49.216540 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:49:49.216751 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:49.216726 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wgk87\"" Nov 03 14:49:49.216804 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:49.216784 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:49:49.218958 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:49.218938 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g5jp5"] Nov 03 14:49:49.273599 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:49.273563 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmnl9\" (UniqueName: \"kubernetes.io/projected/8908b045-1724-4420-90bf-ddd8a559d3bd-kube-api-access-qmnl9\") pod \"maas-api-revoke-sub-5a29efb3-g5jp5\" (UID: \"8908b045-1724-4420-90bf-ddd8a559d3bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g5jp5" Nov 03 14:49:49.273599 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:49.273605 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8908b045-1724-4420-90bf-ddd8a559d3bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-g5jp5\" (UID: \"8908b045-1724-4420-90bf-ddd8a559d3bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g5jp5" Nov 03 14:49:49.273832 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:49.273687 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8908b045-1724-4420-90bf-ddd8a559d3bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-g5jp5\" (UID: \"8908b045-1724-4420-90bf-ddd8a559d3bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g5jp5" Nov 03 14:49:49.374946 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:49.374858 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8908b045-1724-4420-90bf-ddd8a559d3bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-g5jp5\" (UID: \"8908b045-1724-4420-90bf-ddd8a559d3bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g5jp5" Nov 03 14:49:49.375122 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:49.374972 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmnl9\" (UniqueName: \"kubernetes.io/projected/8908b045-1724-4420-90bf-ddd8a559d3bd-kube-api-access-qmnl9\") pod \"maas-api-revoke-sub-5a29efb3-g5jp5\" (UID: \"8908b045-1724-4420-90bf-ddd8a559d3bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g5jp5" Nov 03 14:49:49.375122 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:49.374996 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8908b045-1724-4420-90bf-ddd8a559d3bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-g5jp5\" (UID: \"8908b045-1724-4420-90bf-ddd8a559d3bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g5jp5" Nov 03 14:49:49.375657 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:49.375631 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8908b045-1724-4420-90bf-ddd8a559d3bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-g5jp5\" (UID: \"8908b045-1724-4420-90bf-ddd8a559d3bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g5jp5" Nov 03 14:49:49.377228 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:49.377205 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8908b045-1724-4420-90bf-ddd8a559d3bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-g5jp5\" (UID: \"8908b045-1724-4420-90bf-ddd8a559d3bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g5jp5" Nov 03 14:49:49.428838 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:49.428809 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmnl9\" (UniqueName: \"kubernetes.io/projected/8908b045-1724-4420-90bf-ddd8a559d3bd-kube-api-access-qmnl9\") pod \"maas-api-revoke-sub-5a29efb3-g5jp5\" (UID: \"8908b045-1724-4420-90bf-ddd8a559d3bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g5jp5" Nov 03 14:49:49.435624 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:49.435602 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g5jp5" Nov 03 14:49:49.809757 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:49.809724 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g5jp5"] Nov 03 14:49:49.816674 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:49:49.816646 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8908b045_1724_4420_90bf_ddd8a559d3bd.slice/crio-e9becf85f56fe6029c8f3903af20b38bf8e1603c648ab1a28ee112f4e517b912 WatchSource:0}: Error finding container e9becf85f56fe6029c8f3903af20b38bf8e1603c648ab1a28ee112f4e517b912: Status 404 returned error can't find the container with id e9becf85f56fe6029c8f3903af20b38bf8e1603c648ab1a28ee112f4e517b912 Nov 03 14:49:50.460811 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:50.460732 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g5jp5" event={"ID":"8908b045-1724-4420-90bf-ddd8a559d3bd","Type":"ContainerStarted","Data":"2ce953aae16b30d3bab5b8b502bdd230ceb423da45e55e57193288356cbbbefc"} Nov 03 14:49:50.460811 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:50.460784 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g5jp5" event={"ID":"8908b045-1724-4420-90bf-ddd8a559d3bd","Type":"ContainerStarted","Data":"e9becf85f56fe6029c8f3903af20b38bf8e1603c648ab1a28ee112f4e517b912"} Nov 03 14:49:50.568507 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:50.568445 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g5jp5" podStartSLOduration=1.5684316329999999 podStartE2EDuration="1.568431633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:49:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:49:50.567255978 +0000 UTC m=+6272.577944221" watchObservedRunningTime="2026-11-03 14:49:50.568431633 +0000 UTC m=+6272.579119880" Nov 03 14:49:51.165026 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:51.164994 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:49:51.467260 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:51.467171 5979 generic.go:356] "Generic (PLEG): container finished" podID="8908b045-1724-4420-90bf-ddd8a559d3bd" containerID="2ce953aae16b30d3bab5b8b502bdd230ceb423da45e55e57193288356cbbbefc" exitCode=0 Nov 03 14:49:51.467734 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:51.467257 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g5jp5" event={"ID":"8908b045-1724-4420-90bf-ddd8a559d3bd","Type":"ContainerDied","Data":"2ce953aae16b30d3bab5b8b502bdd230ceb423da45e55e57193288356cbbbefc"} Nov 03 14:49:52.051491 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:52.051437 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-pgjzx"] Nov 03 14:49:52.055171 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:52.055139 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-pgjzx"] Nov 03 14:49:52.118346 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:52.118305 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c6f24ba-019e-47bc-b173-668b20ed1689" path="/var/lib/kubelet/pods/4c6f24ba-019e-47bc-b173-668b20ed1689/volumes" Nov 03 14:49:53.326317 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:53.325971 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g5jp5" Nov 03 14:49:53.400693 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:53.400660 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8908b045-1724-4420-90bf-ddd8a559d3bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8908b045-1724-4420-90bf-ddd8a559d3bd-maas-api-xks-ca\") pod \"8908b045-1724-4420-90bf-ddd8a559d3bd\" (UID: \"8908b045-1724-4420-90bf-ddd8a559d3bd\") " Nov 03 14:49:53.400992 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:53.400961 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8908b045-1724-4420-90bf-ddd8a559d3bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8908b045-1724-4420-90bf-ddd8a559d3bd-maas-api-openshift-ca\") pod \"8908b045-1724-4420-90bf-ddd8a559d3bd\" (UID: \"8908b045-1724-4420-90bf-ddd8a559d3bd\") " Nov 03 14:49:53.401352 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:53.401319 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8908b045-1724-4420-90bf-ddd8a559d3bd-kube-api-access-qmnl9\" (UniqueName: \"kubernetes.io/projected/8908b045-1724-4420-90bf-ddd8a559d3bd-kube-api-access-qmnl9\") pod \"8908b045-1724-4420-90bf-ddd8a559d3bd\" (UID: \"8908b045-1724-4420-90bf-ddd8a559d3bd\") " Nov 03 14:49:53.402807 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:53.402783 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8908b045-1724-4420-90bf-ddd8a559d3bd-maas-api-openshift-ca" pod "8908b045-1724-4420-90bf-ddd8a559d3bd" (UID: "8908b045-1724-4420-90bf-ddd8a559d3bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:49:53.427626 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:53.427561 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8908b045-1724-4420-90bf-ddd8a559d3bd-maas-api-xks-ca" pod "8908b045-1724-4420-90bf-ddd8a559d3bd" (UID: "8908b045-1724-4420-90bf-ddd8a559d3bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:49:53.431679 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:53.431645 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8908b045-1724-4420-90bf-ddd8a559d3bd-kube-api-access-qmnl9" pod "8908b045-1724-4420-90bf-ddd8a559d3bd" (UID: "8908b045-1724-4420-90bf-ddd8a559d3bd"). InnerVolumeSpecName "kube-api-access-qmnl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:49:53.534559 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:53.533620 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmnl9\" (UniqueName: \"kubernetes.io/projected/8908b045-1724-4420-90bf-ddd8a559d3bd-kube-api-access-qmnl9\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:49:53.534559 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:53.533659 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8908b045-1724-4420-90bf-ddd8a559d3bd-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:49:53.534559 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:53.533676 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8908b045-1724-4420-90bf-ddd8a559d3bd-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:49:53.538620 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:53.538587 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g5jp5" event={"ID":"8908b045-1724-4420-90bf-ddd8a559d3bd","Type":"ContainerDied","Data":"e9becf85f56fe6029c8f3903af20b38bf8e1603c648ab1a28ee112f4e517b912"} Nov 03 14:49:53.538895 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:53.538866 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9becf85f56fe6029c8f3903af20b38bf8e1603c648ab1a28ee112f4e517b912" Nov 03 14:49:53.539108 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:49:53.539093 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g5jp5" Nov 03 14:50:01.877633 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:01.877597 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:50:06.656660 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:06.656621 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgt6c"] Nov 03 14:50:06.703046 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:06.703014 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cgt6c"] Nov 03 14:50:06.782799 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:06.782763 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w5x6m"] Nov 03 14:50:06.783328 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:06.783307 5979 cpu_manager.go:414] "containerMap: removing container" podUID="8908b045-1724-4420-90bf-ddd8a559d3bd" containerName="revoke-keys" Nov 03 14:50:06.783434 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:06.783329 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8908b045-1724-4420-90bf-ddd8a559d3bd" containerName="revoke-keys" Nov 03 14:50:06.783434 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:06.783419 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="8908b045-1724-4420-90bf-ddd8a559d3bd" containerName="revoke-keys" Nov 03 14:50:06.787156 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:06.787134 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w5x6m" Nov 03 14:50:06.838178 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:06.838148 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:50:06.838450 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:06.838433 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:50:06.838749 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:06.838731 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:50:06.838958 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:06.838945 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wgk87\"" Nov 03 14:50:06.885610 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:06.885576 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w5x6m"] Nov 03 14:50:06.914774 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:06.914695 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c9e399e-9d8b-42e7-b7bc-876298087344-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-w5x6m\" (UID: \"0c9e399e-9d8b-42e7-b7bc-876298087344\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w5x6m" Nov 03 14:50:06.914774 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:06.914757 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c9e399e-9d8b-42e7-b7bc-876298087344-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-w5x6m\" (UID: \"0c9e399e-9d8b-42e7-b7bc-876298087344\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w5x6m" Nov 03 14:50:06.914982 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:06.914895 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5rs5\" (UniqueName: \"kubernetes.io/projected/0c9e399e-9d8b-42e7-b7bc-876298087344-kube-api-access-n5rs5\") pod \"maas-api-revoke-sub-3d4156f1-w5x6m\" (UID: \"0c9e399e-9d8b-42e7-b7bc-876298087344\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w5x6m" Nov 03 14:50:07.016330 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:07.016288 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5rs5\" (UniqueName: \"kubernetes.io/projected/0c9e399e-9d8b-42e7-b7bc-876298087344-kube-api-access-n5rs5\") pod \"maas-api-revoke-sub-3d4156f1-w5x6m\" (UID: \"0c9e399e-9d8b-42e7-b7bc-876298087344\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w5x6m" Nov 03 14:50:07.016567 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:07.016348 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c9e399e-9d8b-42e7-b7bc-876298087344-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-w5x6m\" (UID: \"0c9e399e-9d8b-42e7-b7bc-876298087344\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w5x6m" Nov 03 14:50:07.016567 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:07.016408 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c9e399e-9d8b-42e7-b7bc-876298087344-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-w5x6m\" (UID: \"0c9e399e-9d8b-42e7-b7bc-876298087344\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w5x6m" Nov 03 14:50:07.017282 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:07.017252 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c9e399e-9d8b-42e7-b7bc-876298087344-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-w5x6m\" (UID: \"0c9e399e-9d8b-42e7-b7bc-876298087344\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w5x6m" Nov 03 14:50:07.018911 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:07.018887 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c9e399e-9d8b-42e7-b7bc-876298087344-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-w5x6m\" (UID: \"0c9e399e-9d8b-42e7-b7bc-876298087344\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w5x6m" Nov 03 14:50:07.070984 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:07.070953 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5rs5\" (UniqueName: \"kubernetes.io/projected/0c9e399e-9d8b-42e7-b7bc-876298087344-kube-api-access-n5rs5\") pod \"maas-api-revoke-sub-3d4156f1-w5x6m\" (UID: \"0c9e399e-9d8b-42e7-b7bc-876298087344\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w5x6m" Nov 03 14:50:07.099800 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:07.099767 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w5x6m" Nov 03 14:50:07.233937 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:50:07.233857 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:50:07.471719 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:07.471686 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w5x6m"] Nov 03 14:50:07.479088 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:50:07.479054 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c9e399e_9d8b_42e7_b7bc_876298087344.slice/crio-a68ea6e26cd9fd86307c7b94d550bd926913b64ce8d9aad7c19074edb397f8f1 WatchSource:0}: Error finding container a68ea6e26cd9fd86307c7b94d550bd926913b64ce8d9aad7c19074edb397f8f1: Status 404 returned error can't find the container with id a68ea6e26cd9fd86307c7b94d550bd926913b64ce8d9aad7c19074edb397f8f1 Nov 03 14:50:07.617371 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:07.617326 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w5x6m" event={"ID":"0c9e399e-9d8b-42e7-b7bc-876298087344","Type":"ContainerStarted","Data":"a68ea6e26cd9fd86307c7b94d550bd926913b64ce8d9aad7c19074edb397f8f1"} Nov 03 14:50:08.115140 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:08.115056 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3867f370-8190-4932-b69f-5d4f7fb2b488" path="/var/lib/kubelet/pods/3867f370-8190-4932-b69f-5d4f7fb2b488/volumes" Nov 03 14:50:08.625342 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:08.625308 5979 generic.go:356] "Generic (PLEG): container finished" podID="0c9e399e-9d8b-42e7-b7bc-876298087344" containerID="732792066df4f26757b5c1be3fe49519ae5f6784f2363fbcb912e1b8c3b6079a" exitCode=0 Nov 03 14:50:08.625552 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:08.625374 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w5x6m" event={"ID":"0c9e399e-9d8b-42e7-b7bc-876298087344","Type":"ContainerDied","Data":"732792066df4f26757b5c1be3fe49519ae5f6784f2363fbcb912e1b8c3b6079a"} Nov 03 14:50:09.775452 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:09.775424 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w5x6m" Nov 03 14:50:09.946691 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:09.946657 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c9e399e-9d8b-42e7-b7bc-876298087344-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c9e399e-9d8b-42e7-b7bc-876298087344-maas-api-xks-ca\") pod \"0c9e399e-9d8b-42e7-b7bc-876298087344\" (UID: \"0c9e399e-9d8b-42e7-b7bc-876298087344\") " Nov 03 14:50:09.946890 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:09.946764 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c9e399e-9d8b-42e7-b7bc-876298087344-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c9e399e-9d8b-42e7-b7bc-876298087344-maas-api-openshift-ca\") pod \"0c9e399e-9d8b-42e7-b7bc-876298087344\" (UID: \"0c9e399e-9d8b-42e7-b7bc-876298087344\") " Nov 03 14:50:09.946890 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:09.946800 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c9e399e-9d8b-42e7-b7bc-876298087344-kube-api-access-n5rs5\" (UniqueName: \"kubernetes.io/projected/0c9e399e-9d8b-42e7-b7bc-876298087344-kube-api-access-n5rs5\") pod \"0c9e399e-9d8b-42e7-b7bc-876298087344\" (UID: \"0c9e399e-9d8b-42e7-b7bc-876298087344\") " Nov 03 14:50:09.947118 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:09.947095 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c9e399e-9d8b-42e7-b7bc-876298087344-maas-api-openshift-ca" pod "0c9e399e-9d8b-42e7-b7bc-876298087344" (UID: "0c9e399e-9d8b-42e7-b7bc-876298087344"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:50:09.947291 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:09.947273 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c9e399e-9d8b-42e7-b7bc-876298087344-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:50:09.948820 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:09.948795 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c9e399e-9d8b-42e7-b7bc-876298087344-maas-api-xks-ca" pod "0c9e399e-9d8b-42e7-b7bc-876298087344" (UID: "0c9e399e-9d8b-42e7-b7bc-876298087344"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:50:09.948954 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:09.948933 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c9e399e-9d8b-42e7-b7bc-876298087344-kube-api-access-n5rs5" pod "0c9e399e-9d8b-42e7-b7bc-876298087344" (UID: "0c9e399e-9d8b-42e7-b7bc-876298087344"). InnerVolumeSpecName "kube-api-access-n5rs5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:50:10.048199 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:10.048163 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c9e399e-9d8b-42e7-b7bc-876298087344-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:50:10.048199 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:10.048196 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5rs5\" (UniqueName: \"kubernetes.io/projected/0c9e399e-9d8b-42e7-b7bc-876298087344-kube-api-access-n5rs5\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:50:10.637808 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:10.637770 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w5x6m" event={"ID":"0c9e399e-9d8b-42e7-b7bc-876298087344","Type":"ContainerDied","Data":"a68ea6e26cd9fd86307c7b94d550bd926913b64ce8d9aad7c19074edb397f8f1"} Nov 03 14:50:10.637808 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:10.637795 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w5x6m" Nov 03 14:50:10.638083 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:10.637820 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a68ea6e26cd9fd86307c7b94d550bd926913b64ce8d9aad7c19074edb397f8f1" Nov 03 14:50:21.355618 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:21.355580 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:50:27.047641 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:27.047605 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vjdcb"] Nov 03 14:50:27.093715 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:27.093674 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-vjdcb"] Nov 03 14:50:28.113632 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:28.113594 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01c5a8f4-4b2d-4d24-a719-ea02e2d7cbc1" path="/var/lib/kubelet/pods/01c5a8f4-4b2d-4d24-a719-ea02e2d7cbc1/volumes" Nov 03 14:50:32.373342 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:32.373310 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-21ea69-67886cfff2lnk6"] Nov 03 14:50:32.373859 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:32.373841 5979 cpu_manager.go:414] "containerMap: removing container" podUID="0c9e399e-9d8b-42e7-b7bc-876298087344" containerName="revoke-keys" Nov 03 14:50:32.373912 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:32.373863 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c9e399e-9d8b-42e7-b7bc-876298087344" containerName="revoke-keys" Nov 03 14:50:32.374025 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:32.374011 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c9e399e-9d8b-42e7-b7bc-876298087344" containerName="revoke-keys" Nov 03 14:50:32.378569 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:32.378547 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-21ea69-67886cfff2lnk6" Nov 03 14:50:32.474966 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:32.474927 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-ed3a3c33-dockercfg-7w2bd\"" Nov 03 14:50:32.474966 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:32.474947 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-21ea69\"" Nov 03 14:50:32.477285 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:32.477264 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-21ea69-67886cfff2lnk6"] Nov 03 14:50:32.514050 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:32.514013 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6btpg\" (UniqueName: \"kubernetes.io/projected/61b9fa8b-c3a7-4aba-adf6-d395b41fb14c-kube-api-access-6btpg\") pod \"payload-processing-e2e-worker-single-main-21ea69-67886cfff2lnk6\" (UID: \"61b9fa8b-c3a7-4aba-adf6-d395b41fb14c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-21ea69-67886cfff2lnk6" Nov 03 14:50:32.514213 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:32.514123 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61b9fa8b-c3a7-4aba-adf6-d395b41fb14c-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-21ea69-67886cfff2lnk6\" (UID: \"61b9fa8b-c3a7-4aba-adf6-d395b41fb14c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-21ea69-67886cfff2lnk6" Nov 03 14:50:32.615212 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:32.615168 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61b9fa8b-c3a7-4aba-adf6-d395b41fb14c-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-21ea69-67886cfff2lnk6\" (UID: \"61b9fa8b-c3a7-4aba-adf6-d395b41fb14c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-21ea69-67886cfff2lnk6" Nov 03 14:50:32.615416 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:32.615261 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6btpg\" (UniqueName: \"kubernetes.io/projected/61b9fa8b-c3a7-4aba-adf6-d395b41fb14c-kube-api-access-6btpg\") pod \"payload-processing-e2e-worker-single-main-21ea69-67886cfff2lnk6\" (UID: \"61b9fa8b-c3a7-4aba-adf6-d395b41fb14c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-21ea69-67886cfff2lnk6" Nov 03 14:50:32.615891 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:32.615863 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61b9fa8b-c3a7-4aba-adf6-d395b41fb14c-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-21ea69-67886cfff2lnk6\" (UID: \"61b9fa8b-c3a7-4aba-adf6-d395b41fb14c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-21ea69-67886cfff2lnk6" Nov 03 14:50:32.669690 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:32.669657 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6btpg\" (UniqueName: \"kubernetes.io/projected/61b9fa8b-c3a7-4aba-adf6-d395b41fb14c-kube-api-access-6btpg\") pod \"payload-processing-e2e-worker-single-main-21ea69-67886cfff2lnk6\" (UID: \"61b9fa8b-c3a7-4aba-adf6-d395b41fb14c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-21ea69-67886cfff2lnk6" Nov 03 14:50:32.692017 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:32.691985 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-21ea69-67886cfff2lnk6" Nov 03 14:50:33.061710 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:33.061633 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-21ea69-67886cfff2lnk6"] Nov 03 14:50:33.069204 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:50:33.069178 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61b9fa8b_c3a7_4aba_adf6_d395b41fb14c.slice/crio-9661fcea68652401402277d66b6e87e7219fc86c43b5f74b83e0858c8340b8e9 WatchSource:0}: Error finding container 9661fcea68652401402277d66b6e87e7219fc86c43b5f74b83e0858c8340b8e9: Status 404 returned error can't find the container with id 9661fcea68652401402277d66b6e87e7219fc86c43b5f74b83e0858c8340b8e9 Nov 03 14:50:33.787092 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:33.787059 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-21ea69-67886cfff2lnk6" event={"ID":"61b9fa8b-c3a7-4aba-adf6-d395b41fb14c","Type":"ContainerStarted","Data":"e306809c35df41b6dfc6e6cc64b62ed5ebb41a3b822e24c2ad4c585bd294365c"} Nov 03 14:50:33.787545 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:33.787113 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-21ea69-67886cfff2lnk6" event={"ID":"61b9fa8b-c3a7-4aba-adf6-d395b41fb14c","Type":"ContainerStarted","Data":"9661fcea68652401402277d66b6e87e7219fc86c43b5f74b83e0858c8340b8e9"} Nov 03 14:50:33.787545 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:33.787232 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-21ea69-67886cfff2lnk6" Nov 03 14:50:33.897194 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:33.897142 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-21ea69-67886cfff2lnk6" podStartSLOduration=1.89712885 podStartE2EDuration="1.89712885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:50:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:50:33.895733502 +0000 UTC m=+6315.906421744" watchObservedRunningTime="2026-11-03 14:50:33.89712885 +0000 UTC m=+6315.907817091" Nov 03 14:50:34.566577 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:34.566540 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:50:37.233872 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:50:37.233832 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:50:44.796442 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:44.796407 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-21ea69-67886cfff2lnk6" Nov 03 14:50:45.763088 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:45.763049 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:50:53.053971 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:53.053932 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-xqc9n"] Nov 03 14:50:53.100549 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:53.100498 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-xqc9n"] Nov 03 14:50:54.115020 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:50:54.114986 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="081897b6-db58-4a1b-85f1-852046335915" path="/var/lib/kubelet/pods/081897b6-db58-4a1b-85f1-852046335915/volumes" Nov 03 14:51:02.908624 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:51:02.908584 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:51:07.044785 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:51:07.044742 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r8gdk"] Nov 03 14:51:07.092213 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:51:07.092175 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-r8gdk"] Nov 03 14:51:07.234458 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:51:07.234424 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:51:08.114031 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:51:08.113999 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e9295fd-84fd-4ae3-88f5-70dedac0bcb3" path="/var/lib/kubelet/pods/1e9295fd-84fd-4ae3-88f5-70dedac0bcb3/volumes" Nov 03 14:51:23.184858 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:51:23.184821 5979 scope.go:122] "RemoveContainer" containerID="f37437f74b8b954a63655f41f07796199fd72a73d7da80aba265e0caa51ca8ed" Nov 03 14:51:33.312308 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:51:33.312271 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:51:37.233907 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:51:37.233867 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:51:42.046520 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:51:42.046455 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vbcp9"] Nov 03 14:51:42.092881 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:51:42.092850 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-vbcp9"] Nov 03 14:51:42.114527 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:51:42.114500 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8750bbf1-49f2-45ef-ad4a-33724ccfd791" path="/var/lib/kubelet/pods/8750bbf1-49f2-45ef-ad4a-33724ccfd791/volumes" Nov 03 14:51:48.087113 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:51:48.087072 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea3723ce-ddnx4"] Nov 03 14:51:48.090936 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:51:48.090913 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea3723ce-ddnx4" Nov 03 14:51:48.187033 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:51:48.186996 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:51:48.187224 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:51:48.187138 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wgk87\"" Nov 03 14:51:48.189093 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:51:48.189072 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea3723ce-ddnx4"] Nov 03 14:51:48.233683 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:51:48.233654 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:51:48.233683 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:51:48.233666 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:51:48.240204 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:51:48.240181 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e90561f-e17a-4d79-b575-ef97ff648f7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea3723ce-ddnx4\" (UID: \"7e90561f-e17a-4d79-b575-ef97ff648f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea3723ce-ddnx4" Nov 03 14:51:48.240264 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:51:48.240220 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bhmk\" (UniqueName: \"kubernetes.io/projected/7e90561f-e17a-4d79-b575-ef97ff648f7f-kube-api-access-8bhmk\") pod \"maas-api-revoke-sub-ea3723ce-ddnx4\" (UID: \"7e90561f-e17a-4d79-b575-ef97ff648f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea3723ce-ddnx4" Nov 03 14:51:48.240382 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:51:48.240368 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e90561f-e17a-4d79-b575-ef97ff648f7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea3723ce-ddnx4\" (UID: \"7e90561f-e17a-4d79-b575-ef97ff648f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea3723ce-ddnx4" Nov 03 14:51:48.341373 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:51:48.341288 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e90561f-e17a-4d79-b575-ef97ff648f7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea3723ce-ddnx4\" (UID: \"7e90561f-e17a-4d79-b575-ef97ff648f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea3723ce-ddnx4" Nov 03 14:51:48.341373 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:51:48.341326 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bhmk\" (UniqueName: \"kubernetes.io/projected/7e90561f-e17a-4d79-b575-ef97ff648f7f-kube-api-access-8bhmk\") pod \"maas-api-revoke-sub-ea3723ce-ddnx4\" (UID: \"7e90561f-e17a-4d79-b575-ef97ff648f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea3723ce-ddnx4" Nov 03 14:51:48.341615 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:51:48.341417 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e90561f-e17a-4d79-b575-ef97ff648f7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea3723ce-ddnx4\" (UID: \"7e90561f-e17a-4d79-b575-ef97ff648f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea3723ce-ddnx4" Nov 03 14:51:48.341995 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:51:48.341970 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e90561f-e17a-4d79-b575-ef97ff648f7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea3723ce-ddnx4\" (UID: \"7e90561f-e17a-4d79-b575-ef97ff648f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea3723ce-ddnx4" Nov 03 14:51:48.343890 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:51:48.343849 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e90561f-e17a-4d79-b575-ef97ff648f7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea3723ce-ddnx4\" (UID: \"7e90561f-e17a-4d79-b575-ef97ff648f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea3723ce-ddnx4" Nov 03 14:51:48.396951 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:51:48.396921 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bhmk\" (UniqueName: \"kubernetes.io/projected/7e90561f-e17a-4d79-b575-ef97ff648f7f-kube-api-access-8bhmk\") pod \"maas-api-revoke-sub-ea3723ce-ddnx4\" (UID: \"7e90561f-e17a-4d79-b575-ef97ff648f7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea3723ce-ddnx4" Nov 03 14:51:48.403613 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:51:48.403591 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea3723ce-ddnx4" Nov 03 14:51:48.772629 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:51:48.772594 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea3723ce-ddnx4"] Nov 03 14:51:48.780309 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:51:48.780285 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e90561f_e17a_4d79_b575_ef97ff648f7f.slice/crio-a32451a678d73a395fc2b7b209b8e982900bc3f3a60a9ad8066c3e1b18e9082c WatchSource:0}: Error finding container a32451a678d73a395fc2b7b209b8e982900bc3f3a60a9ad8066c3e1b18e9082c: Status 404 returned error can't find the container with id a32451a678d73a395fc2b7b209b8e982900bc3f3a60a9ad8066c3e1b18e9082c Nov 03 14:51:49.214605 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:51:49.214568 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea3723ce-ddnx4" event={"ID":"7e90561f-e17a-4d79-b575-ef97ff648f7f","Type":"ContainerStarted","Data":"b1925b25c9811696d21cea1ebe24ee3a28d773362ed0034f7f2c79ab6a8671ae"} Nov 03 14:51:49.215002 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:51:49.214618 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea3723ce-ddnx4" event={"ID":"7e90561f-e17a-4d79-b575-ef97ff648f7f","Type":"ContainerStarted","Data":"a32451a678d73a395fc2b7b209b8e982900bc3f3a60a9ad8066c3e1b18e9082c"} Nov 03 14:51:49.321158 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:51:49.321049 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea3723ce-ddnx4" podStartSLOduration=1.321033121 podStartE2EDuration="1.321033121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:51:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:51:49.320746701 +0000 UTC m=+6391.331434945" watchObservedRunningTime="2026-11-03 14:51:49.321033121 +0000 UTC m=+6391.331721369" Nov 03 14:51:50.221908 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:51:50.221876 5979 generic.go:356] "Generic (PLEG): container finished" podID="7e90561f-e17a-4d79-b575-ef97ff648f7f" containerID="b1925b25c9811696d21cea1ebe24ee3a28d773362ed0034f7f2c79ab6a8671ae" exitCode=0 Nov 03 14:51:50.222310 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:51:50.221914 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea3723ce-ddnx4" event={"ID":"7e90561f-e17a-4d79-b575-ef97ff648f7f","Type":"ContainerDied","Data":"b1925b25c9811696d21cea1ebe24ee3a28d773362ed0034f7f2c79ab6a8671ae"} Nov 03 14:51:51.369114 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:51:51.369087 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea3723ce-ddnx4" Nov 03 14:51:51.472722 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:51:51.472690 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e90561f-e17a-4d79-b575-ef97ff648f7f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e90561f-e17a-4d79-b575-ef97ff648f7f-maas-api-openshift-ca\") pod \"7e90561f-e17a-4d79-b575-ef97ff648f7f\" (UID: \"7e90561f-e17a-4d79-b575-ef97ff648f7f\") " Nov 03 14:51:51.472909 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:51:51.472808 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e90561f-e17a-4d79-b575-ef97ff648f7f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e90561f-e17a-4d79-b575-ef97ff648f7f-maas-api-xks-ca\") pod \"7e90561f-e17a-4d79-b575-ef97ff648f7f\" (UID: \"7e90561f-e17a-4d79-b575-ef97ff648f7f\") " Nov 03 14:51:51.472909 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:51:51.472877 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e90561f-e17a-4d79-b575-ef97ff648f7f-kube-api-access-8bhmk\" (UniqueName: \"kubernetes.io/projected/7e90561f-e17a-4d79-b575-ef97ff648f7f-kube-api-access-8bhmk\") pod \"7e90561f-e17a-4d79-b575-ef97ff648f7f\" (UID: \"7e90561f-e17a-4d79-b575-ef97ff648f7f\") " Nov 03 14:51:51.473141 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:51:51.473118 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e90561f-e17a-4d79-b575-ef97ff648f7f-maas-api-openshift-ca" pod "7e90561f-e17a-4d79-b575-ef97ff648f7f" (UID: "7e90561f-e17a-4d79-b575-ef97ff648f7f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:51:51.473288 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:51:51.473266 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e90561f-e17a-4d79-b575-ef97ff648f7f-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:51:51.474977 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:51:51.474947 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e90561f-e17a-4d79-b575-ef97ff648f7f-maas-api-xks-ca" pod "7e90561f-e17a-4d79-b575-ef97ff648f7f" (UID: "7e90561f-e17a-4d79-b575-ef97ff648f7f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:51:51.475057 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:51:51.475002 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e90561f-e17a-4d79-b575-ef97ff648f7f-kube-api-access-8bhmk" pod "7e90561f-e17a-4d79-b575-ef97ff648f7f" (UID: "7e90561f-e17a-4d79-b575-ef97ff648f7f"). InnerVolumeSpecName "kube-api-access-8bhmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:51:51.573838 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:51:51.573802 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e90561f-e17a-4d79-b575-ef97ff648f7f-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:51:51.573838 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:51:51.573832 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bhmk\" (UniqueName: \"kubernetes.io/projected/7e90561f-e17a-4d79-b575-ef97ff648f7f-kube-api-access-8bhmk\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:51:52.043835 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:51:52.043802 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v6c4j"] Nov 03 14:51:52.046534 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:51:52.046508 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-v6c4j"] Nov 03 14:51:52.114449 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:51:52.114413 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04264336-4e68-424f-93a9-190b9bfe5606" path="/var/lib/kubelet/pods/04264336-4e68-424f-93a9-190b9bfe5606/volumes" Nov 03 14:51:52.235512 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:51:52.235460 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea3723ce-ddnx4" event={"ID":"7e90561f-e17a-4d79-b575-ef97ff648f7f","Type":"ContainerDied","Data":"a32451a678d73a395fc2b7b209b8e982900bc3f3a60a9ad8066c3e1b18e9082c"} Nov 03 14:51:52.235684 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:51:52.235516 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea3723ce-ddnx4" Nov 03 14:51:52.235684 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:51:52.235531 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a32451a678d73a395fc2b7b209b8e982900bc3f3a60a9ad8066c3e1b18e9082c" Nov 03 14:52:02.832529 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:02.832438 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:52:05.922376 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:05.922332 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-21ea69-67886cfff2lnk6"] Nov 03 14:52:07.233786 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:52:07.233751 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:52:10.923277 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:10.923215 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-21ea69-67886cfff2lnk6" podUID="61b9fa8b-c3a7-4aba-adf6-d395b41fb14c" containerName="payload-processing" containerID="cri-o://e306809c35df41b6dfc6e6cc64b62ed5ebb41a3b822e24c2ad4c585bd294365c" gracePeriod=25 Nov 03 14:52:14.794617 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:14.794577 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-21ea69-67886cfff2lnk6" podUID="61b9fa8b-c3a7-4aba-adf6-d395b41fb14c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:52:16.172298 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:16.172265 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:52:23.207634 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:23.207599 5979 scope.go:122] "RemoveContainer" containerID="d861dce9a8f5bea87891b8fbc22c119502cbaf9c416c5604b4d7547ad1495d96" Nov 03 14:52:23.216747 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:23.216719 5979 scope.go:122] "RemoveContainer" containerID="6be055b85bc6a256dc6a68ba7405c86d858a44cfc88994207e2617c1356458a2" Nov 03 14:52:23.225354 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:23.225332 5979 scope.go:122] "RemoveContainer" containerID="bdfeca87978993ca8849654c5de837ebfe754222dbe3c0043f87ccc18738ce2d" Nov 03 14:52:24.794894 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:24.794847 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-21ea69-67886cfff2lnk6" podUID="61b9fa8b-c3a7-4aba-adf6-d395b41fb14c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 14:52:27.044718 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:27.044687 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5sj28"] Nov 03 14:52:27.091131 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:27.091093 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5sj28"] Nov 03 14:52:28.114436 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:28.114396 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1888aae5-3017-43bd-baa1-57623738a1a0" path="/var/lib/kubelet/pods/1888aae5-3017-43bd-baa1-57623738a1a0/volumes" Nov 03 14:52:31.181194 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:31.181131 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-21ea69-67886cfff2lnk6" Nov 03 14:52:31.292763 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:31.292723 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61b9fa8b-c3a7-4aba-adf6-d395b41fb14c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61b9fa8b-c3a7-4aba-adf6-d395b41fb14c-plugins-config-volume\") pod \"61b9fa8b-c3a7-4aba-adf6-d395b41fb14c\" (UID: \"61b9fa8b-c3a7-4aba-adf6-d395b41fb14c\") " Nov 03 14:52:31.292972 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:31.292831 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61b9fa8b-c3a7-4aba-adf6-d395b41fb14c-kube-api-access-6btpg\" (UniqueName: \"kubernetes.io/projected/61b9fa8b-c3a7-4aba-adf6-d395b41fb14c-kube-api-access-6btpg\") pod \"61b9fa8b-c3a7-4aba-adf6-d395b41fb14c\" (UID: \"61b9fa8b-c3a7-4aba-adf6-d395b41fb14c\") " Nov 03 14:52:31.293124 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:31.293095 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61b9fa8b-c3a7-4aba-adf6-d395b41fb14c-plugins-config-volume" pod "61b9fa8b-c3a7-4aba-adf6-d395b41fb14c" (UID: "61b9fa8b-c3a7-4aba-adf6-d395b41fb14c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:52:31.295036 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:31.295011 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61b9fa8b-c3a7-4aba-adf6-d395b41fb14c-kube-api-access-6btpg" pod "61b9fa8b-c3a7-4aba-adf6-d395b41fb14c" (UID: "61b9fa8b-c3a7-4aba-adf6-d395b41fb14c"). InnerVolumeSpecName "kube-api-access-6btpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:52:31.393730 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:31.393692 5979 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61b9fa8b-c3a7-4aba-adf6-d395b41fb14c-plugins-config-volume\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:52:31.393730 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:31.393722 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6btpg\" (UniqueName: \"kubernetes.io/projected/61b9fa8b-c3a7-4aba-adf6-d395b41fb14c-kube-api-access-6btpg\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:52:31.448849 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:31.448760 5979 generic.go:356] "Generic (PLEG): container finished" podID="61b9fa8b-c3a7-4aba-adf6-d395b41fb14c" containerID="e306809c35df41b6dfc6e6cc64b62ed5ebb41a3b822e24c2ad4c585bd294365c" exitCode=0 Nov 03 14:52:31.448849 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:31.448809 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-21ea69-67886cfff2lnk6" Nov 03 14:52:31.449095 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:31.448843 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-21ea69-67886cfff2lnk6" event={"ID":"61b9fa8b-c3a7-4aba-adf6-d395b41fb14c","Type":"ContainerDied","Data":"e306809c35df41b6dfc6e6cc64b62ed5ebb41a3b822e24c2ad4c585bd294365c"} Nov 03 14:52:31.449095 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:31.448917 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-21ea69-67886cfff2lnk6" event={"ID":"61b9fa8b-c3a7-4aba-adf6-d395b41fb14c","Type":"ContainerDied","Data":"9661fcea68652401402277d66b6e87e7219fc86c43b5f74b83e0858c8340b8e9"} Nov 03 14:52:31.449095 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:31.448947 5979 scope.go:122] "RemoveContainer" containerID="e306809c35df41b6dfc6e6cc64b62ed5ebb41a3b822e24c2ad4c585bd294365c" Nov 03 14:52:31.458101 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:31.458081 5979 scope.go:122] "RemoveContainer" containerID="e306809c35df41b6dfc6e6cc64b62ed5ebb41a3b822e24c2ad4c585bd294365c" Nov 03 14:52:31.458346 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:52:31.458326 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e306809c35df41b6dfc6e6cc64b62ed5ebb41a3b822e24c2ad4c585bd294365c\": container with ID starting with e306809c35df41b6dfc6e6cc64b62ed5ebb41a3b822e24c2ad4c585bd294365c not found: ID does not exist" containerID="e306809c35df41b6dfc6e6cc64b62ed5ebb41a3b822e24c2ad4c585bd294365c" Nov 03 14:52:31.458425 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:31.458360 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e306809c35df41b6dfc6e6cc64b62ed5ebb41a3b822e24c2ad4c585bd294365c"} err="failed to get container status \"e306809c35df41b6dfc6e6cc64b62ed5ebb41a3b822e24c2ad4c585bd294365c\": rpc error: code = NotFound desc = could not find container \"e306809c35df41b6dfc6e6cc64b62ed5ebb41a3b822e24c2ad4c585bd294365c\": container with ID starting with e306809c35df41b6dfc6e6cc64b62ed5ebb41a3b822e24c2ad4c585bd294365c not found: ID does not exist" Nov 03 14:52:31.604435 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:31.604404 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-21ea69-67886cfff2lnk6"] Nov 03 14:52:31.609704 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:31.609679 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-21ea69-67886cfff2lnk6"] Nov 03 14:52:32.114163 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:32.114133 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61b9fa8b-c3a7-4aba-adf6-d395b41fb14c" path="/var/lib/kubelet/pods/61b9fa8b-c3a7-4aba-adf6-d395b41fb14c/volumes" Nov 03 14:52:37.232816 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:52:37.232776 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:52:42.800074 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:42.800035 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-97787c8c5-d996t"] Nov 03 14:52:42.800616 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:42.800598 5979 cpu_manager.go:414] "containerMap: removing container" podUID="61b9fa8b-c3a7-4aba-adf6-d395b41fb14c" containerName="payload-processing" Nov 03 14:52:42.800730 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:42.800617 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61b9fa8b-c3a7-4aba-adf6-d395b41fb14c" containerName="payload-processing" Nov 03 14:52:42.800730 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:42.800629 5979 cpu_manager.go:414] "containerMap: removing container" podUID="7e90561f-e17a-4d79-b575-ef97ff648f7f" containerName="revoke-keys" Nov 03 14:52:42.800730 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:42.800634 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e90561f-e17a-4d79-b575-ef97ff648f7f" containerName="revoke-keys" Nov 03 14:52:42.800898 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:42.800732 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="61b9fa8b-c3a7-4aba-adf6-d395b41fb14c" containerName="payload-processing" Nov 03 14:52:42.800898 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:42.800743 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e90561f-e17a-4d79-b575-ef97ff648f7f" containerName="revoke-keys" Nov 03 14:52:42.804052 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:42.804027 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-97787c8c5-d996t" Nov 03 14:52:42.852828 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:42.852796 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:52:42.899527 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:42.899498 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:52:42.901385 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:42.901363 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-97787c8c5-d996t"] Nov 03 14:52:42.907951 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:42.907927 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97109657-c89e-4dab-b607-48f1961f03ea-metrics-certs\") pod \"maas-api-e2e-worker-single-main-21ea69-97787c8c5-d996t\" (UID: \"97109657-c89e-4dab-b607-48f1961f03ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-97787c8c5-d996t" Nov 03 14:52:42.908025 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:42.907972 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97109657-c89e-4dab-b607-48f1961f03ea-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-21ea69-97787c8c5-d996t\" (UID: \"97109657-c89e-4dab-b607-48f1961f03ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-97787c8c5-d996t" Nov 03 14:52:42.908065 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:42.908049 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxqmw\" (UniqueName: \"kubernetes.io/projected/97109657-c89e-4dab-b607-48f1961f03ea-kube-api-access-dxqmw\") pod \"maas-api-e2e-worker-single-main-21ea69-97787c8c5-d996t\" (UID: \"97109657-c89e-4dab-b607-48f1961f03ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-97787c8c5-d996t" Nov 03 14:52:42.908157 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:42.908144 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97109657-c89e-4dab-b607-48f1961f03ea-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-21ea69-97787c8c5-d996t\" (UID: \"97109657-c89e-4dab-b607-48f1961f03ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-97787c8c5-d996t" Nov 03 14:52:42.910671 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:42.910648 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-21ea69-serving-cert\"" Nov 03 14:52:42.910802 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:42.910678 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 14:52:42.910802 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:42.910722 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-dgx9m\"" Nov 03 14:52:43.009090 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:43.009050 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97109657-c89e-4dab-b607-48f1961f03ea-metrics-certs\") pod \"maas-api-e2e-worker-single-main-21ea69-97787c8c5-d996t\" (UID: \"97109657-c89e-4dab-b607-48f1961f03ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-97787c8c5-d996t" Nov 03 14:52:43.009090 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:43.009095 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97109657-c89e-4dab-b607-48f1961f03ea-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-21ea69-97787c8c5-d996t\" (UID: \"97109657-c89e-4dab-b607-48f1961f03ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-97787c8c5-d996t" Nov 03 14:52:43.009381 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:43.009127 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxqmw\" (UniqueName: \"kubernetes.io/projected/97109657-c89e-4dab-b607-48f1961f03ea-kube-api-access-dxqmw\") pod \"maas-api-e2e-worker-single-main-21ea69-97787c8c5-d996t\" (UID: \"97109657-c89e-4dab-b607-48f1961f03ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-97787c8c5-d996t" Nov 03 14:52:43.009381 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:43.009205 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97109657-c89e-4dab-b607-48f1961f03ea-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-21ea69-97787c8c5-d996t\" (UID: \"97109657-c89e-4dab-b607-48f1961f03ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-97787c8c5-d996t" Nov 03 14:52:43.009381 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:52:43.009314 5979 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-serving-cert: secret "maas-api-e2e-worker-single-main-21ea69-serving-cert" not found Nov 03 14:52:43.009597 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:52:43.009384 5979 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97109657-c89e-4dab-b607-48f1961f03ea-maas-api-tls podName:97109657-c89e-4dab-b607-48f1961f03ea nodeName:}" failed. No retries permitted until 2026-11-03 14:52:43.509364349 +0000 UTC m=+6445.520052570 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/97109657-c89e-4dab-b607-48f1961f03ea-maas-api-tls") pod "maas-api-e2e-worker-single-main-21ea69-97787c8c5-d996t" (UID: "97109657-c89e-4dab-b607-48f1961f03ea") : secret "maas-api-e2e-worker-single-main-21ea69-serving-cert" not found Nov 03 14:52:43.009949 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:43.009924 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97109657-c89e-4dab-b607-48f1961f03ea-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-21ea69-97787c8c5-d996t\" (UID: \"97109657-c89e-4dab-b607-48f1961f03ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-97787c8c5-d996t" Nov 03 14:52:43.011580 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:43.011557 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97109657-c89e-4dab-b607-48f1961f03ea-metrics-certs\") pod \"maas-api-e2e-worker-single-main-21ea69-97787c8c5-d996t\" (UID: \"97109657-c89e-4dab-b607-48f1961f03ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-97787c8c5-d996t" Nov 03 14:52:43.067093 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:43.067014 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxqmw\" (UniqueName: \"kubernetes.io/projected/97109657-c89e-4dab-b607-48f1961f03ea-kube-api-access-dxqmw\") pod \"maas-api-e2e-worker-single-main-21ea69-97787c8c5-d996t\" (UID: \"97109657-c89e-4dab-b607-48f1961f03ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-97787c8c5-d996t" Nov 03 14:52:43.513769 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:43.513730 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97109657-c89e-4dab-b607-48f1961f03ea-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-21ea69-97787c8c5-d996t\" (UID: \"97109657-c89e-4dab-b607-48f1961f03ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-97787c8c5-d996t" Nov 03 14:52:43.516435 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:43.516412 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97109657-c89e-4dab-b607-48f1961f03ea-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-21ea69-97787c8c5-d996t\" (UID: \"97109657-c89e-4dab-b607-48f1961f03ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-97787c8c5-d996t" Nov 03 14:52:43.716523 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:43.716462 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-97787c8c5-d996t" Nov 03 14:52:44.086413 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:44.086340 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-97787c8c5-d996t"] Nov 03 14:52:44.093623 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:52:44.093595 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97109657_c89e_4dab_b607_48f1961f03ea.slice/crio-916d638a0638f75451e5a746813e4122861c61272d0e3ac85eb54a86b0fb9fe1 WatchSource:0}: Error finding container 916d638a0638f75451e5a746813e4122861c61272d0e3ac85eb54a86b0fb9fe1: Status 404 returned error can't find the container with id 916d638a0638f75451e5a746813e4122861c61272d0e3ac85eb54a86b0fb9fe1 Nov 03 14:52:44.094999 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:44.094979 5979 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 14:52:44.521960 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:44.521924 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-97787c8c5-d996t" event={"ID":"97109657-c89e-4dab-b607-48f1961f03ea","Type":"ContainerStarted","Data":"916d638a0638f75451e5a746813e4122861c61272d0e3ac85eb54a86b0fb9fe1"} Nov 03 14:52:45.529589 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:45.529496 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-97787c8c5-d996t" event={"ID":"97109657-c89e-4dab-b607-48f1961f03ea","Type":"ContainerStarted","Data":"853f13371254bb89e831b20b02ffb9d89aecb3414113a7a1978f35a4561244b2"} Nov 03 14:52:45.530047 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:45.529609 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-97787c8c5-d996t" Nov 03 14:52:45.638107 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:45.638054 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-97787c8c5-d996t" podStartSLOduration=2.567576544 podStartE2EDuration="3.638040084s" totalImagesPullingTime="1.07046354s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:52:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:52:45.636651641 +0000 UTC m=+6447.647339883" watchObservedRunningTime="2026-11-03 14:52:45.638040084 +0000 UTC m=+6447.648728327" Nov 03 14:52:46.062993 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:46.062722 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-xwwdw"] Nov 03 14:52:46.063803 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:46.063771 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-xwwdw"] Nov 03 14:52:46.114394 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:46.114356 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cea64d6-4955-4f2c-a08c-0f5d7724927b" path="/var/lib/kubelet/pods/5cea64d6-4955-4f2c-a08c-0f5d7724927b/volumes" Nov 03 14:52:49.771729 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:49.771684 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:52:51.540636 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:52:51.540608 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-97787c8c5-d996t" Nov 03 14:53:07.233299 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:53:07.233259 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:53:08.049516 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:08.049454 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wzslf"] Nov 03 14:53:08.095928 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:08.095889 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-wzslf"] Nov 03 14:53:08.114583 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:08.114546 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9e1f087-e6e6-4cc7-b0c9-6ba850c9881f" path="/var/lib/kubelet/pods/b9e1f087-e6e6-4cc7-b0c9-6ba850c9881f/volumes" Nov 03 14:53:12.853819 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:12.853781 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-97787c8c5-d996t"] Nov 03 14:53:12.854302 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:12.854026 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-97787c8c5-d996t" podUID="97109657-c89e-4dab-b607-48f1961f03ea" containerName="maas-api" containerID="cri-o://853f13371254bb89e831b20b02ffb9d89aecb3414113a7a1978f35a4561244b2" gracePeriod=30 Nov 03 14:53:13.124702 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:13.124671 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-97787c8c5-d996t" Nov 03 14:53:13.260314 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:13.260275 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97109657-c89e-4dab-b607-48f1961f03ea-metrics-certs\" (UniqueName: \"kubernetes.io/secret/97109657-c89e-4dab-b607-48f1961f03ea-metrics-certs\") pod \"97109657-c89e-4dab-b607-48f1961f03ea\" (UID: \"97109657-c89e-4dab-b607-48f1961f03ea\") " Nov 03 14:53:13.260314 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:13.260325 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97109657-c89e-4dab-b607-48f1961f03ea-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97109657-c89e-4dab-b607-48f1961f03ea-maas-api-tls\") pod \"97109657-c89e-4dab-b607-48f1961f03ea\" (UID: \"97109657-c89e-4dab-b607-48f1961f03ea\") " Nov 03 14:53:13.260651 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:13.260365 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97109657-c89e-4dab-b607-48f1961f03ea-kube-api-access-dxqmw\" (UniqueName: \"kubernetes.io/projected/97109657-c89e-4dab-b607-48f1961f03ea-kube-api-access-dxqmw\") pod \"97109657-c89e-4dab-b607-48f1961f03ea\" (UID: \"97109657-c89e-4dab-b607-48f1961f03ea\") " Nov 03 14:53:13.260651 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:13.260418 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97109657-c89e-4dab-b607-48f1961f03ea-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97109657-c89e-4dab-b607-48f1961f03ea-openshift-service-ca\") pod \"97109657-c89e-4dab-b607-48f1961f03ea\" (UID: \"97109657-c89e-4dab-b607-48f1961f03ea\") " Nov 03 14:53:13.261244 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:13.261212 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97109657-c89e-4dab-b607-48f1961f03ea-openshift-service-ca" pod "97109657-c89e-4dab-b607-48f1961f03ea" (UID: "97109657-c89e-4dab-b607-48f1961f03ea"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:53:13.265184 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:13.265140 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97109657-c89e-4dab-b607-48f1961f03ea-maas-api-tls" pod "97109657-c89e-4dab-b607-48f1961f03ea" (UID: "97109657-c89e-4dab-b607-48f1961f03ea"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:53:13.265663 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:13.265631 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97109657-c89e-4dab-b607-48f1961f03ea-metrics-certs" pod "97109657-c89e-4dab-b607-48f1961f03ea" (UID: "97109657-c89e-4dab-b607-48f1961f03ea"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:53:13.266132 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:13.266096 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97109657-c89e-4dab-b607-48f1961f03ea-kube-api-access-dxqmw" pod "97109657-c89e-4dab-b607-48f1961f03ea" (UID: "97109657-c89e-4dab-b607-48f1961f03ea"). InnerVolumeSpecName "kube-api-access-dxqmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:53:13.370385 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:13.370255 5979 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97109657-c89e-4dab-b607-48f1961f03ea-metrics-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:53:13.370385 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:13.370299 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97109657-c89e-4dab-b607-48f1961f03ea-maas-api-tls\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:53:13.370385 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:13.370316 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxqmw\" (UniqueName: \"kubernetes.io/projected/97109657-c89e-4dab-b607-48f1961f03ea-kube-api-access-dxqmw\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:53:13.370385 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:13.370332 5979 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97109657-c89e-4dab-b607-48f1961f03ea-openshift-service-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:53:13.707893 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:13.707860 5979 generic.go:356] "Generic (PLEG): container finished" podID="97109657-c89e-4dab-b607-48f1961f03ea" containerID="853f13371254bb89e831b20b02ffb9d89aecb3414113a7a1978f35a4561244b2" exitCode=0 Nov 03 14:53:13.708094 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:13.707909 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-97787c8c5-d996t" Nov 03 14:53:13.708094 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:13.707948 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-97787c8c5-d996t" event={"ID":"97109657-c89e-4dab-b607-48f1961f03ea","Type":"ContainerDied","Data":"853f13371254bb89e831b20b02ffb9d89aecb3414113a7a1978f35a4561244b2"} Nov 03 14:53:13.708094 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:13.708011 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-97787c8c5-d996t" event={"ID":"97109657-c89e-4dab-b607-48f1961f03ea","Type":"ContainerDied","Data":"916d638a0638f75451e5a746813e4122861c61272d0e3ac85eb54a86b0fb9fe1"} Nov 03 14:53:13.708094 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:13.708043 5979 scope.go:122] "RemoveContainer" containerID="853f13371254bb89e831b20b02ffb9d89aecb3414113a7a1978f35a4561244b2" Nov 03 14:53:13.718003 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:13.717983 5979 scope.go:122] "RemoveContainer" containerID="853f13371254bb89e831b20b02ffb9d89aecb3414113a7a1978f35a4561244b2" Nov 03 14:53:13.718263 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:53:13.718240 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"853f13371254bb89e831b20b02ffb9d89aecb3414113a7a1978f35a4561244b2\": container with ID starting with 853f13371254bb89e831b20b02ffb9d89aecb3414113a7a1978f35a4561244b2 not found: ID does not exist" containerID="853f13371254bb89e831b20b02ffb9d89aecb3414113a7a1978f35a4561244b2" Nov 03 14:53:13.718322 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:13.718271 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"853f13371254bb89e831b20b02ffb9d89aecb3414113a7a1978f35a4561244b2"} err="failed to get container status \"853f13371254bb89e831b20b02ffb9d89aecb3414113a7a1978f35a4561244b2\": rpc error: code = NotFound desc = could not find container \"853f13371254bb89e831b20b02ffb9d89aecb3414113a7a1978f35a4561244b2\": container with ID starting with 853f13371254bb89e831b20b02ffb9d89aecb3414113a7a1978f35a4561244b2 not found: ID does not exist" Nov 03 14:53:13.867520 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:13.867483 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-97787c8c5-d996t"] Nov 03 14:53:13.914043 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:13.914008 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-21ea69-97787c8c5-d996t"] Nov 03 14:53:14.114366 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:14.114287 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97109657-c89e-4dab-b607-48f1961f03ea" path="/var/lib/kubelet/pods/97109657-c89e-4dab-b607-48f1961f03ea/volumes" Nov 03 14:53:17.318776 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:17.318744 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w5x6m"] Nov 03 14:53:17.365803 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:17.365765 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-w5x6m"] Nov 03 14:53:17.441108 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:17.441074 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6ws92"] Nov 03 14:53:17.441513 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:17.441499 5979 cpu_manager.go:414] "containerMap: removing container" podUID="97109657-c89e-4dab-b607-48f1961f03ea" containerName="maas-api" Nov 03 14:53:17.441571 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:17.441514 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97109657-c89e-4dab-b607-48f1961f03ea" containerName="maas-api" Nov 03 14:53:17.441610 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:17.441600 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="97109657-c89e-4dab-b607-48f1961f03ea" containerName="maas-api" Nov 03 14:53:17.448096 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:17.448059 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6ws92" Nov 03 14:53:17.500317 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:17.500272 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:53:17.500317 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:17.500272 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:53:17.500626 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:17.500378 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:53:17.500626 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:17.500385 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wgk87\"" Nov 03 14:53:17.502217 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:17.502193 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6ws92"] Nov 03 14:53:17.612598 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:17.612509 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9n72\" (UniqueName: \"kubernetes.io/projected/05787e77-c394-4cdf-8a2a-93a9f2ded1c2-kube-api-access-s9n72\") pod \"maas-api-revoke-sub-3d4156f1-6ws92\" (UID: \"05787e77-c394-4cdf-8a2a-93a9f2ded1c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6ws92" Nov 03 14:53:17.612769 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:17.612653 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05787e77-c394-4cdf-8a2a-93a9f2ded1c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-6ws92\" (UID: \"05787e77-c394-4cdf-8a2a-93a9f2ded1c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6ws92" Nov 03 14:53:17.612769 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:17.612691 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05787e77-c394-4cdf-8a2a-93a9f2ded1c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-6ws92\" (UID: \"05787e77-c394-4cdf-8a2a-93a9f2ded1c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6ws92" Nov 03 14:53:17.713948 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:17.713907 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9n72\" (UniqueName: \"kubernetes.io/projected/05787e77-c394-4cdf-8a2a-93a9f2ded1c2-kube-api-access-s9n72\") pod \"maas-api-revoke-sub-3d4156f1-6ws92\" (UID: \"05787e77-c394-4cdf-8a2a-93a9f2ded1c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6ws92" Nov 03 14:53:17.714148 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:17.714044 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05787e77-c394-4cdf-8a2a-93a9f2ded1c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-6ws92\" (UID: \"05787e77-c394-4cdf-8a2a-93a9f2ded1c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6ws92" Nov 03 14:53:17.714148 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:17.714081 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05787e77-c394-4cdf-8a2a-93a9f2ded1c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-6ws92\" (UID: \"05787e77-c394-4cdf-8a2a-93a9f2ded1c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6ws92" Nov 03 14:53:17.714770 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:17.714744 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05787e77-c394-4cdf-8a2a-93a9f2ded1c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-6ws92\" (UID: \"05787e77-c394-4cdf-8a2a-93a9f2ded1c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6ws92" Nov 03 14:53:17.716677 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:17.716650 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05787e77-c394-4cdf-8a2a-93a9f2ded1c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-6ws92\" (UID: \"05787e77-c394-4cdf-8a2a-93a9f2ded1c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6ws92" Nov 03 14:53:17.767314 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:17.767285 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9n72\" (UniqueName: \"kubernetes.io/projected/05787e77-c394-4cdf-8a2a-93a9f2ded1c2-kube-api-access-s9n72\") pod \"maas-api-revoke-sub-3d4156f1-6ws92\" (UID: \"05787e77-c394-4cdf-8a2a-93a9f2ded1c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6ws92" Nov 03 14:53:18.064342 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:18.064309 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6ws92" Nov 03 14:53:18.118071 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:18.118014 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c9e399e-9d8b-42e7-b7bc-876298087344" path="/var/lib/kubelet/pods/0c9e399e-9d8b-42e7-b7bc-876298087344/volumes" Nov 03 14:53:18.435338 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:18.435306 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6ws92"] Nov 03 14:53:18.443251 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:53:18.443219 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05787e77_c394_4cdf_8a2a_93a9f2ded1c2.slice/crio-15208aa7b23be50ddafe2df7d1480495a6a590db9a6256f575af9f7f1fb42215 WatchSource:0}: Error finding container 15208aa7b23be50ddafe2df7d1480495a6a590db9a6256f575af9f7f1fb42215: Status 404 returned error can't find the container with id 15208aa7b23be50ddafe2df7d1480495a6a590db9a6256f575af9f7f1fb42215 Nov 03 14:53:18.738070 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:18.737993 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6ws92" event={"ID":"05787e77-c394-4cdf-8a2a-93a9f2ded1c2","Type":"ContainerStarted","Data":"15208aa7b23be50ddafe2df7d1480495a6a590db9a6256f575af9f7f1fb42215"} Nov 03 14:53:18.938504 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:18.938455 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:53:19.744758 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:19.744720 5979 generic.go:356] "Generic (PLEG): container finished" podID="05787e77-c394-4cdf-8a2a-93a9f2ded1c2" containerID="2e54b7a302131b14bbd47c6888d163747b15eccd0eaa5c13c07e7bcc8aa0cf91" exitCode=0 Nov 03 14:53:19.745154 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:19.744805 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6ws92" event={"ID":"05787e77-c394-4cdf-8a2a-93a9f2ded1c2","Type":"ContainerDied","Data":"2e54b7a302131b14bbd47c6888d163747b15eccd0eaa5c13c07e7bcc8aa0cf91"} Nov 03 14:53:20.886795 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:20.886766 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6ws92" Nov 03 14:53:21.050227 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:21.050148 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05787e77-c394-4cdf-8a2a-93a9f2ded1c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05787e77-c394-4cdf-8a2a-93a9f2ded1c2-maas-api-openshift-ca\") pod \"05787e77-c394-4cdf-8a2a-93a9f2ded1c2\" (UID: \"05787e77-c394-4cdf-8a2a-93a9f2ded1c2\") " Nov 03 14:53:21.050392 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:21.050258 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05787e77-c394-4cdf-8a2a-93a9f2ded1c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05787e77-c394-4cdf-8a2a-93a9f2ded1c2-maas-api-xks-ca\") pod \"05787e77-c394-4cdf-8a2a-93a9f2ded1c2\" (UID: \"05787e77-c394-4cdf-8a2a-93a9f2ded1c2\") " Nov 03 14:53:21.050392 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:21.050338 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05787e77-c394-4cdf-8a2a-93a9f2ded1c2-kube-api-access-s9n72\" (UniqueName: \"kubernetes.io/projected/05787e77-c394-4cdf-8a2a-93a9f2ded1c2-kube-api-access-s9n72\") pod \"05787e77-c394-4cdf-8a2a-93a9f2ded1c2\" (UID: \"05787e77-c394-4cdf-8a2a-93a9f2ded1c2\") " Nov 03 14:53:21.050578 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:21.050550 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05787e77-c394-4cdf-8a2a-93a9f2ded1c2-maas-api-openshift-ca" pod "05787e77-c394-4cdf-8a2a-93a9f2ded1c2" (UID: "05787e77-c394-4cdf-8a2a-93a9f2ded1c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:53:21.052304 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:21.052273 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05787e77-c394-4cdf-8a2a-93a9f2ded1c2-maas-api-xks-ca" pod "05787e77-c394-4cdf-8a2a-93a9f2ded1c2" (UID: "05787e77-c394-4cdf-8a2a-93a9f2ded1c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:53:21.052420 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:21.052353 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05787e77-c394-4cdf-8a2a-93a9f2ded1c2-kube-api-access-s9n72" pod "05787e77-c394-4cdf-8a2a-93a9f2ded1c2" (UID: "05787e77-c394-4cdf-8a2a-93a9f2ded1c2"). InnerVolumeSpecName "kube-api-access-s9n72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:53:21.151975 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:21.151940 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9n72\" (UniqueName: \"kubernetes.io/projected/05787e77-c394-4cdf-8a2a-93a9f2ded1c2-kube-api-access-s9n72\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:53:21.151975 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:21.151969 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05787e77-c394-4cdf-8a2a-93a9f2ded1c2-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:53:21.152193 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:21.151986 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05787e77-c394-4cdf-8a2a-93a9f2ded1c2-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:53:21.756917 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:21.756876 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6ws92" event={"ID":"05787e77-c394-4cdf-8a2a-93a9f2ded1c2","Type":"ContainerDied","Data":"15208aa7b23be50ddafe2df7d1480495a6a590db9a6256f575af9f7f1fb42215"} Nov 03 14:53:21.756917 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:21.756904 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6ws92" Nov 03 14:53:21.757157 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:21.756927 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15208aa7b23be50ddafe2df7d1480495a6a590db9a6256f575af9f7f1fb42215" Nov 03 14:53:23.256549 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:23.256518 5979 scope.go:122] "RemoveContainer" containerID="f800bad9ce14929b16fb99c223119a4d178c448ecec653bfff48c4b73ac78269" Nov 03 14:53:23.265212 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:23.265191 5979 scope.go:122] "RemoveContainer" containerID="8a1651867885d5c9e1fdecf63705a7a11f118ac710cb209dbbd8a284cb13c6ed" Nov 03 14:53:26.044082 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:26.044047 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jxkv2"] Nov 03 14:53:26.090601 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:26.090562 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-jxkv2"] Nov 03 14:53:26.114265 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:26.114229 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f8bed76-f71e-46ec-ab8f-ebca5a00f714" path="/var/lib/kubelet/pods/0f8bed76-f71e-46ec-ab8f-ebca5a00f714/volumes" Nov 03 14:53:31.778496 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:53:31.778434 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:53:37.237572 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:53:37.237532 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:54:07.232895 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:54:07.232839 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:54:23.298365 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:54:23.298330 5979 scope.go:122] "RemoveContainer" containerID="211464ba6c647fc16ae174d9227a8b10b6015be5956aef449dafd2d0d8bd0236" Nov 03 14:54:23.307290 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:54:23.307267 5979 scope.go:122] "RemoveContainer" containerID="5ff445b9529969f61c99d18bf19dc90d3f0e74f5780c729f0c8cab0ced212d8d" Nov 03 14:54:23.315541 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:54:23.315520 5979 scope.go:122] "RemoveContainer" containerID="9dc508f8b15c9eb4cb7a842b6b07b2655a8599c0e52f43e94a94fbe131f8002c" Nov 03 14:54:23.323276 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:54:23.323257 5979 scope.go:122] "RemoveContainer" containerID="6828f10bf3f227098cd4bd30f041b29fdc122fc71675c4b6f8d1fe8f0ee59175" Nov 03 14:54:37.234837 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:54:37.234801 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:54:42.048790 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:54:42.048756 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w27bv"] Nov 03 14:54:42.095197 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:54:42.095159 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-w27bv"] Nov 03 14:54:42.115074 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:54:42.115036 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ced6c9f-849c-48a3-99c3-bf54e17c6231" path="/var/lib/kubelet/pods/0ced6c9f-849c-48a3-99c3-bf54e17c6231/volumes" Nov 03 14:54:53.045170 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:54:53.045131 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g5jp5"] Nov 03 14:54:53.091879 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:54:53.091840 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-g5jp5"] Nov 03 14:54:54.114128 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:54:54.114092 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8908b045-1724-4420-90bf-ddd8a559d3bd" path="/var/lib/kubelet/pods/8908b045-1724-4420-90bf-ddd8a559d3bd/volumes" Nov 03 14:55:06.366460 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:06.366421 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-n69dm"] Nov 03 14:55:06.366935 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:06.366919 5979 cpu_manager.go:414] "containerMap: removing container" podUID="05787e77-c394-4cdf-8a2a-93a9f2ded1c2" containerName="revoke-keys" Nov 03 14:55:06.366987 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:06.366936 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05787e77-c394-4cdf-8a2a-93a9f2ded1c2" containerName="revoke-keys" Nov 03 14:55:06.367028 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:06.367000 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="05787e77-c394-4cdf-8a2a-93a9f2ded1c2" containerName="revoke-keys" Nov 03 14:55:06.370870 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:06.370845 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-n69dm" Nov 03 14:55:06.395984 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:06.395944 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzxzx\" (UniqueName: \"kubernetes.io/projected/9004440b-f00d-4290-98f9-64bbb932a45d-kube-api-access-hzxzx\") pod \"maas-api-revoke-sub-d576e68b-n69dm\" (UID: \"9004440b-f00d-4290-98f9-64bbb932a45d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-n69dm" Nov 03 14:55:06.396127 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:06.396006 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9004440b-f00d-4290-98f9-64bbb932a45d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-n69dm\" (UID: \"9004440b-f00d-4290-98f9-64bbb932a45d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-n69dm" Nov 03 14:55:06.396127 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:06.396113 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9004440b-f00d-4290-98f9-64bbb932a45d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-n69dm\" (UID: \"9004440b-f00d-4290-98f9-64bbb932a45d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-n69dm" Nov 03 14:55:06.419842 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:06.419810 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:55:06.420063 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:06.420040 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:55:06.420144 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:06.420108 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:55:06.466393 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:06.466354 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wgk87\"" Nov 03 14:55:06.468105 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:06.468075 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-n69dm"] Nov 03 14:55:06.480274 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:06.480238 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:55:06.497482 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:06.497429 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9004440b-f00d-4290-98f9-64bbb932a45d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-n69dm\" (UID: \"9004440b-f00d-4290-98f9-64bbb932a45d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-n69dm" Nov 03 14:55:06.497686 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:06.497570 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzxzx\" (UniqueName: \"kubernetes.io/projected/9004440b-f00d-4290-98f9-64bbb932a45d-kube-api-access-hzxzx\") pod \"maas-api-revoke-sub-d576e68b-n69dm\" (UID: \"9004440b-f00d-4290-98f9-64bbb932a45d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-n69dm" Nov 03 14:55:06.497686 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:06.497621 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9004440b-f00d-4290-98f9-64bbb932a45d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-n69dm\" (UID: \"9004440b-f00d-4290-98f9-64bbb932a45d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-n69dm" Nov 03 14:55:06.498195 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:06.498167 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9004440b-f00d-4290-98f9-64bbb932a45d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-n69dm\" (UID: \"9004440b-f00d-4290-98f9-64bbb932a45d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-n69dm" Nov 03 14:55:06.500012 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:06.499991 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9004440b-f00d-4290-98f9-64bbb932a45d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-n69dm\" (UID: \"9004440b-f00d-4290-98f9-64bbb932a45d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-n69dm" Nov 03 14:55:06.551502 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:06.551442 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzxzx\" (UniqueName: \"kubernetes.io/projected/9004440b-f00d-4290-98f9-64bbb932a45d-kube-api-access-hzxzx\") pod \"maas-api-revoke-sub-d576e68b-n69dm\" (UID: \"9004440b-f00d-4290-98f9-64bbb932a45d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-n69dm" Nov 03 14:55:06.684113 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:06.684081 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-n69dm" Nov 03 14:55:07.052647 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:07.052568 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-n69dm"] Nov 03 14:55:07.060852 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:55:07.060819 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9004440b_f00d_4290_98f9_64bbb932a45d.slice/crio-458e1d1988c17bcdeb795d92f24eb5437b80528c22576a979d07e1adec7ab29e WatchSource:0}: Error finding container 458e1d1988c17bcdeb795d92f24eb5437b80528c22576a979d07e1adec7ab29e: Status 404 returned error can't find the container with id 458e1d1988c17bcdeb795d92f24eb5437b80528c22576a979d07e1adec7ab29e Nov 03 14:55:07.232994 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:55:07.232951 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:55:07.343230 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:07.343142 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-n69dm" event={"ID":"9004440b-f00d-4290-98f9-64bbb932a45d","Type":"ContainerStarted","Data":"458e1d1988c17bcdeb795d92f24eb5437b80528c22576a979d07e1adec7ab29e"} Nov 03 14:55:08.350174 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:08.350071 5979 generic.go:356] "Generic (PLEG): container finished" podID="9004440b-f00d-4290-98f9-64bbb932a45d" containerID="f876f2c3f363780bbc7a069e024fc6c379528e9547effb28feb7e56b9a3c4495" exitCode=0 Nov 03 14:55:08.350174 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:08.350152 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-n69dm" event={"ID":"9004440b-f00d-4290-98f9-64bbb932a45d","Type":"ContainerDied","Data":"f876f2c3f363780bbc7a069e024fc6c379528e9547effb28feb7e56b9a3c4495"} Nov 03 14:55:09.495275 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:09.495250 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-n69dm" Nov 03 14:55:09.531059 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:09.531020 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9004440b-f00d-4290-98f9-64bbb932a45d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9004440b-f00d-4290-98f9-64bbb932a45d-maas-api-xks-ca\") pod \"9004440b-f00d-4290-98f9-64bbb932a45d\" (UID: \"9004440b-f00d-4290-98f9-64bbb932a45d\") " Nov 03 14:55:09.531258 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:09.531075 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9004440b-f00d-4290-98f9-64bbb932a45d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9004440b-f00d-4290-98f9-64bbb932a45d-maas-api-openshift-ca\") pod \"9004440b-f00d-4290-98f9-64bbb932a45d\" (UID: \"9004440b-f00d-4290-98f9-64bbb932a45d\") " Nov 03 14:55:09.531258 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:09.531160 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9004440b-f00d-4290-98f9-64bbb932a45d-kube-api-access-hzxzx\" (UniqueName: \"kubernetes.io/projected/9004440b-f00d-4290-98f9-64bbb932a45d-kube-api-access-hzxzx\") pod \"9004440b-f00d-4290-98f9-64bbb932a45d\" (UID: \"9004440b-f00d-4290-98f9-64bbb932a45d\") " Nov 03 14:55:09.531453 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:09.531429 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9004440b-f00d-4290-98f9-64bbb932a45d-maas-api-openshift-ca" pod "9004440b-f00d-4290-98f9-64bbb932a45d" (UID: "9004440b-f00d-4290-98f9-64bbb932a45d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:55:09.533297 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:09.533266 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9004440b-f00d-4290-98f9-64bbb932a45d-maas-api-xks-ca" pod "9004440b-f00d-4290-98f9-64bbb932a45d" (UID: "9004440b-f00d-4290-98f9-64bbb932a45d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:55:09.533411 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:09.533273 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9004440b-f00d-4290-98f9-64bbb932a45d-kube-api-access-hzxzx" pod "9004440b-f00d-4290-98f9-64bbb932a45d" (UID: "9004440b-f00d-4290-98f9-64bbb932a45d"). InnerVolumeSpecName "kube-api-access-hzxzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:55:09.632603 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:09.632524 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzxzx\" (UniqueName: \"kubernetes.io/projected/9004440b-f00d-4290-98f9-64bbb932a45d-kube-api-access-hzxzx\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:55:09.632603 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:09.632556 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9004440b-f00d-4290-98f9-64bbb932a45d-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:55:09.632603 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:09.632569 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9004440b-f00d-4290-98f9-64bbb932a45d-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:55:10.361655 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:10.361610 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-n69dm" event={"ID":"9004440b-f00d-4290-98f9-64bbb932a45d","Type":"ContainerDied","Data":"458e1d1988c17bcdeb795d92f24eb5437b80528c22576a979d07e1adec7ab29e"} Nov 03 14:55:10.361655 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:10.361644 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-n69dm" Nov 03 14:55:10.361867 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:10.361671 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="458e1d1988c17bcdeb795d92f24eb5437b80528c22576a979d07e1adec7ab29e" Nov 03 14:55:19.161778 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:19.161738 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:55:21.364792 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:21.364744 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-dkw27 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:55:21.365464 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:21.365426 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dkw27" podUID="f313510b-02d8-4c68-accd-920343219917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:55:23.355427 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:23.355393 5979 scope.go:122] "RemoveContainer" containerID="6aaa38469308c8464360ad20356102e844673f8d254feb2889c031d448835ee3" Nov 03 14:55:23.364149 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:23.364128 5979 scope.go:122] "RemoveContainer" containerID="4dfdff6521e41e11d658f0c4c2bd7f80cbe93c19848b9e68f7754f4a555ac5c0" Nov 03 14:55:27.929532 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:27.929496 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9rlnn"] Nov 03 14:55:27.930040 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:27.929921 5979 cpu_manager.go:414] "containerMap: removing container" podUID="9004440b-f00d-4290-98f9-64bbb932a45d" containerName="revoke-keys" Nov 03 14:55:27.930040 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:27.929933 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9004440b-f00d-4290-98f9-64bbb932a45d" containerName="revoke-keys" Nov 03 14:55:27.930040 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:27.930012 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="9004440b-f00d-4290-98f9-64bbb932a45d" containerName="revoke-keys" Nov 03 14:55:27.934701 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:27.934677 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9rlnn" Nov 03 14:55:27.983878 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:27.983843 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:55:28.017232 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:28.017198 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/670d6bc1-efef-4801-9259-00d065eed9ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-9rlnn\" (UID: \"670d6bc1-efef-4801-9259-00d065eed9ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9rlnn" Nov 03 14:55:28.017408 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:28.017277 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/670d6bc1-efef-4801-9259-00d065eed9ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-9rlnn\" (UID: \"670d6bc1-efef-4801-9259-00d065eed9ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9rlnn" Nov 03 14:55:28.017461 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:28.017408 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk8t9\" (UniqueName: \"kubernetes.io/projected/670d6bc1-efef-4801-9259-00d065eed9ab-kube-api-access-mk8t9\") pod \"maas-api-revoke-sub-d2bc93bc-9rlnn\" (UID: \"670d6bc1-efef-4801-9259-00d065eed9ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9rlnn" Nov 03 14:55:28.030360 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:28.030335 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 14:55:28.030512 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:28.030391 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:55:28.030642 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:28.030626 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wgk87\"" Nov 03 14:55:28.032386 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:28.032366 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9rlnn"] Nov 03 14:55:28.118184 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:28.118147 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/670d6bc1-efef-4801-9259-00d065eed9ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-9rlnn\" (UID: \"670d6bc1-efef-4801-9259-00d065eed9ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9rlnn" Nov 03 14:55:28.118377 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:28.118306 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk8t9\" (UniqueName: \"kubernetes.io/projected/670d6bc1-efef-4801-9259-00d065eed9ab-kube-api-access-mk8t9\") pod \"maas-api-revoke-sub-d2bc93bc-9rlnn\" (UID: \"670d6bc1-efef-4801-9259-00d065eed9ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9rlnn" Nov 03 14:55:28.118448 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:28.118384 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/670d6bc1-efef-4801-9259-00d065eed9ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-9rlnn\" (UID: \"670d6bc1-efef-4801-9259-00d065eed9ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9rlnn" Nov 03 14:55:28.118976 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:28.118956 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/670d6bc1-efef-4801-9259-00d065eed9ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-9rlnn\" (UID: \"670d6bc1-efef-4801-9259-00d065eed9ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9rlnn" Nov 03 14:55:28.120582 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:28.120562 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/670d6bc1-efef-4801-9259-00d065eed9ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-9rlnn\" (UID: \"670d6bc1-efef-4801-9259-00d065eed9ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9rlnn" Nov 03 14:55:28.172572 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:28.172539 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk8t9\" (UniqueName: \"kubernetes.io/projected/670d6bc1-efef-4801-9259-00d065eed9ab-kube-api-access-mk8t9\") pod \"maas-api-revoke-sub-d2bc93bc-9rlnn\" (UID: \"670d6bc1-efef-4801-9259-00d065eed9ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9rlnn" Nov 03 14:55:28.247654 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:28.247563 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9rlnn" Nov 03 14:55:28.561451 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:28.561360 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:55:28.616224 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:28.616184 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9rlnn"] Nov 03 14:55:28.624253 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:55:28.624221 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod670d6bc1_efef_4801_9259_00d065eed9ab.slice/crio-ee148b5099ecedbcf6e662cb91d6698f909cde26561c42686cb6d742fb581da9 WatchSource:0}: Error finding container ee148b5099ecedbcf6e662cb91d6698f909cde26561c42686cb6d742fb581da9: Status 404 returned error can't find the container with id ee148b5099ecedbcf6e662cb91d6698f909cde26561c42686cb6d742fb581da9 Nov 03 14:55:29.499884 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:29.499842 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9rlnn" event={"ID":"670d6bc1-efef-4801-9259-00d065eed9ab","Type":"ContainerStarted","Data":"5e6fe7023840a8c11cc2e9d0fd53bbad5acbeba947a02315762c3d9b0cd392e4"} Nov 03 14:55:29.500293 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:29.499911 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9rlnn" event={"ID":"670d6bc1-efef-4801-9259-00d065eed9ab","Type":"ContainerStarted","Data":"ee148b5099ecedbcf6e662cb91d6698f909cde26561c42686cb6d742fb581da9"} Nov 03 14:55:29.608374 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:29.608322 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9rlnn" podStartSLOduration=2.6083087799999998 podStartE2EDuration="2.60830878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:55:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:55:29.6076482 +0000 UTC m=+6611.618336443" watchObservedRunningTime="2026-11-03 14:55:29.60830878 +0000 UTC m=+6611.618997021" Nov 03 14:55:30.506561 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:30.506521 5979 generic.go:356] "Generic (PLEG): container finished" podID="670d6bc1-efef-4801-9259-00d065eed9ab" containerID="5e6fe7023840a8c11cc2e9d0fd53bbad5acbeba947a02315762c3d9b0cd392e4" exitCode=0 Nov 03 14:55:30.506990 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:30.506609 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9rlnn" event={"ID":"670d6bc1-efef-4801-9259-00d065eed9ab","Type":"ContainerDied","Data":"5e6fe7023840a8c11cc2e9d0fd53bbad5acbeba947a02315762c3d9b0cd392e4"} Nov 03 14:55:31.649060 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:31.649028 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9rlnn" Nov 03 14:55:31.756040 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:31.756006 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/670d6bc1-efef-4801-9259-00d065eed9ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/670d6bc1-efef-4801-9259-00d065eed9ab-maas-api-openshift-ca\") pod \"670d6bc1-efef-4801-9259-00d065eed9ab\" (UID: \"670d6bc1-efef-4801-9259-00d065eed9ab\") " Nov 03 14:55:31.756263 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:31.756144 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/670d6bc1-efef-4801-9259-00d065eed9ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/670d6bc1-efef-4801-9259-00d065eed9ab-maas-api-xks-ca\") pod \"670d6bc1-efef-4801-9259-00d065eed9ab\" (UID: \"670d6bc1-efef-4801-9259-00d065eed9ab\") " Nov 03 14:55:31.756263 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:31.756169 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/670d6bc1-efef-4801-9259-00d065eed9ab-kube-api-access-mk8t9\" (UniqueName: \"kubernetes.io/projected/670d6bc1-efef-4801-9259-00d065eed9ab-kube-api-access-mk8t9\") pod \"670d6bc1-efef-4801-9259-00d065eed9ab\" (UID: \"670d6bc1-efef-4801-9259-00d065eed9ab\") " Nov 03 14:55:31.756717 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:31.756582 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/670d6bc1-efef-4801-9259-00d065eed9ab-maas-api-openshift-ca" pod "670d6bc1-efef-4801-9259-00d065eed9ab" (UID: "670d6bc1-efef-4801-9259-00d065eed9ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:55:31.758498 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:31.758460 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/670d6bc1-efef-4801-9259-00d065eed9ab-maas-api-xks-ca" pod "670d6bc1-efef-4801-9259-00d065eed9ab" (UID: "670d6bc1-efef-4801-9259-00d065eed9ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:55:31.758613 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:31.758589 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/670d6bc1-efef-4801-9259-00d065eed9ab-kube-api-access-mk8t9" pod "670d6bc1-efef-4801-9259-00d065eed9ab" (UID: "670d6bc1-efef-4801-9259-00d065eed9ab"). InnerVolumeSpecName "kube-api-access-mk8t9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:55:31.857253 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:31.857213 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/670d6bc1-efef-4801-9259-00d065eed9ab-maas-api-openshift-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:55:31.857253 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:31.857247 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/670d6bc1-efef-4801-9259-00d065eed9ab-maas-api-xks-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:55:31.857253 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:31.857259 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk8t9\" (UniqueName: \"kubernetes.io/projected/670d6bc1-efef-4801-9259-00d065eed9ab-kube-api-access-mk8t9\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:55:32.516996 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:32.516963 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9rlnn" Nov 03 14:55:32.517182 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:32.516989 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9rlnn" event={"ID":"670d6bc1-efef-4801-9259-00d065eed9ab","Type":"ContainerDied","Data":"ee148b5099ecedbcf6e662cb91d6698f909cde26561c42686cb6d742fb581da9"} Nov 03 14:55:32.517182 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:32.517041 5979 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee148b5099ecedbcf6e662cb91d6698f909cde26561c42686cb6d742fb581da9" Nov 03 14:55:36.646055 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:36.646019 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:55:37.234776 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:55:37.234740 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:55:41.152011 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:41.151969 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5b8b4cb57b-wswhj"] Nov 03 14:55:41.152753 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:41.152604 5979 cpu_manager.go:414] "containerMap: removing container" podUID="670d6bc1-efef-4801-9259-00d065eed9ab" containerName="revoke-keys" Nov 03 14:55:41.152753 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:41.152625 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="670d6bc1-efef-4801-9259-00d065eed9ab" containerName="revoke-keys" Nov 03 14:55:41.152753 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:41.152727 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="670d6bc1-efef-4801-9259-00d065eed9ab" containerName="revoke-keys" Nov 03 14:55:41.156438 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:41.156417 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5b8b4cb57b-wswhj" Nov 03 14:55:41.252664 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:41.252627 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 14:55:41.252906 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:41.252674 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 14:55:41.253016 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:41.252748 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38d3fe8f-c7ae-4142-8e9f-7f30c86157f3-openshift-service-ca\") pod \"maas-api-5b8b4cb57b-wswhj\" (UID: \"38d3fe8f-c7ae-4142-8e9f-7f30c86157f3\") " pod="odh-ai-gateway-infra/maas-api-5b8b4cb57b-wswhj" Nov 03 14:55:41.253154 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:41.253138 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38d3fe8f-c7ae-4142-8e9f-7f30c86157f3-metrics-certs\") pod \"maas-api-5b8b4cb57b-wswhj\" (UID: \"38d3fe8f-c7ae-4142-8e9f-7f30c86157f3\") " pod="odh-ai-gateway-infra/maas-api-5b8b4cb57b-wswhj" Nov 03 14:55:41.253316 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:41.253304 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38d3fe8f-c7ae-4142-8e9f-7f30c86157f3-maas-api-tls\") pod \"maas-api-5b8b4cb57b-wswhj\" (UID: \"38d3fe8f-c7ae-4142-8e9f-7f30c86157f3\") " pod="odh-ai-gateway-infra/maas-api-5b8b4cb57b-wswhj" Nov 03 14:55:41.253524 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:41.253501 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2hv6\" (UniqueName: \"kubernetes.io/projected/38d3fe8f-c7ae-4142-8e9f-7f30c86157f3-kube-api-access-f2hv6\") pod \"maas-api-5b8b4cb57b-wswhj\" (UID: \"38d3fe8f-c7ae-4142-8e9f-7f30c86157f3\") " pod="odh-ai-gateway-infra/maas-api-5b8b4cb57b-wswhj" Nov 03 14:55:41.253779 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:41.252761 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-dgx9m\"" Nov 03 14:55:41.253913 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:41.252849 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 14:55:41.253913 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:41.252989 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 14:55:41.254898 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:41.254877 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5b8b4cb57b-wswhj"] Nov 03 14:55:41.354952 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:41.354915 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38d3fe8f-c7ae-4142-8e9f-7f30c86157f3-maas-api-tls\") pod \"maas-api-5b8b4cb57b-wswhj\" (UID: \"38d3fe8f-c7ae-4142-8e9f-7f30c86157f3\") " pod="odh-ai-gateway-infra/maas-api-5b8b4cb57b-wswhj" Nov 03 14:55:41.355138 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:41.354986 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2hv6\" (UniqueName: \"kubernetes.io/projected/38d3fe8f-c7ae-4142-8e9f-7f30c86157f3-kube-api-access-f2hv6\") pod \"maas-api-5b8b4cb57b-wswhj\" (UID: \"38d3fe8f-c7ae-4142-8e9f-7f30c86157f3\") " pod="odh-ai-gateway-infra/maas-api-5b8b4cb57b-wswhj" Nov 03 14:55:41.355138 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:41.355072 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38d3fe8f-c7ae-4142-8e9f-7f30c86157f3-openshift-service-ca\") pod \"maas-api-5b8b4cb57b-wswhj\" (UID: \"38d3fe8f-c7ae-4142-8e9f-7f30c86157f3\") " pod="odh-ai-gateway-infra/maas-api-5b8b4cb57b-wswhj" Nov 03 14:55:41.355138 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:41.355101 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38d3fe8f-c7ae-4142-8e9f-7f30c86157f3-metrics-certs\") pod \"maas-api-5b8b4cb57b-wswhj\" (UID: \"38d3fe8f-c7ae-4142-8e9f-7f30c86157f3\") " pod="odh-ai-gateway-infra/maas-api-5b8b4cb57b-wswhj" Nov 03 14:55:41.355721 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:41.355679 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38d3fe8f-c7ae-4142-8e9f-7f30c86157f3-openshift-service-ca\") pod \"maas-api-5b8b4cb57b-wswhj\" (UID: \"38d3fe8f-c7ae-4142-8e9f-7f30c86157f3\") " pod="odh-ai-gateway-infra/maas-api-5b8b4cb57b-wswhj" Nov 03 14:55:41.357454 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:41.357426 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38d3fe8f-c7ae-4142-8e9f-7f30c86157f3-maas-api-tls\") pod \"maas-api-5b8b4cb57b-wswhj\" (UID: \"38d3fe8f-c7ae-4142-8e9f-7f30c86157f3\") " pod="odh-ai-gateway-infra/maas-api-5b8b4cb57b-wswhj" Nov 03 14:55:41.357454 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:41.357440 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38d3fe8f-c7ae-4142-8e9f-7f30c86157f3-metrics-certs\") pod \"maas-api-5b8b4cb57b-wswhj\" (UID: \"38d3fe8f-c7ae-4142-8e9f-7f30c86157f3\") " pod="odh-ai-gateway-infra/maas-api-5b8b4cb57b-wswhj" Nov 03 14:55:41.409135 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:41.409053 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2hv6\" (UniqueName: \"kubernetes.io/projected/38d3fe8f-c7ae-4142-8e9f-7f30c86157f3-kube-api-access-f2hv6\") pod \"maas-api-5b8b4cb57b-wswhj\" (UID: \"38d3fe8f-c7ae-4142-8e9f-7f30c86157f3\") " pod="odh-ai-gateway-infra/maas-api-5b8b4cb57b-wswhj" Nov 03 14:55:41.471148 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:41.471108 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5b8b4cb57b-wswhj" Nov 03 14:55:41.841921 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:41.841828 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5b8b4cb57b-wswhj"] Nov 03 14:55:41.850678 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:55:41.850648 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38d3fe8f_c7ae_4142_8e9f_7f30c86157f3.slice/crio-ce70bfa3556217260546fec3fbba246923af71204658e1c61bdaac31489c0c8f WatchSource:0}: Error finding container ce70bfa3556217260546fec3fbba246923af71204658e1c61bdaac31489c0c8f: Status 404 returned error can't find the container with id ce70bfa3556217260546fec3fbba246923af71204658e1c61bdaac31489c0c8f Nov 03 14:55:42.568022 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:42.567982 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5b8b4cb57b-wswhj" event={"ID":"38d3fe8f-c7ae-4142-8e9f-7f30c86157f3","Type":"ContainerStarted","Data":"ce70bfa3556217260546fec3fbba246923af71204658e1c61bdaac31489c0c8f"} Nov 03 14:55:43.575261 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:43.575220 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5b8b4cb57b-wswhj" event={"ID":"38d3fe8f-c7ae-4142-8e9f-7f30c86157f3","Type":"ContainerStarted","Data":"6b629705c9a3bf104b693a7412190ff1ac89b4c99bee88cd8a85a94c3b1f6821"} Nov 03 14:55:43.575710 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:43.575289 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5b8b4cb57b-wswhj" Nov 03 14:55:43.684695 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:43.684640 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5b8b4cb57b-wswhj" podStartSLOduration=1.651406828 podStartE2EDuration="2.684626254s" totalImagesPullingTime="1.033219426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:55:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:55:43.683248622 +0000 UTC m=+6625.693936866" watchObservedRunningTime="2026-11-03 14:55:43.684626254 +0000 UTC m=+6625.695314495" Nov 03 14:55:49.586233 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:49.586193 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5b8b4cb57b-wswhj" Nov 03 14:55:49.976254 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:49.976212 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-578f968db4-2n9zm"] Nov 03 14:55:49.980840 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:49.980806 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-578f968db4-2n9zm" Nov 03 14:55:50.037351 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:50.037313 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r97vm\" (UniqueName: \"kubernetes.io/projected/1b7019e1-6304-4aeb-a81e-0a6a72321369-kube-api-access-r97vm\") pod \"maas-api-578f968db4-2n9zm\" (UID: \"1b7019e1-6304-4aeb-a81e-0a6a72321369\") " pod="odh-ai-gateway-infra/maas-api-578f968db4-2n9zm" Nov 03 14:55:50.037572 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:50.037446 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b7019e1-6304-4aeb-a81e-0a6a72321369-metrics-certs\") pod \"maas-api-578f968db4-2n9zm\" (UID: \"1b7019e1-6304-4aeb-a81e-0a6a72321369\") " pod="odh-ai-gateway-infra/maas-api-578f968db4-2n9zm" Nov 03 14:55:50.037572 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:50.037533 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b7019e1-6304-4aeb-a81e-0a6a72321369-openshift-service-ca\") pod \"maas-api-578f968db4-2n9zm\" (UID: \"1b7019e1-6304-4aeb-a81e-0a6a72321369\") " pod="odh-ai-gateway-infra/maas-api-578f968db4-2n9zm" Nov 03 14:55:50.037665 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:50.037596 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b7019e1-6304-4aeb-a81e-0a6a72321369-maas-api-tls\") pod \"maas-api-578f968db4-2n9zm\" (UID: \"1b7019e1-6304-4aeb-a81e-0a6a72321369\") " pod="odh-ai-gateway-infra/maas-api-578f968db4-2n9zm" Nov 03 14:55:50.038913 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:50.038893 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-578f968db4-2n9zm"] Nov 03 14:55:50.138921 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:50.138881 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b7019e1-6304-4aeb-a81e-0a6a72321369-metrics-certs\") pod \"maas-api-578f968db4-2n9zm\" (UID: \"1b7019e1-6304-4aeb-a81e-0a6a72321369\") " pod="odh-ai-gateway-infra/maas-api-578f968db4-2n9zm" Nov 03 14:55:50.138921 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:50.138923 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b7019e1-6304-4aeb-a81e-0a6a72321369-openshift-service-ca\") pod \"maas-api-578f968db4-2n9zm\" (UID: \"1b7019e1-6304-4aeb-a81e-0a6a72321369\") " pod="odh-ai-gateway-infra/maas-api-578f968db4-2n9zm" Nov 03 14:55:50.139213 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:50.138974 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b7019e1-6304-4aeb-a81e-0a6a72321369-maas-api-tls\") pod \"maas-api-578f968db4-2n9zm\" (UID: \"1b7019e1-6304-4aeb-a81e-0a6a72321369\") " pod="odh-ai-gateway-infra/maas-api-578f968db4-2n9zm" Nov 03 14:55:50.139213 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:50.139006 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r97vm\" (UniqueName: \"kubernetes.io/projected/1b7019e1-6304-4aeb-a81e-0a6a72321369-kube-api-access-r97vm\") pod \"maas-api-578f968db4-2n9zm\" (UID: \"1b7019e1-6304-4aeb-a81e-0a6a72321369\") " pod="odh-ai-gateway-infra/maas-api-578f968db4-2n9zm" Nov 03 14:55:50.139655 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:50.139628 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b7019e1-6304-4aeb-a81e-0a6a72321369-openshift-service-ca\") pod \"maas-api-578f968db4-2n9zm\" (UID: \"1b7019e1-6304-4aeb-a81e-0a6a72321369\") " pod="odh-ai-gateway-infra/maas-api-578f968db4-2n9zm" Nov 03 14:55:50.141461 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:50.141438 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b7019e1-6304-4aeb-a81e-0a6a72321369-maas-api-tls\") pod \"maas-api-578f968db4-2n9zm\" (UID: \"1b7019e1-6304-4aeb-a81e-0a6a72321369\") " pod="odh-ai-gateway-infra/maas-api-578f968db4-2n9zm" Nov 03 14:55:50.141582 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:50.141503 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b7019e1-6304-4aeb-a81e-0a6a72321369-metrics-certs\") pod \"maas-api-578f968db4-2n9zm\" (UID: \"1b7019e1-6304-4aeb-a81e-0a6a72321369\") " pod="odh-ai-gateway-infra/maas-api-578f968db4-2n9zm" Nov 03 14:55:50.192931 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:50.192897 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r97vm\" (UniqueName: \"kubernetes.io/projected/1b7019e1-6304-4aeb-a81e-0a6a72321369-kube-api-access-r97vm\") pod \"maas-api-578f968db4-2n9zm\" (UID: \"1b7019e1-6304-4aeb-a81e-0a6a72321369\") " pod="odh-ai-gateway-infra/maas-api-578f968db4-2n9zm" Nov 03 14:55:50.293329 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:50.293242 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-578f968db4-2n9zm" Nov 03 14:55:50.663678 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:50.663645 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-578f968db4-2n9zm"] Nov 03 14:55:50.671610 ip-10-0-130-198 kubenswrapper[5979]: W1103 14:55:50.671580 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b7019e1_6304_4aeb_a81e_0a6a72321369.slice/crio-fc55f229a68ae64566d55e35b8b891689a32b0473cdaec55f944a81a6355b848 WatchSource:0}: Error finding container fc55f229a68ae64566d55e35b8b891689a32b0473cdaec55f944a81a6355b848: Status 404 returned error can't find the container with id fc55f229a68ae64566d55e35b8b891689a32b0473cdaec55f944a81a6355b848 Nov 03 14:55:51.616176 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:51.616134 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-578f968db4-2n9zm" event={"ID":"1b7019e1-6304-4aeb-a81e-0a6a72321369","Type":"ContainerStarted","Data":"d4fa2084e339bf4f352d97d58885f029647c9a0b6c4da3252e91cc6a4be95faa"} Nov 03 14:55:51.616358 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:51.616236 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-578f968db4-2n9zm" Nov 03 14:55:51.616358 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:51.616260 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-578f968db4-2n9zm" event={"ID":"1b7019e1-6304-4aeb-a81e-0a6a72321369","Type":"ContainerStarted","Data":"fc55f229a68ae64566d55e35b8b891689a32b0473cdaec55f944a81a6355b848"} Nov 03 14:55:51.765861 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:51.765761 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-578f968db4-2n9zm" podStartSLOduration=1.941155523 podStartE2EDuration="2.765746673s" totalImagesPullingTime="824.59115ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 14:55:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 14:55:51.764194555 +0000 UTC m=+6633.774882798" watchObservedRunningTime="2026-11-03 14:55:51.765746673 +0000 UTC m=+6633.776434915" Nov 03 14:55:53.563914 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:53.563882 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:55:56.187647 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:56.187600 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-8gjrw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:55:56.188346 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:56.188289 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8gjrw" podUID="e7e12233-9e76-4de5-b4cf-93644996783e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:55:57.626631 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:57.626600 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-578f968db4-2n9zm" Nov 03 14:55:57.785623 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:57.785588 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5b8b4cb57b-wswhj"] Nov 03 14:55:57.785825 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:57.785795 5979 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5b8b4cb57b-wswhj" podUID="38d3fe8f-c7ae-4142-8e9f-7f30c86157f3" containerName="maas-api" containerID="cri-o://6b629705c9a3bf104b693a7412190ff1ac89b4c99bee88cd8a85a94c3b1f6821" gracePeriod=30 Nov 03 14:55:58.048674 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:58.048648 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5b8b4cb57b-wswhj" Nov 03 14:55:58.068160 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:58.068125 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38d3fe8f-c7ae-4142-8e9f-7f30c86157f3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38d3fe8f-c7ae-4142-8e9f-7f30c86157f3-maas-api-tls\") pod \"38d3fe8f-c7ae-4142-8e9f-7f30c86157f3\" (UID: \"38d3fe8f-c7ae-4142-8e9f-7f30c86157f3\") " Nov 03 14:55:58.068337 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:58.068277 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38d3fe8f-c7ae-4142-8e9f-7f30c86157f3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38d3fe8f-c7ae-4142-8e9f-7f30c86157f3-openshift-service-ca\") pod \"38d3fe8f-c7ae-4142-8e9f-7f30c86157f3\" (UID: \"38d3fe8f-c7ae-4142-8e9f-7f30c86157f3\") " Nov 03 14:55:58.068337 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:58.068311 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38d3fe8f-c7ae-4142-8e9f-7f30c86157f3-kube-api-access-f2hv6\" (UniqueName: \"kubernetes.io/projected/38d3fe8f-c7ae-4142-8e9f-7f30c86157f3-kube-api-access-f2hv6\") pod \"38d3fe8f-c7ae-4142-8e9f-7f30c86157f3\" (UID: \"38d3fe8f-c7ae-4142-8e9f-7f30c86157f3\") " Nov 03 14:55:58.068421 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:58.068366 5979 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38d3fe8f-c7ae-4142-8e9f-7f30c86157f3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/38d3fe8f-c7ae-4142-8e9f-7f30c86157f3-metrics-certs\") pod \"38d3fe8f-c7ae-4142-8e9f-7f30c86157f3\" (UID: \"38d3fe8f-c7ae-4142-8e9f-7f30c86157f3\") " Nov 03 14:55:58.068673 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:58.068645 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38d3fe8f-c7ae-4142-8e9f-7f30c86157f3-openshift-service-ca" pod "38d3fe8f-c7ae-4142-8e9f-7f30c86157f3" (UID: "38d3fe8f-c7ae-4142-8e9f-7f30c86157f3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 14:55:58.068804 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:58.068785 5979 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38d3fe8f-c7ae-4142-8e9f-7f30c86157f3-openshift-service-ca\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:55:58.070348 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:58.070319 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38d3fe8f-c7ae-4142-8e9f-7f30c86157f3-maas-api-tls" pod "38d3fe8f-c7ae-4142-8e9f-7f30c86157f3" (UID: "38d3fe8f-c7ae-4142-8e9f-7f30c86157f3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:55:58.070502 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:58.070424 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38d3fe8f-c7ae-4142-8e9f-7f30c86157f3-kube-api-access-f2hv6" pod "38d3fe8f-c7ae-4142-8e9f-7f30c86157f3" (UID: "38d3fe8f-c7ae-4142-8e9f-7f30c86157f3"). InnerVolumeSpecName "kube-api-access-f2hv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 14:55:58.070559 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:58.070513 5979 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38d3fe8f-c7ae-4142-8e9f-7f30c86157f3-metrics-certs" pod "38d3fe8f-c7ae-4142-8e9f-7f30c86157f3" (UID: "38d3fe8f-c7ae-4142-8e9f-7f30c86157f3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 14:55:58.169968 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:58.169892 5979 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2hv6\" (UniqueName: \"kubernetes.io/projected/38d3fe8f-c7ae-4142-8e9f-7f30c86157f3-kube-api-access-f2hv6\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:55:58.169968 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:58.169925 5979 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38d3fe8f-c7ae-4142-8e9f-7f30c86157f3-metrics-certs\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:55:58.169968 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:58.169936 5979 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38d3fe8f-c7ae-4142-8e9f-7f30c86157f3-maas-api-tls\") on node \"ip-10-0-130-198.us-west-2.compute.internal\" DevicePath \"\"" Nov 03 14:55:58.670659 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:58.670623 5979 generic.go:356] "Generic (PLEG): container finished" podID="38d3fe8f-c7ae-4142-8e9f-7f30c86157f3" containerID="6b629705c9a3bf104b693a7412190ff1ac89b4c99bee88cd8a85a94c3b1f6821" exitCode=0 Nov 03 14:55:58.671104 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:58.670699 5979 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5b8b4cb57b-wswhj" Nov 03 14:55:58.671104 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:58.670703 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5b8b4cb57b-wswhj" event={"ID":"38d3fe8f-c7ae-4142-8e9f-7f30c86157f3","Type":"ContainerDied","Data":"6b629705c9a3bf104b693a7412190ff1ac89b4c99bee88cd8a85a94c3b1f6821"} Nov 03 14:55:58.671104 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:58.670787 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5b8b4cb57b-wswhj" event={"ID":"38d3fe8f-c7ae-4142-8e9f-7f30c86157f3","Type":"ContainerDied","Data":"ce70bfa3556217260546fec3fbba246923af71204658e1c61bdaac31489c0c8f"} Nov 03 14:55:58.671104 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:58.670809 5979 scope.go:122] "RemoveContainer" containerID="6b629705c9a3bf104b693a7412190ff1ac89b4c99bee88cd8a85a94c3b1f6821" Nov 03 14:55:58.680012 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:58.679994 5979 scope.go:122] "RemoveContainer" containerID="6b629705c9a3bf104b693a7412190ff1ac89b4c99bee88cd8a85a94c3b1f6821" Nov 03 14:55:58.680273 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:55:58.680250 5979 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b629705c9a3bf104b693a7412190ff1ac89b4c99bee88cd8a85a94c3b1f6821\": container with ID starting with 6b629705c9a3bf104b693a7412190ff1ac89b4c99bee88cd8a85a94c3b1f6821 not found: ID does not exist" containerID="6b629705c9a3bf104b693a7412190ff1ac89b4c99bee88cd8a85a94c3b1f6821" Nov 03 14:55:58.680361 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:58.680282 5979 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b629705c9a3bf104b693a7412190ff1ac89b4c99bee88cd8a85a94c3b1f6821"} err="failed to get container status \"6b629705c9a3bf104b693a7412190ff1ac89b4c99bee88cd8a85a94c3b1f6821\": rpc error: code = NotFound desc = could not find container \"6b629705c9a3bf104b693a7412190ff1ac89b4c99bee88cd8a85a94c3b1f6821\": container with ID starting with 6b629705c9a3bf104b693a7412190ff1ac89b4c99bee88cd8a85a94c3b1f6821 not found: ID does not exist" Nov 03 14:55:58.831412 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:58.831375 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5b8b4cb57b-wswhj"] Nov 03 14:55:58.831777 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:55:58.831434 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5b8b4cb57b-wswhj"] Nov 03 14:56:00.115617 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:56:00.115583 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38d3fe8f-c7ae-4142-8e9f-7f30c86157f3" path="/var/lib/kubelet/pods/38d3fe8f-c7ae-4142-8e9f-7f30c86157f3/volumes" Nov 03 14:56:07.233837 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:56:07.233797 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:56:23.388833 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:56:23.388803 5979 scope.go:122] "RemoveContainer" containerID="2ce953aae16b30d3bab5b8b502bdd230ceb423da45e55e57193288356cbbbefc" Nov 03 14:56:23.397531 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:56:23.397510 5979 scope.go:122] "RemoveContainer" containerID="b62d5bdf0a6c2f9d3e7b92976611bf70315c59da9ee826e88405bc5e6b2f6b04" Nov 03 14:56:23.405175 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:56:23.405154 5979 scope.go:122] "RemoveContainer" containerID="732792066df4f26757b5c1be3fe49519ae5f6784f2363fbcb912e1b8c3b6079a" Nov 03 14:56:31.754718 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:56:31.754682 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:56:37.234110 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:56:37.234073 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:56:43.660024 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:56:43.659985 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:56:52.044986 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:56:52.044952 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea3723ce-ddnx4"] Nov 03 14:56:52.091108 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:56:52.091077 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea3723ce-ddnx4"] Nov 03 14:56:52.115814 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:56:52.115780 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e90561f-e17a-4d79-b575-ef97ff648f7f" path="/var/lib/kubelet/pods/7e90561f-e17a-4d79-b575-ef97ff648f7f/volumes" Nov 03 14:57:05.161047 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:57:05.161000 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:57:07.233889 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:57:07.233852 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:57:14.008638 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:57:14.008589 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-lj2cw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:57:14.009793 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:57:14.009259 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-lj2cw" podUID="7f4e540f-d1f9-4ffe-83ea-dfca92ce7233" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:57:14.454666 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:57:14.454619 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:57:14.454869 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:57:14.454701 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q" podUID="fe445e0e-53c9-4543-9107-ff8f13cbd425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:57:18.976854 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:57:18.976822 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:57:21.342630 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:57:21.342579 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-dkw27 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:57:21.343308 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:57:21.343271 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dkw27" podUID="f313510b-02d8-4c68-accd-920343219917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:57:37.233939 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:57:37.233898 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:57:49.970572 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:57:49.970528 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:58:00.706738 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:58:00.706647 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:58:07.234690 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:58:07.234653 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:58:12.972729 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:58:12.972694 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:58:21.043820 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:58:21.043783 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6ws92"] Nov 03 14:58:21.090391 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:58:21.090353 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-6ws92"] Nov 03 14:58:22.114316 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:58:22.114280 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05787e77-c394-4cdf-8a2a-93a9f2ded1c2" path="/var/lib/kubelet/pods/05787e77-c394-4cdf-8a2a-93a9f2ded1c2/volumes" Nov 03 14:58:23.442452 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:58:23.442415 5979 scope.go:122] "RemoveContainer" containerID="b1925b25c9811696d21cea1ebe24ee3a28d773362ed0034f7f2c79ab6a8671ae" Nov 03 14:58:25.247351 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:58:25.247318 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:58:37.233598 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:58:37.233559 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:58:48.666846 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:58:48.666808 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:58:51.365497 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:58:51.364588 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-dkw27 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:58:51.365497 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:58:51.364677 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dkw27" podUID="f313510b-02d8-4c68-accd-920343219917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:58:59.339929 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:58:59.339888 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:59:06.342716 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:59:06.342639 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-dkw27 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 14:59:06.343266 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:59:06.342764 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dkw27" podUID="f313510b-02d8-4c68-accd-920343219917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 14:59:07.235243 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:59:07.235204 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:59:08.759573 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:59:08.759534 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:59:23.465890 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:59:23.465850 5979 scope.go:122] "RemoveContainer" containerID="2e54b7a302131b14bbd47c6888d163747b15eccd0eaa5c13c07e7bcc8aa0cf91" Nov 03 14:59:31.573668 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:59:31.573633 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 14:59:37.234404 ip-10-0-130-198 kubenswrapper[5979]: E1103 14:59:37.234366 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 14:59:52.638182 ip-10-0-130-198 kubenswrapper[5979]: I1103 14:59:52.638144 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 15:00:07.233840 ip-10-0-130-198 kubenswrapper[5979]: E1103 15:00:07.233799 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 15:00:10.045359 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:00:10.045321 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-n69dm"] Nov 03 15:00:10.091719 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:00:10.091682 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-n69dm"] Nov 03 15:00:10.114360 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:00:10.114325 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9004440b-f00d-4290-98f9-64bbb932a45d" path="/var/lib/kubelet/pods/9004440b-f00d-4290-98f9-64bbb932a45d/volumes" Nov 03 15:00:32.044386 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:00:32.044351 5979 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9rlnn"] Nov 03 15:00:32.091713 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:00:32.091667 5979 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-9rlnn"] Nov 03 15:00:32.116927 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:00:32.116890 5979 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="670d6bc1-efef-4801-9259-00d065eed9ab" path="/var/lib/kubelet/pods/670d6bc1-efef-4801-9259-00d065eed9ab/volumes" Nov 03 15:00:37.234295 ip-10-0-130-198 kubenswrapper[5979]: E1103 15:00:37.234255 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 15:00:59.465676 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:00:59.460818 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:00:59.465676 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:00:59.460897 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-j5s8q" podUID="fe445e0e-53c9-4543-9107-ff8f13cbd425" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:01:07.234327 ip-10-0-130-198 kubenswrapper[5979]: E1103 15:01:07.234286 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 15:01:11.165332 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:01:11.165293 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-8gjrw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:01:11.165983 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:01:11.165950 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-8gjrw" podUID="e7e12233-9e76-4de5-b4cf-93644996783e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:01:21.360636 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:01:21.360592 5979 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b9c49cd7-dkw27 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 15:01:21.361299 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:01:21.361252 5979 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b9c49cd7-dkw27" podUID="f313510b-02d8-4c68-accd-920343219917" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 15:01:23.584089 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:01:23.584056 5979 scope.go:122] "RemoveContainer" containerID="f876f2c3f363780bbc7a069e024fc6c379528e9547effb28feb7e56b9a3c4495" Nov 03 15:01:37.233425 ip-10-0-130-198 kubenswrapper[5979]: E1103 15:01:37.233387 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 15:01:51.058363 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:01:51.058325 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-466qf"] Nov 03 15:02:07.233224 ip-10-0-130-198 kubenswrapper[5979]: E1103 15:02:07.233182 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 15:02:23.520418 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:02:23.520386 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-588d644cc-bhwpl_f6d109e0-3ef7-459b-bf9c-dea5de46b825/manager/0.log" Nov 03 15:02:23.604822 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:02:23.604788 5979 scope.go:122] "RemoveContainer" containerID="5e6fe7023840a8c11cc2e9d0fd53bbad5acbeba947a02315762c3d9b0cd392e4" Nov 03 15:02:25.251016 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:02:25.250983 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-fwvg7_f33ad3b9-499c-41f5-9cf5-029709b82ca6/proxy/0.log" Nov 03 15:02:26.295152 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:02:26.295122 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-578f968db4-2n9zm_1b7019e1-6304-4aeb-a81e-0a6a72321369/maas-api/0.log" Nov 03 15:02:34.050032 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:02:34.049993 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-466qf_f31d565d-15b1-4b03-bbfc-ff43778c7a95/limitador/0.log" Nov 03 15:02:35.557060 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:02:35.557025 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-nvcvb_3a101d3a-a7d5-4f43-a4e8-0a29d9bb5607/discovery/0.log" Nov 03 15:02:35.920869 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:02:35.920841 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-77875f86b8-krvtz_593e2573-815f-4352-8e5f-13ab39dbd37e/kube-auth-proxy/0.log" Nov 03 15:02:36.656321 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:02:36.656291 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b9c49cd7-8gjrw_e7e12233-9e76-4de5-b4cf-93644996783e/istio-proxy/0.log" Nov 03 15:02:36.930274 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:02:36.930181 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b9c49cd7-dkw27_f313510b-02d8-4c68-accd-920343219917/istio-proxy/0.log" Nov 03 15:02:37.233247 ip-10-0-130-198 kubenswrapper[5979]: E1103 15:02:37.233165 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 15:02:37.461180 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:02:37.461134 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b9c49cd7-j5s8q_fe445e0e-53c9-4543-9107-ff8f13cbd425/istio-proxy/0.log" Nov 03 15:02:38.026274 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:02:38.026244 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b9c49cd7-lj2cw_7f4e540f-d1f9-4ffe-83ea-dfca92ce7233/istio-proxy/0.log" Nov 03 15:02:39.270718 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:02:39.270686 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-68d6ccb6b8-8tq9t_706b4485-a4d4-4aa7-a942-9bd37ef907e9/payload-pre-processing/0.log" Nov 03 15:02:41.759748 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:02:41.759717 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8qtv6b_f51c4495-5704-40ba-ad07-0d8b278595d6/main/0.log" Nov 03 15:02:42.262257 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:02:42.262229 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-5p5rm_8407952f-9ae8-40c2-9f22-d6fd2cd68b36/main/0.log" Nov 03 15:03:02.592466 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:02.592430 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-48vd8_2ca5d5b3-4d10-46f6-8f5f-22d0d43e9895/global-pull-secret-syncer/0.log" Nov 03 15:03:06.016795 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:06.016763 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ghgsf_a83707c3-8143-4aad-9131-59986a79968b/konnectivity-agent/0.log" Nov 03 15:03:07.233761 ip-10-0-130-198 kubenswrapper[5979]: E1103 15:03:07.233725 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor" Nov 03 15:03:07.416796 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:07.416766 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-198.us-west-2.compute.internal_069487ba3512639ba31fb8a59a1ce2dd/haproxy/0.log" Nov 03 15:03:09.402245 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:09.402213 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-588d644cc-bhwpl_f6d109e0-3ef7-459b-bf9c-dea5de46b825/manager/0.log" Nov 03 15:03:11.461636 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:11.461607 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-fwvg7_f33ad3b9-499c-41f5-9cf5-029709b82ca6/proxy/0.log" Nov 03 15:03:13.092021 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:13.091984 5979 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wn4x5/perf-node-gather-daemonset-m5pl7"] Nov 03 15:03:13.092664 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:13.092643 5979 cpu_manager.go:414] "containerMap: removing container" podUID="38d3fe8f-c7ae-4142-8e9f-7f30c86157f3" containerName="maas-api" Nov 03 15:03:13.092774 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:13.092665 5979 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38d3fe8f-c7ae-4142-8e9f-7f30c86157f3" containerName="maas-api" Nov 03 15:03:13.092846 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:13.092785 5979 memory_manager.go:365] "RemoveStaleState removing state" podUID="38d3fe8f-c7ae-4142-8e9f-7f30c86157f3" containerName="maas-api" Nov 03 15:03:13.096090 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:13.096069 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-m5pl7" Nov 03 15:03:13.144839 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:13.144783 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wn4x5\"/\"kube-root-ca.crt\"" Nov 03 15:03:13.145074 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:13.145052 5979 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wn4x5\"/\"default-dockercfg-gmmrj\"" Nov 03 15:03:13.145164 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:13.145084 5979 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wn4x5\"/\"openshift-service-ca.crt\"" Nov 03 15:03:13.152190 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:13.152163 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9cd1770d-eee0-4b73-b866-99943822beba-sys\") pod \"perf-node-gather-daemonset-m5pl7\" (UID: \"9cd1770d-eee0-4b73-b866-99943822beba\") " pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-m5pl7" Nov 03 15:03:13.152342 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:13.152196 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9cd1770d-eee0-4b73-b866-99943822beba-podres\") pod \"perf-node-gather-daemonset-m5pl7\" (UID: \"9cd1770d-eee0-4b73-b866-99943822beba\") " pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-m5pl7" Nov 03 15:03:13.152342 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:13.152221 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9cd1770d-eee0-4b73-b866-99943822beba-lib-modules\") pod \"perf-node-gather-daemonset-m5pl7\" (UID: \"9cd1770d-eee0-4b73-b866-99943822beba\") " pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-m5pl7" Nov 03 15:03:13.152342 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:13.152305 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8745t\" (UniqueName: \"kubernetes.io/projected/9cd1770d-eee0-4b73-b866-99943822beba-kube-api-access-8745t\") pod \"perf-node-gather-daemonset-m5pl7\" (UID: \"9cd1770d-eee0-4b73-b866-99943822beba\") " pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-m5pl7" Nov 03 15:03:13.152490 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:13.152353 5979 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9cd1770d-eee0-4b73-b866-99943822beba-proc\") pod \"perf-node-gather-daemonset-m5pl7\" (UID: \"9cd1770d-eee0-4b73-b866-99943822beba\") " pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-m5pl7" Nov 03 15:03:13.193199 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:13.193167 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wn4x5/perf-node-gather-daemonset-m5pl7"] Nov 03 15:03:13.253181 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:13.253146 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9cd1770d-eee0-4b73-b866-99943822beba-sys\") pod \"perf-node-gather-daemonset-m5pl7\" (UID: \"9cd1770d-eee0-4b73-b866-99943822beba\") " pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-m5pl7" Nov 03 15:03:13.253181 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:13.253188 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9cd1770d-eee0-4b73-b866-99943822beba-podres\") pod \"perf-node-gather-daemonset-m5pl7\" (UID: \"9cd1770d-eee0-4b73-b866-99943822beba\") " pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-m5pl7" Nov 03 15:03:13.253426 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:13.253225 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9cd1770d-eee0-4b73-b866-99943822beba-lib-modules\") pod \"perf-node-gather-daemonset-m5pl7\" (UID: \"9cd1770d-eee0-4b73-b866-99943822beba\") " pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-m5pl7" Nov 03 15:03:13.253426 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:13.253270 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8745t\" (UniqueName: \"kubernetes.io/projected/9cd1770d-eee0-4b73-b866-99943822beba-kube-api-access-8745t\") pod \"perf-node-gather-daemonset-m5pl7\" (UID: \"9cd1770d-eee0-4b73-b866-99943822beba\") " pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-m5pl7" Nov 03 15:03:13.253426 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:13.253287 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9cd1770d-eee0-4b73-b866-99943822beba-sys\") pod \"perf-node-gather-daemonset-m5pl7\" (UID: \"9cd1770d-eee0-4b73-b866-99943822beba\") " pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-m5pl7" Nov 03 15:03:13.253426 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:13.253295 5979 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9cd1770d-eee0-4b73-b866-99943822beba-proc\") pod \"perf-node-gather-daemonset-m5pl7\" (UID: \"9cd1770d-eee0-4b73-b866-99943822beba\") " pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-m5pl7" Nov 03 15:03:13.253426 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:13.253304 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9cd1770d-eee0-4b73-b866-99943822beba-podres\") pod \"perf-node-gather-daemonset-m5pl7\" (UID: \"9cd1770d-eee0-4b73-b866-99943822beba\") " pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-m5pl7" Nov 03 15:03:13.253426 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:13.253342 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9cd1770d-eee0-4b73-b866-99943822beba-proc\") pod \"perf-node-gather-daemonset-m5pl7\" (UID: \"9cd1770d-eee0-4b73-b866-99943822beba\") " pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-m5pl7" Nov 03 15:03:13.253426 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:13.253388 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9cd1770d-eee0-4b73-b866-99943822beba-lib-modules\") pod \"perf-node-gather-daemonset-m5pl7\" (UID: \"9cd1770d-eee0-4b73-b866-99943822beba\") " pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-m5pl7" Nov 03 15:03:13.307103 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:13.307067 5979 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8745t\" (UniqueName: \"kubernetes.io/projected/9cd1770d-eee0-4b73-b866-99943822beba-kube-api-access-8745t\") pod \"perf-node-gather-daemonset-m5pl7\" (UID: \"9cd1770d-eee0-4b73-b866-99943822beba\") " pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-m5pl7" Nov 03 15:03:13.408927 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:13.408889 5979 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-m5pl7" Nov 03 15:03:13.779019 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:13.778918 5979 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wn4x5/perf-node-gather-daemonset-m5pl7"] Nov 03 15:03:13.786840 ip-10-0-130-198 kubenswrapper[5979]: W1103 15:03:13.786815 5979 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9cd1770d_eee0_4b73_b866_99943822beba.slice/crio-c1f7f1a70f156d170b0a7feeac49d6272c8e1fbac7f538aa7b89a328a297ccf1 WatchSource:0}: Error finding container c1f7f1a70f156d170b0a7feeac49d6272c8e1fbac7f538aa7b89a328a297ccf1: Status 404 returned error can't find the container with id c1f7f1a70f156d170b0a7feeac49d6272c8e1fbac7f538aa7b89a328a297ccf1 Nov 03 15:03:14.531375 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:14.531343 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-466qf_f31d565d-15b1-4b03-bbfc-ff43778c7a95/limitador/0.log" Nov 03 15:03:14.657862 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:14.657824 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-m5pl7" event={"ID":"9cd1770d-eee0-4b73-b866-99943822beba","Type":"ContainerStarted","Data":"4904a653fd6489ebd2cb506a52b4b726045e6420e9a229af556c01927cf30ea5"} Nov 03 15:03:14.658058 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:14.657887 5979 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-m5pl7" event={"ID":"9cd1770d-eee0-4b73-b866-99943822beba","Type":"ContainerStarted","Data":"c1f7f1a70f156d170b0a7feeac49d6272c8e1fbac7f538aa7b89a328a297ccf1"} Nov 03 15:03:14.658058 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:14.657926 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-m5pl7" Nov 03 15:03:14.765415 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:14.765352 5979 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-m5pl7" podStartSLOduration=1.765334992 podStartE2EDuration="1.765334992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 15:03:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 15:03:14.763777819 +0000 UTC m=+7076.774466074" watchObservedRunningTime="2026-11-03 15:03:14.765334992 +0000 UTC m=+7076.776023227" Nov 03 15:03:15.999784 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:15.999754 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-588d644cc-bhwpl_f6d109e0-3ef7-459b-bf9c-dea5de46b825/manager/0.log" Nov 03 15:03:17.547944 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:17.547915 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0eae77ff-4716-4b3e-94cf-cd767108537c/alertmanager/0.log" Nov 03 15:03:17.748145 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:17.748111 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-fwvg7_f33ad3b9-499c-41f5-9cf5-029709b82ca6/proxy/0.log" Nov 03 15:03:17.997098 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:17.997066 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0eae77ff-4716-4b3e-94cf-cd767108537c/config-reloader/0.log" Nov 03 15:03:18.405282 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:18.405255 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0eae77ff-4716-4b3e-94cf-cd767108537c/kube-rbac-proxy-web/0.log" Nov 03 15:03:18.813127 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:18.813041 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0eae77ff-4716-4b3e-94cf-cd767108537c/kube-rbac-proxy/0.log" Nov 03 15:03:19.222160 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:19.222134 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0eae77ff-4716-4b3e-94cf-cd767108537c/kube-rbac-proxy-metric/0.log" Nov 03 15:03:19.631465 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:19.631391 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0eae77ff-4716-4b3e-94cf-cd767108537c/prom-label-proxy/0.log" Nov 03 15:03:20.041128 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:20.041096 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0eae77ff-4716-4b3e-94cf-cd767108537c/init-config-reloader/0.log" Nov 03 15:03:20.672993 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:20.672967 5979 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wn4x5/perf-node-gather-daemonset-m5pl7" Nov 03 15:03:20.863506 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:20.863448 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-krsxs_b6263f8d-cced-465a-b393-ef51cb902e11/kube-state-metrics/0.log" Nov 03 15:03:21.268332 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:21.268304 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-krsxs_b6263f8d-cced-465a-b393-ef51cb902e11/kube-rbac-proxy-main/0.log" Nov 03 15:03:21.681942 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:21.681912 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-krsxs_b6263f8d-cced-465a-b393-ef51cb902e11/kube-rbac-proxy-self/0.log" Nov 03 15:03:22.039464 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:22.039384 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-588d644cc-bhwpl_f6d109e0-3ef7-459b-bf9c-dea5de46b825/manager/0.log" Nov 03 15:03:22.183369 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:22.183339 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6488577f47-wwpwm_5bfbbeb5-621b-43b0-bad1-0420e8b07b62/metrics-server/0.log" Nov 03 15:03:22.688597 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:22.688569 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-cxrp7_2c1f3b2f-1d7f-44b5-bcba-f9310e7bdb96/monitoring-plugin/0.log" Nov 03 15:03:23.766948 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:23.766919 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-fwvg7_f33ad3b9-499c-41f5-9cf5-029709b82ca6/proxy/0.log" Nov 03 15:03:25.507189 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:25.507149 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zhl5n_0fadb078-8983-42df-afde-c1983044492d/node-exporter/0.log" Nov 03 15:03:25.915192 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:25.915162 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zhl5n_0fadb078-8983-42df-afde-c1983044492d/kube-rbac-proxy/0.log" Nov 03 15:03:26.327530 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:26.327449 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zhl5n_0fadb078-8983-42df-afde-c1983044492d/init-textfile/0.log" Nov 03 15:03:26.739138 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:26.739108 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-fqskz_0d0c5822-ab7b-409d-a755-49e918278fa1/kube-rbac-proxy-main/0.log" Nov 03 15:03:27.149708 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:27.149678 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-fqskz_0d0c5822-ab7b-409d-a755-49e918278fa1/kube-rbac-proxy-self/0.log" Nov 03 15:03:27.643258 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:27.643226 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-fqskz_0d0c5822-ab7b-409d-a755-49e918278fa1/openshift-state-metrics/0.log" Nov 03 15:03:28.064248 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:28.064158 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_16ea9a76-6906-4842-912f-e5cbe2784973/prometheus/0.log" Nov 03 15:03:28.476248 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:28.476215 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_16ea9a76-6906-4842-912f-e5cbe2784973/config-reloader/0.log" Nov 03 15:03:28.578289 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:28.578247 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-nvcvb_3a101d3a-a7d5-4f43-a4e8-0a29d9bb5607/discovery/0.log" Nov 03 15:03:28.692032 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:28.692002 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-77875f86b8-krvtz_593e2573-815f-4352-8e5f-13ab39dbd37e/kube-auth-proxy/0.log" Nov 03 15:03:28.885486 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:28.885452 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_16ea9a76-6906-4842-912f-e5cbe2784973/thanos-sidecar/0.log" Nov 03 15:03:29.296432 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:29.296354 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_16ea9a76-6906-4842-912f-e5cbe2784973/kube-rbac-proxy-web/0.log" Nov 03 15:03:29.338238 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:29.338207 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b9c49cd7-8gjrw_e7e12233-9e76-4de5-b4cf-93644996783e/istio-proxy/0.log" Nov 03 15:03:29.559322 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:29.559248 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b9c49cd7-dkw27_f313510b-02d8-4c68-accd-920343219917/istio-proxy/0.log" Nov 03 15:03:29.704536 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:29.704508 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_16ea9a76-6906-4842-912f-e5cbe2784973/kube-rbac-proxy/0.log" Nov 03 15:03:29.993820 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:29.993786 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b9c49cd7-j5s8q_fe445e0e-53c9-4543-9107-ff8f13cbd425/istio-proxy/0.log" Nov 03 15:03:30.122091 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:30.122061 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_16ea9a76-6906-4842-912f-e5cbe2784973/kube-rbac-proxy-thanos/0.log" Nov 03 15:03:30.424273 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:30.424244 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b9c49cd7-lj2cw_7f4e540f-d1f9-4ffe-83ea-dfca92ce7233/istio-proxy/0.log" Nov 03 15:03:30.522201 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:30.522177 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_16ea9a76-6906-4842-912f-e5cbe2784973/init-config-reloader/0.log" Nov 03 15:03:31.503329 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:31.503298 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-68d6ccb6b8-8tq9t_706b4485-a4d4-4aa7-a942-9bd37ef907e9/payload-pre-processing/0.log" Nov 03 15:03:32.136877 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:32.136849 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-59897cc7df-n8cb9_11a6e3d9-2422-4fdb-8c44-949500637239/telemeter-client/0.log" Nov 03 15:03:32.569965 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:32.569879 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-59897cc7df-n8cb9_11a6e3d9-2422-4fdb-8c44-949500637239/reload/0.log" Nov 03 15:03:33.014141 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:33.014110 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-59897cc7df-n8cb9_11a6e3d9-2422-4fdb-8c44-949500637239/kube-rbac-proxy/0.log" Nov 03 15:03:33.477603 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:33.477569 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86b8856f97-b75hg_0fef60a4-2a84-4318-b1bf-feb8f69ef7e9/thanos-query/0.log" Nov 03 15:03:33.920976 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:33.920943 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86b8856f97-b75hg_0fef60a4-2a84-4318-b1bf-feb8f69ef7e9/kube-rbac-proxy-web/0.log" Nov 03 15:03:34.331035 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:34.330964 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86b8856f97-b75hg_0fef60a4-2a84-4318-b1bf-feb8f69ef7e9/kube-rbac-proxy/0.log" Nov 03 15:03:34.743387 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:34.743359 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86b8856f97-b75hg_0fef60a4-2a84-4318-b1bf-feb8f69ef7e9/prom-label-proxy/0.log" Nov 03 15:03:35.151815 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:35.151791 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86b8856f97-b75hg_0fef60a4-2a84-4318-b1bf-feb8f69ef7e9/kube-rbac-proxy-rules/0.log" Nov 03 15:03:35.560621 ip-10-0-130-198 kubenswrapper[5979]: I1103 15:03:35.560548 5979 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86b8856f97-b75hg_0fef60a4-2a84-4318-b1bf-feb8f69ef7e9/kube-rbac-proxy-metrics/0.log" Nov 03 15:03:37.238054 ip-10-0-130-198 kubenswrapper[5979]: E1103 15:03:37.238009 5979 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-n2nst" containerName="extractor"